Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9126

Assertion failure, crash, and possible lock_sys->wait_mutex or dict_operation_lock deadlock

    XMLWordPrintable

Details

    Description

      A user saw a lot of long lock waits related to lock_sys->wait_mutex and dict_operation_lock. Eventually, this caused an assertion failure and a crash.

      Maybe this is related to MDEV-8899?

      --Thread 127879114209024 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
      S-lock on RW-latch at 0x14879c0 '&dict_operation_lock'
      a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
      number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
      Last time read locked in file row0ins.cc line 1830
      Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
      --Thread 140365365462784 has waited at lock0wait.cc line 77 for 2021.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      --Thread 127879027795712 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
      S-lock on RW-latch at 0x14879c0 '&dict_operation_lock'
      a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
      number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
      Last time read locked in file row0ins.cc line 1830
      Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
      --Thread 140365401642752 has waited at lock0wait.cc line 77 for 2008.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      --Thread 140365363369728 has waited at lock0wait.cc line 77 for 2016.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      --Thread 140365367555840 has waited at lock0wait.cc line 77 for 1967.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      InnoDB: ###### Diagnostic info printed to the standard error stream
      --Thread 140365155309312 has waited at lock0wait.cc line 77 for 2000.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      --Thread 140365430646528 has waited at lock0wait.cc line 77 for 1950.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      --Thread 140365210924800 has waited at lock0wait.cc line 77 for 1983.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      InnoDB: Error: semaphore wait has lasted > 600 seconds
      InnoDB: We intentionally crash the server, because it appears to be hung.
      --Thread 140365354100480 has waited at lock0wait.cc line 77 for 1975.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      2015-11-12 04:15:35 7fdb74ffd700  InnoDB: Assertion failure in thread 140580537489152 in file srv0srv.cc line 2231
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      --Thread 140365122717440 has waited at lock0wait.cc line 77 for 1959.0 seconds the semaphore:
      Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
      waiters flag 1
      --Thread 127879319029504 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
      S-lock on RW-latch at 0x14879c0 '&dict_operation_lock'
      a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
      number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
      Last time read locked in file row0ins.cc line 1830
      Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
      151112  4:15:35 --Thread 127879226337024 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
      [ERROR] mysqld got signal 6 ;
      S-lock onThis could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
       RW-latch at 0x14879c0 '&dict_operation_lock'
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
      Last time read locked in file row0ins.cc line 1830
      Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
      Server version: 10.0.21-MariaDB-wsrep-log

      Attachments

        Activity

          People

            jplindst Jan Lindström (Inactive)
            GeoffMontee Geoff Montee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.