[MDEV-9126] Assertion failure, crash, and possible lock_sys->wait_mutex or dict_operation_lock deadlock Created: 2015-11-12  Updated: 2015-11-17  Resolved: 2015-11-17

Status: Closed
Project: MariaDB Server
Component/s: Galera, wsrep
Affects Version/s: 10.0.21-galera
Fix Version/s: 10.0.22-galera

Type: Bug Priority: Major
Reporter: Geoff Montee (Inactive) Assignee: Jan Lindström (Inactive)
Resolution: Duplicate Votes: 0
Labels: galera


 Description   

A user saw a lot of long lock waits related to lock_sys->wait_mutex and dict_operation_lock. Eventually, this caused an assertion failure and a crash.

Maybe this is related to MDEV-8899?

--Thread 127879114209024 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
S-lock on RW-latch at 0x14879c0 '&dict_operation_lock'
a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
Last time read locked in file row0ins.cc line 1830
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
--Thread 140365365462784 has waited at lock0wait.cc line 77 for 2021.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
--Thread 127879027795712 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
S-lock on RW-latch at 0x14879c0 '&dict_operation_lock'
a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
Last time read locked in file row0ins.cc line 1830
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
--Thread 140365401642752 has waited at lock0wait.cc line 77 for 2008.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
--Thread 140365363369728 has waited at lock0wait.cc line 77 for 2016.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
--Thread 140365367555840 has waited at lock0wait.cc line 77 for 1967.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
InnoDB: ###### Diagnostic info printed to the standard error stream
--Thread 140365155309312 has waited at lock0wait.cc line 77 for 2000.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
--Thread 140365430646528 has waited at lock0wait.cc line 77 for 1950.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
--Thread 140365210924800 has waited at lock0wait.cc line 77 for 1983.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
--Thread 140365354100480 has waited at lock0wait.cc line 77 for 1975.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
2015-11-12 04:15:35 7fdb74ffd700  InnoDB: Assertion failure in thread 140580537489152 in file srv0srv.cc line 2231
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
--Thread 140365122717440 has waited at lock0wait.cc line 77 for 1959.0 seconds the semaphore:
Mutex at 0x7fdc7f4000b8 '&lock_sys->wait_mutex', lock var 1
waiters flag 1
--Thread 127879319029504 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
S-lock on RW-latch at 0x14879c0 '&dict_operation_lock'
a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
Last time read locked in file row0ins.cc line 1830
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
151112  4:15:35 --Thread 127879226337024 has waited at row0ins.cc line 1830 for 344.00 seconds the semaphore:
[ERROR] mysqld got signal 6 ;
S-lock onThis could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 RW-latch at 0x14879c0 '&dict_operation_lock'
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
a writer (thread id 140580554274560) has reserved it in mode  wait exclusive
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
number of readers 31, waiters flag 1, lock_word: ffffffffffffffe1
Last time read locked in file row0ins.cc line 1830
Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.21/storage/xtradb/srv/srv0srv.cc line 2699
Server version: 10.0.21-MariaDB-wsrep-log



 Comments   
Comment by Jan Lindström (Inactive) [ 2015-11-17 ]

Looks similar to MDEV-8869.

Generated at Thu Feb 08 07:32:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.