2021-03-18 3:04:32 0 [Warning] InnoDB: A long semaphore wait: --Thread 140120999970560 has waited at ibuf0ibuf.cc line 4489 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f83859f6228 created in file buf0buf.cc line 1563 a writer (thread id 140120983185152) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time write locked in file ibuf0ibuf.cc line 4489 ... 2021-03-18 3:16:24 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung. 210318 3:16:24 [ERROR] mysqld got signal 6 ; ... # grep ^Thread gdb.thread.apply.all.bt | wc -l 418 # grep '#1 .*_lock' gdb.thread.apply.all.bt #1 0x00007f906c83c025 in __GI___pthread_mutex_lock (mutex=0x55dd53629ca0 ) at ../nptl/pthread_mutex_lock.c:80 #1 0x00007f906c83c025 in __GI___pthread_mutex_lock (mutex=0x55dd53629ca0 ) at ../nptl/pthread_mutex_lock.c:80 #1 0x00007f906c83c19c in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f70b4019bb8) at ../nptl/pthread_mutex_lock.c:135 #1 0x00007f906c83c025 in __GI___pthread_mutex_lock (mutex=0x55dd53629ca0 ) at ../nptl/pthread_mutex_lock.c:80 # grep '#[0-9] .*mutex=[^<]' gdb.thread.apply.all.bt | sed -e 's/.*mutex=\(mutex@entry=\)\?\(0x[0-9a-f]*\).*/\2/' | sort -u | sed -e 'i-ex' -e 's/.*/print\/x &/p' -e 'i-ex' -e 's/print\/x \(.*\)/print *(pthread_mutex_t*)\1/' | xargs -d'\n' gdb `which mysqld` core.6202 -batch Listing: [Current thread is 1 (Thread 0x7f906e32f6c0 (LWP 6202))] $1 = 0x55dd53629ca0 $2 = {__data = {__lock = 2, __count = 0, __owner = 31355, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000{z\000\000\001", '\000' , __align = 2} $3 = 0x55dd53629d20 $4 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 43, __kind = 3, __spins = 11, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "+\000\000\000\003\000\000\000\v", '\000' , __align = 0} $5 = 0x55dd536526a0 $6 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 3, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001\000\000\000\003", '\000' , __align = 0} $7 = 0x55dd53656498 $8 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 3, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001\000\000\000\003", '\000' , __align = 0} $9 = 0x55dd53eac160 $10 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $11 = 0x55dd53ebbb80 $12 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 3, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001\000\000\000\003", '\000' , __align = 0} $13 = 0x55dd548442d8 $14 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $15 = 0x55dd5484dbb8 $16 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $17 = 0x55dd5484e280 $18 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 27, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\033", '\000' , __align = 0} $19 = 0x55dd5484e320 $20 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 6, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\006", '\000' , __align = 0} $21 = 0x55dd54ab5070 $22 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $23 = 0x55dd54ab5350 $24 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $25 = 0x55dd54ab53c0 $26 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $27 = 0x55dd54ab5430 $28 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $29 = 0x55dd54ab54a0 $30 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $31 = 0x55dd54ab55f0 $32 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $33 = 0x55dd54ab56d0 $34 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $35 = 0x55dd54b766a0 $36 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $37 = 0x55dd54b76710 $38 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $39 = 0x55dd54b76940 $40 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $41 = 0x55dd54b769b0 $42 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $43 = 0x55dd54b76a20 $44 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $45 = 0x55dd54b76a90 $46 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $47 = 0x55dd54bef070 $48 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 5, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\005", '\000' , __align = 0} $49 = 0x55dd619a6c20 $50 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $51 = 0x55ddbcf43d00 $52 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 62, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , ">", '\000' , __align = 0} $53 = 0x55ddbcf45090 $54 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 3, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\003", '\000' , __align = 0} $55 = 0x55ddbcf65ff0 $56 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $57 = 0x55ddbcfe2d30 $58 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} $59 = 0x7f70b4019bb8 $60 = {__data = {__lock = 2, __count = 0, __owner = 6848, __nusers = 1, __kind = 3, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "\002\000\000\000\000\000\000\000\300\032\000\000\001\000\000\000\003", '\000' , __align = 2} $61 = 0x7f70c37fdda0 $62 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\001", '\000' , __align = 0} That 0x7f70b4019bb8 is in thread 219: Thread 219 (Thread 0x7f709bfa3700 (LWP 29116)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f906c83c19c in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f70b4019bb8) at ../nptl/pthread_mutex_lock.c:135 #2 0x000055dd526da600 in inline_mysql_mutex_lock (src_file=0x55dd52d57518 "/home/buildbot/buildbot/build/mariadb-10.3.28/sql/wsrep_thd.cc", src_line=761, that=0x7f70b4019bb8) at ./include/mysql/psi/mysql_thread.h:719 #3 wsrep_thd_is_BF (thd=thd@entry=0x7f70b4018138, sync=sync@entry=1 '\001') at ./sql/wsrep_thd.cc:761 #4 0x000055dd52532d68 in thd_need_ordering_with (thd=0x7f6f54000c08, other_thd=0x7f70b4018138) at ./sql/sql_class.cc:4990 #5 0x000055dd52924b39 in lock_rec_has_to_wait (for_locking=, lock_is_on_supremum=, lock2=, type_mode=, trx=) at ./storage/innobase/lock/lock0lock.cc:788 #6 lock_has_to_wait (lock1=0x7f83b8eedbd0, lock2=0x7f83b8f1eaf8) at ./storage/innobase/lock/lock0lock.cc:869 #7 0x000055dd52924d1b in lock_rec_has_to_wait_in_queue (wait_lock=wait_lock@entry=0x7f83b8eedbd0) at ./storage/innobase/lock/lock0lock.cc:2056 #8 0x000055dd529252fe in lock_rec_dequeue_from_page (in_lock=in_lock@entry=0x7f83b8f20a50) at ./storage/innobase/lock/lock0lock.cc:2246 #9 0x000055dd529267f8 in lock_release (trx=trx@entry=0x7f83b8f20988) at ./storage/innobase/lock/lock0lock.cc:4280 #10 0x000055dd52a23ea0 in trx_t::release_locks (this=0x7f83b8f20988) at ./storage/innobase/trx/trx0trx.cc:519 #11 trx_commit_in_memory (mtr=0x7f709bf9f990, trx=0x7f83b8f20988) at ./storage/innobase/trx/trx0trx.cc:1402 #12 trx_commit_low (trx=0x7f83b8f20988, mtr=0x7f709bf9f990) at ./storage/innobase/trx/trx0trx.cc:1609 #13 0x000055dd52a247f0 in trx_commit (trx=trx@entry=0x7f83b8f20988) at ./storage/innobase/trx/trx0trx.cc:1633 #14 0x000055dd52a158f8 in trx_rollback_finish (trx=trx@entry=0x7f83b8f20988) at ./storage/innobase/trx/trx0roll.cc:66 #15 0x000055dd52a16772 in trx_rollback_to_savepoint_low (trx=trx@entry=0x7f83b8f20988, savept=0x0) at ./storage/innobase/trx/trx0roll.cc:148 #16 0x000055dd52a16b96 in trx_rollback_for_mysql_low (trx=0x7f83b8f20988) at ./storage/innobase/trx/trx0roll.cc:209 #17 trx_rollback_for_mysql (trx=trx@entry=0x7f83b8f20988) at ./storage/innobase/trx/trx0roll.cc:286 #18 0x000055dd528d8b70 in innobase_rollback (hton=, thd=0x7f6fc8056898, rollback_trx=) at ./storage/innobase/handler/ha_innodb.cc:4671 #19 0x000055dd5275cef7 in ha_rollback_trans (thd=thd@entry=0x7f6fc8056898, all=all@entry=true) at ./sql/handler.cc:1751 #20 0x000055dd5275d5f4 in ha_commit_trans (thd=thd@entry=0x7f6fc8056898, all=all@entry=true) at ./sql/handler.cc:1571 #21 0x000055dd52661e0b in trans_commit (thd=thd@entry=0x7f6fc8056898) at ./sql/transaction.cc:292 #22 0x000055dd52578847 in mysql_execute_command (thd=thd@entry=0x7f6fc8056898) at ./sql/sql_parse.cc:5554 #23 0x000055dd5257d4fa in mysql_parse (thd=thd@entry=0x7f6fc8056898, rawbuf=rawbuf@entry=0x7f6fc8065130 "COMMIT", length=length@entry=6, parser_state=parser_state@entry=0x7f709bfa2550, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7841 #24 0x000055dd5257ddc6 in wsrep_mysql_parse (thd=0x7f6fc8056898, rawbuf=0x7f6fc8065130 "COMMIT", length=6, parser_state=0x7f709bfa2550, is_com_multi=false, is_next_command=false) at ./sql/sql_parse.cc:7633 #25 0x000055dd5257f183 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f6fc8056898, packet=packet@entry=0x7f6fc8106139 "COMMIT", packet_length=packet_length@entry=6, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:1938 #26 0x000055dd52580ba6 in do_command (thd=0x7f6fc8056898) at ./sql/sql_parse.cc:1398 #27 0x000055dd52655097 in do_handle_one_connection (connect=connect@entry=0x55ddbd0fbb48) at ./sql/sql_connect.cc:1403 #28 0x000055dd5265528d in handle_one_connection (arg=arg@entry=0x55ddbd0fbb48) at ./sql/sql_connect.cc:1308 #29 0x000055dd52c6d12a in pfs_spawn_thread (arg=0x55ddbd0fbda8) at ./storage/perfschema/pfs.cc:1869 #30 0x00007f906c8396db in start_thread (arg=0x7f709bfa3700) at pthread_create.c:463 #31 0x00007f906be3b71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 219 is waiting for 97: (gdb) thread find 6848 Thread 97 has target id 'Thread 0x7f70c2283700 (LWP 6848)' (gdb) thread 97 [Switching to thread 97 (Thread 0x7f70c2283700 (LWP 6848))] #0 0x00007f906c83fad3 in futex_wait_cancelable (private=, expected=0, futex_word=0x55ddbcf43d50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 88 ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory. (gdb) bt #0 0x00007f906c83fad3 in futex_wait_cancelable (private=, expected=0, futex_word=0x55ddbcf43d50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x55ddbcf43d00, cond=0x55ddbcf43d28) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=cond@entry=0x55ddbcf43d28, mutex=mutex@entry=0x55ddbcf43d00) at pthread_cond_wait.c:655 #3 0x000055dd52956230 in os_event::wait (this=0x55ddbcf43cf0) at ./storage/innobase/os/os0event.cc:158 #4 os_event::wait_low (reset_sig_count=3, this=0x55ddbcf43cf0) at ./storage/innobase/os/os0event.cc:325 #5 os_event_wait_low (event=0x55ddbcf43cf0, reset_sig_count=) at ./storage/innobase/os/os0event.cc:502 #6 0x000055dd529f282c in sync_array_wait_event (arr=0x55dd549e06a0, cell=@0x7f70c2282cd8: 0x55dd549e0ca0) at ./storage/innobase/sync/sync0arr.cc:471 #7 0x000055dd528f3cb7 in TTASEventMutex::enter (line=18772, filename=0x55dd52e5b130 "/home/buildbot/buildbot/build/mariadb-10.3.28/storage/innobase/handler/ha_innodb.cc", max_delay=4, max_spins=, this=0x55dd536574c0 ) at ./storage/innobase/include/ib0mutex.h:471 #8 PolicyMutex >::enter (this=0x55dd536574c0 , n_spins=30, n_delay=4, name=name@entry=0x55dd52e5b130 "/home/buildbot/buildbot/build/mariadb-10.3.28/storage/innobase/handler/ha_innodb.cc", line=line@entry=18772) at ./storage/innobase/include/ib0mutex.h:592 #9 0x000055dd528e7798 in bg_wsrep_kill_trx (void_arg=0x7f6f54061260) at ./storage/innobase/handler/ha_innodb.cc:18772 #10 0x000055dd5256d5d3 in handle_manager (arg=arg@entry=0x0) at ./sql/sql_manager.cc:112 #11 0x000055dd52c6d12a in pfs_spawn_thread (arg=0x55ddbd090718) at ./storage/perfschema/pfs.cc:1869 #12 0x00007f906c8396db in start_thread (arg=0x7f70c2283700) at pthread_create.c:463 #13 0x00007f906be3b71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 97 is waiting for a condition on 0x55ddbcf43d00, and so are XXX threads: # grep '__pthread_cond_wait_common.*0x55ddbcf43d00' gdb.thread.apply.all.bt | wc -l 62 (gdb) print *(pthread_mutex_t*)0x55ddbcf43d00 $1 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 62, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , ">", '\000' , __align = 0} There were 3 threads waiting on 0x55dd53629ca0 : Thread 370 (Thread 0x7f709a464700 (LWP 6751)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f906c83c025 in __GI___pthread_mutex_lock (mutex=0x55dd53629ca0 ) at ../nptl/pthread_mutex_lock.c:80 #2 0x000055dd525d25c5 in inline_mysql_mutex_lock (that=0x55dd53629ca0 , src_file=src_file@entry=0x55dd52d36400 "/home/buildbot/buildbot/build/mariadb-10.3.28/sql/sql_show.cc", src_line=src_line@entry=7943) at ./include/mysql/psi/mysql_thread.h:719 -- Thread 257 (Thread 0x7f70c03c0700 (LWP 24782)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f906c83c025 in __GI___pthread_mutex_lock (mutex=0x55dd53629ca0 ) at ../nptl/pthread_mutex_lock.c:80 #2 0x000055dd525d25c5 in inline_mysql_mutex_lock (that=0x55dd53629ca0 , src_file=src_file@entry=0x55dd52d36400 "/home/buildbot/buildbot/build/mariadb-10.3.28/sql/sql_show.cc", src_line=src_line@entry=7943) at ./include/mysql/psi/mysql_thread.h:719 -- Thread 148 (Thread 0x7f709c78c700 (LWP 28053)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f906c83c025 in __GI___pthread_mutex_lock (mutex=0x55dd53629ca0 ) at ../nptl/pthread_mutex_lock.c:80 #2 0x000055dd525d25c5 in inline_mysql_mutex_lock (that=0x55dd53629ca0 , src_file=src_file@entry=0x55dd52d36400 "/home/buildbot/buildbot/build/mariadb-10.3.28/sql/sql_show.cc", src_line=src_line@entry=7943) at ./include/mysql/psi/mysql_thread.h:719 But that one is held by thread 394: (gdb) thread find 31355 Thread 394 has target id 'Thread 0x7f709f573700 (LWP 31355)' (gdb) bt #0 0x00007f906be2ecb9 in __GI___poll (fds=fds@entry=0x7ffd941928f0, nfds=nfds@entry=2, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f906be4eaae in __poll_chk (fds=fds@entry=0x7ffd941928f0, nfds=nfds@entry=2, timeout=timeout@entry=-1, fdslen=fdslen@entry=24) at poll_chk.c:27 #2 0x000055dd524c4ba7 in poll (__timeout=-1, __nfds=2, __fds=0x7ffd941928f0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41 #3 handle_connections_sockets () at ./sql/mysqld.cc:6815 #4 0x000055dd524c62bd in mysqld_main (argc=, argv=) at ./sql/mysqld.cc:6286 #5 0x00007f906bd3bbf7 in __libc_start_main (main=0x55dd524a4d10 , argc=3, argv=0x7ffd94192c98, init=, fini=, rtld_fini=, stack_end=0x7ffd94192c88) at ../csu/libc-start.c:310 #6 0x000055dd524b873a in _start ()