[#MDEV-14486] InnoDB hang on shutdown

[MDEV-14486] InnoDB hang on shutdown Created: 2017-11-23 Updated: 2023-08-31
Status:	Confirmed
Project:	MariaDB Server
Component/s:	Storage Engine - InnoDB
Affects Version/s:	10.2, 10.3, 10.4
Fix Version/s:	10.4

Type:

Bug

Priority:

Major

Reporter:

Elena Stepanova

Assignee:

Thirunarayanan Balathandayuthapani

Resolution:

Unresolved

Votes:

Labels:

need_rr

Issue Links:

Duplicate
is duplicated by	~~MDEV-20158~~	innodb_zip.innochecksum failed in bui...	Closed
Relates
relates to	~~MDEV-14080~~	InnoDB shutdown sometimes hangs	Closed
relates to	~~MDEV-22577~~	innodb_fast_shutdown=0 fails to repor...	Closed
relates to	MDEV-26264	server wont start after clean shutdown	Open
relates to	MDEV-27125	galera_new_cluster hangs with galera_...	Open
relates to	MDEV-32033	all queries get stuck until a timeout...	Open
relates to	MDEV-26327	Assertion `!"purge was not shut down"...	Open

Description

http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-x86/builds/3235/steps/mtr/logs/stdio

innodb.innodb_zip_innochecksum3 '16k,innodb' w4 [ fail ]

        Test ended at 2017-11-14 17:37:32

CURRENT_TEST: innodb.innodb_zip_innochecksum3

Error: Unable to lock file:: /dev/shm/var/4/mysqld.1/data//test/tab1.ibd

fcntl: Resource temporarily unavailable

mysqltest: At line 109: exec of '/usr/bin/innochecksum  --write=CRC32 /dev/shm/var/4/mysqld.1/data//test/tab1.ibd' failed, error: 256, status: 1, errno: 32

Output from before failure:

[1(b)]: Rewrite into new checksum=crc32 for all *.ibd file and ibdata1

The result from queries just before the failure was:

< snip >

VALUES (@col_1,@col_2,@col_3,@col_4,@cl_5,@col_6,1);

SELECT * FROM tab2 ORDER BY col_7;

# stop the server

[1(a)]: Rewrite into new checksum=InnoDB for all *.ibd file and ibdata1

: start the server with innodb_checksum_algorithm=strict_innodb

INSERT INTO tab1 (pk, linestring_key, linestring_nokey)

VALUES (2, ST_GeomFromText('LINESTRING(10 10,20 20,30 30)'), ST_GeomFromText('LINESTRING(10 10,20 20,30 30)'));

SET @col_1 = repeat('a', 5);

SET @col_2 = repeat('b', 20);

SET @col_3 = repeat('c', 100);

SET @col_4 = repeat('d', 100);

SET @col_5 = repeat('e', 100);

SET @col_6 = repeat('f', 100);

INSERT INTO tab2(col_1,col_2,col_3,col_4,col_5,col_6,col_7)

VALUES (@col_1,@col_2,@col_3,@col_4,@cl_5,@col_6,6);

SELECT pk,ST_AsText(linestring_key),ST_AsText(linestring_nokey)

FROM tab1 ORDER BY pk;

SELECT * FROM tab2 ORDER BY col_7;

# stop the server

[1(b)]: Rewrite into new checksum=crc32 for all *.ibd file and ibdata1

More results from queries before failure can be found in /dev/shm/var/4/log/innodb_zip_innochecksum3.log

Comments

Comment by Marko Mäkelä [ 2017-11-30 ]

While this test does not use innodb_fast_shutdown=0, it seems that the server can hang during the shutdown. As noted in ~~MDEV-14080~~, in 10.3 this would result in an external SIGABRT and a crash report message would be picked up by mysql-test-run:

innodb.innodb_zip_innochecksum3 '4k,innodb' w1 [ fail ]  Found warnings/errors in server log file!

        Test ended at 2017-11-30 13:47:37

line

Attempting backtrace. You can use the following information to find out

^ Found warnings in /run/shm/var/1/log/mysqld.1.err

The mysqld.1.err says:

Version: '10.3.3-MariaDB-10.3.3+maria~wheezy-log'  socket: '/run/shm/var/tmp/1/mysqld.1.sock'  port: 16020  mariadb.org binary distribution

2017-11-30 13:46:24 0 [Note] /usr/sbin/mysqld (root[root] @ localhost []): Normal shutdown

2017-11-30 13:46:24 0 [Note] Event Scheduler: Purging the queue. 0 events

2017-11-30 13:46:24 0 [Note] InnoDB: FTS optimize thread exiting.

171130 13:47:24 [ERROR] mysqld got signal 6 ;

Note the exactly 60 seconds between the two messages.

Comment by Elena Stepanova [ 2017-11-30 ]

While trying to reproduce ~~MDEV-14080~~ on my local machine, I got a hang on shutdown, but it wasn't the "read-only" restart, and that's the only one which is done with slow shutdown. Instead, I believe it happened on the last restart in the test. So, despite the fact that it was innodb.row_format_redundant, probably the stack trace belongs here.

10.3 RelWithDebInfo 7cb3520c0632ad91
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/data/bld/10.3-rel/bin/mysqld --defaults-group-suffix=.1 --defaults-file=/data/'.
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57

Thread 21 (Thread 0x7efe05f36700 (LWP 24275)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000055a068c3600e in inline_mysql_cond_wait (src_file=0x55a0693c5dc4 "/data/src/10.3/sql/mysqld.cc", src_line=1824, mutex=0x55a069cd7280 <LOCK_thread_count>, that=0x55a069cd6b60 <COND_thread_count>) at /data/src/10.3/include/mysql/psi/mysql_thread.h:1138
#2 close_connections () at /data/src/10.3/sql/mysqld.cc:1824
#3 kill_server (sig_ptr=0x0) at /data/src/10.3/sql/mysqld.cc:2007
#4 0x000055a068c3642e in kill_server_thread (arg=arg@entry=0x7efe0c049dc8) at /data/src/10.3/sql/mysqld.cc:2039
#5 0x000055a06907f8e4 in pfs_spawn_thread (arg=0x7efdbc001048) at /data/src/10.3/storage/perfschema/pfs.cc:1863
#6 0x00007efe14553494 in start_thread (arg=0x7efe05f36700) at pthread_create.c:333
#7 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 20 (Thread 0x7efdfffff700 (LWP 24260)):
#0 0x00007efe1455c7dd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1 0x000055a06910977c in os_thread_sleep (tm=tm@entry=1000000) at /data/src/10.3/storage/innobase/os/os0thread.cc:230
#2 0x000055a06919317c in srv_master_sleep () at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2436
#3 srv_master_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2481
#4 0x00007efe14553494 in start_thread (arg=0x7efdfffff700) at pthread_create.c:333
#5 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 19 (Thread 0x7efe05619700 (LWP 24252)):
#0 0x00007efe1434a6ea in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x000055a0691049c6 in LinuxAIOHandler::collect (this=this@entry=0x7efe05618ca0) at /data/src/10.3/storage/innobase/os/os0file.cc:1893
#2 0x000055a069104e24 in LinuxAIOHandler::poll (this=this@entry=0x7efe05618ca0, m1=m1@entry=0x7efe05618d40, m2=m2@entry=0x7efe05618d48, request=request@entry=0x7efe05618d50) at /data/src/10.3/storage/innobase/os/os0file.cc:2038
#3 0x000055a06910867b in os_aio_linux_handler (request=0x7efe05618d50, m2=0x7efe05618d48, m1=0x7efe05618d40, global_segment=5) at /data/src/10.3/storage/innobase/os/os0file.cc:2092
#4 os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7efe05618d40, m2=m2@entry=0x7efe05618d48, request=request@entry=0x7efe05618d50) at /data/src/10.3/storage/innobase/os/os0file.cc:5720
#5 0x000055a06927bee3 in fil_aio_wait (segment=segment@entry=5) at /data/src/10.3/storage/innobase/fil/fil0fil.cc:5317
#6 0x000055a069194e48 in io_handler_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0start.cc:340
#7 0x00007efe14553494 in start_thread (arg=0x7efe05619700) at pthread_create.c:333
#8 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 18 (Thread 0x7efe06fcc700 (LWP 24249)):
#0 0x00007efe1434a6ea in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x000055a0691049c6 in LinuxAIOHandler::collect (this=this@entry=0x7efe06fcbca0) at /data/src/10.3/storage/innobase/os/os0file.cc:1893
#2 0x000055a069104e24 in LinuxAIOHandler::poll (this=this@entry=0x7efe06fcbca0, m1=m1@entry=0x7efe06fcbd40, m2=m2@entry=0x7efe06fcbd48, request=request@entry=0x7efe06fcbd50) at /data/src/10.3/storage/innobase/os/os0file.cc:2038
#3 0x000055a06910867b in os_aio_linux_handler (request=0x7efe06fcbd50, m2=0x7efe06fcbd48, m1=0x7efe06fcbd40, global_segment=2) at /data/src/10.3/storage/innobase/os/os0file.cc:2092
#4 os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7efe06fcbd40, m2=m2@entry=0x7efe06fcbd48, request=request@entry=0x7efe06fcbd50) at /data/src/10.3/storage/innobase/os/os0file.cc:5720
#5 0x000055a06927bee3 in fil_aio_wait (segment=segment@entry=2) at /data/src/10.3/storage/innobase/fil/fil0fil.cc:5317
#6 0x000055a069194e48 in io_handler_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0start.cc:340
#7 0x00007efe14553494 in start_thread (arg=0x7efe06fcc700) at pthread_create.c:333
#8 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 17 (Thread 0x7efde97fa700 (LWP 24267)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000055a069109116 in wait (this=0x55a06abf2860) at /data/src/10.3/storage/innobase/os/os0event.cc:166
#2 os_event::wait_low (this=0x55a06abf2860, reset_sig_count=1, reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:336
#3 0x000055a0691095b5 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:535
#4 0x000055a06921f992 in buf_resize_thread () at /data/src/10.3/storage/innobase/buf/buf0buf.cc:3076
#5 0x00007efe14553494 in start_thread (arg=0x7efde97fa700) at pthread_create.c:333
#6 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 16 (Thread 0x7efe0d480700 (LWP 24246)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x000055a069011104 in inline_mysql_cond_timedwait (src_file=0x55a0694cc1f8 "/data/src/10.3/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7efe0d47fe00, mutex=0x55a06a506e60 <LOCK_checkpoint>, that=0x55a06a506e20 <COND_checkpoint>) at /data/src/10.3/include/mysql/psi/mysql_thread.h:1175
#2 my_service_thread_sleep (control=control@entry=0x55a069c49860 <checkpoint_control>, sleep_time=<optimized out>) at /data/src/10.3/storage/maria/ma_servicethread.c:115
#3 0x000055a06900aa0a in ma_checkpoint_background (arg=arg@entry=0x1e) at /data/src/10.3/storage/maria/ma_checkpoint.c:709
#4 0x000055a06907f8e4 in pfs_spawn_thread (arg=0x55a06aae8728) at /data/src/10.3/storage/perfschema/pfs.cc:1863
#5 0x00007efe14553494 in start_thread (arg=0x7efe0d480700) at pthread_create.c:333
#6 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 15 (Thread 0x7efdfeffd700 (LWP 24256)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x000055a069108fee in os_event::timed_wait (this=this@entry=0x55a06abf2660, abstime=abstime@entry=0x7efdfeffccb0) at /data/src/10.3/storage/innobase/os/os0event.cc:286
#2 0x000055a069109241 in os_event::wait_time_low (this=0x55a06abf2660, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /data/src/10.3/storage/innobase/os/os0event.cc:413
#3 0x000055a0691095a5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /data/src/10.3/storage/innobase/os/os0event.cc:518
#4 0x000055a06919157a in srv_error_monitor_thread () at /data/src/10.3/storage/innobase/srv/srv0srv.cc:1947
#5 0x00007efe14553494 in start_thread (arg=0x7efdfeffd700) at pthread_create.c:333
#6 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 14 (Thread 0x7efe04e18700 (LWP 24253)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x000055a069108fee in os_event::timed_wait (this=this@entry=0x55a06abf27e0, abstime=abstime@entry=0x7efe04e17b60) at /data/src/10.3/storage/innobase/os/os0event.cc:286
#2 0x000055a069109241 in os_event::wait_time_low (this=0x55a06abf27e0, time_in_usec=<optimized out>, reset_sig_count=reset_sig_count@entry=2) at /data/src/10.3/storage/innobase/os/os0event.cc:413
#3 0x000055a0691095a5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=<optimized out>, reset_sig_count=reset_sig_count@entry=2) at /data/src/10.3/storage/innobase/os/os0event.cc:518
#4 0x000055a0692308f4 in pc_sleep_if_needed (sig_count=2, next_loop_time=1512065963382) at /data/src/10.3/storage/innobase/buf/buf0flu.cc:2710
#5 buf_flush_page_cleaner_coordinator () at /data/src/10.3/storage/innobase/buf/buf0flu.cc:3207
#6 0x00007efe14553494 in start_thread (arg=0x7efe04e18700) at pthread_create.c:333
#7 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 13 (Thread 0x7efdea7fc700 (LWP 24265)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000055a069109116 in wait (this=0x55a06abf2760) at /data/src/10.3/storage/innobase/os/os0event.cc:166
#2 os_event::wait_low (this=0x55a06abf2760, reset_sig_count=1, reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:336
#3 0x000055a0691095b5 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:535
#4 0x000055a069226c6e in buf_dump_thread () at /data/src/10.3/storage/innobase/buf/buf0dump.cc:777
#5 0x00007efe14553494 in start_thread (arg=0x7efdea7fc700) at pthread_create.c:333
#6 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 12 (Thread 0x7efe05e1a700 (LWP 24251)):
#0 0x00007efe1434a6ea in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x000055a0691049c6 in LinuxAIOHandler::collect (this=this@entry=0x7efe05e19ca0) at /data/src/10.3/storage/innobase/os/os0file.cc:1893
#2 0x000055a069104e24 in LinuxAIOHandler::poll (this=this@entry=0x7efe05e19ca0, m1=m1@entry=0x7efe05e19d40, m2=m2@entry=0x7efe05e19d48, request=request@entry=0x7efe05e19d50) at /data/src/10.3/storage/innobase/os/os0file.cc:2038
#3 0x000055a06910867b in os_aio_linux_handler (request=0x7efe05e19d50, m2=0x7efe05e19d48, m1=0x7efe05e19d40, global_segment=4) at /data/src/10.3/storage/innobase/os/os0file.cc:2092
#4 os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7efe05e19d40, m2=m2@entry=0x7efe05e19d48, request=request@entry=0x7efe05e19d50) at /data/src/10.3/storage/innobase/os/os0file.cc:5720
#5 0x000055a06927bee3 in fil_aio_wait (segment=segment@entry=4) at /data/src/10.3/storage/innobase/fil/fil0fil.cc:5317
#6 0x000055a069194e48 in io_handler_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0start.cc:340
#7 0x00007efe14553494 in start_thread (arg=0x7efe05e1a700) at pthread_create.c:333
#8 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 11 (Thread 0x7efdeaffd700 (LWP 24264)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000055a069109116 in wait (this=0x55a06abf25e0) at /data/src/10.3/storage/innobase/os/os0event.cc:166
#2 os_event::wait_low (this=0x55a06abf25e0, reset_sig_count=20, reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:336
#3 0x000055a0691095b5 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:535
#4 0x000055a06918ec73 in srv_resume_thread (slot=slot@entry=0x55a069c59368 <srv_sys+360>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:918
#5 0x000055a0691938ed in srv_worker_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2625
#6 0x00007efe14553494 in start_thread (arg=0x7efdeaffd700) at pthread_create.c:333
#7 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 10 (Thread 0x7efe0e6eb700 (LWP 24245)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x000055a069367c7e in inline_mysql_cond_timedwait (that=0x55a06a567500 <COND_timer>, mutex=0x55a06a567540 <LOCK_timer>, src_file=0x55a06951c2c8 "/data/src/10.3/mysys/thr_timer.c", src_line=292, abstime=0x7efe0e6eae70) at /data/src/10.3/include/mysql/psi/mysql_thread.h:1175
#2 timer_handler (arg=arg@entry=0x0) at /data/src/10.3/mysys/thr_timer.c:292
#3 0x000055a06907f8e4 in pfs_spawn_thread (arg=0x55a06aadcd58) at /data/src/10.3/storage/perfschema/pfs.cc:1863
#4 0x00007efe14553494 in start_thread (arg=0x7efe0e6eb700) at pthread_create.c:333
#5 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 9 (Thread 0x7efe077cd700 (LWP 24248)):
#0 0x00007efe1434a6ea in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x000055a0691049c6 in LinuxAIOHandler::collect (this=this@entry=0x7efe077ccca0) at /data/src/10.3/storage/innobase/os/os0file.cc:1893
#2 0x000055a069104e24 in LinuxAIOHandler::poll (this=this@entry=0x7efe077ccca0, m1=m1@entry=0x7efe077ccd40, m2=m2@entry=0x7efe077ccd48, request=request@entry=0x7efe077ccd50) at /data/src/10.3/storage/innobase/os/os0file.cc:2038
#3 0x000055a06910867b in os_aio_linux_handler (request=0x7efe077ccd50, m2=0x7efe077ccd48, m1=0x7efe077ccd40, global_segment=1) at /data/src/10.3/storage/innobase/os/os0file.cc:2092
#4 os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7efe077ccd40, m2=m2@entry=0x7efe077ccd48, request=request@entry=0x7efe077ccd50) at /data/src/10.3/storage/innobase/os/os0file.cc:5720
#5 0x000055a06927bee3 in fil_aio_wait (segment=segment@entry=1) at /data/src/10.3/storage/innobase/fil/fil0fil.cc:5317
#6 0x000055a069194e48 in io_handler_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0start.cc:340
#7 0x00007efe14553494 in start_thread (arg=0x7efe077cd700) at pthread_create.c:333
#8 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7efe0c04a700 (LWP 24269)):
#0 do_sigwait (set=<optimized out>, set@entry=0x7efe0c049e20, sig=sig@entry=0x7efe0c049dc8) at ../sysdeps/unix/sysv/linux/sigwait.c:64
#1 0x00007efe1455cced in __sigwait (set=0x7efe0c049e20, sig=0x7efe0c049dc8) at ../sysdeps/unix/sysv/linux/sigwait.c:96
#2 0x000055a068c30be3 in signal_hand (arg=arg@entry=0x0) at /data/src/10.3/sql/mysqld.cc:3519
#3 0x000055a06907f8e4 in pfs_spawn_thread (arg=0x55a06afdadb8) at /data/src/10.3/storage/perfschema/pfs.cc:1863
#4 0x00007efe14553494 in start_thread (arg=0x7efe0c04a700) at pthread_create.c:333
#5 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7efde9ffb700 (LWP 24266)):
#0 0x00007efe1455c7dd in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1 0x000055a06910977c in os_thread_sleep (tm=tm@entry=1000000) at /data/src/10.3/storage/innobase/os/os0thread.cc:230
#2 0x000055a069210f4a in btr_defragment_thread () at /data/src/10.3/storage/innobase/btr/btr0defragment.cc:772
#3 0x00007efe14553494 in start_thread (arg=0x7efde9ffb700) at pthread_create.c:333
#4 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7efdeb7fe700 (LWP 24263)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x000055a069109116 in wait (this=0x55a06abf24e0) at /data/src/10.3/storage/innobase/os/os0event.cc:166
#2 os_event::wait_low (this=0x55a06abf24e0, reset_sig_count=21, reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:336
#3 0x000055a0691095b5 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /data/src/10.3/storage/innobase/os/os0event.cc:535
#4 0x000055a06918ec73 in srv_resume_thread (slot=slot@entry=0x55a069c592f8 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:918
#5 0x000055a0691938ed in srv_worker_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2625
#6 0x00007efe14553494 in start_thread (arg=0x7efdeb7fe700) at pthread_create.c:333
#7 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7efdfe7fc700 (LWP 24257)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x000055a069108fee in os_event::timed_wait (this=this@entry=0x55a06abf26e0, abstime=abstime@entry=0x7efdfe7fbe90) at /data/src/10.3/storage/innobase/os/os0event.cc:286
#2 0x000055a069109241 in os_event::wait_time_low (this=0x55a06abf26e0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /data/src/10.3/storage/innobase/os/os0event.cc:413
#3 0x000055a0691095a5 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /data/src/10.3/storage/innobase/os/os0event.cc:518
#4 0x000055a069192cc4 in srv_monitor_thread () at /data/src/10.3/storage/innobase/srv/srv0srv.cc:1722
#5 0x00007efe14553494 in start_thread (arg=0x7efdfe7fc700) at pthread_create.c:333
#6 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7efe067cb700 (LWP 24250)):
#0 0x00007efe1434a6ea in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x000055a0691049c6 in LinuxAIOHandler::collect (this=this@entry=0x7efe067caca0) at /data/src/10.3/storage/innobase/os/os0file.cc:1893
#2 0x000055a069104e24 in LinuxAIOHandler::poll (this=this@entry=0x7efe067caca0, m1=m1@entry=0x7efe067cad40, m2=m2@entry=0x7efe067cad48, request=request@entry=0x7efe067cad50) at /data/src/10.3/storage/innobase/os/os0file.cc:2038
#3 0x000055a06910867b in os_aio_linux_handler (request=0x7efe067cad50, m2=0x7efe067cad48, m1=0x7efe067cad40, global_segment=3) at /data/src/10.3/storage/innobase/os/os0file.cc:2092
#4 os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7efe067cad40, m2=m2@entry=0x7efe067cad48, request=request@entry=0x7efe067cad50) at /data/src/10.3/storage/innobase/os/os0file.cc:5720
#5 0x000055a06927bee3 in fil_aio_wait (segment=segment@entry=3) at /data/src/10.3/storage/innobase/fil/fil0fil.cc:5317
#6 0x000055a069194e48 in io_handler_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0start.cc:340
#7 0x00007efe14553494 in start_thread (arg=0x7efe067cb700) at pthread_create.c:333
#8 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7efe07fce700 (LWP 24247)):
#0 0x00007efe1434a6ea in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
#1 0x000055a0691049c6 in LinuxAIOHandler::collect (this=this@entry=0x7efe07fcdca0) at /data/src/10.3/storage/innobase/os/os0file.cc:1893
#2 0x000055a069104e24 in LinuxAIOHandler::poll (this=this@entry=0x7efe07fcdca0, m1=m1@entry=0x7efe07fcdd40, m2=m2@entry=0x7efe07fcdd48, request=request@entry=0x7efe07fcdd50) at /data/src/10.3/storage/innobase/os/os0file.cc:2038
#3 0x000055a06910867b in os_aio_linux_handler (request=0x7efe07fcdd50, m2=0x7efe07fcdd48, m1=0x7efe07fcdd40, global_segment=0) at /data/src/10.3/storage/innobase/os/os0file.cc:2092
#4 os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7efe07fcdd40, m2=m2@entry=0x7efe07fcdd48, request=request@entry=0x7efe07fcdd50) at /data/src/10.3/storage/innobase/os/os0file.cc:5720
#5 0x000055a06927bee3 in fil_aio_wait (segment=segment@entry=0) at /data/src/10.3/storage/innobase/fil/fil0fil.cc:5317
#6 0x000055a069194e48 in io_handler_thread (arg=<optimized out>) at /data/src/10.3/storage/innobase/srv/srv0start.cc:340
#7 0x00007efe14553494 in start_thread (arg=0x7efe07fce700) at pthread_create.c:333
#8 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7efdff7fe700 (LWP 24255)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x000055a069108fee in os_event::timed_wait (this=this@entry=0x55a06b1f6fe0, abstime=abstime@entry=0x7efdff7fdd80) at /data/src/10.3/storage/innobase/os/os0event.cc:286
#2 0x000055a069109241 in os_event::wait_time_low (this=this@entry=0x55a06b1f6fe0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=1) at /data/src/10.3/storage/innobase/os/os0event.cc:413
#3 0x000055a0691095a5 in os_event_wait_time_low (event=event@entry=0x55a06b1f6fe0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=1) at /data/src/10.3/storage/innobase/os/os0event.cc:518
#4 0x000055a0690e2602 in lock_wait_timeout_thread () at /data/src/10.3/storage/innobase/lock/lock0wait.cc:529
#5 0x00007efe14553494 in start_thread (arg=0x7efdff7fe700) at pthread_create.c:333
#6 0x00007efe1293993f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7efe1496cb80 (LWP 24244)):
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x000055a068e860a4 in handle_fatal_signal (sig=24244) at /data/src/10.3/sql/signal_handler.cc:303
#2 <signal handler called>
#3 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#4 0x000055a068c3af1b in inline_mysql_cond_wait (src_file=<optimized out>, src_line=<optimized out>, mutex=<optimized out>, that=<optimized out>) at /data/src/10.3/include/mysql/psi/mysql_thread.h:1138
#5 mysqld_main (argc=149, argv=0x55a06a9a19a0) at /data/src/10.3/sql/mysqld.cc:6156
#6 0x00007efe128712b1 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x000055a068c2ddea in _start ()

Comment by Elena Stepanova [ 2017-12-01 ]

After fixing ~~MDEV-14080~~ in 10.2 and 10.3 the problem is expected to go away. We will monitor tests for a while, and if the symptoms don't appear anymore, will close it as a duplicate of ~~MDEV-14080~~.

Comment by Elena Stepanova [ 2017-12-15 ]

There was at least one occasion, on 10.3, Dec 10, 8f581e8bf1d400be08995b1cf8c11e3b0f7ae283 – which should be after ~~MDEV-14080~~ made it to 10.3:

Version: '10.3.3-MariaDB-debug-log'  socket: '/opt/buildbot-slave/mariadb/starfsmenn1/build/mysql-test/var/tmp/1/mysqld.1.sock'  port: 16020  Source distribution

2017-12-11  0:50:41 0 [Note] /opt/buildbot-slave/mariadb/starfsmenn1/build/sql/mysqld (root[root] @ localhost []): Normal shutdown

2017-12-11  0:50:41 0 [Note] InnoDB: FTS optimize thread exiting.

2017-12-11  0:50:41 0 [Note] Event Scheduler: Purging the queue. 0 events

171211  0:51:41 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.3.3-MariaDB-debug-log

key_buffer_size=1048576

read_buffer_size=131072

max_used_connections=1

max_threads=153

thread_count=2

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63188 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x0 thread_stack 0x49000

/opt/buildbot-slave/mariadb/starfsmenn1/build/sql/mysqld(my_print_stacktrace+0x40)[0x7fe22059b7df]

/opt/buildbot-slave/mariadb/starfsmenn1/build/sql/mysqld(handle_fatal_signal+0x3a3)[0x7fe21fdc5f08]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fe21e3de330]

/lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0xc4)[0x7fe21e3da404]

mysys/stacktrace.c:269(my_print_stacktrace)[0x7fe2205a0e78]

psi/mysql_thread.h:1138(inline_mysql_cond_wait)[0x7fe21f9fe9b4]

sql/mysqld.cc:6160(mysqld_main(int, char**))[0x7fe21fa09e50]

sql/main.cc:26(main)[0x7fe21f9fe2e5]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fe21d806f45]

/opt/buildbot-slave/mariadb/starfsmenn1/build/sql/mysqld(+0x55c1d9)[0x7fe21f9fe1d9]

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

It can, of course, be a different hang on shutdown. marko, what do you think?

Comment by Marko Mäkelä [ 2019-09-24 ]

I observed another innodb_fast_shutdown=0 hang today:

10.5 edef6a007428599fd249815f1dc59a02428090f4 with some local changes
encryption.innodb-force-corrupt 'ctr,innodb' w19 [ fail ] Found warnings/errors in server log file!
Test ended at 2019-09-24 15:16:33
line
Attempting backtrace. You can use the following information to find out
^ Found warnings in /dev/shm/10.5o/mysql-test/var/19/log/mysqld.1.err
…
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
56 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f8efb7a37c0 (LWP 8320))]
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x0000556bea14c967 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.5o/sql/signal_handler.cc:343
#2 <signal handler called>
#3 0x00007f8efb9e71d3 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7ffed73f3158) at ../sysdeps/unix/sysv/linux/select.c:41
#4 0x0000556bea8f057e in my_sleep (m_seconds=<optimized out>) at /mariadb/10.5o/mysys/my_sleep.c:29
#5 0x0000556be9df2353 in close_connections () at /mariadb/10.5o/sql/mysqld.cc:1797
#6 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.5o/sql/mysqld.cc:5763
#7 0x00007f8efb91bbbb in __libc_start_main (main=0x556be9ded0a0 <main(int, char**)>, argc=29, argv=0x7ffed73f33a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffed73f3398) at ../csu/libc-start.c:308
#8 0x0000556be9decfda in _start ()

Thread 19 (Thread 0x7f8ec1ffb700 (LWP 10539)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed304fc8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed304f78, cond=0x556bed304fa0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed304fa0, mutex=0x556bed304f78) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed304f60, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea745f74 in buf_resize_thread () at /mariadb/10.5o/storage/innobase/buf/buf0buf.cc:3208
#6 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f8ef486e700 (LWP 10376)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed387418) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed3873c8, cond=0x556bed3873f0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed3873f0, mutex=0x556bed3873c8) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed3873b0, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea548761 in os_aio_simulated_handler (global_segment=0, m1=0x7f8ef486dc50, m2=0x7f8ef486dc58, type=0x7f8ef486ddf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:7088
#6 os_aio_handler (segment=0, m1=<optimized out>, m2=<optimized out>, request=0x7f8ef486ddf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:5525
#7 0x0000556bea7f5f53 in fil_aio_wait (segment=0) at /mariadb/10.5o/storage/innobase/fil/fil0fil.cc:4361
#8 0x0000556bea664698 in io_handler_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0start.cc:324
#9 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f8ec27fc700 (LWP 10538)):
#0 0x00007f8efc580995 in __GI___nanosleep (requested_time=0x7f8ec27fb748, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000556bea5500ff in os_thread_sleep (tm=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0thread.cc:225
#2 0x0000556bea740b1a in btr_defragment_thread () at /mariadb/10.5o/storage/innobase/btr/btr0defragment.cc:703
#3 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#4 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f8ef4a6e700 (LWP 10594)):
#0 0x00007f8efb92fdd2 in __GI___sigtimedwait (set=0x7f8ef4a6dda8, info=0x7f8ef4a6dd28, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1 0x0000556be9df014b in my_sigwait (set=0x86005, sig=<optimized out>, code=<optimized out>) at /mariadb/10.5o/include/my_pthread.h:198
#2 signal_hand (arg=<optimized out>) at /mariadb/10.5o/sql/mysqld.cc:3088
#3 0x0000556bea423936 in pfs_spawn_thread (arg=0x556bed5b2e88) at /mariadb/10.5o/storage/perfschema/pfs.cc:1862
#4 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f8ec2ffd700 (LWP 10537)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed304ec8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed304e78, cond=0x556bed304ea0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed304ea0, mutex=0x556bed304e78) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed304e60, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea768dca in buf_dump_thread () at /mariadb/10.5o/storage/innobase/buf/buf0dump.cc:833
#6 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f8ecb7fe700 (LWP 10413)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f8ecb7fdc40, expected=0, futex_word=0x556bed304dc8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f8ecb7fdc40, mutex=0x556bed304d78, cond=0x556bed304da0) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x556bed304da0, mutex=0x556bed304d78, abstime=0x7f8ecb7fdc40) at pthread_cond_wait.c:667
#3 0x0000556bea54f56b in os_event::timed_wait (this=<optimized out>, abstime=0x7f8ecb7fdc40) at /mariadb/10.5o/storage/innobase/os/os0event.cc:275
#4 0x0000556bea54f8d6 in os_event::wait_time_low (this=0x556bed304d60, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:385
#5 0x0000556bea660f6d in srv_error_monitor_thread () at /mariadb/10.5o/storage/innobase/srv/srv0srv.cc:1879
#6 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f8eeffff700 (LWP 10378)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed387498) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed387448, cond=0x556bed387470) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed387470, mutex=0x556bed387448) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed387430, reset_sig_count=5) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea548761 in os_aio_simulated_handler (global_segment=1, m1=0x7f8eefffec50, m2=0x7f8eefffec58, type=0x7f8eefffedf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:7088
#6 os_aio_handler (segment=1, m1=<optimized out>, m2=<optimized out>, request=0x7f8eefffedf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:5525
#7 0x0000556bea7f5f53 in fil_aio_wait (segment=1) at /mariadb/10.5o/storage/innobase/fil/fil0fil.cc:4361
#8 0x0000556bea664698 in io_handler_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0start.cc:324
#9 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f8ef73d9700 (LWP 8465)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f8ef73d8de0, expected=0, futex_word=0x556bebb11308 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f8ef73d8de0, mutex=0x556bebb11258 <LOCK_timer+40>, cond=0x556bebb112e0 <COND_timer>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x556bebb112e0 <COND_timer>, mutex=0x556bebb11258 <LOCK_timer+40>, abstime=0x7f8ef73d8de0) at pthread_cond_wait.c:667
#3 0x0000556bea8f8e44 in safe_cond_timedwait (cond=0x556bebb112e0 <COND_timer>, mp=0x556bebb11230 <LOCK_timer>, abstime=0x7f8ef73d8de0, file=0x556bea97be7f "/mariadb/10.5o/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.5o/mysys/thr_mutex.c:546
#4 0x0000556bea8f9924 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=<optimized out>, src_line=292, src_file=<optimized out>) at /mariadb/10.5o/include/mysql/psi/mysql_thread.h:1204
#5 timer_handler (arg=<optimized out>) at /mariadb/10.5o/mysys/thr_timer.c:292
#6 0x0000556bea423936 in pfs_spawn_thread (arg=0x556bed20ed78) at /mariadb/10.5o/storage/perfschema/pfs.cc:1862
#7 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f8ec8ff9700 (LWP 10454)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed304d48) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed304cf8, cond=0x556bed304d20) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed304d20, mutex=0x556bed304cf8) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed304ce0, reset_sig_count=787) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea662950 in srv_resume_thread (slot=0x556beb1e89b0 <srv_sys+560>, sig_count=0, wait=<optimized out>, timeout_usec=0) at /mariadb/10.5o/storage/innobase/srv/srv0srv.cc:898
#6 0x0000556bea662c29 in srv_worker_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0srv.cc:2536
#7 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f8eecdf4700 (LWP 10452)):
#0 0x00007f8efc580995 in __GI___nanosleep (requested_time=0x7f8eecdf3c48, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000556bea5500ff in os_thread_sleep (tm=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0thread.cc:225
#2 0x0000556bea66171e in srv_master_sleep () at /mariadb/10.5o/storage/innobase/srv/srv0srv.cc:2362
#3 srv_master_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0srv.cc:2403
#4 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f8ecbfff700 (LWP 10412)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f8ecbffed80, expected=0, futex_word=0x556bed4b6b78) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f8ecbffed80, mutex=0x556bed4b6b28, cond=0x556bed4b6b50) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x556bed4b6b50, mutex=0x556bed4b6b28, abstime=0x7f8ecbffed80) at pthread_cond_wait.c:667
#3 0x0000556bea54f56b in os_event::timed_wait (this=<optimized out>, abstime=0x7f8ecbffed80) at /mariadb/10.5o/storage/innobase/os/os0event.cc:275
#4 0x0000556bea54f8d6 in os_event::wait_time_low (this=0x556bed4b6b10, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:385
#5 0x0000556bea51b980 in lock_wait_timeout_thread () at /mariadb/10.5o/storage/innobase/lock/lock0wait.cc:522
#6 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f8ecaffd700 (LWP 10414)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f8ecaffce00, expected=0, futex_word=0x556bed304e48) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f8ecaffce00, mutex=0x556bed304df8, cond=0x556bed304e20) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x556bed304e20, mutex=0x556bed304df8, abstime=0x7f8ecaffce00) at pthread_cond_wait.c:667
#3 0x0000556bea54f56b in os_event::timed_wait (this=<optimized out>, abstime=0x7f8ecaffce00) at /mariadb/10.5o/storage/innobase/os/os0event.cc:275
#4 0x0000556bea54f8d6 in os_event::wait_time_low (this=0x556bed304de0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:385
#5 0x0000556bea6609d0 in srv_monitor_thread () at /mariadb/10.5o/storage/innobase/srv/srv0srv.cc:1715
#6 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f8eee7fc700 (LWP 10381)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed387618) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed3875c8, cond=0x556bed3875f0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed3875f0, mutex=0x556bed3875c8) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed3875b0, reset_sig_count=3) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea548761 in os_aio_simulated_handler (global_segment=4, m1=0x7f8eee7fbc50, m2=0x7f8eee7fbc58, type=0x7f8eee7fbdf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:7088
#6 os_aio_handler (segment=4, m1=<optimized out>, m2=<optimized out>, request=0x7f8eee7fbdf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:5525
#7 0x0000556bea7f5f53 in fil_aio_wait (segment=4) at /mariadb/10.5o/storage/innobase/fil/fil0fil.cc:4361
#8 0x0000556bea664698 in io_handler_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0start.cc:324
#9 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f8eed7fa700 (LWP 10384)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f8eed7f9ba0, expected=0, futex_word=0x556bed304f4c) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f8eed7f9ba0, mutex=0x556bed304ef8, cond=0x556bed304f20) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x556bed304f20, mutex=0x556bed304ef8, abstime=0x7f8eed7f9ba0) at pthread_cond_wait.c:667
#3 0x0000556bea54f56b in os_event::timed_wait (this=<optimized out>, abstime=0x7f8eed7f9ba0) at /mariadb/10.5o/storage/innobase/os/os0event.cc:275
#4 0x0000556bea54f8d6 in os_event::wait_time_low (this=0x556bed304ee0, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/10.5o/storage/innobase/os/os0event.cc:385
#5 0x0000556bea773e53 in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=<optimized out>, cur_time=<optimized out>) at /mariadb/10.5o/storage/innobase/buf/buf0flu.cc:2643
#6 buf_flush_page_cleaner_coordinator () at /mariadb/10.5o/storage/innobase/buf/buf0flu.cc:3102
#7 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f8eeeffd700 (LWP 10380)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed387598) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed387548, cond=0x556bed387570) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed387570, mutex=0x556bed387548) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed387530, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea548761 in os_aio_simulated_handler (global_segment=3, m1=0x7f8eeeffcc50, m2=0x7f8eeeffcc58, type=0x7f8eeeffcdf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:7088
#6 os_aio_handler (segment=3, m1=<optimized out>, m2=<optimized out>, request=0x7f8eeeffcdf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:5525
#7 0x0000556bea7f5f53 in fil_aio_wait (segment=3) at /mariadb/10.5o/storage/innobase/fil/fil0fil.cc:4361
#8 0x0000556bea664698 in io_handler_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0start.cc:324
#9 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f8eedffb700 (LWP 10383)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed38769c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed387648, cond=0x556bed387670) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed387670, mutex=0x556bed387648) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed387630, reset_sig_count=2) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea548761 in os_aio_simulated_handler (global_segment=5, m1=0x7f8eedffac50, m2=0x7f8eedffac58, type=0x7f8eedffadf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:7088
#6 os_aio_handler (segment=5, m1=<optimized out>, m2=<optimized out>, request=0x7f8eedffadf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:5525
#7 0x0000556bea7f5f53 in fil_aio_wait (segment=5) at /mariadb/10.5o/storage/innobase/fil/fil0fil.cc:4361
#8 0x0000556bea664698 in io_handler_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0start.cc:324
#9 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f8eef7fe700 (LWP 10379)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x556bed387518) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x556bed3874c8, cond=0x556bed3874f0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x556bed3874f0, mutex=0x556bed3874c8) at pthread_cond_wait.c:655
#3 0x0000556bea54f719 in os_event::wait (this=<optimized out>) at /mariadb/10.5o/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x556bed3874b0, reset_sig_count=1) at /mariadb/10.5o/storage/innobase/os/os0event.cc:325
#5 0x0000556bea548761 in os_aio_simulated_handler (global_segment=2, m1=0x7f8eef7fdc50, m2=0x7f8eef7fdc58, type=0x7f8eef7fddf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:7088
#6 os_aio_handler (segment=2, m1=<optimized out>, m2=<optimized out>, request=0x7f8eef7fddf0) at /mariadb/10.5o/storage/innobase/os/os0file.cc:5525
#7 0x0000556bea7f5f53 in fil_aio_wait (segment=2) at /mariadb/10.5o/storage/innobase/fil/fil0fil.cc:4361
#8 0x0000556bea664698 in io_handler_thread (arg=<optimized out>) at /mariadb/10.5o/storage/innobase/srv/srv0start.cc:324
#9 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f8ef6070700 (LWP 8475)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f8ef606fd50, expected=0, futex_word=0x556bebaa3098 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f8ef606fd50, mutex=0x556bebaa2fe0 <LOCK_checkpoint+40>, cond=0x556bebaa3070 <COND_checkpoint>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x556bebaa3070 <COND_checkpoint>, mutex=0x556bebaa2fe0 <LOCK_checkpoint+40>, abstime=0x7f8ef606fd50) at pthread_cond_wait.c:667
#3 0x0000556bea8f8e44 in safe_cond_timedwait (cond=0x556bebaa3070 <COND_checkpoint>, mp=0x556bebaa2fb8 <LOCK_checkpoint>, abstime=0x7f8ef606fd50, file=0x556bea97be7f "/mariadb/10.5o/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.5o/mysys/thr_mutex.c:546
#4 0x0000556bea37f505 in inline_mysql_cond_timedwait (that=0x556bebaa3070 <COND_checkpoint>, mutex=0x556bebaa2fb8 <LOCK_checkpoint>, abstime=<optimized out>, src_line=116, src_file=<optimized out>) at /mariadb/10.5o/include/mysql/psi/mysql_thread.h:1204
#5 my_service_thread_sleep (control=0x556beb1dcf10 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/10.5o/storage/maria/ma_servicethread.c:115
#6 0x0000556bea376ac5 in ma_checkpoint_background (arg=<optimized out>) at /mariadb/10.5o/storage/maria/ma_checkpoint.c:707
#7 0x0000556bea423936 in pfs_spawn_thread (arg=0x556bed2bee28) at /mariadb/10.5o/storage/perfschema/pfs.cc:1862
#8 0x00007f8efc576fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f8efb9ef2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f8efb7a37c0 (LWP 8320)):
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x0000556bea14c967 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.5o/sql/signal_handler.cc:343
#2 <signal handler called>
#3 0x00007f8efb9e71d3 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7ffed73f3158) at ../sysdeps/unix/sysv/linux/select.c:41
#4 0x0000556bea8f057e in my_sleep (m_seconds=<optimized out>) at /mariadb/10.5o/mysys/my_sleep.c:29
#5 0x0000556be9df2353 in close_connections () at /mariadb/10.5o/sql/mysqld.cc:1797
#6 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.5o/sql/mysqld.cc:5763
#7 0x00007f8efb91bbbb in __libc_start_main (main=0x556be9ded0a0 <main(int, char**)>, argc=29, argv=0x7ffed73f33a8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffed73f3398) at ../csu/libc-start.c:308
#8 0x0000556be9decfda in _start ()

Notably, we have exactly one srv_worker_thread running. The coordinator exited, but two worker threads did: srv_n_purge_threads was at the default value 4.

The thd_destructor_proxy did exit, and we have srv_undo_sources=false as expected. Apparently, the shutdown is hung because the purge worker thread is holding a THD. Only after all THD were released, innobase_end() could be invoked.

The problem appears to be this function:

/** Shut down the purge threads. */

void srv_purge_shutdown()

	do {

		ut_ad(!srv_undo_sources);

		srv_purge_wakeup();

	} while (srv_sys.sys_threads[SRV_PURGE_SLOT].in_use);

This is only making sure that the purge coordinator thread (SRV_PURGE_SLOT=1) ends. The purge worker thread is using slot 4, which was allocated dynamically. srv_sys.tasks is empty, and apparently nobody will wake up the worker thread, which is using the last nonzero slot:

(gdb) p srv_sys.sys_threads

$10 = {{type = SRV_MASTER, in_use = 1, suspended = 0, suspend_time = 0, wait_timeout = 0, event = 0x556bed304ae0, thr = 0x0}, {type = SRV_PURGE, in_use = 0, suspended = 1, suspend_time = 0, wait_timeout = 0,

    event = 0x556bed304b60, thr = 0x0}, {type = SRV_WORKER, in_use = 0, suspended = 1, suspend_time = 0, wait_timeout = 0, event = 0x556bed304be0, thr = 0x0}, {type = SRV_WORKER, in_use = 0, suspended = 1,

    suspend_time = 0, wait_timeout = 0, event = 0x556bed304c60, thr = 0x0}, {type = SRV_WORKER, in_use = 1, suspended = 1, suspend_time = 0, wait_timeout = 0, event = 0x556bed304ce0, thr = 0x0}, {

    type = SRV_NONE, in_use = 0, suspended = 0, suspend_time = 0, wait_timeout = 0, event = 0x0, thr = 0x0} <repeats 28 times>}

We have srv_sys.n_threads_active[3]==1, and all other counts are 0.

srv_purge_wakeup() should have been invoking srv_release_threads(SRV_WORKER, n_workers=4) during the loop. Either that call is ‘leaking’, or we should use a more accurate termination condition in srv_purge_shutdown(). We want a condition that cannot ‘oscillate’.

Comment by Marko Mäkelä [ 2019-09-26 ]

I just experienced the hang without innodb_fast_shutdown=0:

10.4 e3c39c0be82e69acb813e7c5e0f2aea7d5546e31 with minor changes
innodb_zip.innochecksum_3 '4k,crc32,innodb' w27 [ fail ] Found warnings/errors in server log file!
Test ended at 2019-09-26 18:36:24
line
Attempting backtrace. You can use the following information to find out
…
Version: '10.4.9-MariaDB-debug-log' socket: '/dev/shm/10.4/mysql-test/var/tmp/27/mysqld.1.sock' port: 16480 Source distribution
2019-09-26 18:35:19 0 [Note] /dev/shm/10.4/sql/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2019-09-26 18:35:19 0 [Note] InnoDB: FTS optimize thread exiting.
2019-09-26 18:35:19 0 [Note] Event Scheduler: Purging the queue. 0 events
190926 18:36:19 [ERROR] mysqld got signal 6 ;
…
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
56 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f989efc0c00 (LWP 19674))]
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x0000555f7182b1c7 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.4/sql/signal_handler.cc:343
#2 <signal handler called>
#3 0x00007f989f2031d3 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7ffd1c9f2998) at ../sysdeps/unix/sysv/linux/select.c:41
#4 0x0000555f71fb09ee in my_sleep (m_seconds=<optimized out>) at /mariadb/10.4/mysys/my_sleep.c:29
#5 0x0000555f714b59f3 in close_connections () at /mariadb/10.4/sql/mysqld.cc:1798
#6 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.4/sql/mysqld.cc:5921
#7 0x00007f989f137bbb in __libc_start_main (main=0x555f714b0340 <main(int, char**)>, argc=27, argv=0x7ffd1c9f2f58, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd1c9f2f48) at ../csu/libc-start.c:308
#8 0x0000555f714b027a in _start ()

Thread 19 (Thread 0x7f98717fa700 (LWP 20711)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x555f73c35d78) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x555f73c35d28, cond=0x555f73c35d50) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x555f73c35d50, mutex=0x555f73c35d28) at pthread_cond_wait.c:655
#3 0x0000555f71b097e9 in os_event::wait (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x555f73c35d10, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:325
#5 0x0000555f71d00fe4 in buf_resize_thread () at /mariadb/10.4/storage/innobase/buf/buf0buf.cc:3207
#6 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f9898288700 (LWP 20731)):
#0 0x00007f989f14bdd2 in __GI___sigtimedwait (set=0x7f9898287da8, info=0x7f9898287d28, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1 0x0000555f714b37db in my_sigwait (set=0x86005, sig=<optimized out>, code=<optimized out>) at /mariadb/10.4/include/my_pthread.h:198
#2 signal_hand (arg=<optimized out>) at /mariadb/10.4/sql/mysqld.cc:3250
#3 0x0000555f71f54f16 in pfs_spawn_thread (arg=0x555f7400e928) at /mariadb/10.4/storage/perfschema/pfs.cc:1862
#4 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f98917fa700 (LWP 20492)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f989fafc27a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000555f71afd3de in LinuxAIOHandler::collect (this=0x7f98917f9bb0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1809
#3 0x0000555f71afdad1 in LinuxAIOHandler::poll (this=0x7f98917f9bb0, m1=0x7f98917f9c50, m2=0x7f98917f9c58, request=0x7f98917f9df0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1968
#4 0x0000555f71b02649 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f98917f9c50, m2=0x7f98917f9c58, request=0x7f98917f9df0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2022
#5 os_aio_handler (segment=<optimized out>, m1=0x7f98917f9c50, m2=0x7f98917f9c58, request=0x7f98917f9df0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5561
#6 0x0000555f71db0883 in fil_aio_wait (segment=5) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4357
#7 0x0000555f71c1ef48 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f9871ffb700 (LWP 20710)):
#0 0x00007f989fd9c995 in __GI___nanosleep (requested_time=0x7f9871ffa748, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000555f71b0a1cf in os_thread_sleep (tm=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0thread.cc:225
#2 0x0000555f71cfbb8a in btr_defragment_thread () at /mariadb/10.4/storage/innobase/btr/btr0defragment.cc:705
#3 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#4 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f98737fe700 (LWP 20636)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x555f73c35af8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x555f73c35aa8, cond=0x555f73c35ad0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x555f73c35ad0, mutex=0x555f73c35aa8) at pthread_cond_wait.c:655
#3 0x0000555f71b097e9 in os_event::wait (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x555f73c35a90, reset_sig_count=17) at /mariadb/10.4/storage/innobase/os/os0event.cc:325
#5 0x0000555f71c1d200 in srv_resume_thread (slot=0x555f72895db0 <srv_sys+560>, sig_count=0, wait=<optimized out>, timeout_usec=0) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:904
#6 0x0000555f71c1d4d9 in srv_worker_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2503
#7 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f98727fc700 (LWP 20709)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x555f73c35c78) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x555f73c35c28, cond=0x555f73c35c50) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x555f73c35c50, mutex=0x555f73c35c28) at pthread_cond_wait.c:655
#3 0x0000555f71b097e9 in os_event::wait (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x555f73c35c10, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:325
#5 0x0000555f71d23dfa in buf_dump_thread () at /mariadb/10.4/storage/innobase/buf/buf0dump.cc:833
#6 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f989abf4700 (LWP 19914)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f989abf3de0, expected=0, futex_word=0x555f731b4af8 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f989abf3de0, mutex=0x555f731b4a48 <LOCK_timer+40>, cond=0x555f731b4ad0 <COND_timer>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x555f731b4ad0 <COND_timer>, mutex=0x555f731b4a48 <LOCK_timer+40>, abstime=0x7f989abf3de0) at pthread_cond_wait.c:667
#3 0x0000555f71fb91f4 in safe_cond_timedwait (cond=0x555f731b4ad0 <COND_timer>, mp=0x555f731b4a20 <LOCK_timer>, abstime=0x7f989abf3de0, file=0x555f72032cc8 "/mariadb/10.4/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.4/mysys/thr_mutex.c:546
#4 0x0000555f71fb9cd4 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=<optimized out>, src_line=292, src_file=<optimized out>) at /mariadb/10.4/include/mysql/psi/mysql_thread.h:1204
#5 timer_handler (arg=<optimized out>) at /mariadb/10.4/mysys/thr_timer.c:292
#6 0x0000555f71f54f16 in pfs_spawn_thread (arg=0x555f73b41828) at /mariadb/10.4/storage/perfschema/pfs.cc:1862
#7 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f98937fe700 (LWP 20488)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f989fafc27a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000555f71afd3de in LinuxAIOHandler::collect (this=0x7f98937fdbb0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1809
#3 0x0000555f71afdad1 in LinuxAIOHandler::poll (this=0x7f98937fdbb0, m1=0x7f98937fdc50, m2=0x7f98937fdc58, request=0x7f98937fddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1968
#4 0x0000555f71b02649 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f98937fdc50, m2=0x7f98937fdc58, request=0x7f98937fddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2022
#5 os_aio_handler (segment=<optimized out>, m1=0x7f98937fdc50, m2=0x7f98937fdc58, request=0x7f98937fddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5561
#6 0x0000555f71db0883 in fil_aio_wait (segment=1) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4357
#7 0x0000555f71c1ef48 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f9892ffd700 (LWP 20489)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f989fafc27a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000555f71afd3de in LinuxAIOHandler::collect (this=0x7f9892ffcbb0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1809
#3 0x0000555f71afdad1 in LinuxAIOHandler::poll (this=0x7f9892ffcbb0, m1=0x7f9892ffcc50, m2=0x7f9892ffcc58, request=0x7f9892ffcdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1968
#4 0x0000555f71b02649 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f9892ffcc50, m2=0x7f9892ffcc58, request=0x7f9892ffcdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2022
#5 os_aio_handler (segment=<optimized out>, m1=0x7f9892ffcc50, m2=0x7f9892ffcc58, request=0x7f9892ffcdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5561
#6 0x0000555f71db0883 in fil_aio_wait (segment=2) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4357
#7 0x0000555f71c1ef48 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f98857fa700 (LWP 20631)):
#0 0x00007f989fd9c995 in __GI___nanosleep (requested_time=0x7f98857f9c48, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000555f71b0a1cf in os_thread_sleep (tm=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0thread.cc:225
#2 0x0000555f71c1bfce in srv_master_sleep () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2329
#3 srv_master_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2370
#4 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f98867fc700 (LWP 20522)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f98867fbe00, expected=0, futex_word=0x555f73c35bf8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f98867fbe00, mutex=0x555f73c35ba8, cond=0x555f73c35bd0) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x555f73c35bd0, mutex=0x555f73c35ba8, abstime=0x7f98867fbe00) at pthread_cond_wait.c:667
#3 0x0000555f71b0963b in os_event::timed_wait (this=<optimized out>, abstime=0x7f98867fbe00) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000555f71b099a6 in os_event::wait_time_low (this=0x555f73c35b90, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000555f71c1b280 in srv_monitor_thread () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:1682
#6 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f9886ffd700 (LWP 20520)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f9886ffcc40, expected=0, futex_word=0x555f73c35b78) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f9886ffcc40, mutex=0x555f73c35b28, cond=0x555f73c35b50) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x555f73c35b50, mutex=0x555f73c35b28, abstime=0x7f9886ffcc40) at pthread_cond_wait.c:667
#3 0x0000555f71b0963b in os_event::timed_wait (this=<optimized out>, abstime=0x7f9886ffcc40) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000555f71b099a6 in os_event::wait_time_low (this=0x555f73c35b10, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000555f71c1b81d in srv_error_monitor_thread () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:1846
#6 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f98877fe700 (LWP 20516)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f98877fdd80, expected=0, futex_word=0x555f73f06c48) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f98877fdd80, mutex=0x555f73f06bf8, cond=0x555f73f06c20) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x555f73f06c20, mutex=0x555f73f06bf8, abstime=0x7f98877fdd80) at pthread_cond_wait.c:667
#3 0x0000555f71b0963b in os_event::timed_wait (this=<optimized out>, abstime=0x7f98877fdd80) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000555f71b099a6 in os_event::wait_time_low (this=0x555f73f06be0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000555f71ad61a0 in lock_wait_timeout_thread () at /mariadb/10.4/storage/innobase/lock/lock0wait.cc:522
#6 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f9890ff9700 (LWP 20493)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f9890ff8ba0, expected=0, futex_word=0x555f73c35cfc) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f9890ff8ba0, mutex=0x555f73c35ca8, cond=0x555f73c35cd0) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x555f73c35cd0, mutex=0x555f73c35ca8, abstime=0x7f9890ff8ba0) at pthread_cond_wait.c:667
#3 0x0000555f71b0963b in os_event::timed_wait (this=<optimized out>, abstime=0x7f9890ff8ba0) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000555f71b099a6 in os_event::wait_time_low (this=0x555f73c35c90, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000555f71d2ee63 in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=<optimized out>, cur_time=<optimized out>) at /mariadb/10.4/storage/innobase/buf/buf0flu.cc:2640
#6 buf_flush_page_cleaner_coordinator () at /mariadb/10.4/storage/innobase/buf/buf0flu.cc:3099
#7 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f98927fc700 (LWP 20490)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f989fafc27a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000555f71afd3de in LinuxAIOHandler::collect (this=0x7f98927fbbb0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1809
#3 0x0000555f71afdad1 in LinuxAIOHandler::poll (this=0x7f98927fbbb0, m1=0x7f98927fbc50, m2=0x7f98927fbc58, request=0x7f98927fbdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1968
#4 0x0000555f71b02649 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f98927fbc50, m2=0x7f98927fbc58, request=0x7f98927fbdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2022
#5 os_aio_handler (segment=<optimized out>, m1=0x7f98927fbc50, m2=0x7f98927fbc58, request=0x7f98927fbdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5561
#6 0x0000555f71db0883 in fil_aio_wait (segment=3) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4357
#7 0x0000555f71c1ef48 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f9891ffb700 (LWP 20491)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f989fafc27a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000555f71afd3de in LinuxAIOHandler::collect (this=0x7f9891ffabb0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1809
#3 0x0000555f71afdad1 in LinuxAIOHandler::poll (this=0x7f9891ffabb0, m1=0x7f9891ffac50, m2=0x7f9891ffac58, request=0x7f9891ffadf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1968
#4 0x0000555f71b02649 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f9891ffac50, m2=0x7f9891ffac58, request=0x7f9891ffadf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2022
#5 os_aio_handler (segment=<optimized out>, m1=0x7f9891ffac50, m2=0x7f9891ffac58, request=0x7f9891ffadf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5561
#6 0x0000555f71db0883 in fil_aio_wait (segment=4) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4357
#7 0x0000555f71c1ef48 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f9893fff700 (LWP 20487)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f989fafc27a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000555f71afd3de in LinuxAIOHandler::collect (this=0x7f9893ffebb0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1809
#3 0x0000555f71afdad1 in LinuxAIOHandler::poll (this=0x7f9893ffebb0, m1=0x7f9893ffec50, m2=0x7f9893ffec58, request=0x7f9893ffedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1968
#4 0x0000555f71b02649 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f9893ffec50, m2=0x7f9893ffec58, request=0x7f9893ffedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2022
#5 os_aio_handler (segment=<optimized out>, m1=0x7f9893ffec50, m2=0x7f9893ffec58, request=0x7f9893ffedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5561
#6 0x0000555f71db0883 in fil_aio_wait (segment=0) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4357
#7 0x0000555f71c1ef48 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f989988a700 (LWP 19924)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f9899889d50, expected=0, futex_word=0x555f731a2778 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f9899889d50, mutex=0x555f731a26c0 <LOCK_checkpoint+40>, cond=0x555f731a2750 <COND_checkpoint>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x555f731a2750 <COND_checkpoint>, mutex=0x555f731a26c0 <LOCK_checkpoint+40>, abstime=0x7f9899889d50) at pthread_cond_wait.c:667
#3 0x0000555f71fb91f4 in safe_cond_timedwait (cond=0x555f731a2750 <COND_checkpoint>, mp=0x555f731a2698 <LOCK_checkpoint>, abstime=0x7f9899889d50, file=0x555f72032cc8 "/mariadb/10.4/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.4/mysys/thr_mutex.c:546
#4 0x0000555f71e80885 in inline_mysql_cond_timedwait (that=0x555f731a2750 <COND_checkpoint>, mutex=0x555f731a2698 <LOCK_checkpoint>, abstime=<optimized out>, src_line=116, src_file=<optimized out>) at /mariadb/10.4/include/mysql/psi/mysql_thread.h:1204
#5 my_service_thread_sleep (control=0x555f72899790 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/10.4/storage/maria/ma_servicethread.c:115
#6 0x0000555f71e77e85 in ma_checkpoint_background (arg=<optimized out>) at /mariadb/10.4/storage/maria/ma_checkpoint.c:707
#7 0x0000555f71f54f16 in pfs_spawn_thread (arg=0x555f73bed818) at /mariadb/10.4/storage/perfschema/pfs.cc:1862
#8 0x00007f989fd92fb7 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f989f20b2ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f989efc0c00 (LWP 19674)):
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x0000555f7182b1c7 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.4/sql/signal_handler.cc:343
#2 <signal handler called>
#3 0x00007f989f2031d3 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7ffd1c9f2998) at ../sysdeps/unix/sysv/linux/select.c:41
#4 0x0000555f71fb09ee in my_sleep (m_seconds=<optimized out>) at /mariadb/10.4/mysys/my_sleep.c:29
#5 0x0000555f714b59f3 in close_connections () at /mariadb/10.4/sql/mysqld.cc:1798
#6 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.4/sql/mysqld.cc:5921
#7 0x00007f989f137bbb in __libc_start_main (main=0x555f714b0340 <main(int, char**)>, argc=27, argv=0x7ffd1c9f2f58, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd1c9f2f48) at ../csu/libc-start.c:308
#8 0x0000555f714b027a in _start ()

Also in this case, only one srv_worker_thread exists, and thd_destructor_proxy() had completed execution.

Comment by Marko Mäkelä [ 2020-05-05 ]

Another occurrence on 10.4, a hang during the slave shutdown of the test rpl.rpl_upgrade_master_info,stmt:

10.4 37a01aceca9a79c48e8d066dee55a1c321d322b6 with a merge of 10.3 8648b9bed86e9f52c027daec760d6ab5ce52e889
2020-05-05 11:50:38 0 [Note] /dev/shm/10.4/sql/mysqld (initiated by: root[root] @ localhost [127.0.0.1]): Normal shutdown
2020-05-05 11:50:38 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-05-05 11:50:38 0 [Note] InnoDB: FTS optimize thread exiting.
200505 11:51:38 [ERROR] mysqld got signal 6 ;
…
Thread 19 (Thread 0x7fdb9affd700 (LWP 1556478)):
#0 0x00007fdbbd98f575 in __GI___nanosleep (requested_time=0x7fdb9affcc48, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000562f6c1c09cf in os_thread_sleep (tm=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0thread.cc:231
#2 0x0000562f6c2d6a6b in srv_master_sleep () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2336
#3 srv_master_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2377
#4 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#5 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7fdb7f7fe700 (LWP 1556553)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x562f6e087708) at ../sysdeps/unix/sysv/linux/futex-internal.h:80
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x562f6e0876b8, cond=0x562f6e0876e0) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x562f6e0876e0, mutex=0x562f6e0876b8) at pthread_cond_wait.c:638
#3 0x0000562f6c1bfff9 in os_event::wait (this=0x562f6e0876a0) at /mariadb/10.4/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x562f6e0876a0, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:325
#5 0x0000562f6c1c0655 in os_event_wait_low (event=0x562f6e087708, reset_sig_count=128) at /mariadb/10.4/storage/innobase/os/os0event.cc:502
#6 0x0000562f6c3b2ac0 in buf_resize_thread () at /mariadb/10.4/storage/innobase/buf/buf0buf.cc:3160
#7 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#8 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7fdbb1fa5700 (LWP 1556578)):
#0 0x00007fdbbcf254b2 in __GI___sigtimedwait (set=0x7fdbb1fa4da8, info=0x7fdbb1fa4d28, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1 0x0000562f6bb6ddfb in my_sigwait (set=0x7fdbb1fa4da8, sig=0x7fdbb1fa4cf0, code=<optimized out>) at /mariadb/10.4/include/my_pthread.h:196
#2 signal_hand (arg=<optimized out>) at /mariadb/10.4/sql/mysqld.cc:3254
#3 0x0000562f6c605146 in pfs_spawn_thread (arg=0x562f6e3a7418) at /mariadb/10.4/storage/perfschema/pfs.cc:1869
#4 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#5 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7fdb9bfff700 (LWP 1556467)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7fdb9bffee00, clockid=<optimized out>, expected=0, futex_word=0x562f6e087588) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
#1 __pthread_cond_wait_common (abstime=0x7fdb9bffee00, clockid=<optimized out>, mutex=0x562f6e087538, cond=0x562f6e087560) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x562f6e087560, mutex=0x562f6e087538, abstime=0x7fdb9bffee00) at pthread_cond_wait.c:656
#3 0x0000562f6c1bfe4b in os_event::timed_wait (this=<optimized out>, abstime=0x7fdb9bffee00) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000562f6c1c01a6 in os_event::wait_time_low (this=0x562f6e087520, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000562f6c1c0645 in os_event_wait_time_low (event=0x562f6e087588, time_in_usec=393, reset_sig_count=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:485
#6 0x0000562f6c2d5d64 in srv_monitor_thread () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:1689
#7 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#8 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7fdb77fff700 (LWP 1556552)):
#0 0x00007fdbbd98f575 in __GI___nanosleep (requested_time=0x7fdb77ffe788, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x0000562f6c1c09cf in os_thread_sleep (tm=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0thread.cc:231
#2 0x0000562f6c3ad46a in btr_defragment_thread () at /mariadb/10.4/storage/innobase/btr/btr0defragment.cc:705
#3 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#4 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7fdb7ffff700 (LWP 1556551)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x562f6e087608) at ../sysdeps/unix/sysv/linux/futex-internal.h:80
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x562f6e0875b8, cond=0x562f6e0875e0) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x562f6e0875e0, mutex=0x562f6e0875b8) at pthread_cond_wait.c:638
#3 0x0000562f6c1bfff9 in os_event::wait (this=0x562f6e0875a0) at /mariadb/10.4/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x562f6e0875a0, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:325
#5 0x0000562f6c1c0655 in os_event_wait_low (event=0x562f6e087608, reset_sig_count=128) at /mariadb/10.4/storage/innobase/os/os0event.cc:502
#6 0x0000562f6c3d5f80 in buf_dump_thread () at /mariadb/10.4/storage/innobase/buf/buf0dump.cc:832
#7 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#8 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7fdb98ff9700 (LWP 1556482)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x562f6e08748c) at ../sysdeps/unix/sysv/linux/futex-internal.h:80
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x562f6e087438, cond=0x562f6e087460) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x562f6e087460, mutex=0x562f6e087438) at pthread_cond_wait.c:638
#3 0x0000562f6c1bfff9 in os_event::wait (this=0x562f6e087420) at /mariadb/10.4/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x562f6e087420, reset_sig_count=26) at /mariadb/10.4/storage/innobase/os/os0event.cc:325
#5 0x0000562f6c1c0655 in os_event_wait_low (event=0x562f6e08748c, reset_sig_count=128) at /mariadb/10.4/storage/innobase/os/os0event.cc:502
#6 0x0000562f6c2d7d20 in srv_resume_thread (slot=0x562f6cf722d0 <srv_sys+1360>, sig_count=0, wait=true, timeout_usec=0) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:911
#7 0x0000562f6c2d80e6 in srv_worker_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2519
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7fdba8ff9700 (LWP 1556465)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7fdba8ff8c40, clockid=<optimized out>, expected=0, futex_word=0x562f6e087508) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
#1 __pthread_cond_wait_common (abstime=0x7fdba8ff8c40, clockid=<optimized out>, mutex=0x562f6e0874b8, cond=0x562f6e0874e0) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x562f6e0874e0, mutex=0x562f6e0874b8, abstime=0x7fdba8ff8c40) at pthread_cond_wait.c:656
#3 0x0000562f6c1bfe4b in os_event::timed_wait (this=<optimized out>, abstime=0x7fdba8ff8c40) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000562f6c1c01a6 in os_event::wait_time_low (this=0x562f6e0874a0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000562f6c1c0645 in os_event_wait_time_low (event=0x562f6e087508, time_in_usec=393, reset_sig_count=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:485
#6 0x0000562f6c2d614f in srv_error_monitor_thread () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:1853
#7 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#8 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7fdbb1df0700 (LWP 1556434)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fdbbd8e8277 in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000562f6c1b36f0 in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0file.cc:1819
#3 0x0000562f6c1b3db9 in LinuxAIOHandler::poll (this=0x7fdbb1defbb0, m1=0x7fdbb1defc50, m2=0x7fdbb1defc58, request=0x7fdbb1defdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1980
#4 0x0000562f6c1b8ca9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7fdbb1defc50, m2=0x7fdbb1defc58, request=0x7fdbb1defdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2034
#5 os_aio_handler (segment=<optimized out>, m1=0x7fdbb1defc50, m2=0x7fdbb1defc58, request=0x7fdbb1defdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5604
#6 0x0000562f6c466438 in fil_aio_wait (segment=0) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4350
#7 0x0000562f6c2d9d18 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fdbabfff700 (LWP 1556437)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fdbbd8e8277 in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000562f6c1b36f0 in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0file.cc:1819
#3 0x0000562f6c1b3db9 in LinuxAIOHandler::poll (this=0x7fdbabffebb0, m1=0x7fdbabffec50, m2=0x7fdbabffec58, request=0x7fdbabffedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1980
#4 0x0000562f6c1b8ca9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7fdbabffec50, m2=0x7fdbabffec58, request=0x7fdbabffedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2034
#5 os_aio_handler (segment=<optimized out>, m1=0x7fdbabffec50, m2=0x7fdbabffec58, request=0x7fdbabffedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5604
#6 0x0000562f6c466438 in fil_aio_wait (segment=3) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4350
#7 0x0000562f6c2d9d18 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fdba97fa700 (LWP 1556464)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7fdba97f9d80, clockid=<optimized out>, expected=0, futex_word=0x562f6e2c1f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
#1 __pthread_cond_wait_common (abstime=0x7fdba97f9d80, clockid=<optimized out>, mutex=0x562f6e2c1ed8, cond=0x562f6e2c1f00) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x562f6e2c1f00, mutex=0x562f6e2c1ed8, abstime=0x7fdba97f9d80) at pthread_cond_wait.c:656
#3 0x0000562f6c1bfe4b in os_event::timed_wait (this=<optimized out>, abstime=0x7fdba97f9d80) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000562f6c1c01a6 in os_event::wait_time_low (this=0x562f6e2c1ec0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000562f6c1c0645 in os_event_wait_time_low (event=0x562f6e2c1f28, time_in_usec=393, reset_sig_count=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:485
#6 0x0000562f6c18d419 in lock_wait_timeout_thread () at /mariadb/10.4/storage/innobase/lock/lock0wait.cc:522
#7 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#8 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fdbaaffd700 (LWP 1556439)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fdbbd8e8277 in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000562f6c1b36f0 in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0file.cc:1819
#3 0x0000562f6c1b3db9 in LinuxAIOHandler::poll (this=0x7fdbaaffcbb0, m1=0x7fdbaaffcc50, m2=0x7fdbaaffcc58, request=0x7fdbaaffcdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1980
#4 0x0000562f6c1b8ca9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7fdbaaffcc50, m2=0x7fdbaaffcc58, request=0x7fdbaaffcdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2034
#5 os_aio_handler (segment=<optimized out>, m1=0x7fdbaaffcc50, m2=0x7fdbaaffcc58, request=0x7fdbaaffcdf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5604
#6 0x0000562f6c466438 in fil_aio_wait (segment=5) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4350
#7 0x0000562f6c2d9d18 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fdbaa7fc700 (LWP 1556440)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7fdbaa7fbbc0, clockid=<optimized out>, expected=0, futex_word=0x562f6e08768c) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
#1 __pthread_cond_wait_common (abstime=0x7fdbaa7fbbc0, clockid=<optimized out>, mutex=0x562f6e087638, cond=0x562f6e087660) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x562f6e087660, mutex=0x562f6e087638, abstime=0x7fdbaa7fbbc0) at pthread_cond_wait.c:656
#3 0x0000562f6c1bfe4b in os_event::timed_wait (this=<optimized out>, abstime=0x7fdbaa7fbbc0) at /mariadb/10.4/storage/innobase/os/os0event.cc:275
#4 0x0000562f6c1c01a6 in os_event::wait_time_low (this=0x562f6e087620, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/10.4/storage/innobase/os/os0event.cc:385
#5 0x0000562f6c1c0645 in os_event_wait_time_low (event=0x562f6e08768c, time_in_usec=393, reset_sig_count=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:485
#6 0x0000562f6c3e13c6 in pc_sleep_if_needed (next_loop_time=19318810, sig_count=2, cur_time=19317810) at /mariadb/10.4/storage/innobase/buf/buf0flu.cc:2639
#7 buf_flush_page_cleaner_coordinator () at /mariadb/10.4/storage/innobase/buf/buf0flu.cc:3098
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fdbb89a6700 (LWP 1555073)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7fdbb89a5de0, clockid=<optimized out>, expected=0, futex_word=0x562f6d892978 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
#1 __pthread_cond_wait_common (abstime=0x7fdbb89a5de0, clockid=<optimized out>, mutex=0x562f6d8928c8 <LOCK_timer+40>, cond=0x562f6d892950 <COND_timer>) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x562f6d892950 <COND_timer>, mutex=0x562f6d8928c8 <LOCK_timer+40>, abstime=0x7fdbb89a5de0) at pthread_cond_wait.c:656
#3 0x0000562f6c668484 in safe_cond_timedwait (cond=0x562f6d892950 <COND_timer>, mp=0x562f6d8928a0 <LOCK_timer>, abstime=0x7fdbb89a5de0, file=0x562f6c6e3cac "/mariadb/10.4/include/mysql/psi/mysql_thread.h", line=1211) at /mariadb/10.4/mysys/thr_mutex.c:546
#4 0x0000562f6c668f90 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=0x7fdbb89a5de0, src_file=<optimized out>, src_line=292) at /mariadb/10.4/include/mysql/psi/mysql_thread.h:1211
#5 timer_handler (arg=<optimized out>) at /mariadb/10.4/mysys/thr_timer.c:292
#6 0x0000562f6c605146 in pfs_spawn_thread (arg=0x562f6df815c8) at /mariadb/10.4/storage/perfschema/pfs.cc:1869
#7 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#8 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fdbb35f2700 (LWP 1555085)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7fdbb35f1d50, clockid=<optimized out>, expected=0, futex_word=0x562f6d8805f8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:208
#1 __pthread_cond_wait_common (abstime=0x7fdbb35f1d50, clockid=<optimized out>, mutex=0x562f6d880540 <LOCK_checkpoint+40>, cond=0x562f6d8805d0 <COND_checkpoint>) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x562f6d8805d0 <COND_checkpoint>, mutex=0x562f6d880540 <LOCK_checkpoint+40>, abstime=0x7fdbb35f1d50) at pthread_cond_wait.c:656
#3 0x0000562f6c668484 in safe_cond_timedwait (cond=0x562f6d8805d0 <COND_checkpoint>, mp=0x562f6d880518 <LOCK_checkpoint>, abstime=0x7fdbb35f1d50, file=0x562f6c6e3cac "/mariadb/10.4/include/mysql/psi/mysql_thread.h", line=1211) at /mariadb/10.4/mysys/thr_mutex.c:546
#4 0x0000562f6c536673 in inline_mysql_cond_timedwait (that=0x562f6d8805d0 <COND_checkpoint>, mutex=0x562f6d880518 <LOCK_checkpoint>, abstime=0x7fdbb35f1d50, src_file=<optimized out>, src_line=116) at /mariadb/10.4/include/mysql/psi/mysql_thread.h:1211
#5 my_service_thread_sleep (control=0x562f6cf77310 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/10.4/storage/maria/ma_servicethread.c:115
#6 0x0000562f6c52da11 in ma_checkpoint_background (arg=<optimized out>) at /mariadb/10.4/storage/maria/ma_checkpoint.c:707
#7 0x0000562f6c605146 in pfs_spawn_thread (arg=0x562f6e03eec8) at /mariadb/10.4/storage/perfschema/pfs.cc:1869
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fdbb0dee700 (LWP 1556436)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fdbbd8e8277 in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000562f6c1b36f0 in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0file.cc:1819
#3 0x0000562f6c1b3db9 in LinuxAIOHandler::poll (this=0x7fdbb0dedbb0, m1=0x7fdbb0dedc50, m2=0x7fdbb0dedc58, request=0x7fdbb0deddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1980
#4 0x0000562f6c1b8ca9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7fdbb0dedc50, m2=0x7fdbb0dedc58, request=0x7fdbb0deddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2034
#5 os_aio_handler (segment=<optimized out>, m1=0x7fdbb0dedc50, m2=0x7fdbb0dedc58, request=0x7fdbb0deddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5604
#6 0x0000562f6c466438 in fil_aio_wait (segment=2) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4350
#7 0x0000562f6c2d9d18 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fdbab7fe700 (LWP 1556438)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fdbbd8e8277 in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000562f6c1b36f0 in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0file.cc:1819
#3 0x0000562f6c1b3db9 in LinuxAIOHandler::poll (this=0x7fdbab7fdbb0, m1=0x7fdbab7fdc50, m2=0x7fdbab7fdc58, request=0x7fdbab7fddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1980
#4 0x0000562f6c1b8ca9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7fdbab7fdc50, m2=0x7fdbab7fdc58, request=0x7fdbab7fddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2034
#5 os_aio_handler (segment=<optimized out>, m1=0x7fdbab7fdc50, m2=0x7fdbab7fdc58, request=0x7fdbab7fddf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5604
#6 0x0000562f6c466438 in fil_aio_wait (segment=4) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4350
#7 0x0000562f6c2d9d18 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fdbb15ef700 (LWP 1556435)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007fdbbd8e8277 in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000562f6c1b36f0 in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.4/storage/innobase/os/os0file.cc:1819
#3 0x0000562f6c1b3db9 in LinuxAIOHandler::poll (this=0x7fdbb15eebb0, m1=0x7fdbb15eec50, m2=0x7fdbb15eec58, request=0x7fdbb15eedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:1980
#4 0x0000562f6c1b8ca9 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7fdbb15eec50, m2=0x7fdbb15eec58, request=0x7fdbb15eedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:2034
#5 os_aio_handler (segment=<optimized out>, m1=0x7fdbb15eec50, m2=0x7fdbb15eec58, request=0x7fdbb15eedf0) at /mariadb/10.4/storage/innobase/os/os0file.cc:5604
#6 0x0000562f6c466438 in fil_aio_wait (segment=1) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4350
#7 0x0000562f6c2d9d18 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324
#8 0x00007fdbbd984f27 in start_thread (arg=<optimized out>) at pthread_create.c:479
#9 0x00007fdbbcfe62ef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fdbbcd9b340 (LWP 1554947)):
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x0000562f6bedd707 in handle_fatal_signal (sig=6) at /mariadb/10.4/sql/signal_handler.cc:343
#2 <signal handler called>
#3 0x00007fdbbcfde023 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7ffe37048ce8) at ../sysdeps/unix/sysv/linux/select.c:41
#4 0x0000562f6c65fdde in my_sleep (m_seconds=<optimized out>) at /mariadb/10.4/mysys/my_sleep.c:29
#5 0x0000562f6bb6fff3 in close_connections () at /mariadb/10.4/sql/mysqld.cc:1802
#6 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.4/sql/mysqld.cc:5936
#7 0x00007fdbbcf0fe0b in __libc_start_main (main=0x562f6bb6ab00 <main(int, char**)>, argc=8, argv=0x7ffe37048f38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe37048f28) at ../csu/libc-start.c:308
#8 0x0000562f6bb6aa3a in _start

Note: there is a srv_worker_thread even though the purge coordinator thread has already exited.

Comment by Marko Mäkelä [ 2020-09-22 ]

I just got this locally on 10.4:

10.4 55e48b77226fec7619986fc1206930bcf63439fe
rpl.rpl_heartbeat_basic 'mix' w46 [ fail ] Found warnings/errors in server log file!
Test ended at 2020-09-22 15:00:06
line
Attempting backtrace. You can use the following information to find out

Only one srv_worker_thread exists:

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x55a37df97fd8) at ../sysdeps/nptl/futex-internal.h:183

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55a37df97f88, cond=0x55a37df97fb0) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x55a37df97fb0, mutex=0x55a37df97f88) at pthread_cond_wait.c:638

#3  0x000055a37bc0ef39 in os_event::wait (this=0x55a37df97f70) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=0x55a37df97f70, reset_sig_count=7) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x000055a37bc0f595 in os_event_wait_low (event=0x55a37df97fd8, reset_sig_count=128) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x000055a37bd26450 in srv_resume_thread (slot=0x55a37c966fb0 <srv_sys+560>, sig_count=0, wait=true, timeout_usec=0) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:910

#7  0x000055a37bd26756 in srv_worker_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2517

#8  0x00007fa8909f0ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

#9  0x00007fa89004ceaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The purge coordinator thread is nowhere to be seen. srv_shutdown_state has not advanced from SRV_SHUTDOWN_INITIATED.

Comment by Marko Mäkelä [ 2021-03-05 ]

We got a recent occurrence on a 10.4-based branch:

innodb.innodb-get-fk 'innodb'            w3 [ fail ]

        Test ended at 2021-03-04 18:03:53

CURRENT_TEST: innodb.innodb-get-fk

mysqltest: At line 41: query 'ALTER TABLE `repro`.`crew_role_assigned` COMMENT = 'innodb_read_only'' failed with wrong errno 1286: 'Unknown storage engine 'InnoDB'', instead of 1036...

…

Thread 19 (Thread 0xae7fdb40 (LWP 25014)):

#0  0xb63f54f4 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb63f5548 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6cd0282 in LinuxAIOHandler::collect (this=this@entry=0xae7fd1b0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1819

#3  0xb6cd648a in LinuxAIOHandler::poll (this=this@entry=0xae7fd1b0, m1=m1@entry=0xae7fd240, m2=m2@entry=0xae7fd244, request=request@entry=0xae7fd250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1980

#4  0xb6cda4ab in os_aio_linux_handler (request=0xae7fd250, m2=0xae7fd244, m1=0xae7fd240, global_segment=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2034

#5  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0xae7fd240, m2=m2@entry=0xae7fd244, request=request@entry=0xae7fd250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5607

#6  0xb6e65ec0 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:4315

#7  0xb6d6d120 in io_handler_thread (arg=0xb793d944 <n+4>) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:324

#8  0xb63aef72 in start_thread (arg=0xae7fdb40) at pthread_create.c:312

#9  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 18 (Thread 0xa6bf1b40 (LWP 25030)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b2d0b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6cdb50f in wait (this=<optimized out>) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158

#3  wait_low (reset_sig_count=<optimized out>, this=0xb9dcfab8) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325

#4  os_event_wait_low (event=0xb9dcfab8, reset_sig_count=0) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:502

#5  0xb6d6af30 in srv_resume_thread (sig_count=0, timeout_usec=0, wait=true, slot=0xb787e9c4 <srv_sys+196>) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:907

#6  srv_worker_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2514

#7  0xb63aef72 in start_thread (arg=0xa6bf1b40) at pthread_create.c:312

#8  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 17 (Thread 0xa4ffdb40 (LWP 25033)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b2d0b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6cdb50f in wait (this=<optimized out>) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158

#3  wait_low (reset_sig_count=<optimized out>, this=0xb9dcfc70) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325

#4  os_event_wait_low (event=0xb9dcfc70, reset_sig_count=0) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:502

#5  0xb6dff370 in buf_resize_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0buf.cc:3142

#6  0xb63aef72 in start_thread (arg=0xa4ffdb40) at pthread_create.c:312

#7  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 16 (Thread 0xb09cab40 (LWP 25012)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b30ed in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6f19280 in inline_mysql_cond_timedwait (src_file=0xb735af04 "/home/buildbot/buildbot/build/storage/maria/ma_servicethread.c", src_line=116, abstime=0xb09ca224, mutex=0xb814bf34 <LOCK_checkpoint>, that=0xb814bf00 <COND_checkpoint>) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1211

#3  my_service_thread_sleep (control=control@entry=0xb7880384 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/buildbot/build/storage/maria/ma_servicethread.c:115

#4  0xb6f0e84d in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/buildbot/buildbot/build/storage/maria/ma_checkpoint.c:707

#5  0xb7012fcd in pfs_spawn_thread (arg=0xb9da4f08) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869

#6  0xb63aef72 in start_thread (arg=0xb09cab40) at pthread_create.c:312

#7  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 15 (Thread 0xac7f9b40 (LWP 25018)):

#0  0xb63f54f4 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb63f5548 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6cd0282 in LinuxAIOHandler::collect (this=this@entry=0xac7f91b0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1819

#3  0xb6cd648a in LinuxAIOHandler::poll (this=this@entry=0xac7f91b0, m1=m1@entry=0xac7f9240, m2=m2@entry=0xac7f9244, request=request@entry=0xac7f9250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1980

#4  0xb6cda4ab in os_aio_linux_handler (request=0xac7f9250, m2=0xac7f9244, m1=0xac7f9240, global_segment=5) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2034

#5  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0xac7f9240, m2=m2@entry=0xac7f9244, request=request@entry=0xac7f9250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5607

#6  0xb6e65ec0 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:4315

#7  0xb6d6d120 in io_handler_thread (arg=0xb793d954 <n+20>) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:324

#8  0xb63aef72 in start_thread (arg=0xac7f9b40) at pthread_create.c:312

#9  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 14 (Thread 0xaabf9b40 (LWP 25021)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b30ed in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6cdafe2 in timed_wait (abstime=0xaabf9194, this=0xb9edad70) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275

#3  os_event::wait_time_low (this=0xb9edad70, time_in_usec=1000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385

#4  0xb6cdb499 in os_event_wait_time_low (event=event@entry=0xb9edad70, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:485

#5  0xb6cae4ce in lock_wait_timeout_thread () at /home/buildbot/buildbot/build/storage/innobase/lock/lock0wait.cc:522

#6  0xb63aef72 in start_thread (arg=0xaabf9b40) at pthread_create.c:312

#7  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 13 (Thread 0xabff8b40 (LWP 25019)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b30ed in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6cdafe2 in timed_wait (abstime=0xabff80f4, this=0xb9dcfc18) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275

#3  os_event::wait_time_low (this=0xb9dcfc18, time_in_usec=1000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385

#4  0xb6cdb499 in os_event_wait_time_low (event=0xb9dcfc18, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=61) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:485

#5  0xb6e0db92 in pc_sleep_if_needed (cur_time=2057248, sig_count=61, next_loop_time=2058248) at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:2638

#6  buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0flu.cc:3097

#7  0xb63aef72 in start_thread (arg=0xabff8b40) at pthread_create.c:312

#8  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 12 (Thread 0xaa3f8b40 (LWP 25022)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b30ed in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6cdafe2 in timed_wait (abstime=0xaa3f8164, this=0xb9dcfb10) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275

#3  os_event::wait_time_low (this=0xb9dcfb10, time_in_usec=1000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385

#4  0xb6cdb499 in os_event_wait_time_low (event=0xb9dcfb10, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:485

#5  0xb6d697b2 in srv_error_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1851

#6  0xb63aef72 in start_thread (arg=0xaa3f8b40) at pthread_create.c:312

#7  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 11 (Thread 0xab3fab40 (LWP 25026)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b65b6 in nanosleep () at ../sysdeps/unix/syscall-template.S:81

#2  0xb6cdb858 in os_thread_sleep (tm=tm@entry=1000000) at /home/buildbot/buildbot/build/storage/innobase/os/os0thread.cc:231

#3  0xb6d6a487 in srv_master_sleep () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2338

#4  srv_master_thread (arg=0x0) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:2378

#5  0xb63aef72 in start_thread (arg=0xab3fab40) at pthread_create.c:312

#6  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 10 (Thread 0xa5fffb40 (LWP 25031)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b2d0b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb6cdb50f in wait (this=<optimized out>) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:158

#3  wait_low (reset_sig_count=<optimized out>, this=0xb9dcfbc0) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:325

#4  os_event_wait_low (event=0xb9dcfbc0, reset_sig_count=0) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:502

#5  0xb6e0735e in buf_dump_thread () at /home/buildbot/buildbot/build/storage/innobase/buf/buf0dump.cc:832

#6  0xb63aef72 in start_thread (arg=0xa5fffb40) at pthread_create.c:312

#7  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 9 (Thread 0xb01c4b40 (LWP 25041)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb60e87ba in do_sigwaitinfo (set=0xfffffffc, set@entry=0xb01c41e4, info=info@entry=0xb01c4264) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwaitinfo.c:54

#2  0xb60e8859 in __GI___sigwaitinfo (set=0xb01c41e4, info=0xb01c4264) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwaitinfo.c:81

#3  0xb6773330 in my_sigwait (code=<synthetic pointer>, sig=0xb01c41e0, set=0xb01c41e4) at /home/buildbot/buildbot/build/include/my_pthread.h:196

#4  signal_hand (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/sql/mysqld.cc:3224

#5  0xb7012fcd in pfs_spawn_thread (arg=0xb9f58d50) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869

#6  0xb63aef72 in start_thread (arg=0xb01c4b40) at pthread_create.c:312

#7  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 8 (Thread 0xb017ab40 (LWP 25042)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b2d0b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187

#2  0xb683dcb5 in inline_mysql_cond_wait (that=0xb78e81a0 <COND_manager>, mutex=0xb78e81d4 <LOCK_manager>, src_file=0xb71d488c "/home/buildbot/buildbot/build/sql/sql_manager.cc", src_line=102) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1174

#3  handle_manager (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/sql/sql_manager.cc:102

#4  0xb7012fcd in pfs_spawn_thread (arg=0xb9fcec70) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869

#5  0xb63aef72 in start_thread (arg=0xb017ab40) at pthread_create.c:312

#6  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 7 (Thread 0xadffcb40 (LWP 25015)):

#0  0xb63f54f4 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb63f5548 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6cd0282 in LinuxAIOHandler::collect (this=this@entry=0xadffc1b0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1819

#3  0xb6cd648a in LinuxAIOHandler::poll (this=this@entry=0xadffc1b0, m1=m1@entry=0xadffc240, m2=m2@entry=0xadffc244, request=request@entry=0xadffc250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1980

#4  0xb6cda4ab in os_aio_linux_handler (request=0xadffc250, m2=0xadffc244, m1=0xadffc240, global_segment=2) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2034

#5  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0xadffc240, m2=m2@entry=0xadffc244, request=request@entry=0xadffc250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5607

#6  0xb6e65ec0 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:4315

#7  0xb6d6d120 in io_handler_thread (arg=0xb793d948 <n+8>) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:324

#8  0xb63aef72 in start_thread (arg=0xadffcb40) at pthread_create.c:312

#9  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 6 (Thread 0xa9bf7b40 (LWP 25023)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b30ed in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb6cdafe2 in timed_wait (abstime=0xa9bf71e4, this=0xb9dcfb68) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:275

#3  os_event::wait_time_low (this=0xb9dcfb68, time_in_usec=5000000, reset_sig_count=<optimized out>) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:385

#4  0xb6cdb499 in os_event_wait_time_low (event=0xb9dcfb68, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/buildbot/buildbot/build/storage/innobase/os/os0event.cc:485

#5  0xb6d6911c in srv_monitor_thread () at /home/buildbot/buildbot/build/storage/innobase/srv/srv0srv.cc:1687

#6  0xb63aef72 in start_thread (arg=0xa9bf7b40) at pthread_create.c:312

#7  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 5 (Thread 0xad7fbb40 (LWP 25016)):

#0  0xb63f54f4 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb63f5548 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6cd0282 in LinuxAIOHandler::collect (this=this@entry=0xad7fb1b0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1819

#3  0xb6cd648a in LinuxAIOHandler::poll (this=this@entry=0xad7fb1b0, m1=m1@entry=0xad7fb240, m2=m2@entry=0xad7fb244, request=request@entry=0xad7fb250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1980

#4  0xb6cda4ab in os_aio_linux_handler (request=0xad7fb250, m2=0xad7fb244, m1=0xad7fb240, global_segment=3) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2034

#5  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0xad7fb240, m2=m2@entry=0xad7fb244, request=request@entry=0xad7fb250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5607

#6  0xb6e65ec0 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:4315

#7  0xb6d6d120 in io_handler_thread (arg=0xb793d94c <n+12>) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:324

#8  0xb63aef72 in start_thread (arg=0xad7fbb40) at pthread_create.c:312

#9  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 4 (Thread 0xacffab40 (LWP 25017)):

#0  0xb63f54f4 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb63f5548 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6cd0282 in LinuxAIOHandler::collect (this=this@entry=0xacffa1b0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1819

#3  0xb6cd648a in LinuxAIOHandler::poll (this=this@entry=0xacffa1b0, m1=m1@entry=0xacffa240, m2=m2@entry=0xacffa244, request=request@entry=0xacffa250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1980

#4  0xb6cda4ab in os_aio_linux_handler (request=0xacffa250, m2=0xacffa244, m1=0xacffa240, global_segment=4) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2034

#5  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0xacffa240, m2=m2@entry=0xacffa244, request=request@entry=0xacffa250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5607

#6  0xb6e65ec0 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:4315

#7  0xb6d6d120 in io_handler_thread (arg=0xb793d950 <n+16>) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:324

#8  0xb63aef72 in start_thread (arg=0xacffab40) at pthread_create.c:312

#9  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 3 (Thread 0xaeffeb40 (LWP 25013)):

#0  0xb63f54f4 in ?? () from /lib/i386-linux-gnu/libaio.so.1

#1  0xb63f5548 in io_getevents () from /lib/i386-linux-gnu/libaio.so.1

#2  0xb6cd0282 in LinuxAIOHandler::collect (this=this@entry=0xaeffe1b0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1819

#3  0xb6cd648a in LinuxAIOHandler::poll (this=this@entry=0xaeffe1b0, m1=m1@entry=0xaeffe240, m2=m2@entry=0xaeffe244, request=request@entry=0xaeffe250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:1980

#4  0xb6cda4ab in os_aio_linux_handler (request=0xaeffe250, m2=0xaeffe244, m1=0xaeffe240, global_segment=0) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:2034

#5  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0xaeffe240, m2=m2@entry=0xaeffe244, request=request@entry=0xaeffe250) at /home/buildbot/buildbot/build/storage/innobase/os/os0file.cc:5607

#6  0xb6e65ec0 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/storage/innobase/fil/fil0fil.cc:4315

#7  0xb6d6d120 in io_handler_thread (arg=0xb793d940 <n>) at /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc:324

#8  0xb63aef72 in start_thread (arg=0xaeffeb40) at pthread_create.c:312

#9  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 2 (Thread 0xb1dffb40 (LWP 25011)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b30ed in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:245

#2  0xb7104bc2 in inline_mysql_cond_timedwait (that=0xb815d480 <COND_timer>, mutex=0xb815d4b4 <LOCK_timer>, src_file=0xb737edb0 "/home/buildbot/buildbot/build/mysys/thr_timer.c", src_line=292, abstime=0xb1dff2d4) at /home/buildbot/buildbot/build/include/mysql/psi/mysql_thread.h:1211

#3  timer_handler (arg=arg@entry=0x0) at /home/buildbot/buildbot/build/mysys/thr_timer.c:292

#4  0xb7012fcd in pfs_spawn_thread (arg=0xb9d254e8) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1869

#5  0xb63aef72 in start_thread (arg=0xb1dffb40) at pthread_create.c:312

#6  0xb61a743e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

Thread 1 (Thread 0xb609a940 (LWP 25010)):

#0  0xb63fe424 in __kernel_vsyscall ()

#1  0xb63b3fb3 in __pthread_kill (threadid=3054086464, signo=signo@entry=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61

#2  0xb71001d7 in my_write_core (sig=6) at /home/buildbot/buildbot/build/mysys/stacktrace.c:386

#3  0xb6a93dc6 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/sql/signal_handler.cc:344

#4  <signal handler called>

#5  0xb63fe424 in __kernel_vsyscall ()

#6  0xb619f661 in select () at ../sysdeps/unix/syscall-template.S:81

#7  0xb70fda8a in my_sleep (m_seconds=1000) at /home/buildbot/buildbot/build/mysys/my_sleep.c:29

#8  0xb677dd0c in close_connections () at /home/buildbot/buildbot/build/sql/mysqld.cc:1775

#9  mysqld_main (argc=147, argv=0xbf9b0dd4) at /home/buildbot/buildbot/build/sql/mysqld.cc:5924

#10 0xb67431a7 in main (argc=23, argv=0xbf9b0dd4) at /home/buildbot/buildbot/build/sql/main.cc:25

The reason for failing to start up the server (as well as for the stack trace output) was that the server had been forcibly killed due to a shutdown timeout:

CURRENT_TEST: innodb.innodb-get-fk

2021-03-04 18:02:49 0 [Note] /usr/local/mariadb-10.4.19-linux-i686/bin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown

2021-03-04 18:02:49 0 [Note] Event Scheduler: Purging the queue. 0 events

2021-03-04 18:02:49 0 [Note] InnoDB: FTS optimize thread exiting.

210304 18:03:49 [ERROR] mysqld got signal 6 ;

…

2021-03-04 18:03:52 0 [Warning] InnoDB: innodb_read_only prevents crash recovery

2021-03-04 18:03:52 0 [ERROR] InnoDB: Plugin initialization aborted with error Read only transaction

2021-03-04 18:03:52 0 [Note] InnoDB: Starting shutdown...

2021-03-04 18:03:52 0 [ERROR] Plugin 'InnoDB' init function returned error.

2021-03-04 18:03:52 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

Among the stack traces, we can observe that a single srv_worker_thread is left, and no purge coordinator. So, it looks like this issue again.

Comment by Marko Mäkelä [ 2021-09-22 ]

10.4 9024498e88f01f235f56db1dc18389baaef5b0b2
binlog_encryption.rpl_binlog_errors 'innodb,stmt' w44 [ fail ] Found warnings/errors in server log file!
Test ended at 2021-09-22 18:19:09
line
Attempting backtrace. You can use the following information to find out

In mysqld.2.err we have the following:

CURRENT_TEST: binlog_encryption.rpl_binlog_errors

…

2021-09-22 18:18:01 0 [Note] /dev/shm/10.4/sql/mysqld (initiated by: root[root] @ localhost [127.0.0.1]): Normal shutdown

2021-09-22 18:18:01 0 [Note] Event Scheduler: Purging the queue. 0 events

2021-09-22 18:18:01 0 [Note] InnoDB: FTS optimize thread exiting.

210922 18:19:02 [ERROR] mysqld got signal 6 ;

Note the 61-second delay between the 2 last messages. That is, shutdown_server in the test environment delivered the SIGABRT.

We have Thread 10 waiting for srv_shutdown_state <= SRV_SHUTDOWN_INITIATED (it is SRV_SHUTDOWN_INITIATED. The purge coordinator has exited, but a worker is left:

Thread 19 (Thread 0x7f3c137fe640 (LWP 1670932)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c26e6398) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c26e6348, cond=0x5556c26e6370) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c26e6370, mutex=0x5556c26e6348) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c26e6330) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=0x5556c26e6330, reset_sig_count=1, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c26e6398, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c03f8447 in buf_resize_thread () at /mariadb/10.4/storage/innobase/buf/buf0buf.cc:3142

#7  0x00007f3c61a8deae in start_thread (arg=0x7f3c137fe640) at pthread_create.c:463

#8  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f3c28ff9640 (LWP 1670930)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c26e6298) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c26e6248, cond=0x5556c26e6270) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c26e6270, mutex=0x5556c26e6248) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c26e6230) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=0x5556c26e6230, reset_sig_count=1, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c26e6298, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c041a280 in buf_dump_thread () at /mariadb/10.4/storage/innobase/buf/buf0dump.cc:832

#7  0x00007f3c61a8deae in start_thread (arg=0x7f3c28ff9640) at pthread_create.c:463

#8  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f3c5c053640 (LWP 1670940)):

#0  0x00007f3c616cfbd2 in __GI___sigtimedwait (set=0x7f3c5c052d08, info=0x7f3c5c052c88, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:36

#1  0x00005556bfc180eb in my_sigwait (set=0x7f3c5c052d08, sig=0x7f3c5c052c70, code=<optimized out>) at /mariadb/10.4/include/my_pthread.h:196

#2  signal_hand (arg=<optimized out>) at /mariadb/10.4/sql/mysqld.cc:3219

#3  0x00005556c062a5a6 in pfs_spawn_thread (arg=0x5556c29f2478) at /mariadb/10.4/storage/perfschema/pfs.cc:1869

#4  0x00007f3c61a8deae in start_thread (arg=0x7f3c5c053640) at pthread_create.c:463

#5  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f3c44ff9640 (LWP 1670903)):

#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f3c44ff8d60, clockid=1157598000, expected=0, futex_word=0x5556c26e6218) at ../sysdeps/nptl/futex-internal.h:323

#1  __pthread_cond_wait_common (abstime=0x7f3c44ff8d60, clockid=1157598000, mutex=0x5556c26e61c8, cond=0x5556c26e61f0) at pthread_cond_wait.c:520

#2  __pthread_cond_timedwait (cond=0x5556c26e61f0, mutex=0x5556c26e61c8, abstime=0x7f3c44ff8d60) at pthread_cond_wait.c:656

#3  0x00005556c0216ad9 in os_event::timed_wait (this=this@entry=0x5556c26e61b0, abstime=abstime@entry=0x7f3c44ff8d60) at /mariadb/10.4/storage/innobase/os/os0event.cc:275

#4  0x00005556c0216e6b in os_event::wait_time_low (this=0x5556c26e61b0, time_in_usec=<optimized out>, time_in_usec@entry=5000000, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385

#5  0x00005556c0217325 in os_event_wait_time_low (event=0x5556c26e6218, time_in_usec=393, time_in_usec@entry=5000000, reset_sig_count=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:485

#6  0x00005556c0327710 in srv_monitor_thread () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:1684

#7  0x00007f3c61a8deae in start_thread (arg=0x7f3c44ff9640) at pthread_create.c:463

#8  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f3c457fa640 (LWP 1670902)):

#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f3c457f9b80, clockid=1165990224, expected=0, futex_word=0x5556c26e6198) at ../sysdeps/nptl/futex-internal.h:323

#1  __pthread_cond_wait_common (abstime=0x7f3c457f9b80, clockid=1165990224, mutex=0x5556c26e6148, cond=0x5556c26e6170) at pthread_cond_wait.c:520

#2  __pthread_cond_timedwait (cond=0x5556c26e6170, mutex=0x5556c26e6148, abstime=0x7f3c457f9b80) at pthread_cond_wait.c:656

#3  0x00005556c0216ad9 in os_event::timed_wait (this=this@entry=0x5556c26e6130, abstime=abstime@entry=0x7f3c457f9b80) at /mariadb/10.4/storage/innobase/os/os0event.cc:275

#4  0x00005556c0216e6b in os_event::wait_time_low (this=0x5556c26e6130, time_in_usec=<optimized out>, time_in_usec@entry=1000000, reset_sig_count=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385

#5  0x00005556c0217325 in os_event_wait_time_low (event=0x5556c26e6198, time_in_usec=393, time_in_usec@entry=1000000, reset_sig_count=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:485

#6  0x00005556c0327af2 in srv_error_monitor_thread () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:1848

#7  0x00007f3c61a8deae in start_thread (arg=0x7f3c457fa640) at pthread_create.c:463

#8  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f3c45ffb640 (LWP 1670901)):

#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f3c45fface0, clockid=1174383280, expected=0, futex_word=0x5556c291abe8) at ../sysdeps/nptl/futex-internal.h:323

#1  __pthread_cond_wait_common (abstime=0x7f3c45fface0, clockid=1174383280, mutex=0x5556c291ab98, cond=0x5556c291abc0) at pthread_cond_wait.c:520

#2  __pthread_cond_timedwait (cond=0x5556c291abc0, mutex=0x5556c291ab98, abstime=0x7f3c45fface0) at pthread_cond_wait.c:656

#3  0x00005556c0216ad9 in os_event::timed_wait (this=this@entry=0x5556c291ab80, abstime=abstime@entry=0x7f3c45fface0) at /mariadb/10.4/storage/innobase/os/os0event.cc:275

#4  0x00005556c0216e6b in os_event::wait_time_low (this=this@entry=0x5556c291ab80, time_in_usec=<optimized out>, time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:385

#5  0x00005556c0217325 in os_event_wait_time_low (event=0x5556c291abe8, event@entry=0x5556c291ab80, time_in_usec=393, time_in_usec@entry=1000000, reset_sig_count=0, reset_sig_count@entry=1) at /mariadb/10.4/storage/innobase/os/os0event.cc:485

#6  0x00005556c01e3a49 in lock_wait_timeout_thread () at /mariadb/10.4/storage/innobase/lock/lock0wait.cc:527

#7  0x00007f3c61a8deae in start_thread (arg=0x7f3c45ffb640) at pthread_create.c:463

#8  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f3c477fe640 (LWP 1670894)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c276835c) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c2768308, cond=0x5556c2768330) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c2768330, mutex=0x5556c2768308) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c27682f0) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=this@entry=0x5556c27682f0, reset_sig_count=2, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c276835c, event@entry=0x5556c27682f0, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c020f70a in os_aio_simulated_handler (global_segment=5, m1=<optimized out>, m2=<optimized out>, type=0x7f3c477fdd50) at /mariadb/10.4/storage/innobase/os/os0file.cc:6942

#7  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f3c477fdba0, m2=m2@entry=0x7f3c477fdbb0, request=request@entry=0x7f3c477fdd50) at /mariadb/10.4/storage/innobase/os/os0file.cc:5378

#8  0x00005556c049f956 in fil_aio_wait (segment=segment@entry=5) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4315

#9  0x00005556c032b4d8 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324

#10 0x00007f3c61a8deae in start_thread (arg=0x7f3c477fe640) at pthread_create.c:463

#11 0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f3c56659640 (LWP 1670941)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c0ff2928 <COND_manager+40>) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c0ff2878 <LOCK_manager+40>, cond=0x5556c0ff2900 <COND_manager>) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=cond@entry=0x5556c0ff2900 <COND_manager>, mutex=mutex@entry=0x5556c0ff2878 <LOCK_manager+40>) at pthread_cond_wait.c:638

#3  0x00005556c068b3f7 in safe_cond_wait (cond=0x5556c0ff2928 <COND_manager+40>, mp=0x5556c0ff2850 <LOCK_manager>, file=0x5556c0705e8c "/mariadb/10.4/include/mysql/psi/mysql_thread.h", line=1174) at /mariadb/10.4/mysys/thr_mutex.c:492

#4  0x00005556bfcf7f28 in inline_mysql_cond_wait (that=0x5556c0ff2900 <COND_manager>, src_line=102, mutex=<optimized out>, src_file=<optimized out>) at /mariadb/10.4/include/mysql/psi/mysql_thread.h:1174

#5  handle_manager (arg=<optimized out>) at /mariadb/10.4/sql/sql_manager.cc:102

#6  0x00005556c062a5a6 in pfs_spawn_thread (arg=0x5556c29f1008) at /mariadb/10.4/storage/perfschema/pfs.cc:1869

#7  0x00007f3c61a8deae in start_thread (arg=0x7f3c56659640) at pthread_create.c:463

#8  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f3c297fa640 (LWP 1670918)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c26e611c) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c26e60c8, cond=0x5556c26e60f0) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c26e60f0, mutex=0x5556c26e60c8) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c26e60b0) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=0x5556c26e60b0, reset_sig_count=8, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c26e611c, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c0329710 in srv_resume_thread (slot=slot@entry=0x5556c0f696b0 <srv_sys+1328>, sig_count=sig_count@entry=0, wait=true, timeout_usec=timeout_usec@entry=0) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:904

#7  0x00005556c0329ad5 in srv_worker_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2518

#8  0x00007f3c61a8deae in start_thread (arg=0x7f3c297fa640) at pthread_create.c:463

#9  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f3c2b7fe640 (LWP 1670914)):

#0  0x00007f3c61757b21 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7f3c2b7fdba8, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:46

#1  0x00007f3c6175cec3 in __GI___nanosleep (requested_time=<optimized out>, remaining=<optimized out>) at nanosleep.c:27

#2  0x00005556c02176b1 in os_thread_sleep (tm=tm@entry=1000000) at /mariadb/10.4/storage/innobase/os/os0thread.cc:231

#3  0x00005556c0328409 in srv_master_sleep () at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2335

#4  srv_master_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0srv.cc:2375

#5  0x00007f3c61a8deae in start_thread (arg=0x7f3c2b7fe640) at pthread_create.c:463

#6  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f3c55c58640 (LWP 1670890)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c276815c) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c2768108, cond=0x5556c2768130) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c2768130, mutex=0x5556c2768108) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c27680f0) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=this@entry=0x5556c27680f0, reset_sig_count=2, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c276815c, event@entry=0x5556c27680f0, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c020f70a in os_aio_simulated_handler (global_segment=1, m1=<optimized out>, m2=<optimized out>, type=0x7f3c55c57d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:6942

#7  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f3c55c57ba0, m2=m2@entry=0x7f3c55c57bb0, request=request@entry=0x7f3c55c57d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:5378

#8  0x00005556c049f956 in fil_aio_wait (segment=segment@entry=1) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4315

#9  0x00005556c032b4d8 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324

#10 0x00007f3c61a8deae in start_thread (arg=0x7f3c55c58640) at pthread_create.c:463

#11 0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f3c5785a640 (LWP 1669810)):

#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f3c57859cd0, clockid=1468374016, expected=0, futex_word=0x5556c1879898 <COND_checkpoint+40>) at ../sysdeps/nptl/futex-internal.h:323

#1  __pthread_cond_wait_common (abstime=0x7f3c57859cd0, clockid=1468374016, mutex=0x5556c18797e0 <LOCK_checkpoint+40>, cond=0x5556c1879870 <COND_checkpoint>) at pthread_cond_wait.c:520

#2  __pthread_cond_timedwait (cond=cond@entry=0x5556c1879870 <COND_checkpoint>, mutex=mutex@entry=0x5556c18797e0 <LOCK_checkpoint+40>, abstime=abstime@entry=0x7f3c57859cd0) at pthread_cond_wait.c:656

#3  0x00005556c068b62f in safe_cond_timedwait (cond=cond@entry=0x5556c1879870 <COND_checkpoint>, mp=mp@entry=0x5556c18797b8 <LOCK_checkpoint>, abstime=abstime@entry=0x7f3c57859cd0, file=0x5556c0705e8c "/mariadb/10.4/include/mysql/psi/mysql_thread.h", line=line@entry=1211) at /mariadb/10.4/mysys/thr_mutex.c:546

#4  0x00005556c0564a8f in inline_mysql_cond_timedwait (that=0x5556c1879870 <COND_checkpoint>, mutex=0x5556c18797b8 <LOCK_checkpoint>, abstime=0x7f3c57859cd0, src_line=116, src_file=<optimized out>) at /mariadb/10.4/include/mysql/psi/mysql_thread.h:1211

#5  my_service_thread_sleep (control=control@entry=0x5556c0f6e590 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/10.4/storage/maria/ma_servicethread.c:115

#6  0x00005556c055c671 in ma_checkpoint_background (arg=<optimized out>, arg@entry=0x1e) at /mariadb/10.4/storage/maria/ma_checkpoint.c:707

#7  0x00005556c062a5a6 in pfs_spawn_thread (arg=0x5556c269df68) at /mariadb/10.4/storage/perfschema/pfs.cc:1869

#8  0x00007f3c61a8deae in start_thread (arg=0x7f3c5785a640) at pthread_create.c:463

#9  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f3c54c56640 (LWP 1670892)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c2768258) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c2768208, cond=0x5556c2768230) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c2768230, mutex=0x5556c2768208) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c27681f0) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=this@entry=0x5556c27681f0, reset_sig_count=1, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c2768258, event@entry=0x5556c27681f0, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c020f70a in os_aio_simulated_handler (global_segment=3, m1=<optimized out>, m2=<optimized out>, type=0x7f3c54c55d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:6942

#7  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f3c54c55ba0, m2=m2@entry=0x7f3c54c55bb0, request=request@entry=0x7f3c54c55d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:5378

#8  0x00005556c049f956 in fil_aio_wait (segment=segment@entry=3) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4315

#9  0x00005556c032b4d8 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324

#10 0x00007f3c61a8deae in start_thread (arg=0x7f3c54c56640) at pthread_create.c:463

#11 0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f3c55457640 (LWP 1670891)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c27681d8) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c2768188, cond=0x5556c27681b0) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c27681b0, mutex=0x5556c2768188) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c2768170) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=this@entry=0x5556c2768170, reset_sig_count=1, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c27681d8, event@entry=0x5556c2768170, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c020f70a in os_aio_simulated_handler (global_segment=2, m1=<optimized out>, m2=<optimized out>, type=0x7f3c55456d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:6942

#7  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f3c55456ba0, m2=m2@entry=0x7f3c55456bb0, request=request@entry=0x7f3c55456d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:5378

#8  0x00005556c049f956 in fil_aio_wait (segment=segment@entry=2) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4315

#9  0x00005556c032b4d8 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324

#10 0x00007f3c61a8deae in start_thread (arg=0x7f3c55457640) at pthread_create.c:463

#11 0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f3c56459640 (LWP 1670889)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c27680d8) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c2768088, cond=0x5556c27680b0) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c27680b0, mutex=0x5556c2768088) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c2768070) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=this@entry=0x5556c2768070, reset_sig_count=1, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c27680d8, event@entry=0x5556c2768070, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c020f70a in os_aio_simulated_handler (global_segment=0, m1=<optimized out>, m2=<optimized out>, type=0x7f3c56458d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:6942

#7  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f3c56458ba0, m2=m2@entry=0x7f3c56458bb0, request=request@entry=0x7f3c56458d50) at /mariadb/10.4/storage/innobase/os/os0file.cc:5378

#8  0x00005556c049f956 in fil_aio_wait (segment=segment@entry=0) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4315

#9  0x00005556c032b4d8 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324

#10 0x00007f3c61a8deae in start_thread (arg=0x7f3c56459640) at pthread_create.c:463

#11 0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f3c5d078640 (LWP 1669804)):

#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f3c5d077d40, clockid=1560771696, expected=0, futex_word=0x5556c188c8b8 <COND_timer+40>) at ../sysdeps/nptl/futex-internal.h:323

#1  __pthread_cond_wait_common (abstime=0x7f3c5d077d40, clockid=1560771696, mutex=0x5556c188c808 <LOCK_timer+40>, cond=0x5556c188c890 <COND_timer>) at pthread_cond_wait.c:520

#2  __pthread_cond_timedwait (cond=cond@entry=0x5556c188c890 <COND_timer>, mutex=mutex@entry=0x5556c188c808 <LOCK_timer+40>, abstime=abstime@entry=0x7f3c5d077d40) at pthread_cond_wait.c:656

#3  0x00005556c068b62f in safe_cond_timedwait (cond=0x5556c188c890 <COND_timer>, mp=0x5556c188c7e0 <LOCK_timer>, abstime=abstime@entry=0x7f3c5d077d40, file=0x5556c0705e8c "/mariadb/10.4/include/mysql/psi/mysql_thread.h", line=line@entry=1211) at /mariadb/10.4/mysys/thr_mutex.c:546

#4  0x00005556c068bfca in inline_mysql_cond_timedwait (abstime=0x7f3c5d077d40, src_line=292, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /mariadb/10.4/include/mysql/psi/mysql_thread.h:1211

#5  timer_handler (arg=arg@entry=0x0) at /mariadb/10.4/mysys/thr_timer.c:292

#6  0x00005556c062a5a6 in pfs_spawn_thread (arg=0x5556c25e07b8) at /mariadb/10.4/storage/perfschema/pfs.cc:1869

#7  0x00007f3c61a8deae in start_thread (arg=0x7f3c5d078640) at pthread_create.c:463

#8  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f3c46ffd640 (LWP 1670895)):

#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f3c46ffcb20, clockid=1191168240, expected=0, futex_word=0x5556c26e631c) at ../sysdeps/nptl/futex-internal.h:323

#1  __pthread_cond_wait_common (abstime=0x7f3c46ffcb20, clockid=1191168240, mutex=0x5556c26e62c8, cond=0x5556c26e62f0) at pthread_cond_wait.c:520

#2  __pthread_cond_timedwait (cond=0x5556c26e62f0, mutex=0x5556c26e62c8, abstime=0x7f3c46ffcb20) at pthread_cond_wait.c:656

#3  0x00005556c0216ad9 in os_event::timed_wait (this=this@entry=0x5556c26e62b0, abstime=abstime@entry=0x7f3c46ffcb20) at /mariadb/10.4/storage/innobase/os/os0event.cc:275

#4  0x00005556c0216e6b in os_event::wait_time_low (this=0x5556c26e62b0, time_in_usec=<optimized out>, reset_sig_count=reset_sig_count@entry=63) at /mariadb/10.4/storage/innobase/os/os0event.cc:385

#5  0x00005556c0217325 in os_event_wait_time_low (event=0x5556c26e631c, time_in_usec=393, reset_sig_count=0, reset_sig_count@entry=63) at /mariadb/10.4/storage/innobase/os/os0event.cc:485

#6  0x00005556c0424b21 in pc_sleep_if_needed (next_loop_time=40622825, sig_count=63, cur_time=40621825) at /mariadb/10.4/storage/innobase/buf/buf0flu.cc:2637

#7  buf_flush_page_cleaner_coordinator () at /mariadb/10.4/storage/innobase/buf/buf0flu.cc:3096

#8  0x00007f3c61a8deae in start_thread (arg=0x7f3c46ffd640) at pthread_create.c:463

#9  0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f3c47fff640 (LWP 1670893)):

#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x5556c27682dc) at ../sysdeps/nptl/futex-internal.h:186

#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5556c2768288, cond=0x5556c27682b0) at pthread_cond_wait.c:508

#2  __pthread_cond_wait (cond=0x5556c27682b0, mutex=0x5556c2768288) at pthread_cond_wait.c:638

#3  0x00005556c0216c9b in os_event::wait (this=0x5556c2768270) at /mariadb/10.4/storage/innobase/os/os0event.cc:158

#4  os_event::wait_low (this=this@entry=0x5556c2768270, reset_sig_count=2, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:325

#5  0x00005556c0217335 in os_event_wait_low (event=0x5556c27682dc, event@entry=0x5556c2768270, reset_sig_count=128, reset_sig_count@entry=0) at /mariadb/10.4/storage/innobase/os/os0event.cc:502

#6  0x00005556c020f70a in os_aio_simulated_handler (global_segment=4, m1=<optimized out>, m2=<optimized out>, type=0x7f3c47ffed50) at /mariadb/10.4/storage/innobase/os/os0file.cc:6942

#7  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f3c47ffeba0, m2=m2@entry=0x7f3c47ffebb0, request=request@entry=0x7f3c47ffed50) at /mariadb/10.4/storage/innobase/os/os0file.cc:5378

#8  0x00005556c049f956 in fil_aio_wait (segment=segment@entry=4) at /mariadb/10.4/storage/innobase/fil/fil0fil.cc:4315

#9  0x00005556c032b4d8 in io_handler_thread (arg=<optimized out>) at /mariadb/10.4/storage/innobase/srv/srv0start.cc:324

#10 0x00007f3c61a8deae in start_thread (arg=0x7f3c47fff640) at pthread_create.c:463

#11 0x00007f3c6178fa5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f3c614427c0 (LWP 1669790)):

#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:54

#1  0x00005556bff58bf7 in handle_fatal_signal (sig=6) at /mariadb/10.4/sql/signal_handler.cc:356

#2  <signal handler called>

#3  0x00007f3c61787543 in __GI___select (nfds=nfds@entry=0, readfds=readfds@entry=0x0, writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x7ffd0aa889a8) at ../sysdeps/unix/sysv/linux/select.c:41

#4  0x00005556c0683920 in my_sleep (m_seconds=<optimized out>) at /mariadb/10.4/mysys/my_sleep.c:29

#5  0x00005556bfc1ef7a in close_connections () at /mariadb/10.4/sql/mysqld.cc:1769

#6  0x00005556bfc19636 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.4/sql/mysqld.cc:5922

#7  0x00007f3c616b9e4a in __libc_start_main (main=0x5556bfc151f0 <main(int, char**)>, argc=25, argv=0x7ffd0aa88c28, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd0aa88c18) at ../csu/libc-start.c:314

#8  0x00005556bfc1512a in _start () at /mariadb/10.4/sql/my_decimal.h:143

Comment by Marko Mäkelä [ 2022-12-16 ]

I just reproduced this on 10.3 for the test innodb.doublewrite,32k. There was no srv_purge_coordinator_thread, but one srv_worker_thread was waiting:

#4  ___pthread_cond_wait (cond=0x55980841ea50, mutex=0x55980841ea28) at ./nptl/pthread_cond_wait.c:618

#5  0x0000559806288eef in os_event::wait (this=this@entry=0x55980841ea10) at /mariadb/10.3-MDEV-25004/storage/innobase/os/os0event.cc:158

#6  0x0000559806288b45 in os_event::wait_low (this=0x55980841ea10, reset_sig_count=169, reset_sig_count@entry=0) at /mariadb/10.3-MDEV-25004/storage/innobase/os/os0event.cc:325

#7  0x0000559806288c13 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /mariadb/10.3-MDEV-25004/storage/innobase/os/os0event.cc:502

#8  0x000055980635baca in srv_resume_thread (slot=slot@entry=0x55980669e870 <srv_sys+1264>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /mariadb/10.3-MDEV-25004/storage/innobase/srv/srv0srv.cc:909

#9  0x000055980635c30f in srv_worker_thread (arg=<optimized out>) at /mariadb/10.3-MDEV-25004/storage/innobase/srv/srv0srv.cc:2505

#10 0x00007fb1393c8fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442

#11 0x00007fb13944966c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

A number of io_handler_thread and many other InnoDB threads were idle. srv_shutdown_state was SRV_SHUTDOWN_INITIATED.

I suspect that this may have been fixed by ~~MDEV-16264~~ in 10.5 already (closed on 2019-11-15, 2 months after I had set the 10.5 version on this ticket).

Generated at Thu Feb 08 08:13:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.

[MDEV-14486] InnoDB hang on shutdown Created: 2017-11-23 Updated: 2023-08-31