[MDEV-20528] innodb.purge_secondary_mdev-16222 failed in buildbot, debug sync point wait timed out - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.3(EOL)
Fix Version/s: 10.3.23, 10.4.13
Component/s: Tests
Labels:
None

Description

http://buildbot.askmonty.org/buildbot/builders/win32-debug/builds/15203

10.3 e8de75db88acdf228237fbebad7f4c8f
innodb.purge_secondary_mdev-16222 'innodb' w3 [ fail ]
Test ended at 2019-08-21 09:26:19

CURRENT_TEST: innodb.purge_secondary_mdev-16222
--- D:/win32-debug/build/src/mysql-test/suite/innodb/r/purge_secondary_mdev-16222.result 2019-08-21 09:03:50.847375800 +0000
+++ D:\win32-debug\build\src\mysql-test\suite\innodb\r\purge_secondary_mdev-16222.reject 2019-08-21 09:26:18.243301400 +0000
@@ -13,9 +13,13 @@
select * into outfile 'load.data' from t1;
load data infile 'load.data' replace into table t1;
set debug_sync= "now WAIT_FOR latch_released";
+Warnings:
+Warning 1639 debug sync point wait timed out
set global debug_dbug= "-d,ib_purge_virtual_mdev_16222_1";
drop table t1;
set debug_sync= "now SIGNAL drop_started WAIT_FOR got_no_such_table";
+Warnings:
+Warning 1639 debug sync point wait timed out
create table t1 (
pk serial, vb tinyblob as (b) virtual, b tinyblob,
primary key(pk), index (vb(64)))

mysqltest: Result length mismatch

Attachments

Issue Links

is caused by

MDEV-16222 Assertion `0' failed in row_purge_remove_sec_if_poss_leaf on table with virtual columns and indexes

Closed

Activity

Ascending order - Click to sort in descending order

Marko Mäkelä added a comment - 2019-09-09 05:44

The test is using DEBUG_SYNC despite no have_debug_sync.inc. It is also introducing a new DBUG_EXECUTE_IF fault injection point. Was there really no other way to achieve this?

The failure does not repeat easily, and I did not review the change that introduced this test. It seems to me that there could be some missing synchronization in the test. Usually, in purge-related tests you need wait_all_purged.inc. Sometimes, you must also prevent purge until a specific time, by creating a read view and holding it open until the critical moment.

I repeated the hang by running the test with --repeat=1000 and several instances of the test in parallel with itself. After the test output ceased at the end, there were a few hung tests. I did killall -ABRT mysqld to get stack traces:

10.3 604f80e77c054758aa449064cdc29dfa13a71922
Program terminated with signal SIGABRT, Aborted.
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
56 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7f6683ac7c00 (LWP 26397))]
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x00005589086876a7 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.3/sql/signal_handler.cc:343
#2 <signal handler called>
#3 0x00007f6683d05819 in __GI___poll (fds=0x7ffe802bec00, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#4 0x0000558908342f02 in handle_connections_sockets () at /mariadb/10.3/sql/mysqld.cc:6761
#5 0x000055890833d3f2 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.3/sql/mysqld.cc:6230
#6 0x00007f6683c3b09b in __libc_start_main (main=0x558908336ec0 <main(int, char**)>, argc=24, argv=0x7ffe802befe8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe802befd8) at ../csu/libc-start.c:308
#7 0x0000558908336dfa in _start ()

Thread 24 (Thread 0x7f6652ffd700 (LWP 26938)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6652ffcd30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6652ffcd68, cond=0x7f6652ffcd08) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x7f6652ffcd08, mutex=0x7f6652ffcd68) at pthread_cond_wait.c:655
#3 0x0000558908e72e2c in safe_cond_wait (cond=0x7f6652ffcd08, mp=0x7f6652ffcd40, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1178) at /mariadb/10.3/mysys/thr_mutex.c:492
#4 0x000055890886793d in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_line=305, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1178
#5 thd_destructor_proxy () at /mariadb/10.3/storage/innobase/handler/ha_innodb.cc:305
#6 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b404ea8) at /mariadb/10.3/storage/perfschema/pfs.cc:1862
#7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 23 (Thread 0x7f66627fc700 (LWP 26725)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f66627fbbd0, expected=0, futex_word=0x55890b3ea818) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f66627fbbd0, mutex=0x55890b3ea7c8, cond=0x55890b3ea7f0) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x55890b3ea7f0, mutex=0x55890b3ea7c8, abstime=0x7f66627fbbd0) at pthread_cond_wait.c:667
#3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f66627fbbd0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275
#4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b3ea7b0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385
#5 0x0000558908afc9cc in ib_wqueue_timedwait (wq=0x55890b467c80, wait_in_usecs=5000000) at /mariadb/10.3/storage/innobase/ut/ut0wqueue.cc:161
#6 0x0000558908c9b772 in fts_optimize_thread (arg=0x55890b467c80) at /mariadb/10.3/storage/innobase/fts/fts0opt.cc:2834
#7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 22 (Thread 0x7f667e3e6700 (LWP 26533)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f667e3e5d50, expected=0, futex_word=0x558909fc67f8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f667e3e5d50, mutex=0x558909fc6740 <LOCK_checkpoint+40>, cond=0x558909fc67d0 <COND_checkpoint>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x558909fc67d0 <COND_checkpoint>, mutex=0x558909fc6740 <LOCK_checkpoint+40>, abstime=0x7f667e3e5d50) at pthread_cond_wait.c:667
#3 0x0000558908e73074 in safe_cond_timedwait (cond=0x558909fc67d0 <COND_checkpoint>, mp=0x558909fc6718 <LOCK_checkpoint>, abstime=0x7f667e3e5d50, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.3/mysys/thr_mutex.c:546
#4 0x0000558908d2fe95 in inline_mysql_cond_timedwait (that=0x558909fc67d0 <COND_checkpoint>, mutex=0x558909fc6718 <LOCK_checkpoint>, abstime=<optimized out>, src_line=116, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1204
#5 my_service_thread_sleep (control=0x5589096c63c0 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/10.3/storage/maria/ma_servicethread.c:115
#6 0x0000558908d270b5 in ma_checkpoint_background (arg=<optimized out>) at /mariadb/10.3/storage/maria/ma_checkpoint.c:707
#7 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b21a058) at /mariadb/10.3/storage/perfschema/pfs.cc:1862
#8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7f667ccc7700 (LWP 27171)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f667ccc5170, expected=0, futex_word=0x55890974b2fc <debug_sync_global+76>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f667ccc5170, mutex=0x55890974b330 <debug_sync_global+128>, cond=0x55890974b2d0 <debug_sync_global+32>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x55890974b2d0 <debug_sync_global+32>, mutex=0x55890974b330 <debug_sync_global+128>, abstime=0x7f667ccc5170) at pthread_cond_wait.c:667
#3 0x0000558908e73074 in safe_cond_timedwait (cond=0x55890974b2d0 <debug_sync_global+32>, mp=0x55890974b308 <debug_sync_global+88>, abstime=0x7f667ccc5170, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.3/mysys/thr_mutex.c:546
#4 0x0000558908454f5a in inline_mysql_cond_timedwait (abstime=0x5d75e5fd, src_line=1444, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1204
#5 debug_sync_execute (thd=<optimized out>, action=0x7f66342535a8) at /mariadb/10.3/sql/debug_sync.cc:1442
#6 debug_sync (thd=0x7f6634000cf8, sync_point_name=<optimized out>, name_len=<optimized out>) at /mariadb/10.3/sql/debug_sync.cc:1552
#7 0x0000558908456c35 in debug_sync_set_action (thd=<optimized out>, action=<optimized out>) at /mariadb/10.3/sql/debug_sync.cc:776
#8 debug_sync_eval_action (thd=<optimized out>, action_str=<optimized out>, action_end=<optimized out>) at /mariadb/10.3/sql/debug_sync.cc:1225
#9 0x0000558908455bab in debug_sync_update (thd=0x7f6634000cf8, val_str=0x7f6634011f20 "now", len=27) at /mariadb/10.3/sql/debug_sync.cc:1261
#10 0x0000558908359f2d in sys_var::update (this=0x55890975f9f0 <Sys_debug_sync>, thd=0x7f6634000cf8, var=0x7f6634011ed0) at /mariadb/10.3/sql/set_var.cc:213
#11 0x000055890835b6ef in set_var::update (this=0x7f6634011ed0, thd=0x7f6634000cf8) at /mariadb/10.3/sql/set_var.cc:295
#12 0x000055890835b387 in sql_set_variables (thd=0x7f6634000cf8, var_list=0x7f66340059c0, free=<optimized out>) at /mariadb/10.3/sql/set_var.cc:740
#13 0x0000558908431641 in mysql_execute_command (thd=0x7f6634000cf8) at /mariadb/10.3/sql/sql_parse.cc:4877
#14 0x000055890842adbd in mysql_parse (thd=0x7f6634000cf8, rawbuf=0x7f6634011cf0 "set debug_sync= \"now WAIT_FOR latch_released\"", length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /mariadb/10.3/sql/sql_parse.cc:7830
#15 0x0000558908426b07 in dispatch_command (command=COM_QUERY, thd=0x7f6634000cf8, packet=0x7f6634011cf0 "set debug_sync= \"now WAIT_FOR latch_released\"", packet_length=872432936, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /mariadb/10.3/sql/sql_parse.cc:1855
#16 0x0000558908428fc8 in do_command (thd=0x7f6634000cf8) at /mariadb/10.3/sql/sql_parse.cc:1401
#17 0x00005589085499e0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/10.3/sql/sql_connect.cc:1403
#18 0x0000558908549733 in handle_one_connection (arg=0x55890b535a58) at /mariadb/10.3/sql/sql_connect.cc:1308
#19 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b4b9b78) at /mariadb/10.3/storage/perfschema/pfs.cc:1862
#20 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#21 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7f6662ffd700 (LWP 26724)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6662ffcdd0, expected=0, futex_word=0x55890b454168) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f6662ffcdd0, mutex=0x55890b454118, cond=0x55890b454140) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x55890b454140, mutex=0x55890b454118, abstime=0x7f6662ffcdd0) at pthread_cond_wait.c:667
#3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f6662ffcdd0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275
#4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b454100, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385
#5 0x0000558908c34f23 in dict_stats_thread () at /mariadb/10.3/storage/innobase/dict/dict0stats_bg.cc:484
#6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7f667cd5d700 (LWP 27034)):
#0 0x00007f6683c4f4cc in __GI___sigtimedwait (set=set@entry=0x7f667cd5cda8, info=info@entry=0x7f667cd5cc70, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1 0x00007f66848e32bc in __sigwait (set=0x7f667cd5cda8, sig=0x7f667cd5cd20) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2 0x000055890833b65b in signal_hand (arg=<optimized out>) at /mariadb/10.3/sql/mysqld.cc:3554
#3 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b4acd98) at /mariadb/10.3/storage/perfschema/pfs.cc:1862
#4 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f6660ff9700 (LWP 26931)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x55890b2612e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x55890b261298, cond=0x55890b2612c0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x55890b2612c0, mutex=0x55890b261298) at pthread_cond_wait.c:655
#3 0x000055890895dcd6 in os_event::wait (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x55890b261280, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:325
#5 os_event_wait_low (event=0x55890b261280, reset_sig_count=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:502
#6 0x0000558908bafbba in buf_dump_thread () at /mariadb/10.3/storage/innobase/buf/buf0dump.cc:835
#7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f66777fe700 (LWP 26630)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922
#3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f66777fdb90, m1=0x7f66777fdc50, m2=0x7f66777fdc58, request=0x7f66777fdde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081
#4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f66777fdc50, m2=0x7f66777fdc58, request=0x7f66777fdde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135
#5 os_aio_handler (segment=<optimized out>, m1=0x7f66777fdc50, m2=0x7f66777fdc58, request=0x7f66777fdde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682
#6 0x0000558908c4fe9d in fil_aio_wait (segment=2) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439
#7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326
#8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f667cd12700 (LWP 27042)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x5589097412fc <COND_slave_background+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x558909744560 <LOCK_slave_background+40>, cond=0x5589097412d0 <COND_slave_background>) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x5589097412d0 <COND_slave_background>, mutex=0x558909744560 <LOCK_slave_background+40>) at pthread_cond_wait.c:655
#3 0x0000558908e72e2c in safe_cond_wait (cond=0x5589097412d0 <COND_slave_background>, mp=0x558909744538 <LOCK_slave_background>, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1167) at /mariadb/10.3/mysys/thr_mutex.c:492
#4 0x000055890835db8a in inline_mysql_cond_wait (that=0x5589097412d0 <COND_slave_background>, mutex=0x558909744538 <LOCK_slave_background>, src_line=513, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1167
#5 handle_slave_background (arg=<optimized out>) at /mariadb/10.3/sql/slave.cc:513
#6 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b4ab928) at /mariadb/10.3/storage/perfschema/pfs.cc:1862
#7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f6676ffd700 (LWP 26631)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922
#3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f6676ffcb90, m1=0x7f6676ffcc50, m2=0x7f6676ffcc58, request=0x7f6676ffcde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081
#4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6676ffcc50, m2=0x7f6676ffcc58, request=0x7f6676ffcde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135
#5 os_aio_handler (segment=<optimized out>, m1=0x7f6676ffcc50, m2=0x7f6676ffcc58, request=0x7f6676ffcde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682
#6 0x0000558908c4fe9d in fil_aio_wait (segment=3) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439
#7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326
#8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f66767fc700 (LWP 26632)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922
#3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f66767fbb90, m1=0x7f66767fbc50, m2=0x7f66767fbc58, request=0x7f66767fbde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081
#4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f66767fbc50, m2=0x7f66767fbc58, request=0x7f66767fbde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135
#5 os_aio_handler (segment=<optimized out>, m1=0x7f66767fbc50, m2=0x7f66767fbc58, request=0x7f66767fbde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682
#6 0x0000558908c4fe9d in fil_aio_wait (segment=4) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439
#7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326
#8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f667cb63700 (LWP 26627)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922
#3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f667cb62b90, m1=0x7f667cb62c50, m2=0x7f667cb62c58, request=0x7f667cb62de0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081
#4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f667cb62c50, m2=0x7f667cb62c58, request=0x7f667cb62de0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135
#5 os_aio_handler (segment=<optimized out>, m1=0x7f667cb62c50, m2=0x7f667cb62c58, request=0x7f667cb62de0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682
#6 0x0000558908c4fe9d in fil_aio_wait (segment=0) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439
#7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326
#8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f6675ffb700 (LWP 26633)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922
#3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f6675ffab90, m1=0x7f6675ffac50, m2=0x7f6675ffac58, request=0x7f6675ffade0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081
#4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6675ffac50, m2=0x7f6675ffac58, request=0x7f6675ffade0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135
#5 os_aio_handler (segment=<optimized out>, m1=0x7f6675ffac50, m2=0x7f6675ffac58, request=0x7f6675ffade0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682
#6 0x0000558908c4fe9d in fil_aio_wait (segment=5) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439
#7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326
#8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f6661ffb700 (LWP 26783)):
#0 0x00007f66848e2bf0 in __GI___nanosleep (requested_time=0x7f6661ffac48, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x000055890895df6f in os_thread_sleep (tm=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0thread.cc:225
#2 0x0000558908a98c3e in srv_master_sleep () at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2344
#3 srv_master_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2385
#4 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#5 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f66637fe700 (LWP 26723)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f66637fddb0, expected=0, futex_word=0x55890b261268) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f66637fddb0, mutex=0x55890b261218, cond=0x55890b261240) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x55890b261240, mutex=0x55890b261218, abstime=0x7f66637fddb0) at pthread_cond_wait.c:667
#3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f66637fddb0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275
#4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b261200, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385
#5 0x0000558908a97890 in srv_monitor_thread () at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:1694
#6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f66537fe700 (LWP 26937)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x55890b2613e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x55890b261398, cond=0x55890b2613c0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x55890b2613c0, mutex=0x55890b261398) at pthread_cond_wait.c:655
#3 0x000055890895dcd6 in os_event::wait (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x55890b261380, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:325
#5 os_event_wait_low (event=0x55890b261380, reset_sig_count=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:502
#6 0x0000558908b881f4 in buf_resize_thread () at /mariadb/10.3/storage/innobase/buf/buf0buf.cc:3230
#7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f6677fff700 (LWP 26629)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1
#2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922
#3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f6677ffeb90, m1=0x7f6677ffec50, m2=0x7f6677ffec58, request=0x7f6677ffede0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081
#4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6677ffec50, m2=0x7f6677ffec58, request=0x7f6677ffede0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135
#5 os_aio_handler (segment=<optimized out>, m1=0x7f6677ffec50, m2=0x7f6677ffec58, request=0x7f6677ffede0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682
#6 0x0000558908c4fe9d in fil_aio_wait (segment=1) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439
#7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326
#8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f66617fa700 (LWP 26784)):
#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x55890b261168) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x55890b261118, cond=0x55890b261140) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x55890b261140, mutex=0x55890b261118) at pthread_cond_wait.c:655
#3 0x000055890895dcd6 in os_event::wait (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:158
#4 os_event::wait_low (this=0x55890b261100, reset_sig_count=1337) at /mariadb/10.3/storage/innobase/os/os0event.cc:325
#5 os_event_wait_low (event=0x55890b261100, reset_sig_count=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:502
#6 0x0000558908a99d4f in srv_resume_thread (slot=0x5589096c2b48 <srv_sys+328>, sig_count=1337, wait=<optimized out>, timeout_usec=0) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:915
#7 0x0000558908a9aa2e in srv_purge_coordinator_suspend (slot=<optimized out>, rseg_history_len=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2654
#8 srv_purge_coordinator_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2731
#9 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#10 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f6674ff9700 (LWP 26722)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6674ff8c40, expected=0, futex_word=0x55890b2611e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f6674ff8c40, mutex=0x55890b261198, cond=0x55890b2611c0) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x55890b2611c0, mutex=0x55890b261198, abstime=0x7f6674ff8c40) at pthread_cond_wait.c:667
#3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f6674ff8c40) at /mariadb/10.3/storage/innobase/os/os0event.cc:275
#4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b261180, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385
#5 0x0000558908a9833d in srv_error_monitor_thread () at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:1858
#6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f66757fa700 (LWP 26634)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f66757f9b80, expected=0, futex_word=0x55890b26136c) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f66757f9b80, mutex=0x55890b261318, cond=0x55890b261340) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x55890b261340, mutex=0x55890b261318, abstime=0x7f66757f9b80) at pthread_cond_wait.c:667
#3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f66757f9b80) at /mariadb/10.3/storage/innobase/os/os0event.cc:275
#4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b261300, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/10.3/storage/innobase/os/os0event.cc:385
#5 0x0000558908bbc8ab in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=<optimized out>, cur_time=<optimized out>) at /mariadb/10.3/storage/innobase/buf/buf0flu.cc:2679
#6 buf_flush_page_cleaner_coordinator () at /mariadb/10.3/storage/innobase/buf/buf0flu.cc:3138
#7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f667f750700 (LWP 26481)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f667f74fde0, expected=0, futex_word=0x558909fd7ab8 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f667f74fde0, mutex=0x558909fd7a08 <LOCK_timer+40>, cond=0x558909fd7a90 <COND_timer>) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x558909fd7a90 <COND_timer>, mutex=0x558909fd7a08 <LOCK_timer+40>, abstime=0x7f667f74fde0) at pthread_cond_wait.c:667
#3 0x0000558908e73074 in safe_cond_timedwait (cond=0x558909fd7a90 <COND_timer>, mp=0x558909fd79e0 <LOCK_timer>, abstime=0x7f667f74fde0, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.3/mysys/thr_mutex.c:546
#4 0x0000558908e73bc4 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=<optimized out>, src_line=292, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1204
#5 timer_handler (arg=<optimized out>) at /mariadb/10.3/mysys/thr_timer.c:292
#6 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b141d48) at /mariadb/10.3/storage/perfschema/pfs.cc:1862
#7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f6653fff700 (LWP 26936)):
#0 0x00007f66848e2bf0 in __GI___nanosleep (requested_time=0x7f6653ffe748, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1 0x000055890895df6f in os_thread_sleep (tm=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0thread.cc:225
#2 0x0000558908b825ad in btr_defragment_thread () at /mariadb/10.3/storage/innobase/btr/btr0defragment.cc:708
#3 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#4 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f6663fff700 (LWP 26721)):
#0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6663ffeda0, expected=0, futex_word=0x55890b422bc8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 __pthread_cond_wait_common (abstime=0x7f6663ffeda0, mutex=0x55890b422b78, cond=0x55890b422ba0) at pthread_cond_wait.c:539
#2 __pthread_cond_timedwait (cond=0x55890b422ba0, mutex=0x55890b422b78, abstime=0x7f6663ffeda0) at pthread_cond_wait.c:667
#3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f6663ffeda0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275
#4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b422b60, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385
#5 0x0000558908926770 in lock_wait_timeout_thread () at /mariadb/10.3/storage/innobase/lock/lock0wait.cc:522
#6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f6683ac7c00 (LWP 26397)):
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1 0x00005589086876a7 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.3/sql/signal_handler.cc:343
#2 <signal handler called>
#3 0x00007f6683d05819 in __GI___poll (fds=0x7ffe802bec00, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#4 0x0000558908342f02 in handle_connections_sockets () at /mariadb/10.3/sql/mysqld.cc:6761
#5 0x000055890833d3f2 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.3/sql/mysqld.cc:6230
#6 0x00007f6683c3b09b in __libc_start_main (main=0x558908336ec0 <main(int, char**)>, argc=24, argv=0x7ffe802befe8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe802befd8) at ../csu/libc-start.c:308
#7 0x0000558908336dfa in _start ()

Marko Mäkelä added a comment - 2019-09-09 05:44 The test is using DEBUG_SYNC despite no have_debug_sync.inc . It is also introducing a new DBUG_EXECUTE_IF fault injection point. Was there really no other way to achieve this? The failure does not repeat easily, and I did not review the change that introduced this test. It seems to me that there could be some missing synchronization in the test. Usually, in purge-related tests you need wait_all_purged.inc . Sometimes, you must also prevent purge until a specific time, by creating a read view and holding it open until the critical moment. I repeated the hang by running the test with --repeat=1000 and several instances of the test in parallel with itself. After the test output ceased at the end, there were a few hung tests. I did killall -ABRT mysqld to get stack traces: 10.3 604f80e77c054758aa449064cdc29dfa13a71922 Program terminated with signal SIGABRT, Aborted. #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56 56 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory. [Current thread is 1 (Thread 0x7f6683ac7c00 (LWP 26397))] #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56 #1 0x00005589086876a7 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.3/sql/signal_handler.cc:343 #2 <signal handler called> #3 0x00007f6683d05819 in __GI___poll (fds=0x7ffe802bec00, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #4 0x0000558908342f02 in handle_connections_sockets () at /mariadb/10.3/sql/mysqld.cc:6761 #5 0x000055890833d3f2 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.3/sql/mysqld.cc:6230 #6 0x00007f6683c3b09b in __libc_start_main (main=0x558908336ec0 <main(int, char**)>, argc=24, argv=0x7ffe802befe8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe802befd8) at ../csu/libc-start.c:308 #7 0x0000558908336dfa in _start () Thread 24 (Thread 0x7f6652ffd700 (LWP 26938)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x7f6652ffcd30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x7f6652ffcd68, cond=0x7f6652ffcd08) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x7f6652ffcd08, mutex=0x7f6652ffcd68) at pthread_cond_wait.c:655 #3 0x0000558908e72e2c in safe_cond_wait (cond=0x7f6652ffcd08, mp=0x7f6652ffcd40, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1178) at /mariadb/10.3/mysys/thr_mutex.c:492 #4 0x000055890886793d in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_line=305, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1178 #5 thd_destructor_proxy () at /mariadb/10.3/storage/innobase/handler/ha_innodb.cc:305 #6 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b404ea8) at /mariadb/10.3/storage/perfschema/pfs.cc:1862 #7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 23 (Thread 0x7f66627fc700 (LWP 26725)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f66627fbbd0, expected=0, futex_word=0x55890b3ea818) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f66627fbbd0, mutex=0x55890b3ea7c8, cond=0x55890b3ea7f0) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x55890b3ea7f0, mutex=0x55890b3ea7c8, abstime=0x7f66627fbbd0) at pthread_cond_wait.c:667 #3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f66627fbbd0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275 #4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b3ea7b0, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385 #5 0x0000558908afc9cc in ib_wqueue_timedwait (wq=0x55890b467c80, wait_in_usecs=5000000) at /mariadb/10.3/storage/innobase/ut/ut0wqueue.cc:161 #6 0x0000558908c9b772 in fts_optimize_thread (arg=0x55890b467c80) at /mariadb/10.3/storage/innobase/fts/fts0opt.cc:2834 #7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 22 (Thread 0x7f667e3e6700 (LWP 26533)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f667e3e5d50, expected=0, futex_word=0x558909fc67f8 <COND_checkpoint+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f667e3e5d50, mutex=0x558909fc6740 <LOCK_checkpoint+40>, cond=0x558909fc67d0 <COND_checkpoint>) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x558909fc67d0 <COND_checkpoint>, mutex=0x558909fc6740 <LOCK_checkpoint+40>, abstime=0x7f667e3e5d50) at pthread_cond_wait.c:667 #3 0x0000558908e73074 in safe_cond_timedwait (cond=0x558909fc67d0 <COND_checkpoint>, mp=0x558909fc6718 <LOCK_checkpoint>, abstime=0x7f667e3e5d50, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.3/mysys/thr_mutex.c:546 #4 0x0000558908d2fe95 in inline_mysql_cond_timedwait (that=0x558909fc67d0 <COND_checkpoint>, mutex=0x558909fc6718 <LOCK_checkpoint>, abstime=<optimized out>, src_line=116, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1204 #5 my_service_thread_sleep (control=0x5589096c63c0 <checkpoint_control>, sleep_time=<optimized out>) at /mariadb/10.3/storage/maria/ma_servicethread.c:115 #6 0x0000558908d270b5 in ma_checkpoint_background (arg=<optimized out>) at /mariadb/10.3/storage/maria/ma_checkpoint.c:707 #7 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b21a058) at /mariadb/10.3/storage/perfschema/pfs.cc:1862 #8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 21 (Thread 0x7f667ccc7700 (LWP 27171)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f667ccc5170, expected=0, futex_word=0x55890974b2fc <debug_sync_global+76>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f667ccc5170, mutex=0x55890974b330 <debug_sync_global+128>, cond=0x55890974b2d0 <debug_sync_global+32>) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x55890974b2d0 <debug_sync_global+32>, mutex=0x55890974b330 <debug_sync_global+128>, abstime=0x7f667ccc5170) at pthread_cond_wait.c:667 #3 0x0000558908e73074 in safe_cond_timedwait (cond=0x55890974b2d0 <debug_sync_global+32>, mp=0x55890974b308 <debug_sync_global+88>, abstime=0x7f667ccc5170, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.3/mysys/thr_mutex.c:546 #4 0x0000558908454f5a in inline_mysql_cond_timedwait (abstime=0x5d75e5fd, src_line=1444, that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1204 #5 debug_sync_execute (thd=<optimized out>, action=0x7f66342535a8) at /mariadb/10.3/sql/debug_sync.cc:1442 #6 debug_sync (thd=0x7f6634000cf8, sync_point_name=<optimized out>, name_len=<optimized out>) at /mariadb/10.3/sql/debug_sync.cc:1552 #7 0x0000558908456c35 in debug_sync_set_action (thd=<optimized out>, action=<optimized out>) at /mariadb/10.3/sql/debug_sync.cc:776 #8 debug_sync_eval_action (thd=<optimized out>, action_str=<optimized out>, action_end=<optimized out>) at /mariadb/10.3/sql/debug_sync.cc:1225 #9 0x0000558908455bab in debug_sync_update (thd=0x7f6634000cf8, val_str=0x7f6634011f20 "now", len=27) at /mariadb/10.3/sql/debug_sync.cc:1261 #10 0x0000558908359f2d in sys_var::update (this=0x55890975f9f0 <Sys_debug_sync>, thd=0x7f6634000cf8, var=0x7f6634011ed0) at /mariadb/10.3/sql/set_var.cc:213 #11 0x000055890835b6ef in set_var::update (this=0x7f6634011ed0, thd=0x7f6634000cf8) at /mariadb/10.3/sql/set_var.cc:295 #12 0x000055890835b387 in sql_set_variables (thd=0x7f6634000cf8, var_list=0x7f66340059c0, free=<optimized out>) at /mariadb/10.3/sql/set_var.cc:740 #13 0x0000558908431641 in mysql_execute_command (thd=0x7f6634000cf8) at /mariadb/10.3/sql/sql_parse.cc:4877 #14 0x000055890842adbd in mysql_parse (thd=0x7f6634000cf8, rawbuf=0x7f6634011cf0 "set debug_sync= \"now WAIT_FOR latch_released\"", length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /mariadb/10.3/sql/sql_parse.cc:7830 #15 0x0000558908426b07 in dispatch_command (command=COM_QUERY, thd=0x7f6634000cf8, packet=0x7f6634011cf0 "set debug_sync= \"now WAIT_FOR latch_released\"", packet_length=872432936, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /mariadb/10.3/sql/sql_parse.cc:1855 #16 0x0000558908428fc8 in do_command (thd=0x7f6634000cf8) at /mariadb/10.3/sql/sql_parse.cc:1401 #17 0x00005589085499e0 in do_handle_one_connection (connect=<optimized out>) at /mariadb/10.3/sql/sql_connect.cc:1403 #18 0x0000558908549733 in handle_one_connection (arg=0x55890b535a58) at /mariadb/10.3/sql/sql_connect.cc:1308 #19 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b4b9b78) at /mariadb/10.3/storage/perfschema/pfs.cc:1862 #20 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #21 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 20 (Thread 0x7f6662ffd700 (LWP 26724)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6662ffcdd0, expected=0, futex_word=0x55890b454168) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f6662ffcdd0, mutex=0x55890b454118, cond=0x55890b454140) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x55890b454140, mutex=0x55890b454118, abstime=0x7f6662ffcdd0) at pthread_cond_wait.c:667 #3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f6662ffcdd0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275 #4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b454100, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385 #5 0x0000558908c34f23 in dict_stats_thread () at /mariadb/10.3/storage/innobase/dict/dict0stats_bg.cc:484 #6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 19 (Thread 0x7f667cd5d700 (LWP 27034)): #0 0x00007f6683c4f4cc in __GI___sigtimedwait (set=set@entry=0x7f667cd5cda8, info=info@entry=0x7f667cd5cc70, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29 #1 0x00007f66848e32bc in __sigwait (set=0x7f667cd5cda8, sig=0x7f667cd5cd20) at ../sysdeps/unix/sysv/linux/sigwait.c:28 #2 0x000055890833b65b in signal_hand (arg=<optimized out>) at /mariadb/10.3/sql/mysqld.cc:3554 #3 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b4acd98) at /mariadb/10.3/storage/perfschema/pfs.cc:1862 #4 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #5 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 18 (Thread 0x7f6660ff9700 (LWP 26931)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x55890b2612e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x55890b261298, cond=0x55890b2612c0) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x55890b2612c0, mutex=0x55890b261298) at pthread_cond_wait.c:655 #3 0x000055890895dcd6 in os_event::wait (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:158 #4 os_event::wait_low (this=0x55890b261280, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:325 #5 os_event_wait_low (event=0x55890b261280, reset_sig_count=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:502 #6 0x0000558908bafbba in buf_dump_thread () at /mariadb/10.3/storage/innobase/buf/buf0dump.cc:835 #7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 17 (Thread 0x7f66777fe700 (LWP 26630)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922 #3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f66777fdb90, m1=0x7f66777fdc50, m2=0x7f66777fdc58, request=0x7f66777fdde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081 #4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f66777fdc50, m2=0x7f66777fdc58, request=0x7f66777fdde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135 #5 os_aio_handler (segment=<optimized out>, m1=0x7f66777fdc50, m2=0x7f66777fdc58, request=0x7f66777fdde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682 #6 0x0000558908c4fe9d in fil_aio_wait (segment=2) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439 #7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326 #8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 16 (Thread 0x7f667cd12700 (LWP 27042)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x5589097412fc <COND_slave_background+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x558909744560 <LOCK_slave_background+40>, cond=0x5589097412d0 <COND_slave_background>) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x5589097412d0 <COND_slave_background>, mutex=0x558909744560 <LOCK_slave_background+40>) at pthread_cond_wait.c:655 #3 0x0000558908e72e2c in safe_cond_wait (cond=0x5589097412d0 <COND_slave_background>, mp=0x558909744538 <LOCK_slave_background>, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1167) at /mariadb/10.3/mysys/thr_mutex.c:492 #4 0x000055890835db8a in inline_mysql_cond_wait (that=0x5589097412d0 <COND_slave_background>, mutex=0x558909744538 <LOCK_slave_background>, src_line=513, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1167 #5 handle_slave_background (arg=<optimized out>) at /mariadb/10.3/sql/slave.cc:513 #6 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b4ab928) at /mariadb/10.3/storage/perfschema/pfs.cc:1862 #7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 15 (Thread 0x7f6676ffd700 (LWP 26631)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922 #3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f6676ffcb90, m1=0x7f6676ffcc50, m2=0x7f6676ffcc58, request=0x7f6676ffcde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081 #4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6676ffcc50, m2=0x7f6676ffcc58, request=0x7f6676ffcde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6676ffcc50, m2=0x7f6676ffcc58, request=0x7f6676ffcde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682 #6 0x0000558908c4fe9d in fil_aio_wait (segment=3) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439 #7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326 #8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 14 (Thread 0x7f66767fc700 (LWP 26632)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922 #3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f66767fbb90, m1=0x7f66767fbc50, m2=0x7f66767fbc58, request=0x7f66767fbde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081 #4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f66767fbc50, m2=0x7f66767fbc58, request=0x7f66767fbde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135 #5 os_aio_handler (segment=<optimized out>, m1=0x7f66767fbc50, m2=0x7f66767fbc58, request=0x7f66767fbde0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682 #6 0x0000558908c4fe9d in fil_aio_wait (segment=4) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439 #7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326 #8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 13 (Thread 0x7f667cb63700 (LWP 26627)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922 #3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f667cb62b90, m1=0x7f667cb62c50, m2=0x7f667cb62c58, request=0x7f667cb62de0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081 #4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f667cb62c50, m2=0x7f667cb62c58, request=0x7f667cb62de0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135 #5 os_aio_handler (segment=<optimized out>, m1=0x7f667cb62c50, m2=0x7f667cb62c58, request=0x7f667cb62de0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682 #6 0x0000558908c4fe9d in fil_aio_wait (segment=0) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439 #7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326 #8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 12 (Thread 0x7f6675ffb700 (LWP 26633)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922 #3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f6675ffab90, m1=0x7f6675ffac50, m2=0x7f6675ffac58, request=0x7f6675ffade0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081 #4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6675ffac50, m2=0x7f6675ffac58, request=0x7f6675ffade0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6675ffac50, m2=0x7f6675ffac58, request=0x7f6675ffade0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682 #6 0x0000558908c4fe9d in fil_aio_wait (segment=5) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439 #7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326 #8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 11 (Thread 0x7f6661ffb700 (LWP 26783)): #0 0x00007f66848e2bf0 in __GI___nanosleep (requested_time=0x7f6661ffac48, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 #1 0x000055890895df6f in os_thread_sleep (tm=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0thread.cc:225 #2 0x0000558908a98c3e in srv_master_sleep () at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2344 #3 srv_master_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2385 #4 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #5 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 10 (Thread 0x7f66637fe700 (LWP 26723)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f66637fddb0, expected=0, futex_word=0x55890b261268) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f66637fddb0, mutex=0x55890b261218, cond=0x55890b261240) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x55890b261240, mutex=0x55890b261218, abstime=0x7f66637fddb0) at pthread_cond_wait.c:667 #3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f66637fddb0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275 #4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b261200, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385 #5 0x0000558908a97890 in srv_monitor_thread () at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:1694 #6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 9 (Thread 0x7f66537fe700 (LWP 26937)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x55890b2613e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x55890b261398, cond=0x55890b2613c0) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x55890b2613c0, mutex=0x55890b261398) at pthread_cond_wait.c:655 #3 0x000055890895dcd6 in os_event::wait (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:158 #4 os_event::wait_low (this=0x55890b261380, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:325 #5 os_event_wait_low (event=0x55890b261380, reset_sig_count=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:502 #6 0x0000558908b881f4 in buf_resize_thread () at /mariadb/10.3/storage/innobase/buf/buf0buf.cc:3230 #7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 8 (Thread 0x7f6677fff700 (LWP 26629)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f668464327a in ?? () from /usr/lib/x86_64-linux-gnu/libaio.so.1 #2 0x0000558908951bad in LinuxAIOHandler::collect (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0file.cc:1922 #3 0x00005589089522d0 in LinuxAIOHandler::poll (this=0x7f6677ffeb90, m1=0x7f6677ffec50, m2=0x7f6677ffec58, request=0x7f6677ffede0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2081 #4 0x0000558908957055 in os_aio_linux_handler (global_segment=<optimized out>, m1=0x7f6677ffec50, m2=0x7f6677ffec58, request=0x7f6677ffede0) at /mariadb/10.3/storage/innobase/os/os0file.cc:2135 #5 os_aio_handler (segment=<optimized out>, m1=0x7f6677ffec50, m2=0x7f6677ffec58, request=0x7f6677ffede0) at /mariadb/10.3/storage/innobase/os/os0file.cc:5682 #6 0x0000558908c4fe9d in fil_aio_wait (segment=1) at /mariadb/10.3/storage/innobase/fil/fil0fil.cc:4439 #7 0x0000558908a9c798 in io_handler_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0start.cc:326 #8 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #9 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 7 (Thread 0x7f66617fa700 (LWP 26784)): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0x55890b261168) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0x55890b261118, cond=0x55890b261140) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0x55890b261140, mutex=0x55890b261118) at pthread_cond_wait.c:655 #3 0x000055890895dcd6 in os_event::wait (this=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:158 #4 os_event::wait_low (this=0x55890b261100, reset_sig_count=1337) at /mariadb/10.3/storage/innobase/os/os0event.cc:325 #5 os_event_wait_low (event=0x55890b261100, reset_sig_count=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0event.cc:502 #6 0x0000558908a99d4f in srv_resume_thread (slot=0x5589096c2b48 <srv_sys+328>, sig_count=1337, wait=<optimized out>, timeout_usec=0) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:915 #7 0x0000558908a9aa2e in srv_purge_coordinator_suspend (slot=<optimized out>, rseg_history_len=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2654 #8 srv_purge_coordinator_thread (arg=<optimized out>) at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:2731 #9 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #10 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7f6674ff9700 (LWP 26722)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6674ff8c40, expected=0, futex_word=0x55890b2611e8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f6674ff8c40, mutex=0x55890b261198, cond=0x55890b2611c0) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x55890b2611c0, mutex=0x55890b261198, abstime=0x7f6674ff8c40) at pthread_cond_wait.c:667 #3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f6674ff8c40) at /mariadb/10.3/storage/innobase/os/os0event.cc:275 #4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b261180, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385 #5 0x0000558908a9833d in srv_error_monitor_thread () at /mariadb/10.3/storage/innobase/srv/srv0srv.cc:1858 #6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 5 (Thread 0x7f66757fa700 (LWP 26634)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f66757f9b80, expected=0, futex_word=0x55890b26136c) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f66757f9b80, mutex=0x55890b261318, cond=0x55890b261340) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x55890b261340, mutex=0x55890b261318, abstime=0x7f66757f9b80) at pthread_cond_wait.c:667 #3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f66757f9b80) at /mariadb/10.3/storage/innobase/os/os0event.cc:275 #4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b261300, time_in_usec=<optimized out>, reset_sig_count=2) at /mariadb/10.3/storage/innobase/os/os0event.cc:385 #5 0x0000558908bbc8ab in pc_sleep_if_needed (next_loop_time=<optimized out>, sig_count=<optimized out>, cur_time=<optimized out>) at /mariadb/10.3/storage/innobase/buf/buf0flu.cc:2679 #6 buf_flush_page_cleaner_coordinator () at /mariadb/10.3/storage/innobase/buf/buf0flu.cc:3138 #7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 4 (Thread 0x7f667f750700 (LWP 26481)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f667f74fde0, expected=0, futex_word=0x558909fd7ab8 <COND_timer+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f667f74fde0, mutex=0x558909fd7a08 <LOCK_timer+40>, cond=0x558909fd7a90 <COND_timer>) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x558909fd7a90 <COND_timer>, mutex=0x558909fd7a08 <LOCK_timer+40>, abstime=0x7f667f74fde0) at pthread_cond_wait.c:667 #3 0x0000558908e73074 in safe_cond_timedwait (cond=0x558909fd7a90 <COND_timer>, mp=0x558909fd79e0 <LOCK_timer>, abstime=0x7f667f74fde0, file=0x558908ecdadf "/mariadb/10.3/include/mysql/psi/mysql_thread.h", line=1204) at /mariadb/10.3/mysys/thr_mutex.c:546 #4 0x0000558908e73bc4 in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=<optimized out>, src_line=292, src_file=<optimized out>) at /mariadb/10.3/include/mysql/psi/mysql_thread.h:1204 #5 timer_handler (arg=<optimized out>) at /mariadb/10.3/mysys/thr_timer.c:292 #6 0x0000558908e09856 in pfs_spawn_thread (arg=0x55890b141d48) at /mariadb/10.3/storage/perfschema/pfs.cc:1862 #7 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #8 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 3 (Thread 0x7f6653fff700 (LWP 26936)): #0 0x00007f66848e2bf0 in __GI___nanosleep (requested_time=0x7f6653ffe748, remaining=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28 #1 0x000055890895df6f in os_thread_sleep (tm=<optimized out>) at /mariadb/10.3/storage/innobase/os/os0thread.cc:225 #2 0x0000558908b825ad in btr_defragment_thread () at /mariadb/10.3/storage/innobase/btr/btr0defragment.cc:708 #3 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #4 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 2 (Thread 0x7f6663fff700 (LWP 26721)): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x7f6663ffeda0, expected=0, futex_word=0x55890b422bc8) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 __pthread_cond_wait_common (abstime=0x7f6663ffeda0, mutex=0x55890b422b78, cond=0x55890b422ba0) at pthread_cond_wait.c:539 #2 __pthread_cond_timedwait (cond=0x55890b422ba0, mutex=0x55890b422b78, abstime=0x7f6663ffeda0) at pthread_cond_wait.c:667 #3 0x000055890895d7db in os_event::timed_wait (this=<optimized out>, abstime=0x7f6663ffeda0) at /mariadb/10.3/storage/innobase/os/os0event.cc:275 #4 0x000055890895da66 in os_event::wait_time_low (this=0x55890b422b60, time_in_usec=<optimized out>, reset_sig_count=1) at /mariadb/10.3/storage/innobase/os/os0event.cc:385 #5 0x0000558908926770 in lock_wait_timeout_thread () at /mariadb/10.3/storage/innobase/lock/lock0wait.cc:522 #6 0x00007f66848d8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486 #7 0x00007f6683d104cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 1 (Thread 0x7f6683ac7c00 (LWP 26397)): #0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56 #1 0x00005589086876a7 in handle_fatal_signal (sig=<optimized out>) at /mariadb/10.3/sql/signal_handler.cc:343 #2 <signal handler called> #3 0x00007f6683d05819 in __GI___poll (fds=0x7ffe802bec00, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #4 0x0000558908342f02 in handle_connections_sockets () at /mariadb/10.3/sql/mysqld.cc:6761 #5 0x000055890833d3f2 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.3/sql/mysqld.cc:6230 #6 0x00007f6683c3b09b in __libc_start_main (main=0x558908336ec0 <main(int, char**)>, argc=24, argv=0x7ffe802befe8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe802befd8) at ../csu/libc-start.c:308 #7 0x0000558908336dfa in _start ()

Aleksey Midenkov added a comment - 2020-01-04 15:41

Test moved to `gcol.innodb_virtual_debug_purge`.

Added wait_all_purged.inc like was suggested. Nonetheless no such failures recently:

http://buildbot.askmonty.org/buildbot/reports/cross_reference#branch=&revision=&platform=&dt=&bbnum=&typ=&info=&fail_name=^gcol.innodb_virtual_debug_purge&fail_variant=&fail_info_short=&fail_info_full=got_no_such_table&limit=25

Though somewhat related segfault crash:

http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64-debug/builds/3154

Aleksey Midenkov added a comment - 2020-01-04 15:41 Test moved to `gcol.innodb_virtual_debug_purge`. Added wait_all_purged.inc like was suggested. Nonetheless no such failures recently: http://buildbot.askmonty.org/buildbot/reports/cross_reference#branch=&revision=&platform=&dt=&bbnum=&typ=&info=&fail_name= ^gcol.innodb_virtual_debug_purge&fail_variant=&fail_info_short=&fail_info_full=got_no_such_table&limit=25 Though somewhat related segfault crash: http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64-debug/builds/3154

People

Assignee:: Aleksey Midenkov

Reporter:: Elena Stepanova

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2019-09-08 14:37

Updated:: 2020-02-03 12:03

Resolved:: 2020-02-03 12:03

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server