Checking supported features... MariaDB Version 10.4.3-MariaDB-debug - SSL connections supported - binaries are debug compiled Using suites: galera_sr Collecting tests... Installing system database... Using parallel: 4 worker[3] - 'localhost:16000' was not free worker[3] Using MTR_BUILD_THREAD 303, with reserved ports 16060..16079 worker[4] - 'localhost:16000' was not free worker[4] Using MTR_BUILD_THREAD 304, with reserved ports 16080..16099 worker[1] Using MTR_BUILD_THREAD 302, with reserved ports 16040..16059 worker[2] Using MTR_BUILD_THREAD 301, with reserved ports 16020..16039 worker[1] Trying to dump core for [mysqltest - pid: 226217, winpid: 226217] worker[1] Trying to dump core for [mysqld.1 - pid: 223866, winpid: 223866] worker[1] Trying to dump core for [mysqld.2 - pid: 224402, winpid: 224402] worker[3] Trying to dump core for [mysqltest - pid: 228210, winpid: 228210] worker[3] Trying to dump core for [mysqld.1 - pid: 226888, winpid: 226888] worker[3] Trying to dump core for [mysqld.2 - pid: 227619, winpid: 227619] ============================================================================== TEST WORKER RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- galera_sr.GCF-574 [ disabled ] CTAS is not supported together with SR galera_sr.galera_sr_sbr [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.mysql-wsrep-features#29 [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.galera_sr_gtid 'innodb' w4 [ pass ] 7685 galera_sr.mysql-wsrep-features#32 'innodb' w2 [ pass ] 9457 galera_sr.galera_sr_log_bin 'innodb' w3 [ pass ] 13811 galera_sr.mysql-wsrep-features#136 'innodb' w4 [ pass ] 6664 galera_sr.galera_sr_v1_row_events 'innodb' w2 [ pass ] 6592 galera_sr.GCF-623 'innodb' w4 [ pass ] 8285 galera_sr.galera_sr_conflict_with_rollback_master 'innodb' w2 [ pass ] 7657 galera_sr.GCF-627 'innodb' w4 [ pass ] 8247 galera_sr.galera_sr_ddl_master 'innodb' w2 [ pass ] 11368 galera_sr.GCF-845 'innodb' w4 [ pass ] 10361 galera_sr.galera_sr_ddl_schema 'innodb' w2 [ pass ] 9390 galera_sr.GCF-851 'innodb' w4 [ pass ] 7881 galera_sr.galera_sr_ddl_slave 'innodb' w2 [ pass ] 12702 galera_sr.galera_sr_ddl_unrelated 'innodb' w2 [ pass ] 13982 galera_sr.galera_sr_dupkey_error 'innodb' w2 [ pass ] 13600 galera_sr.GCF-867 'innodb' w4 [ pass ] 52089 galera_sr.galera_sr_fk_conflict 'innodb' w2 [ pass ] 24853 galera_sr.GCF-889 'innodb' w4 [ pass ] 7376 galera_sr.GCF-900 'innodb' w4 [ pass ] 9945 galera_sr.galera_sr_large_fragment 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 12:32:03 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr_large_fragment.log == connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1024 * 1024 * 10; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; connection node_2; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT COUNT(*) > 50000 FROM t1; COUNT(*) > 50000 1 connection node_1; ROLLBACK; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.223869' (0/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.1/data/core.223869 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 223869] [New LWP 223909] [New LWP 223919] [New LWP 223924] [New LWP 223879] [New LWP 223936] [New LWP 223967] [New LWP 223966] [New LWP 223968] [New LWP 224086] [New LWP 224082] [New LWP 224087] [New LWP 224088] [New LWP 224094] [New LWP 224096] [New LWP 223964] [New LWP 223965] [New LWP 223914] [New LWP 223962] [New LWP 226309] [New LWP 224255] [New LWP 224254] [New LWP 224092] [New LWP 224090] [New LWP 224089] [New LWP 224085] [New LWP 224095] [New LWP 224093] [New LWP 224084] [New LWP 224083] [New LWP 223963] [New LWP 224091] [New LWP 223929] [New LWP 223930] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007ffa4dc589d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007ffa4dc589d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056191ecf1368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056191e52f503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ffa4c00720d in poll () from /lib64/libc.so.6 #5 0x000056191e0cfa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056191e0ced25 in mysqld_main (argc=163, argv=0x5619214c6d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056191e0c2ec5 in main (argc=25, argv=0x7fffa4cf32a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7ffa44268700 (LWP 223930)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffa45a56132 in fifo_lock_get (q=q@entry=0x561921673bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007ffa45a567a1 in gu_fifo_get_head (q=0x561921673bd0, err=0x7ffa4426706c) at galerautils/src/gu_fifo.c:330 #3 0x00007ffa45b380cc in gcs_recv (conn=0x56192164d550, action=0x7ffa442670b0) at gcs/src/gcs.cpp:2024 #4 0x00007ffa45b6516c in galera::GcsActionSource::process (this=0x56192164cf30, recv_ctx=0x7ffa442674d0, exit_loop=@0x7ffa4426729e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007ffa45b84033 in galera::ReplicatorSMM::async_recv (this=0x561921645280, recv_ctx=0x7ffa442674d0) at galera/src/replicator_smm.cpp:383 #6 0x000056191edab276 in wsrep::wsrep_provider_v26::run_applier (this=0x5619215cc360, applier_ctx=0x7ffa442674d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000056191e44e636 in wsrep_replication_process (thd=0x7ffa2c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000056191e440282 in start_wsrep_THD (arg=0x5619216a6d70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7ffa442b3700 (LWP 223929)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191ecf6997 in safe_cond_wait (cond=0x7ffa28015608, mp=0x7ffa28015558, file=0x56191ee60b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056191e44e088 in inline_mysql_cond_wait (that=0x7ffa28015608, mutex=0x7ffa28015558, src_file=0x56191ee60e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056191e450d20 in Wsrep_thd_queue::pop_front (this=0x7ffa28015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056191e44fb00 in wsrep_post_rollback_process (post_rollbacker=0x7ffa28000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000056191e440282 in start_wsrep_THD (arg=0x561921687810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ffa09ffb700 (LWP 224091)): #0 0x00007ffa4dc5ad1d in fsync () from /lib64/libpthread.so.0 #1 0x000056191e82dacb in os_file_fsync_posix (file=51) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056191e82de21 in os_file_flush_func (file=51) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056191ea763a2 in pfs_os_file_flush_func (file=..., src_file=0x56191f0f0a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056191ea7911d in fil_flush_low (space=0x561921b2f390, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056191ea84b11 in fil_flush (space=0x561921b2f390) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4468 #6 0x000056191ea09919 in buf_flush_write_block_low (bpage=0x7ffa273b5db0, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1027 #7 0x000056191ea09fb6 in buf_flush_page (buf_pool=0x56192187b160, bpage=0x7ffa273b5db0, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1165 #8 0x000056191ea0ca9b in buf_flush_single_page_from_LRU (buf_pool=0x56192187b160) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2138 #9 0x000056191ea17a06 in buf_LRU_get_free_block (buf_pool=0x56192187b160) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1194 #10 0x000056191e9f4d13 in buf_page_init_for_read (err=0x7ffa09ff9194, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5332 #11 0x000056191ea1e0be in buf_read_page_low (err=0x7ffa09ff9194, sync=true, type=0, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #12 0x000056191ea1ec9e in buf_read_page (page_id=..., zip_size=0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:412 #13 0x000056191e9f1bda in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x56191f0a7f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=8113, mtr=0x7ffa09ff9620, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4340 #14 0x000056191e9c6117 in btr_free_externally_stored_field (index=0x561921b30f48, field_ref=0x7ffa27a8c1e4 "", rec=0x7ffa27a8c19f "eca169b3-2de5-11e9-b28c-cef43aef6f27\200", offsets=0x7ff9ec0134b8, page_zip=0x0, i=6, rollback=false, local_mtr=0x7ffa09ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8113 #15 0x000056191e9c665a in btr_rec_free_externally_stored_fields (index=0x561921b30f48, rec=0x7ffa27a8c19f "eca169b3-2de5-11e9-b28c-cef43aef6f27\200", offsets=0x7ff9ec0134b8, page_zip=0x0, rollback=false, mtr=0x7ffa09ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8215 #16 0x000056191e9c0cb1 in btr_cur_pessimistic_delete (err=0x7ffa09ff9e5c, has_reserved_extents=0, cursor=0x5619219b73b0, flags=0, rollback=false, mtr=0x7ffa09ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5905 #17 0x000056191e8e6084 in row_purge_remove_clust_if_poss_low (node=0x5619219b7318, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:175 #18 0x000056191e8e6209 in row_purge_remove_clust_if_poss (node=0x5619219b7318) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #19 0x000056191e8e7a36 in row_purge_del_mark (node=0x5619219b7318) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #20 0x000056191e8e8fbe in row_purge_record_func (node=0x5619219b7318, undo_rec=0x5619219b7ee8 "", thr=0x5619219b6e70, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #21 0x000056191e8e92fb in row_purge (node=0x5619219b7318, undo_rec=0x5619219b7ee8 "", thr=0x5619219b6e70) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #22 0x000056191e8e96a6 in row_purge_step (thr=0x5619219b6e70) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #23 0x000056191e86ba9a in que_thr_step (thr=0x5619219b6e70) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #24 0x000056191e86bcca in que_run_threads_low (thr=0x5619219b6e70) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #25 0x000056191e86beba in que_run_threads (thr=0x5619219b6e70) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #26 0x000056191e928855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #27 0x000056191e9289fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #28 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ffa267ad700 (LWP 223963)): #0 0x00007ffa4d824644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056191e82c41c in LinuxAIOHandler::collect (this=0x7ffa267acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056191e82c945 in LinuxAIOHandler::poll (this=0x7ffa267acb00, m1=0x7ffa267acba8, m2=0x7ffa267acbb0, request=0x7ffa267acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056191e82caa8 in os_aio_linux_handler (global_segment=1, m1=0x7ffa267acba8, m2=0x7ffa267acbb0, request=0x7ffa267acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056191e8315cc in os_aio_handler (segment=1, m1=0x7ffa267acba8, m2=0x7ffa267acbb0, request=0x7ffa267acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056191ea844a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056191e92a909 in io_handler_thread (arg=0x56191fab1628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ffa19ffb700 (LWP 224083)): #0 0x00007ffa4dc5ae3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056191e839401 in os_thread_sleep (tm=30000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056191e934068 in sync_array_print_long_waits (waiter=0x7ffa19ffac28, sema=0x7ffa19ffac30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:1099 #3 0x000056191e926e88 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1801 #4 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ffa197fa700 (LWP 224084)): #0 0x00007ffa4dc57d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e838a1a in os_event::timed_wait (this=0x561921795b40, abstime=0x7ffa197f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056191e838d31 in os_event::wait_time_low (this=0x561921795b40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056191e838ed1 in os_event_wait_time_low (event=0x561921795b40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056191e926ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ffa08ff9700 (LWP 224093)): #0 0x00007ffa4dc5ae3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056191e839401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056191e9dfa3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ffa037fe700 (LWP 224095)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191ecf6997 in safe_cond_wait (cond=0x7ffa037fdc70, mp=0x7ffa037fdcb0, file=0x56191efc4c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056191e750487 in inline_mysql_cond_wait (that=0x7ffa037fdc70, mutex=0x7ffa037fdcb0, src_file=0x56191efc65d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056191e7550c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000056191ec8945f in pfs_spawn_thread (arg=0x5619219d6700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ffa18ff9700 (LWP 224085)): #0 0x00007ffa4dc57d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e838a1a in os_event::timed_wait (this=0x56192199fa10, abstime=0x7ffa18ff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056191e838d31 in os_event::wait_time_low (this=0x56192199fa10, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056191e838ed1 in os_event_wait_time_low (event=0x56192199fa10, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056191ea6e142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ffa0affd700 (LWP 224089)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e8390ff in os_event::wait (this=0x5619219ab890) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056191e838b85 in os_event::wait_low (this=0x5619219ab890, reset_sig_count=6) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056191e838ef6 in os_event_wait_low (event=0x5619219ab890, reset_sig_count=6) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056191e932bfc in sync_array_wait_event (arr=0x56192176cb40, cell=@0x7ffa0affcb18: 0x56192176cd30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056191e935911 in rw_lock_s_lock_spin (lock=0x56192199ff40, pass=0, file_name=0x56191f058d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000056191e8e3236 in rw_lock_s_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f058d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000056191e8e366f in pfs_rw_lock_s_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f058d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000056191e8e8a54 in row_purge_parse_undo_rec (node=0x5619219b6cc0, undo_rec=0x5619219b7198 "", updated_extern=0x7ffa0affcc9e, thr=0x5619219b6c08) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1077 #9 0x000056191e8e93df in row_purge (node=0x5619219b6cc0, undo_rec=0x5619219b7198 "", thr=0x5619219b6c08) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1258 #10 0x000056191e8e96a6 in row_purge_step (thr=0x5619219b6c08) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #11 0x000056191e86ba9a in que_thr_step (thr=0x5619219b6c08) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #12 0x000056191e86bcca in que_run_threads_low (thr=0x5619219b6c08) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000056191e86beba in que_run_threads (thr=0x5619219b6c08) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x000056191e928855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #15 0x000056191e9289fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #16 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ffa0a7fc700 (LWP 224090)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e8390ff in os_event::wait (this=0x5619219ab890) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056191e838b85 in os_event::wait_low (this=0x5619219ab890, reset_sig_count=6) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056191e838ef6 in os_event_wait_low (event=0x5619219ab890, reset_sig_count=6) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056191e932bfc in sync_array_wait_event (arr=0x56192176cb40, cell=@0x7ffa0a7fbb18: 0x56192176ccf0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056191e935911 in rw_lock_s_lock_spin (lock=0x56192199ff40, pass=0, file_name=0x56191f058d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000056191e8e3236 in rw_lock_s_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f058d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000056191e8e366f in pfs_rw_lock_s_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f058d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000056191e8e8a54 in row_purge_parse_undo_rec (node=0x5619219b6a58, undo_rec=0x5619219b7008 "", updated_extern=0x7ffa0a7fbc9e, thr=0x5619219b6888) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1077 #9 0x000056191e8e93df in row_purge (node=0x5619219b6a58, undo_rec=0x5619219b7008 "", thr=0x5619219b6888) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1258 #10 0x000056191e8e96a6 in row_purge_step (thr=0x5619219b6888) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #11 0x000056191e86ba9a in que_thr_step (thr=0x5619219b6888) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #12 0x000056191e86bcca in que_run_threads_low (thr=0x5619219b6888) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000056191e86beba in que_run_threads (thr=0x5619219b6888) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x000056191e928855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #15 0x000056191e9289fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #16 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ffa097fa700 (LWP 224092)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e8390ff in os_event::wait (this=0x561921795bc0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056191e838b85 in os_event::wait_low (this=0x561921795bc0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056191e838ef6 in os_event_wait_low (event=0x561921795bc0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056191ea04eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ffa3c1a2700 (LWP 224254)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191ecf6997 in safe_cond_wait (cond=0x56191fa361e0 , mp=0x56191fa39560 , file=0x56191edc1cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056191e0f44ee in inline_mysql_cond_wait (that=0x56191fa361e0 , mutex=0x56191fa39560 , src_file=0x56191edc2830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000056191e0f5d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056191ec8945f in pfs_spawn_thread (arg=0x561921a936f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ffa3c157700 (LWP 224255)): #0 0x00007ffa4dc57d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191ecf6c99 in safe_cond_timedwait (cond=0x56191fa361a0 , mp=0x56191fa38ba0 , abstime=0x7ffa3c156c60, file=0x56191edb24c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056191e0c3845 in inline_mysql_cond_timedwait (that=0x56191fa361a0 , mutex=0x56191fa38ba0 , abstime=0x7ffa3c156c60, src_file=0x56191edb4538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056191e0c861b in cache_thread (thd=0x7ff9d8000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000056191e0c88c9 in one_thread_per_connection_end (thd=0x7ff9d8000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000056191e3650f3 in do_handle_one_connection (connect=0x561921b044b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000056191e364dad in handle_one_connection (arg=0x561921b044b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000056191ec8945f in pfs_spawn_thread (arg=0x561921b26130) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ffa3c10c700 (LWP 226309)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e8390ff in os_event::wait (this=0x5619219ab890) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056191e838b85 in os_event::wait_low (this=0x5619219ab890, reset_sig_count=6) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056191e838ef6 in os_event_wait_low (event=0x5619219ab890, reset_sig_count=6) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056191e932bfc in sync_array_wait_event (arr=0x56192176cb40, cell=@0x7ffa3c109198: 0x56192176cc30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056191e935911 in rw_lock_s_lock_spin (lock=0x56192199ff40, pass=0, file_name=0x56191f066718 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000056191e8be034 in rw_lock_s_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f066718 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000056191e8be4a5 in pfs_rw_lock_s_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f066718 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000056191e8c733e in row_mysql_freeze_data_dictionary_func (trx=0x7ffa27c01268, file=0x56191f066718 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:2123 #9 0x000056191e90a64b in row_undo (node=0x7ff9ccff2c58, thr=0x7ff9cc23aa18) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc:429 #10 0x000056191e90a8b8 in row_undo_step (thr=0x7ff9cc23aa18) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc:497 #11 0x000056191e86ba7e in que_thr_step (thr=0x7ff9cc23aa18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1040 #12 0x000056191e86bcca in que_run_threads_low (thr=0x7ff9cc23aa18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000056191e86beba in que_run_threads (thr=0x7ff9cc23aa18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x000056191e9662bd in trx_rollback_to_savepoint_low (trx=0x7ffa27c01268, savept=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:137 #15 0x000056191e966776 in trx_rollback_for_mysql_low (trx=0x7ffa27c01268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:211 #16 0x000056191e966ab5 in trx_rollback_for_mysql (trx=0x7ffa27c01268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:237 #17 0x000056191e75b777 in innobase_rollback (hton=0x561921769c00, thd=0x7ff9cc000b00, rollback_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4718 #18 0x000056191e53546e in ha_rollback_trans (thd=0x7ff9cc000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1850 #19 0x000056191e37a41c in trans_rollback (thd=0x7ff9cc000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:418 #20 0x000056191e203b27 in mysql_execute_command (thd=0x7ff9cc000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5866 #21 0x000056191e20a935 in mysql_parse (thd=0x7ff9cc000b00, rawbuf=0x7ff9cc013e58 "ROLLBACK", length=8, parser_state=0x7ffa3c10b0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #22 0x000056191e209f68 in wsrep_mysql_parse (thd=0x7ff9cc000b00, rawbuf=0x7ff9cc013e58 "ROLLBACK", length=8, parser_state=0x7ffa3c10b0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #23 0x000056191e1f5e25 in dispatch_command (command=COM_QUERY, thd=0x7ff9cc000b00, packet=0x7ff9cc00a431 "ROLLBACK", packet_length=8, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #24 0x000056191e1f480d in do_command (thd=0x7ff9cc000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #25 0x000056191e365069 in do_handle_one_connection (connect=0x561921a2cb20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #26 0x000056191e364dad in handle_one_connection (arg=0x561921a2cb20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #27 0x000056191ec8945f in pfs_spawn_thread (arg=0x5619219e6020) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #28 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ffa26fae700 (LWP 223962)): #0 0x00007ffa4d824644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056191e82c41c in LinuxAIOHandler::collect (this=0x7ffa26fadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056191e82c945 in LinuxAIOHandler::poll (this=0x7ffa26fadb00, m1=0x7ffa26fadba8, m2=0x7ffa26fadbb0, request=0x7ffa26fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056191e82caa8 in os_aio_linux_handler (global_segment=0, m1=0x7ffa26fadba8, m2=0x7ffa26fadbb0, request=0x7ffa26fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056191e8315cc in os_aio_handler (segment=0, m1=0x7ffa26fadba8, m2=0x7ffa26fadbb0, request=0x7ffa26fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056191ea844a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056191e92a909 in io_handler_thread (arg=0x56191fab1620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ffa3e37c700 (LWP 223914)): #0 0x00007ffa4c012483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ffa45b036f8 in asio::detail::epoll_reactor::run (this=0x56192168dfc0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007ffa45b07df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x56192168dee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007ffa45b080a2 in asio::detail::task_io_service::run (this=0x56192168dee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007ffa45b1fafd in asio::io_service::run (this=0x56192168cfe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007ffa45b20251 in gcomm::AsioProtonet::event_loop (this=0x56192168cf40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007ffa45b3e2ed in GCommConn::run (this=0x56192168d6c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007ffa45b41399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ffa257ab700 (LWP 223965)): #0 0x00007ffa4d824644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056191e82c41c in LinuxAIOHandler::collect (this=0x7ffa257aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056191e82c945 in LinuxAIOHandler::poll (this=0x7ffa257aab00, m1=0x7ffa257aaba8, m2=0x7ffa257aabb0, request=0x7ffa257aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056191e82caa8 in os_aio_linux_handler (global_segment=3, m1=0x7ffa257aaba8, m2=0x7ffa257aabb0, request=0x7ffa257aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056191e8315cc in os_aio_handler (segment=3, m1=0x7ffa257aaba8, m2=0x7ffa257aabb0, request=0x7ffa257aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056191ea844a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056191e92a909 in io_handler_thread (arg=0x56191fab1638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ffa25fac700 (LWP 223964)): #0 0x00007ffa4d824644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056191e82c41c in LinuxAIOHandler::collect (this=0x7ffa25fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056191e82c945 in LinuxAIOHandler::poll (this=0x7ffa25fabb00, m1=0x7ffa25fabba8, m2=0x7ffa25fabbb0, request=0x7ffa25fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056191e82caa8 in os_aio_linux_handler (global_segment=2, m1=0x7ffa25fabba8, m2=0x7ffa25fabbb0, request=0x7ffa25fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056191e8315cc in os_aio_handler (segment=2, m1=0x7ffa25fabba8, m2=0x7ffa25fabbb0, request=0x7ffa25fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056191ea844a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056191e92a909 in io_handler_thread (arg=0x56191fab1630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ffa3c1ed700 (LWP 224096)): #0 0x00007ffa4bf4b40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000056191e0c2f44 in my_sigwait (set=0x7ffa3c1eccf0, sig=0x7ffa3c1eccb4, code=0x7ffa3c1eccb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000056191e0c9584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000056191ec8945f in pfs_spawn_thread (arg=0x561921a12a00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ffa03fff700 (LWP 224094)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e8390ff in os_event::wait (this=0x561921795cc0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056191e838b85 in os_event::wait_low (this=0x561921795cc0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056191e838ef6 in os_event_wait_low (event=0x561921795cc0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056191e9edb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ffa0b7fe700 (LWP 224088)): #0 0x00007ffa4bff6d47 in sched_yield () from /lib64/libc.so.6 #1 0x000056191e839393 in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x000056191e952bd9 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x000056191e952de1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x000056191e928cca in srv_do_purge (n_total_purged=0x7ffa0b7fdd98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x000056191e9290f7 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ffa0bfff700 (LWP 224087)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e8390ff in os_event::wait (this=0x5619219ab910) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056191e838b85 in os_event::wait_low (this=0x5619219ab910, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056191e838ef6 in os_event_wait_low (event=0x5619219ab910, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056191e932bfc in sync_array_wait_event (arr=0x56192176cb40, cell=@0x7ffa0bffeb98: 0x56192176cd70) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056191e935ae2 in rw_lock_x_lock_wait_func (lock=0x56192199ff40, pass=0, threshold=0, file_name=0x56191f076b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:459 #6 0x000056191e935c14 in rw_lock_x_lock_low (lock=0x56192199ff40, pass=0, file_name=0x56191f076b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:514 #7 0x000056191e936032 in rw_lock_x_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f076b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:673 #8 0x000056191e924443 in pfs_rw_lock_x_lock_func (lock=0x56192199ff40, pass=0, file_name=0x56191f076b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:552 #9 0x000056191e927486 in srv_master_evict_from_table_cache (pct_check=100) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1999 #10 0x000056191e927f82 in srv_master_do_idle_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2237 #11 0x000056191e928558 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2358 #12 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ffa1a7fc700 (LWP 224082)): #0 0x00007ffa4dc57d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e838a1a in os_event::timed_wait (this=0x56192196d950, abstime=0x7ffa1a7fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056191e838d31 in os_event::wait_time_low (this=0x56192196d950, time_in_usec=1000000, reset_sig_count=16) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056191e838ed1 in os_event_wait_time_low (event=0x56192196d950, time_in_usec=1000000, reset_sig_count=16) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056191e80588e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ffa1affd700 (LWP 224086)): #0 0x00007ffa4dc57d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e838a1a in os_event::timed_wait (this=0x561921935c20, abstime=0x7ffa1affcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056191e838d31 in os_event::wait_time_low (this=0x561921935c20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056191e838ed1 in os_event_wait_time_low (event=0x561921935c20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056191e9872fb in ib_wqueue_timedwait (wq=0x561921935af0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056191eac7ea4 in fts_optimize_thread (arg=0x561921935af0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ffa1b7fe700 (LWP 223968)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191e8390ff in os_event::wait (this=0x561921887480) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056191e838b85 in os_event::wait_low (this=0x561921887480, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056191e838ef6 in os_event_wait_low (event=0x561921887480, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056191e932bfc in sync_array_wait_event (arr=0x56192176cb40, cell=@0x7ffa1b7fd680: 0x56192176cc70) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056191e93647b in rw_lock_sx_lock_func (lock=0x7ffa273b7e28, pass=2, file_name=0x56191f0c7528 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x000056191ea05837 in pfs_rw_lock_sx_lock_func (lock=0x7ffa273b7e28, pass=2, file_name=0x56191f0c7528 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x000056191ea09eb4 in buf_flush_page (buf_pool=0x56192187b160, bpage=0x7ffa273b7d60, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1144 #8 0x000056191ea0a8fe in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=72, n_to_flush=200) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1389 #9 0x000056191ea0ac81 in buf_flush_page_and_try_neighbors (bpage=0x7ffa273b7d60, flush_type=BUF_FLUSH_LIST, n_to_flush=200, count=0x7ffa1b7fd8b8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1461 #10 0x000056191ea0b89a in buf_do_flush_list_batch (buf_pool=0x56192187b160, min_n=200, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1719 #11 0x000056191ea0be92 in buf_flush_batch (buf_pool=0x56192187b160, flush_type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7ffa1b7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1789 #12 0x000056191ea0c552 in buf_flush_do_batch (buf_pool=0x56192187b160, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7ffa1b7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1947 #13 0x000056191ea0c826 in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7ffa1b7fdbd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2053 #14 0x000056191ea10571 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3205 #15 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ffa24faa700 (LWP 223966)): #0 0x00007ffa4d824644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056191e82c41c in LinuxAIOHandler::collect (this=0x7ffa24fa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056191e82c945 in LinuxAIOHandler::poll (this=0x7ffa24fa9b00, m1=0x7ffa24fa9ba8, m2=0x7ffa24fa9bb0, request=0x7ffa24fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056191e82caa8 in os_aio_linux_handler (global_segment=4, m1=0x7ffa24fa9ba8, m2=0x7ffa24fa9bb0, request=0x7ffa24fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056191e8315cc in os_aio_handler (segment=4, m1=0x7ffa24fa9ba8, m2=0x7ffa24fa9bb0, request=0x7ffa24fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056191ea844a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056191e92a909 in io_handler_thread (arg=0x56191fab1640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ffa1bfff700 (LWP 223967)): #0 0x00007ffa4d824644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056191e82c41c in LinuxAIOHandler::collect (this=0x7ffa1bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056191e82c945 in LinuxAIOHandler::poll (this=0x7ffa1bffeb00, m1=0x7ffa1bffeba8, m2=0x7ffa1bffebb0, request=0x7ffa1bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056191e82caa8 in os_aio_linux_handler (global_segment=5, m1=0x7ffa1bffeba8, m2=0x7ffa1bffebb0, request=0x7ffa1bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056191e8315cc in os_aio_handler (segment=5, m1=0x7ffa1bffeba8, m2=0x7ffa1bffebb0, request=0x7ffa1bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056191ea844a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056191e92a909 in io_handler_thread (arg=0x56191fab1648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ffa3c9ee700 (LWP 223936)): #0 0x00007ffa4dc57d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191ecf6c99 in safe_cond_timedwait (cond=0x5619202c2ea0 , mp=0x5619202c2de0 , abstime=0x7ffa3c9edcb0, file=0x56191f13a610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056191eb60acb in inline_mysql_cond_timedwait (that=0x5619202c2ea0 , mutex=0x5619202c2de0 , abstime=0x7ffa3c9edcb0, src_file=0x56191f13a658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056191eb60fef in my_service_thread_sleep (control=0x56191f877b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056191eb55488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056191ec8945f in pfs_spawn_thread (arg=0x56192174d6d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ffa466ac700 (LWP 223879)): #0 0x00007ffa4dc57d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191ecf6c99 in safe_cond_timedwait (cond=0x5619202d5640 , mp=0x5619202d5580 , abstime=0x7ffa466abd70, file=0x56191f17e840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056191ecf7c53 in inline_mysql_cond_timedwait (that=0x5619202d5640 , mutex=0x5619202d5580 , abstime=0x7ffa466abd70, src_file=0x56191f17e888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056191ecf87b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056191ec8945f in pfs_spawn_thread (arg=0x56192160e730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ffa442fe700 (LWP 223924)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056191ecf6997 in safe_cond_wait (cond=0x7ffa34016858, mp=0x7ffa340167a8, file=0x56191ee60b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056191e44e088 in inline_mysql_cond_wait (that=0x7ffa34016858, mutex=0x7ffa340167a8, src_file=0x56191ee60e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056191e450d20 in Wsrep_thd_queue::pop_front (this=0x7ffa34016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056191e44f740 in wsrep_rollback_process (rollbacker=0x7ffa34001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000056191e440282 in start_wsrep_THD (arg=0x56192169d340) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ffa3db7b700 (LWP 223919)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffa45b3c867 in gu::Lock::wait (cond=..., this=0x7ffa3db7a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007ffa45b3f3ab in RecvBuf::front (this=this@entry=0x56192168d8a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007ffa45b3cf46 in gcomm_recv (backend=, msg=0x56192164d830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007ffa45b30eec in core_msg_recv (backend=backend@entry=0x56192164d978, recv_msg=recv_msg@entry=0x56192164d830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007ffa45b34164 in gcs_core_recv (conn=0x56192164d7d0, recv_act=recv_act@entry=0x7ffa3db7ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007ffa45b3b3d8 in gcs_recv_thread (arg=0x56192164d550) at gcs/src/gcs.cpp:1398 #7 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ffa44aff700 (LWP 223909)): #0 0x00007ffa4dc57965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffa45b61fd7 in gu::Lock::wait (cond=..., this=0x7ffa44afebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007ffa45b62069 in galera::ServiceThd::thd_func (arg=0x561921645998) at galera/src/galera_service_thd.cpp:30 #3 0x00007ffa4dc53dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffa4c011ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ffa4e064880 (LWP 223869)): #0 0x00007ffa4dc589d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056191ecf1368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056191e52f503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ffa4c00720d in poll () from /lib64/libc.so.6 #5 0x000056191e0cfa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056191e0ced25 in mysqld_main (argc=163, argv=0x5619214c6d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056191e0c2ec5 in main (argc=25, argv=0x7fffa4cf32a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.1/data/core.223869 - found 'core.224404' (1/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.2/data/core.224404 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 224404] [New LWP 224507] [New LWP 224446] [New LWP 224841] [New LWP 224838] [New LWP 224840] [New LWP 226139] [New LWP 226157] [New LWP 226141] [New LWP 226164] [New LWP 226145] [New LWP 226202] [New LWP 226163] [New LWP 226206] [New LWP 226172] [New LWP 226210] [New LWP 226175] [New LWP 226162] [New LWP 224839] [New LWP 226137] [New LWP 226174] [New LWP 226173] [New LWP 226161] [New LWP 226160] [New LWP 226159] [New LWP 226158] [New LWP 226156] [New LWP 226144] [New LWP 226143] [New LWP 226142] [New LWP 226140] [New LWP 224596] [New LWP 226165] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fc6b32bd9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fc6b32bd9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056041ded1368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056041d70f503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fc6b166c20d in poll () from /lib64/libc.so.6 #5 0x000056041d2afa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056041d2aed25 in mysqld_main (argc=164, argv=0x5604204d0d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056041d2a2ec5 in main (argc=25, argv=0x7fff923cbb98) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fc65affd700 (LWP 226165)): #0 0x00007fc6b32bfd1d in fsync () from /lib64/libpthread.so.0 #1 0x000056041da0dacb in os_file_fsync_posix (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056041da0de21 in os_file_flush_func (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056041dc563a2 in pfs_os_file_flush_func (file=..., src_file=0x56041e2d0a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056041dc5911d in fil_flush_low (space=0x5604209b2140, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056041dc649f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000056041dbe26b6 in buf_dblwr_write_single_page (bpage=0x7fc698417a70, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1309 #7 0x000056041dbe988a in buf_flush_write_block_low (bpage=0x7fc698417a70, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1014 #8 0x000056041dbe9fb6 in buf_flush_page (buf_pool=0x56042088de30, bpage=0x7fc698417a70, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1165 #9 0x000056041dbeca9b in buf_flush_single_page_from_LRU (buf_pool=0x56042088de30) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2138 #10 0x000056041dbf7a06 in buf_LRU_get_free_block (buf_pool=0x56042088de30) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1194 #11 0x000056041dbd4d13 in buf_page_init_for_read (err=0x7fc65affb194, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5332 #12 0x000056041dbfe0be in buf_read_page_low (err=0x7fc65affb194, sync=true, type=0, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #13 0x000056041dbfec9e in buf_read_page (page_id=..., zip_size=0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:412 #14 0x000056041dbd1bda in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x56041e287f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=8113, mtr=0x7fc65affb620, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4340 #15 0x000056041dba6117 in btr_free_externally_stored_field (index=0x7fc67c02fb78, field_ref=0x7fc698a74184 "", rec=0x7fc698a7413f "eca169b3-2de5-11e9-b28c-cef43aef6f27\200", offsets=0x7fc64000a7a8, page_zip=0x0, i=6, rollback=false, local_mtr=0x7fc65affc1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8113 #16 0x000056041dba665a in btr_rec_free_externally_stored_fields (index=0x7fc67c02fb78, rec=0x7fc698a7413f "eca169b3-2de5-11e9-b28c-cef43aef6f27\200", offsets=0x7fc64000a7a8, page_zip=0x0, rollback=false, mtr=0x7fc65affc1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8215 #17 0x000056041dba0cb1 in btr_cur_pessimistic_delete (err=0x7fc65affbe5c, has_reserved_extents=0, cursor=0x5604209ccd20, flags=0, rollback=false, mtr=0x7fc65affc1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5905 #18 0x000056041dac6084 in row_purge_remove_clust_if_poss_low (node=0x5604209ccc88, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:175 #19 0x000056041dac6209 in row_purge_remove_clust_if_poss (node=0x5604209ccc88) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #20 0x000056041dac7a36 in row_purge_del_mark (node=0x5604209ccc88) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #21 0x000056041dac8fbe in row_purge_record_func (node=0x5604209ccc88, undo_rec=0x5604209cd858 "", thr=0x5604209cc7e0, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #22 0x000056041dac92fb in row_purge (node=0x5604209ccc88, undo_rec=0x5604209cd858 "", thr=0x5604209cc7e0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #23 0x000056041dac96a6 in row_purge_step (thr=0x5604209cc7e0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #24 0x000056041da4ba9a in que_thr_step (thr=0x5604209cc7e0) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #25 0x000056041da4bcca in que_run_threads_low (thr=0x5604209cc7e0) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #26 0x000056041da4beba in que_run_threads (thr=0x5604209cc7e0) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #27 0x000056041db08855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #28 0x000056041db089fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #29 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fc69bffe700 (LWP 224596)): #0 0x00007fc6b1677483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fc6ab1686f8 in asio::detail::epoll_reactor::run (this=0x560420698030, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fc6ab16cdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x560420697ed0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fc6ab16d0a2 in asio::detail::task_io_service::run (this=0x560420697ed0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fc6ab184afd in asio::io_service::run (this=0x560420696bf0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fc6ab185251 in gcomm::AsioProtonet::event_loop (this=0x560420696b50, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fc6ab1a32ed in GCommConn::run (this=0x560420697660) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fc6ab1a6399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fc6737fe700 (LWP 226140)): #0 0x00007fc6b2e89644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056041da0c41c in LinuxAIOHandler::collect (this=0x7fc6737fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056041da0c945 in LinuxAIOHandler::poll (this=0x7fc6737fdb00, m1=0x7fc6737fdba8, m2=0x7fc6737fdbb0, request=0x7fc6737fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056041da0caa8 in os_aio_linux_handler (global_segment=1, m1=0x7fc6737fdba8, m2=0x7fc6737fdbb0, request=0x7fc6737fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056041da115cc in os_aio_handler (segment=1, m1=0x7fc6737fdba8, m2=0x7fc6737fdbb0, request=0x7fc6737fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056041dc644a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056041db0a909 in io_handler_thread (arg=0x56041ec91628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fc6727fc700 (LWP 226142)): #0 0x00007fc6b2e89644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056041da0c41c in LinuxAIOHandler::collect (this=0x7fc6727fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056041da0c945 in LinuxAIOHandler::poll (this=0x7fc6727fbb00, m1=0x7fc6727fbba8, m2=0x7fc6727fbbb0, request=0x7fc6727fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056041da0caa8 in os_aio_linux_handler (global_segment=3, m1=0x7fc6727fbba8, m2=0x7fc6727fbbb0, request=0x7fc6727fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056041da115cc in os_aio_handler (segment=3, m1=0x7fc6727fbba8, m2=0x7fc6727fbbb0, request=0x7fc6727fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056041dc644a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056041db0a909 in io_handler_thread (arg=0x56041ec91638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fc671ffb700 (LWP 226143)): #0 0x00007fc6b2e89644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056041da0c41c in LinuxAIOHandler::collect (this=0x7fc671ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056041da0c945 in LinuxAIOHandler::poll (this=0x7fc671ffab00, m1=0x7fc671ffaba8, m2=0x7fc671ffabb0, request=0x7fc671ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056041da0caa8 in os_aio_linux_handler (global_segment=4, m1=0x7fc671ffaba8, m2=0x7fc671ffabb0, request=0x7fc671ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056041da115cc in os_aio_handler (segment=4, m1=0x7fc671ffaba8, m2=0x7fc671ffabb0, request=0x7fc671ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056041dc644a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056041db0a909 in io_handler_thread (arg=0x56041ec91640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fc6717fa700 (LWP 226144)): #0 0x00007fc6b2e89644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056041da0c41c in LinuxAIOHandler::collect (this=0x7fc6717f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056041da0c945 in LinuxAIOHandler::poll (this=0x7fc6717f9b00, m1=0x7fc6717f9ba8, m2=0x7fc6717f9bb0, request=0x7fc6717f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056041da0caa8 in os_aio_linux_handler (global_segment=5, m1=0x7fc6717f9ba8, m2=0x7fc6717f9bb0, request=0x7fc6717f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056041da115cc in os_aio_handler (segment=5, m1=0x7fc6717f9ba8, m2=0x7fc6717f9bb0, request=0x7fc6717f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056041dc644a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056041db0a909 in io_handler_thread (arg=0x56041ec91648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fc66b7fe700 (LWP 226156)): #0 0x00007fc6b32bcd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da18a1a in os_event::timed_wait (this=0x560420980a60, abstime=0x7fc66b7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056041da18d31 in os_event::wait_time_low (this=0x560420980a60, time_in_usec=1000000, reset_sig_count=14) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056041da18ed1 in os_event_wait_time_low (event=0x560420980a60, time_in_usec=1000000, reset_sig_count=14) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056041d9e588e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fc66affd700 (LWP 226158)): #0 0x00007fc6b32bcd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da18a1a in os_event::timed_wait (this=0x5604207a85b0, abstime=0x7fc66affcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056041da18d31 in os_event::wait_time_low (this=0x5604207a85b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056041da18ed1 in os_event_wait_time_low (event=0x5604207a85b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056041db06ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fc66a7fc700 (LWP 226159)): #0 0x00007fc6b32bcd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da18a1a in os_event::timed_wait (this=0x5604209b2b20, abstime=0x7fc66a7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056041da18d31 in os_event::wait_time_low (this=0x5604209b2b20, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056041da18ed1 in os_event_wait_time_low (event=0x5604209b2b20, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056041dc4e142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fc669ffb700 (LWP 226160)): #0 0x00007fc6b32bcd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da18a1a in os_event::timed_wait (this=0x560420950d70, abstime=0x7fc669ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056041da18d31 in os_event::wait_time_low (this=0x560420950d70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056041da18ed1 in os_event_wait_time_low (event=0x560420950d70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056041db672fb in ib_wqueue_timedwait (wq=0x560420950cc0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056041dca7ea4 in fts_optimize_thread (arg=0x560420950cc0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fc6697fa700 (LWP 226161)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da190ff in os_event::wait (this=0x5604209ba410) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056041da18b85 in os_event::wait_low (this=0x5604209ba410, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056041da18ef6 in os_event_wait_low (event=0x5604209ba410, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056041db12bfc in sync_array_wait_event (arr=0x56042077f880, cell=@0x7fc6697f9b98: 0x56042077f9f0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056041db15ae2 in rw_lock_x_lock_wait_func (lock=0x5604209b5950, pass=0, threshold=0, file_name=0x56041e256b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:459 #6 0x000056041db15c14 in rw_lock_x_lock_low (lock=0x5604209b5950, pass=0, file_name=0x56041e256b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:514 #7 0x000056041db16032 in rw_lock_x_lock_func (lock=0x5604209b5950, pass=0, file_name=0x56041e256b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:673 #8 0x000056041db04443 in pfs_rw_lock_x_lock_func (lock=0x5604209b5950, pass=0, file_name=0x56041e256b48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:552 #9 0x000056041db07486 in srv_master_evict_from_table_cache (pct_check=100) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1999 #10 0x000056041db07f82 in srv_master_do_idle_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2237 #11 0x000056041db08558 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2358 #12 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fc659ffb700 (LWP 226173)): #0 0x00007fc6b32bfe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056041da19401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056041dbbfa3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fc6597fa700 (LWP 226174)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da190ff in os_event::wait (this=0x5604207a8730) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056041da18b85 in os_event::wait_low (this=0x5604207a8730, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056041da18ef6 in os_event_wait_low (event=0x5604207a8730, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056041dbcdb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fc699855700 (LWP 226137)): #0 0x00007fc6b32bcd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041ded6c99 in safe_cond_timedwait (cond=0x56041f4a2ea0 , mp=0x56041f4a2de0 , abstime=0x7fc699854cb0, file=0x56041e31a610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056041dd40acb in inline_mysql_cond_timedwait (that=0x56041f4a2ea0 , mutex=0x56041f4a2de0 , abstime=0x7fc699854cb0, src_file=0x56041e31a658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056041dd40fef in my_service_thread_sleep (control=0x56041ea57b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056041dd35488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056041de6945f in pfs_spawn_thread (arg=0x560420760510) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fc6a86e1700 (LWP 224839)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041ded6997 in safe_cond_wait (cond=0x7fc690015608, mp=0x7fc690015558, file=0x56041e040b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056041d62e088 in inline_mysql_cond_wait (that=0x7fc690015608, mutex=0x7fc690015558, src_file=0x56041e040e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056041d630d20 in Wsrep_thd_queue::pop_front (this=0x7fc690015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056041d62f740 in wsrep_rollback_process (rollbacker=0x7fc690000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000056041d620282 in start_wsrep_THD (arg=0x5604206b8ec0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fc668ff9700 (LWP 226162)): #0 0x00007fc6b165bd47 in sched_yield () from /lib64/libc.so.6 #1 0x000056041da19393 in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x000056041db32bd9 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x000056041db32de1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x000056041db08cca in srv_do_purge (n_total_purged=0x7fc668ff8d98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x000056041db090f7 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fc658ff9700 (LWP 226175)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041ded6997 in safe_cond_wait (cond=0x7fc658ff8c70, mp=0x7fc658ff8cb0, file=0x56041e1a4c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056041d930487 in inline_mysql_cond_wait (that=0x7fc658ff8c70, mutex=0x7fc658ff8cb0, src_file=0x56041e1a65d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056041d9350c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000056041de6945f in pfs_spawn_thread (arg=0x560420963740) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fc69836d700 (LWP 226210)): #0 0x00007fc6b32bcd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041ded6c99 in safe_cond_timedwait (cond=0x56041ec161a0 , mp=0x56041ec18ba0 , abstime=0x7fc69836cc60, file=0x56041df924c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056041d2a3845 in inline_mysql_cond_timedwait (that=0x56041ec161a0 , mutex=0x56041ec18ba0 , abstime=0x7fc69836cc60, src_file=0x56041df94538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056041d2a861b in cache_thread (thd=0x7fc67c000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000056041d2a88c9 in one_thread_per_connection_end (thd=0x7fc67c000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000056041d5450f3 in do_handle_one_connection (connect=0x560420a57620) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000056041d544dad in handle_one_connection (arg=0x560420a57620) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000056041de6945f in pfs_spawn_thread (arg=0x560420b2f2d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fc65a7fc700 (LWP 226172)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da190ff in os_event::wait (this=0x5604207a8630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056041da18b85 in os_event::wait_low (this=0x5604207a8630, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056041da18ef6 in os_event_wait_low (event=0x5604207a8630, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056041dbe4eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fc6983b8700 (LWP 226206)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041ded6997 in safe_cond_wait (cond=0x56041ec161e0 , mp=0x56041ec19560 , file=0x56041dfa1cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056041d2d44ee in inline_mysql_cond_wait (that=0x56041ec161e0 , mutex=0x56041ec19560 , src_file=0x56041dfa2830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000056041d2d5d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056041de6945f in pfs_spawn_thread (arg=0x560420b0ad60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc65bfff700 (LWP 226163)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da190ff in os_event::wait (this=0x5604209ba390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056041da18b85 in os_event::wait_low (this=0x5604209ba390, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056041da18ef6 in os_event_wait_low (event=0x5604209ba390, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056041db12bfc in sync_array_wait_event (arr=0x56042077f880, cell=@0x7fc65bffeb18: 0x56042077f9b0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056041db15911 in rw_lock_s_lock_spin (lock=0x5604209b5950, pass=0, file_name=0x56041e238d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000056041dac3236 in rw_lock_s_lock_func (lock=0x5604209b5950, pass=0, file_name=0x56041e238d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000056041dac366f in pfs_rw_lock_s_lock_func (lock=0x5604209b5950, pass=0, file_name=0x56041e238d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000056041dac8a54 in row_purge_parse_undo_rec (node=0x5604209cc630, undo_rec=0x5604209ccb08 "", updated_extern=0x7fc65bffec9e, thr=0x5604209cc578) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1077 #9 0x000056041dac93df in row_purge (node=0x5604209cc630, undo_rec=0x5604209ccb08 "", thr=0x5604209cc578) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1258 #10 0x000056041dac96a6 in row_purge_step (thr=0x5604209cc578) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #11 0x000056041da4ba9a in que_thr_step (thr=0x5604209cc578) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #12 0x000056041da4bcca in que_run_threads_low (thr=0x5604209cc578) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000056041da4beba in que_run_threads (thr=0x5604209cc578) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x000056041db08855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #15 0x000056041db089fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #16 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc698403700 (LWP 226202)): #0 0x00007fc6b15b040a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000056041d2a2f44 in my_sigwait (set=0x7fc698402cf0, sig=0x7fc698402cb4, code=0x7fc698402cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000056041d2a9584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000056041de6945f in pfs_spawn_thread (arg=0x56042076bac0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc670ff9700 (LWP 226145)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da190ff in os_event::wait (this=0x5604208b2300) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056041da18b85 in os_event::wait_low (this=0x5604208b2300, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056041da18ef6 in os_event_wait_low (event=0x5604208b2300, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056041db12bfc in sync_array_wait_event (arr=0x56042077f880, cell=@0x7fc670ff8680: 0x56042077fab0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056041db1647b in rw_lock_sx_lock_func (lock=0x7fc69841f518, pass=2, file_name=0x56041e2a7528 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x000056041dbe5837 in pfs_rw_lock_sx_lock_func (lock=0x7fc69841f518, pass=2, file_name=0x56041e2a7528 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x000056041dbe9eb4 in buf_flush_page (buf_pool=0x56042088de30, bpage=0x7fc69841f450, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1144 #8 0x000056041dbea8fe in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=149, n_to_flush=200) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1389 #9 0x000056041dbeac81 in buf_flush_page_and_try_neighbors (bpage=0x7fc69841f450, flush_type=BUF_FLUSH_LIST, n_to_flush=200, count=0x7fc670ff88b8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1461 #10 0x000056041dbeb89a in buf_do_flush_list_batch (buf_pool=0x56042088de30, min_n=200, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1719 #11 0x000056041dbebe92 in buf_flush_batch (buf_pool=0x56042088de30, flush_type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7fc670ff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1789 #12 0x000056041dbec552 in buf_flush_do_batch (buf_pool=0x56042088de30, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7fc670ff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1947 #13 0x000056041dbec826 in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7fc670ff8bd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2053 #14 0x000056041dbf0571 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3205 #15 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc65b7fe700 (LWP 226164)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041da190ff in os_event::wait (this=0x5604209ba390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056041da18b85 in os_event::wait_low (this=0x5604209ba390, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056041da18ef6 in os_event_wait_low (event=0x5604209ba390, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056041db12bfc in sync_array_wait_event (arr=0x56042077f880, cell=@0x7fc65b7fdb18: 0x56042077fa70) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056041db15911 in rw_lock_s_lock_spin (lock=0x5604209b5950, pass=0, file_name=0x56041e238d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000056041dac3236 in rw_lock_s_lock_func (lock=0x5604209b5950, pass=0, file_name=0x56041e238d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000056041dac366f in pfs_rw_lock_s_lock_func (lock=0x5604209b5950, pass=0, file_name=0x56041e238d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000056041dac8a54 in row_purge_parse_undo_rec (node=0x5604209cc3c8, undo_rec=0x5604209cc978 "", updated_extern=0x7fc65b7fdc9e, thr=0x5604209cc1f8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1077 #9 0x000056041dac93df in row_purge (node=0x5604209cc3c8, undo_rec=0x5604209cc978 "", thr=0x5604209cc1f8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1258 #10 0x000056041dac96a6 in row_purge_step (thr=0x5604209cc1f8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #11 0x000056041da4ba9a in que_thr_step (thr=0x5604209cc1f8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #12 0x000056041da4bcca in que_run_threads_low (thr=0x5604209cc1f8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000056041da4beba in que_run_threads (thr=0x5604209cc1f8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x000056041db08855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #15 0x000056041db089fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #16 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc672ffd700 (LWP 226141)): #0 0x00007fc6b2e89644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056041da0c41c in LinuxAIOHandler::collect (this=0x7fc672ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056041da0c945 in LinuxAIOHandler::poll (this=0x7fc672ffcb00, m1=0x7fc672ffcba8, m2=0x7fc672ffcbb0, request=0x7fc672ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056041da0caa8 in os_aio_linux_handler (global_segment=2, m1=0x7fc672ffcba8, m2=0x7fc672ffcbb0, request=0x7fc672ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056041da115cc in os_aio_handler (segment=2, m1=0x7fc672ffcba8, m2=0x7fc672ffcbb0, request=0x7fc672ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056041dc644a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056041db0a909 in io_handler_thread (arg=0x56041ec91630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc66bfff700 (LWP 226157)): #0 0x00007fc6b32bfe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056041da19401 in os_thread_sleep (tm=30000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056041db14068 in sync_array_print_long_waits (waiter=0x7fc66bffec28, sema=0x7fc66bffec30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:1099 #3 0x000056041db06e88 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1801 #4 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc673fff700 (LWP 226139)): #0 0x00007fc6b2e89644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056041da0c41c in LinuxAIOHandler::collect (this=0x7fc673ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056041da0c945 in LinuxAIOHandler::poll (this=0x7fc673ffeb00, m1=0x7fc673ffeba8, m2=0x7fc673ffebb0, request=0x7fc673ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056041da0caa8 in os_aio_linux_handler (global_segment=0, m1=0x7fc673ffeba8, m2=0x7fc673ffebb0, request=0x7fc673ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056041da115cc in os_aio_handler (segment=0, m1=0x7fc673ffeba8, m2=0x7fc673ffebb0, request=0x7fc673ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056041dc644a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056041db0a909 in io_handler_thread (arg=0x56041ec91620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc6a8696700 (LWP 224840)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041ded6997 in safe_cond_wait (cond=0x7fc684017b68, mp=0x7fc684017ab8, file=0x56041e040b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056041d62e088 in inline_mysql_cond_wait (that=0x7fc684017b68, mutex=0x7fc684017ab8, src_file=0x56041e040e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056041d630d20 in Wsrep_thd_queue::pop_front (this=0x7fc684017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056041d62fb00 in wsrep_post_rollback_process (post_rollbacker=0x7fc684001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000056041d620282 in start_wsrep_THD (arg=0x5604206b9610) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc69b7fd700 (LWP 224838)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6ab1a1867 in gu::Lock::wait (cond=..., this=0x7fc69b7fc850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fc6ab1a43ab in RecvBuf::front (this=this@entry=0x560420697848, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fc6ab1a1f46 in gcomm_recv (backend=, msg=0x560420657750, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fc6ab195eec in core_msg_recv (backend=backend@entry=0x560420657898, recv_msg=recv_msg@entry=0x560420657750, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fc6ab199164 in gcs_core_recv (conn=0x5604206576f0, recv_act=recv_act@entry=0x7fc69b7fcd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fc6ab1a03d8 in gcs_recv_thread (arg=0x560420657470) at gcs/src/gcs.cpp:1398 #7 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc6a864b700 (LWP 224841)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6ab0bb132 in fifo_lock_get (q=q@entry=0x56042067daf0) at galerautils/src/gu_fifo.c:262 #2 0x00007fc6ab0bb7a1 in gu_fifo_get_head (q=0x56042067daf0, err=0x7fc6a864a06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fc6ab19d0cc in gcs_recv (conn=0x560420657470, action=0x7fc6a864a0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fc6ab1ca16c in galera::GcsActionSource::process (this=0x56042068fe90, recv_ctx=0x7fc6a864a4d0, exit_loop=@0x7fc6a864a29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fc6ab1e9033 in galera::ReplicatorSMM::async_recv (this=0x56042064f240, recv_ctx=0x7fc6a864a4d0) at galera/src/replicator_smm.cpp:383 #6 0x000056041df8b276 in wsrep::wsrep_provider_v26::run_applier (this=0x5604205c4ee0, applier_ctx=0x7fc6a864a4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000056041d62e636 in wsrep_replication_process (thd=0x7fc688000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000056041d620282 in start_wsrep_THD (arg=0x56042066fda0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc6abd11700 (LWP 224446)): #0 0x00007fc6b32bcd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056041ded6c99 in safe_cond_timedwait (cond=0x56041f4b5640 , mp=0x56041f4b5580 , abstime=0x7fc6abd10d70, file=0x56041e35e840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056041ded7c53 in inline_mysql_cond_timedwait (that=0x56041f4b5640 , mutex=0x56041f4b5580 , abstime=0x7fc6abd10d70, src_file=0x56041e35e888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056041ded87b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056041de6945f in pfs_spawn_thread (arg=0x560420618860) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc6aab65700 (LWP 224507)): #0 0x00007fc6b32bc965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc6ab1c6fd7 in gu::Lock::wait (cond=..., this=0x7fc6aab64bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fc6ab1c7069 in galera::ServiceThd::thd_func (arg=0x56042064f958) at galera/src/galera_service_thd.cpp:30 #3 0x00007fc6b32b8dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc6b1676ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc6b36c9880 (LWP 224404)): #0 0x00007fc6b32bd9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056041ded1368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056041d70f503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fc6b166c20d in poll () from /lib64/libc.so.6 #5 0x000056041d2afa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056041d2aed25 in mysqld_main (argc=164, argv=0x5604204d0d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056041d2a2ec5 in main (argc=25, argv=0x7fff923cbb98) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.2/data/core.224404 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_large_fragment-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_large_fragment-innodb/' galera_sr.GCF-1018B 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 12:32:41 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/GCF-1018B.log == connection node_2; connection node_1; connection node_1; SET SESSION wsrep_sync_wait = 0; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 64; SET SESSION innodb_lock_wait_timeout = 1000; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 64; SET SESSION innodb_lock_wait_timeout = 1000; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.226889' (2/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.1/data/core.226889 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 226889] [New LWP 227031] [New LWP 227030] [New LWP 226973] [New LWP 227116] [New LWP 227117] [New LWP 227118] [New LWP 227141] [New LWP 227121] [New LWP 227132] [New LWP 227143] [New LWP 227122] [New LWP 227133] [New LWP 227130] [New LWP 227139] [New LWP 227136] [New LWP 227151] [New LWP 227137] [New LWP 227567] [New LWP 227138] [New LWP 227120] [New LWP 227142] [New LWP 227119] [New LWP 228272] [New LWP 226991] [New LWP 227568] [New LWP 227144] [New LWP 227134] [New LWP 227131] [New LWP 227029] [New LWP 227028] [New LWP 226984] [New LWP 227135] [New LWP 227123] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f6d3434a9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f6d3434a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556446d55368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556446593503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6d326f920d in poll () from /lib64/libc.so.6 #5 0x0000556446133a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000556446132d25 in mysqld_main (argc=161, argv=0x5564492e5d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000556446126ec5 in main (argc=23, argv=0x7fff80e605c8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f6d027fc700 (LWP 227123)): #0 0x00007f6d3434cd1d in fsync () from /lib64/libpthread.so.0 #1 0x0000556446891acb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000556446891e21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000556446ada3a2 in pfs_os_file_flush_func (file=..., src_file=0x556447154a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000556446add11d in fil_flush_low (space=0x5564497b63b0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000556446ae89f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000556446a65cbd in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x0000556446a703a5 in buf_flush_end (buf_pool=0x5564496920a0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x0000556446a70563 in buf_flush_do_batch (buf_pool=0x5564496920a0, type=BUF_FLUSH_LIST, min_n=2, lsn_limit=18446744073709551615, n=0x7f6d027fbb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x0000556446a7324d in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x0000556446a7419b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f6cfaffd700 (LWP 227135)): #0 0x00007f6d3434cd1d in fsync () from /lib64/libpthread.so.0 #1 0x0000556446891acb in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000556446891e21 in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000556446ada3a2 in pfs_os_file_flush_func (file=..., src_file=0x556447154a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000556446add11d in fil_flush_low (space=0x5564497b6750, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000556446ae89f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055644686d1cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055644686dc31 in log_write_up_to (lsn=2561999, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055644686dd61 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x000055644698b430 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #10 0x000055644698b9f3 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #11 0x000055644698c551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f6d26f7d700 (LWP 226984)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d27cb5fd7 in gu::Lock::wait (cond=..., this=0x7f6d26f7cbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6d27cb6069 in galera::ServiceThd::thd_func (arg=0x556449464998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f6d242f8700 (LWP 227028)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d27c90867 in gu::Lock::wait (cond=..., this=0x7f6d242f7850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6d27c933ab in RecvBuf::front (this=this@entry=0x5564494a48a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f6d27c90f46 in gcomm_recv (backend=, msg=0x55644946c830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f6d27c84eec in core_msg_recv (backend=backend@entry=0x55644946c978, recv_msg=recv_msg@entry=0x55644946c830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f6d27c88164 in gcs_core_recv (conn=0x55644946c7d0, recv_act=recv_act@entry=0x7f6d242f7d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f6d27c8f3d8 in gcs_recv_thread (arg=0x55644946c550) at gcs/src/gcs.cpp:1398 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f6d2c273700 (LWP 227029)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5a997 in safe_cond_wait (cond=0x7f6d18015608, mp=0x7f6d18015558, file=0x556446ec4b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564464b2088 in inline_mysql_cond_wait (that=0x7f6d18015608, mutex=0x7f6d18015558, src_file=0x556446ec4e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005564464b4d20 in Wsrep_thd_queue::pop_front (this=0x7f6d18015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005564464b3740 in wsrep_rollback_process (rollbacker=0x7f6d18000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005564464a4282 in start_wsrep_THD (arg=0x5564494b4320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f6d00df4700 (LWP 227131)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689ca1a in os_event::timed_wait (this=0x5564495ac7a0, abstime=0x7f6d00df3ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055644689cd31 in os_event::wait_time_low (this=0x5564495ac7a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055644689ced1 in os_event_wait_time_low (event=0x5564495ac7a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055644698affc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f6cfb7fe700 (LWP 227134)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689ca1a in os_event::timed_wait (this=0x55644974ca20, abstime=0x7f6cfb7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055644689cd31 in os_event::wait_time_low (this=0x55644974ca20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055644689ced1 in os_event_wait_time_low (event=0x55644974ca20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005564469eb2fb in ib_wqueue_timedwait (wq=0x55644974c8f0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000556446b2bea4 in fts_optimize_thread (arg=0x55644974c8f0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f6cde7fc700 (LWP 227144)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5a997 in safe_cond_wait (cond=0x7f6cde7fbc70, mp=0x7f6cde7fbcb0, file=0x556447028c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564467b4487 in inline_mysql_cond_wait (that=0x7f6cde7fbc70, mutex=0x7f6cde7fbcb0, src_file=0x55644702a5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005564467b90c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000556446ced45f in pfs_spawn_thread (arg=0x5564497ed8e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f6d21a02700 (LWP 227568)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5ac99 in safe_cond_timedwait (cond=0x556447a9a1a0 , mp=0x556447a9cba0 , abstime=0x7f6d21a01c60, file=0x556446e164c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556446127845 in inline_mysql_cond_timedwait (that=0x556447a9a1a0 , mutex=0x556447a9cba0 , abstime=0x7f6d21a01c60, src_file=0x556446e18538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055644612c61b in cache_thread (thd=0x7f6cc0000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055644612c8c9 in one_thread_per_connection_end (thd=0x7f6cc0000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005564463c90f3 in do_handle_one_connection (connect=0x55644991b830) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005564463c8dad in handle_one_connection (arg=0x55644991b830) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000556446ced45f in pfs_spawn_thread (arg=0x55644993d500) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f6d24af9700 (LWP 226991)): #0 0x00007f6d32704483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6d27c576f8 in asio::detail::epoll_reactor::run (this=0x5564494a5040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f6d27c5bdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5564494a4ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f6d27c5c0a2 in asio::detail::task_io_service::run (this=0x5564494a4ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f6d27c73afd in asio::io_service::run (this=0x5564494a3fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f6d27c74251 in gcomm::AsioProtonet::event_loop (this=0x5564494a3f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f6d27c922ed in GCommConn::run (this=0x5564494a46c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f6d27c95399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f6d219b7700 (LWP 228272)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5ac99 in safe_cond_timedwait (cond=0x556447a9a1a0 , mp=0x556447a9cba0 , abstime=0x7f6d219b6c60, file=0x556446e164c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556446127845 in inline_mysql_cond_timedwait (that=0x556447a9a1a0 , mutex=0x556447a9cba0 , abstime=0x7f6d219b6c60, src_file=0x556446e18538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055644612c61b in cache_thread (thd=0x7f6cb8000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055644612c8c9 in one_thread_per_connection_end (thd=0x7f6cb8000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005564463c90f3 in do_handle_one_connection (connect=0x556449570c10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005564463c8dad in handle_one_connection (arg=0x556449570c10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000556446ced45f in pfs_spawn_thread (arg=0x556449420a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f6d20896700 (LWP 227119)): #0 0x00007f6d33f16644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055644689041c in LinuxAIOHandler::collect (this=0x7f6d20895b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556446890945 in LinuxAIOHandler::poll (this=0x7f6d20895b00, m1=0x7f6d20895ba8, m2=0x7f6d20895bb0, request=0x7f6d20895be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556446890aa8 in os_aio_linux_handler (global_segment=2, m1=0x7f6d20895ba8, m2=0x7f6d20895bb0, request=0x7f6d20895be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564468955cc in os_aio_handler (segment=2, m1=0x7f6d20895ba8, m2=0x7f6d20895bb0, request=0x7f6d20895be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556446ae84a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055644698e909 in io_handler_thread (arg=0x556447b15630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f6cdf7fe700 (LWP 227142)): #0 0x00007f6d3434ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055644689d401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000556446a43a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f6d03fff700 (LWP 227120)): #0 0x00007f6d33f16644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055644689041c in LinuxAIOHandler::collect (this=0x7f6d03ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556446890945 in LinuxAIOHandler::poll (this=0x7f6d03ffeb00, m1=0x7f6d03ffeba8, m2=0x7f6d03ffebb0, request=0x7f6d03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556446890aa8 in os_aio_linux_handler (global_segment=3, m1=0x7f6d03ffeba8, m2=0x7f6d03ffebb0, request=0x7f6d03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564468955cc in os_aio_handler (segment=3, m1=0x7f6d03ffeba8, m2=0x7f6d03ffebb0, request=0x7f6d03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556446ae84a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055644698e909 in io_handler_thread (arg=0x556447b15638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f6cf97fa700 (LWP 227138)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689d0ff in os_event::wait (this=0x5564495ac620) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055644689cb85 in os_event::wait_low (this=0x5564495ac620, reset_sig_count=8552) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055644689cef6 in os_event_wait_low (event=0x5564495ac620, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556446988df3 in srv_resume_thread (slot=0x5564478d7780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055644698c9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f6d21a4d700 (LWP 227567)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5a997 in safe_cond_wait (cond=0x556447a9a1e0 , mp=0x556447a9d560 , file=0x556446e25cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564461584ee in inline_mysql_cond_wait (that=0x556447a9a1e0 , mutex=0x556447a9d560 , src_file=0x556446e26830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000556446159d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000556446ced45f in pfs_spawn_thread (arg=0x5564495704a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f6cf9ffb700 (LWP 227137)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689d0ff in os_event::wait (this=0x5564495ac6a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055644689cb85 in os_event::wait_low (this=0x5564495ac6a0, reset_sig_count=8052) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055644689cef6 in os_event_wait_low (event=0x5564495ac6a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556446988df3 in srv_resume_thread (slot=0x5564478d77b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055644698c9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f6d21a98700 (LWP 227151)): #0 0x00007f6d3263d40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000556446126f44 in my_sigwait (set=0x7f6d21a97cf0, sig=0x7f6d21a97cb4, code=0x7f6d21a97cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055644612d584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000556446ced45f in pfs_spawn_thread (arg=0x55644956ffb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f6cfa7fc700 (LWP 227136)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689d0ff in os_event::wait (this=0x5564495ac5a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055644689cb85 in os_event::wait_low (this=0x5564495ac5a0, reset_sig_count=2126) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055644689cef6 in os_event_wait_low (event=0x5564495ac5a0, reset_sig_count=2126) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556446988df3 in srv_resume_thread (slot=0x5564478d7748 , sig_count=2126, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055644698ce04 in srv_purge_coordinator_suspend (slot=0x5564478d7748 , rseg_history_len=3) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055644698d09a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f6cf8ff9700 (LWP 227139)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689d0ff in os_event::wait (this=0x5564495ac720) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055644689cb85 in os_event::wait_low (this=0x5564495ac720, reset_sig_count=8042) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055644689cef6 in os_event_wait_low (event=0x5564495ac720, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556446988df3 in srv_resume_thread (slot=0x5564478d77f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055644698c9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f6d015f5700 (LWP 227130)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689ca1a in os_event::timed_wait (this=0x556449784cd0, abstime=0x7f6d015f4d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055644689cd31 in os_event::wait_time_low (this=0x556449784cd0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055644689ced1 in os_event_wait_time_low (event=0x556449784cd0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055644686988e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f6cfbfff700 (LWP 227133)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689ca1a in os_event::timed_wait (this=0x5564497b6d90, abstime=0x7f6cfbffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055644689cd31 in os_event::wait_time_low (this=0x5564497b6d90, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055644689ced1 in os_event_wait_time_low (event=0x5564497b6d90, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556446ad2142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f6d02ffd700 (LWP 227122)): #0 0x00007f6d33f16644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055644689041c in LinuxAIOHandler::collect (this=0x7f6d02ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556446890945 in LinuxAIOHandler::poll (this=0x7f6d02ffcb00, m1=0x7f6d02ffcba8, m2=0x7f6d02ffcbb0, request=0x7f6d02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556446890aa8 in os_aio_linux_handler (global_segment=5, m1=0x7f6d02ffcba8, m2=0x7f6d02ffcbb0, request=0x7f6d02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564468955cc in os_aio_handler (segment=5, m1=0x7f6d02ffcba8, m2=0x7f6d02ffcbb0, request=0x7f6d02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556446ae84a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055644698e909 in io_handler_thread (arg=0x556447b15648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f6cdeffd700 (LWP 227143)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689d0ff in os_event::wait (this=0x5564495ac9a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055644689cb85 in os_event::wait_low (this=0x5564495ac9a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055644689cef6 in os_event_wait_low (event=0x5564495ac9a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556446a51b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f6d01df6700 (LWP 227132)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689ca1a in os_event::timed_wait (this=0x5564495ac820, abstime=0x7f6d01df5d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055644689cd31 in os_event::wait_time_low (this=0x5564495ac820, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055644689ced1 in os_event_wait_time_low (event=0x5564495ac820, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055644698aada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f6d037fe700 (LWP 227121)): #0 0x00007f6d33f16644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055644689041c in LinuxAIOHandler::collect (this=0x7f6d037fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556446890945 in LinuxAIOHandler::poll (this=0x7f6d037fdb00, m1=0x7f6d037fdba8, m2=0x7f6d037fdbb0, request=0x7f6d037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556446890aa8 in os_aio_linux_handler (global_segment=4, m1=0x7f6d037fdba8, m2=0x7f6d037fdbb0, request=0x7f6d037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564468955cc in os_aio_handler (segment=4, m1=0x7f6d037fdba8, m2=0x7f6d037fdbb0, request=0x7f6d037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556446ae84a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055644698e909 in io_handler_thread (arg=0x556447b15640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f6cdffff700 (LWP 227141)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055644689d0ff in os_event::wait (this=0x5564495ac8a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055644689cb85 in os_event::wait_low (this=0x5564495ac8a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055644689cef6 in os_event_wait_low (event=0x5564495ac8a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556446a68eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f6d21097700 (LWP 227118)): #0 0x00007f6d33f16644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055644689041c in LinuxAIOHandler::collect (this=0x7f6d21096b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556446890945 in LinuxAIOHandler::poll (this=0x7f6d21096b00, m1=0x7f6d21096ba8, m2=0x7f6d21096bb0, request=0x7f6d21096be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556446890aa8 in os_aio_linux_handler (global_segment=1, m1=0x7f6d21096ba8, m2=0x7f6d21096bb0, request=0x7f6d21096be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564468955cc in os_aio_handler (segment=1, m1=0x7f6d21096ba8, m2=0x7f6d21096bb0, request=0x7f6d21096be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556446ae84a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055644698e909 in io_handler_thread (arg=0x556447b15628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f6d21898700 (LWP 227117)): #0 0x00007f6d33f16644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055644689041c in LinuxAIOHandler::collect (this=0x7f6d21897b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556446890945 in LinuxAIOHandler::poll (this=0x7f6d21897b00, m1=0x7f6d21897ba8, m2=0x7f6d21897bb0, request=0x7f6d21897be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556446890aa8 in os_aio_linux_handler (global_segment=0, m1=0x7f6d21897ba8, m2=0x7f6d21897bb0, request=0x7f6d21897be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564468955cc in os_aio_handler (segment=0, m1=0x7f6d21897ba8, m2=0x7f6d21897bb0, request=0x7f6d21897be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556446ae84a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055644698e909 in io_handler_thread (arg=0x556447b15620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f6d230ea700 (LWP 227116)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5ac99 in safe_cond_timedwait (cond=0x556448326ea0 , mp=0x556448326de0 , abstime=0x7f6d230e9cb0, file=0x55644719e610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556446bc4acb in inline_mysql_cond_timedwait (that=0x556448326ea0 , mutex=0x556448326de0 , abstime=0x7f6d230e9cb0, src_file=0x55644719e658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556446bc4fef in my_service_thread_sleep (control=0x5564478dbb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000556446bb9488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000556446ced45f in pfs_spawn_thread (arg=0x556449564880) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f6d2cd9e700 (LWP 226973)): #0 0x00007f6d34349d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5ac99 in safe_cond_timedwait (cond=0x556448339640 , mp=0x556448339580 , abstime=0x7f6d2cd9dd70, file=0x5564471e2840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556446d5bc53 in inline_mysql_cond_timedwait (that=0x556448339640 , mutex=0x556448339580 , abstime=0x7f6d2cd9dd70, src_file=0x5564471e2888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556446d5c7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000556446ced45f in pfs_spawn_thread (arg=0x55644942d730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f6d2c228700 (LWP 227030)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556446d5a997 in safe_cond_wait (cond=0x7f6d0c017b68, mp=0x7f6d0c017ab8, file=0x556446ec4b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564464b2088 in inline_mysql_cond_wait (that=0x7f6d0c017b68, mutex=0x7f6d0c017ab8, src_file=0x556446ec4e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005564464b4d20 in Wsrep_thd_queue::pop_front (this=0x7f6d0c017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005564464b3b00 in wsrep_post_rollback_process (post_rollbacker=0x7f6d0c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005564464a4282 in start_wsrep_THD (arg=0x5564494bde90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f6d2c1dd700 (LWP 227031)): #0 0x00007f6d34349965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d27baa132 in fifo_lock_get (q=q@entry=0x556449492bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f6d27baa7a1 in gu_fifo_get_head (q=0x556449492bd0, err=0x7f6d2c1dc06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f6d27c8c0cc in gcs_recv (conn=0x55644946c550, action=0x7f6d2c1dc0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f6d27cb916c in galera::GcsActionSource::process (this=0x55644946bf30, recv_ctx=0x7f6d2c1dc4d0, exit_loop=@0x7f6d2c1dc29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f6d27cd8033 in galera::ReplicatorSMM::async_recv (this=0x556449464280, recv_ctx=0x7f6d2c1dc4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000556446e0f276 in wsrep::wsrep_provider_v26::run_applier (this=0x5564493eb360, applier_ctx=0x7f6d2c1dc4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005564464b2636 in wsrep_replication_process (thd=0x7f6d10000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005564464a4282 in start_wsrep_THD (arg=0x5564494bde70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f6d34345dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f6d32703ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f6d34756880 (LWP 226889)): #0 0x00007f6d3434a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556446d55368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556446593503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6d326f920d in poll () from /lib64/libc.so.6 #5 0x0000556446133a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000556446132d25 in mysqld_main (argc=161, argv=0x5564492e5d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000556446126ec5 in main (argc=23, argv=0x7fff80e605c8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.1/data/core.226889 - found 'core.227621' (3/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.2/data/core.227621 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 227621] [New LWP 227742] [New LWP 227685] [New LWP 227740] [New LWP 228146] [New LWP 227624] [New LWP 228147] [New LWP 228145] [New LWP 228149] [New LWP 228148] [New LWP 228152] [New LWP 228150] [New LWP 228179] [New LWP 228180] [New LWP 228183] [New LWP 228181] [New LWP 228184] [New LWP 228185] [New LWP 228186] [New LWP 228188] [New LWP 228189] [New LWP 228190] [New LWP 228191] [New LWP 228192] [New LWP 228193] [New LWP 228194] [New LWP 228197] [New LWP 228187] [New LWP 228182] [New LWP 228151] [New LWP 227648] [New LWP 227739] [New LWP 227741] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fcc4462f9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fcc4462f9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055577c8ef368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055577c12d503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fcc429de20d in poll () from /lib64/libc.so.6 #5 0x000055577bccda8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055577bcccd25 in mysqld_main (argc=162, argv=0x55577f43cd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055577bcc0ec5 in main (argc=23, argv=0x7ffc85184638) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fcc3c094700 (LWP 227741)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c8f4997 in safe_cond_wait (cond=0x7fcc20017b68, mp=0x7fcc20017ab8, file=0x55577ca5eb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055577c04c088 in inline_mysql_cond_wait (that=0x7fcc20017b68, mutex=0x7fcc20017ab8, src_file=0x55577ca5ee80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055577c04ed20 in Wsrep_thd_queue::pop_front (this=0x7fcc20017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055577c04db00 in wsrep_post_rollback_process (post_rollbacker=0x7fcc20001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055577c03e282 in start_wsrep_THD (arg=0x55577f6174a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fcc2d171700 (LWP 227739)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcc3c513867 in gu::Lock::wait (cond=..., this=0x7fcc2d170850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fcc3c5163ab in RecvBuf::front (this=this@entry=0x55577f5f5818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fcc3c513f46 in gcomm_recv (backend=, msg=0x55577f5c3720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fcc3c507eec in core_msg_recv (backend=backend@entry=0x55577f5c3868, recv_msg=recv_msg@entry=0x55577f5c3720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fcc3c50b164 in gcs_core_recv (conn=0x55577f5c36c0, recv_act=recv_act@entry=0x7fcc2d170d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fcc3c5123d8 in gcs_recv_thread (arg=0x55577f5c3440) at gcs/src/gcs.cpp:1398 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fcc2fdf6700 (LWP 227648)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcc3c538fd7 in gu::Lock::wait (cond=..., this=0x7fcc2fdf5bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fcc3c539069 in galera::ServiceThd::thd_func (arg=0x55577f5bb908) at galera/src/galera_service_thd.cpp:30 #3 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fcc067fc700 (LWP 228151)): #0 0x00007fcc441fb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055577c42a41c in LinuxAIOHandler::collect (this=0x7fcc067fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055577c42a945 in LinuxAIOHandler::poll (this=0x7fcc067fbb00, m1=0x7fcc067fbba8, m2=0x7fcc067fbbb0, request=0x7fcc067fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055577c42aaa8 in os_aio_linux_handler (global_segment=5, m1=0x7fcc067fbba8, m2=0x7fcc067fbbb0, request=0x7fcc067fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055577c42f5cc in os_aio_handler (segment=5, m1=0x7fcc067fbba8, m2=0x7fcc067fbbb0, request=0x7fcc067fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055577c6824a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055577c528909 in io_handler_thread (arg=0x55577d6af648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fcbf2ffd700 (LWP 228182)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c436a1a in os_event::timed_wait (this=0x55577f910ad0, abstime=0x7fcbf2ffcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055577c436d31 in os_event::wait_time_low (this=0x55577f910ad0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055577c436ed1 in os_event_wait_time_low (event=0x55577f910ad0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055577c66c142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fcbf0ff9700 (LWP 228187)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c4370ff in os_event::wait (this=0x55577f7063e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055577c436b85 in os_event::wait_low (this=0x55577f7063e0, reset_sig_count=8097) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055577c436ef6 in os_event_wait_low (event=0x55577f7063e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055577c522df3 in srv_resume_thread (slot=0x55577d4717b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055577c5269f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fcc0d6f9700 (LWP 228197)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c8f4c99 in safe_cond_timedwait (cond=0x55577d6341a0 , mp=0x55577d636ba0 , abstime=0x7fcc0d6f8c60, file=0x55577c9b04c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055577bcc1845 in inline_mysql_cond_timedwait (that=0x55577d6341a0 , mutex=0x55577d636ba0 , abstime=0x7fcc0d6f8c60, src_file=0x55577c9b2538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055577bcc661b in cache_thread (thd=0x7fcbc4000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055577bcc68c9 in one_thread_per_connection_end (thd=0x7fcbc4000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055577bf630f3 in do_handle_one_connection (connect=0x55577f9581c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055577bf62dad in handle_one_connection (arg=0x55577f9581c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055577c88745f in pfs_spawn_thread (arg=0x55577f987cd0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fcc0d744700 (LWP 228194)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c8f4997 in safe_cond_wait (cond=0x55577d6341e0 , mp=0x55577d637560 , file=0x55577c9bfcc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055577bcf24ee in inline_mysql_cond_wait (that=0x55577d6341e0 , mutex=0x55577d637560 , src_file=0x55577c9c0830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055577bcf3d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055577c88745f in pfs_spawn_thread (arg=0x55577fa056d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fcc2c054700 (LWP 228193)): #0 0x00007fcc4292240a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055577bcc0f44 in my_sigwait (set=0x7fcc2c053cf0, sig=0x7fcc2c053cb4, code=0x7fcc2c053cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055577bcc7584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055577c88745f in pfs_spawn_thread (arg=0x55577f9848d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fcbe9ffb700 (LWP 228192)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c8f4997 in safe_cond_wait (cond=0x7fcbe9ffac70, mp=0x7fcbe9ffacb0, file=0x55577cbc2c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055577c34e487 in inline_mysql_cond_wait (that=0x7fcbe9ffac70, mutex=0x7fcbe9ffacb0, src_file=0x55577cbc45d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055577c3530c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055577c88745f in pfs_spawn_thread (arg=0x55577f8c12d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fcbea7fc700 (LWP 228191)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c4370ff in os_event::wait (this=0x55577f7066e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055577c436b85 in os_event::wait_low (this=0x55577f7066e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055577c436ef6 in os_event_wait_low (event=0x55577f7066e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055577c5ebb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fcbeaffd700 (LWP 228190)): #0 0x00007fcc44631e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055577c437401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055577c5dda3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fcbeb7fe700 (LWP 228189)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c4370ff in os_event::wait (this=0x55577f7065e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055577c436b85 in os_event::wait_low (this=0x55577f7065e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055577c436ef6 in os_event_wait_low (event=0x55577f7065e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055577c602eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fcbebfff700 (LWP 228188)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c4370ff in os_event::wait (this=0x55577f706460) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055577c436b85 in os_event::wait_low (this=0x55577f706460, reset_sig_count=7828) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055577c436ef6 in os_event_wait_low (event=0x55577f706460, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055577c522df3 in srv_resume_thread (slot=0x55577d4717f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055577c5269f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fcbf17fa700 (LWP 228186)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c4370ff in os_event::wait (this=0x55577f706360) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055577c436b85 in os_event::wait_low (this=0x55577f706360, reset_sig_count=8518) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055577c436ef6 in os_event_wait_low (event=0x55577f706360, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055577c522df3 in srv_resume_thread (slot=0x55577d471780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055577c5269f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fcbf1ffb700 (LWP 228185)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c4370ff in os_event::wait (this=0x55577f7062e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055577c436b85 in os_event::wait_low (this=0x55577f7062e0, reset_sig_count=2379) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055577c436ef6 in os_event_wait_low (event=0x55577f7062e0, reset_sig_count=2379) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055577c522df3 in srv_resume_thread (slot=0x55577d471748 , sig_count=2379, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055577c526e04 in srv_purge_coordinator_suspend (slot=0x55577d471748 , rseg_history_len=4) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055577c52709a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fcc057fa700 (LWP 228184)): #0 0x00007fcc44631e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055577c437401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055577c52635b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055577c52648b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fcbf37fe700 (LWP 228181)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c436a1a in os_event::timed_wait (this=0x55577f706560, abstime=0x7fcbf37fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055577c436d31 in os_event::wait_time_low (this=0x55577f706560, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055577c436ed1 in os_event_wait_time_low (event=0x55577f706560, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055577c524ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fcbf27fc700 (LWP 228183)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c436a1a in os_event::timed_wait (this=0x55577f8aea50, abstime=0x7fcbf27fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055577c436d31 in os_event::wait_time_low (this=0x55577f8aea50, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055577c436ed1 in os_event_wait_time_low (event=0x55577f8aea50, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055577c5852fb in ib_wqueue_timedwait (wq=0x55577f8ae920, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055577c6c5ea4 in fts_optimize_thread (arg=0x55577f8ae920) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fcbf3fff700 (LWP 228180)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c436a1a in os_event::timed_wait (this=0x55577f7064e0, abstime=0x7fcbf3ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055577c436d31 in os_event::wait_time_low (this=0x55577f7064e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055577c436ed1 in os_event_wait_time_low (event=0x55577f7064e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055577c524ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fcc04ff9700 (LWP 228179)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c436a1a in os_event::timed_wait (this=0x55577f8dea10, abstime=0x7fcc04ff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055577c436d31 in os_event::wait_time_low (this=0x55577f8dea10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055577c436ed1 in os_event_wait_time_low (event=0x55577f8dea10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055577c40388e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fcc06ffd700 (LWP 228150)): #0 0x00007fcc441fb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055577c42a41c in LinuxAIOHandler::collect (this=0x7fcc06ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055577c42a945 in LinuxAIOHandler::poll (this=0x7fcc06ffcb00, m1=0x7fcc06ffcba8, m2=0x7fcc06ffcbb0, request=0x7fcc06ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055577c42aaa8 in os_aio_linux_handler (global_segment=4, m1=0x7fcc06ffcba8, m2=0x7fcc06ffcbb0, request=0x7fcc06ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055577c42f5cc in os_aio_handler (segment=4, m1=0x7fcc06ffcba8, m2=0x7fcc06ffcbb0, request=0x7fcc06ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055577c6824a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055577c528909 in io_handler_thread (arg=0x55577d6af640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fcc05ffb700 (LWP 228152)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c436a1a in os_event::timed_wait (this=0x55577f706660, abstime=0x7fcc05ffaaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055577c436d31 in os_event::wait_time_low (this=0x55577f706660, time_in_usec=405000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055577c436ed1 in os_event_wait_time_low (event=0x55577f706660, time_in_usec=405000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055577c60cd1d in pc_sleep_if_needed (next_loop_time=1549881091811, sig_count=2, cur_time=1549881091406) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055577c60dc56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fcc07fff700 (LWP 228148)): #0 0x00007fcc441fb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055577c42a41c in LinuxAIOHandler::collect (this=0x7fcc07ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055577c42a945 in LinuxAIOHandler::poll (this=0x7fcc07ffeb00, m1=0x7fcc07ffeba8, m2=0x7fcc07ffebb0, request=0x7fcc07ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055577c42aaa8 in os_aio_linux_handler (global_segment=2, m1=0x7fcc07ffeba8, m2=0x7fcc07ffebb0, request=0x7fcc07ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055577c42f5cc in os_aio_handler (segment=2, m1=0x7fcc07ffeba8, m2=0x7fcc07ffebb0, request=0x7fcc07ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055577c6824a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055577c528909 in io_handler_thread (arg=0x55577d6af630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fcc077fe700 (LWP 228149)): #0 0x00007fcc441fb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055577c42a41c in LinuxAIOHandler::collect (this=0x7fcc077fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055577c42a945 in LinuxAIOHandler::poll (this=0x7fcc077fdb00, m1=0x7fcc077fdba8, m2=0x7fcc077fdbb0, request=0x7fcc077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055577c42aaa8 in os_aio_linux_handler (global_segment=3, m1=0x7fcc077fdba8, m2=0x7fcc077fdbb0, request=0x7fcc077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055577c42f5cc in os_aio_handler (segment=3, m1=0x7fcc077fdba8, m2=0x7fcc077fdbb0, request=0x7fcc077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055577c6824a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055577c528909 in io_handler_thread (arg=0x55577d6af638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fcc0ed96700 (LWP 228145)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c8f4c99 in safe_cond_timedwait (cond=0x55577dec0ea0 , mp=0x55577dec0de0 , abstime=0x7fcc0ed95cb0, file=0x55577cd38610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055577c75eacb in inline_mysql_cond_timedwait (that=0x55577dec0ea0 , mutex=0x55577dec0de0 , abstime=0x7fcc0ed95cb0, src_file=0x55577cd38658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055577c75efef in my_service_thread_sleep (control=0x55577d475b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055577c753488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055577c88745f in pfs_spawn_thread (arg=0x55577f6be540) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fcc0cd43700 (LWP 228147)): #0 0x00007fcc441fb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055577c42a41c in LinuxAIOHandler::collect (this=0x7fcc0cd42b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055577c42a945 in LinuxAIOHandler::poll (this=0x7fcc0cd42b00, m1=0x7fcc0cd42ba8, m2=0x7fcc0cd42bb0, request=0x7fcc0cd42be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055577c42aaa8 in os_aio_linux_handler (global_segment=1, m1=0x7fcc0cd42ba8, m2=0x7fcc0cd42bb0, request=0x7fcc0cd42be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055577c42f5cc in os_aio_handler (segment=1, m1=0x7fcc0cd42ba8, m2=0x7fcc0cd42bb0, request=0x7fcc0cd42be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055577c6824a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055577c528909 in io_handler_thread (arg=0x55577d6af628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fcc3d083700 (LWP 227624)): #0 0x00007fcc4462ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c8f4c99 in safe_cond_timedwait (cond=0x55577ded3640 , mp=0x55577ded3580 , abstime=0x7fcc3d082d70, file=0x55577cd7c840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055577c8f5c53 in inline_mysql_cond_timedwait (that=0x55577ded3640 , mutex=0x55577ded3580 , abstime=0x7fcc3d082d70, src_file=0x55577cd7c888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055577c8f67b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055577c88745f in pfs_spawn_thread (arg=0x55577f5847d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fcc0d544700 (LWP 228146)): #0 0x00007fcc441fb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055577c42a41c in LinuxAIOHandler::collect (this=0x7fcc0d543b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055577c42a945 in LinuxAIOHandler::poll (this=0x7fcc0d543b00, m1=0x7fcc0d543ba8, m2=0x7fcc0d543bb0, request=0x7fcc0d543be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055577c42aaa8 in os_aio_linux_handler (global_segment=0, m1=0x7fcc0d543ba8, m2=0x7fcc0d543bb0, request=0x7fcc0d543be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055577c42f5cc in os_aio_handler (segment=0, m1=0x7fcc0d543ba8, m2=0x7fcc0d543bb0, request=0x7fcc0d543be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055577c6824a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055577c528909 in io_handler_thread (arg=0x55577d6af620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fcc3c0df700 (LWP 227740)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055577c8f4997 in safe_cond_wait (cond=0x7fcc18015608, mp=0x7fcc18015558, file=0x55577ca5eb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055577c04c088 in inline_mysql_cond_wait (that=0x7fcc18015608, mutex=0x7fcc18015558, src_file=0x55577ca5ee80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055577c04ed20 in Wsrep_thd_queue::pop_front (this=0x7fcc18015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055577c04d740 in wsrep_rollback_process (rollbacker=0x7fcc18000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055577c03e282 in start_wsrep_THD (arg=0x55577f617de0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fcc2d972700 (LWP 227685)): #0 0x00007fcc429e9483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fcc3c4da6f8 in asio::detail::epoll_reactor::run (this=0x55577f5f6000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fcc3c4dedf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55577f5f5ea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fcc3c4df0a2 in asio::detail::task_io_service::run (this=0x55577f5f5ea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fcc3c4f6afd in asio::io_service::run (this=0x55577f5f4bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fcc3c4f7251 in gcomm::AsioProtonet::event_loop (this=0x55577f5f4b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fcc3c5152ed in GCommConn::run (this=0x55577f5f5630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fcc3c518399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fcc2c970700 (LWP 227742)): #0 0x00007fcc4462e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcc3c42d132 in fifo_lock_get (q=q@entry=0x55577f5e9ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007fcc3c42d7a1 in gu_fifo_get_head (q=0x55577f5e9ac0, err=0x7fcc2c96f06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fcc3c50f0cc in gcs_recv (conn=0x55577f5c3440, action=0x7fcc2c96f0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fcc3c53c16c in galera::GcsActionSource::process (this=0x55577f5ede60, recv_ctx=0x7fcc2c96f4d0, exit_loop=@0x7fcc2c96f29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fcc3c55b033 in galera::ReplicatorSMM::async_recv (this=0x55577f5bb1f0, recv_ctx=0x7fcc2c96f4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055577c9a9276 in wsrep::wsrep_provider_v26::run_applier (this=0x55577f530e50, applier_ctx=0x7fcc2c96f4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055577c04c636 in wsrep_replication_process (thd=0x7fcc1c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055577c03e282 in start_wsrep_THD (arg=0x55577f60dbf0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fcc4462add5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fcc429e8ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fcc44a3b880 (LWP 227621)): #0 0x00007fcc4462f9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055577c8ef368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055577c12d503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fcc429de20d in poll () from /lib64/libc.so.6 #5 0x000055577bccda8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055577bcccd25 in mysqld_main (argc=162, argv=0x55577f43cd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055577bcc0ec5 in main (argc=23, argv=0x7ffc85184638) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 worker[1] mysql-test-run: WARNING: Test reserved for w3 picked up by w1 worker[2] Trying to dump core for [mysqltest - pid: 233421, winpid: 233421] worker[2] Trying to dump core for [mysqld.1 - pid: 228538, winpid: 228538] worker[2] Trying to dump core for [mysqld.2 - pid: 228906, winpid: 228906] worker[4] Trying to dump core for [mysqltest - pid: 233806, winpid: 233806] worker[4] Trying to dump core for [mysqld.1 - pid: 228349, winpid: 228349] worker[4] Trying to dump core for [mysqld.2 - pid: 228668, winpid: 228668] Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.2/data/core.227621 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1018B-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_large_fragment 2019-02-11 12:23:32 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:24:03 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:24:03 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:24:34 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:24:34 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:05 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:05 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:05 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:05 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:05 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:36 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:36 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:26:07 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:26:07 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:26:38 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:26:38 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:27:09 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:27:09 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:27:40 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:27:40 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:11 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:11 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:42 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:42 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:13 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:13 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:44 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:44 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:44 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:44 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:44 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:15 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:15 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:15 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:15 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:15 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:46 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:46 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:46 0 [Warning] InnoDB: A long semaphore wait: Attempting backtrace. You can use the following information to find out 2019-02-11 12:24:41 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:12 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:25:43 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:26:14 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:26:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:27:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:27:47 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:18 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:49 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:49 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:49 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:28:49 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:20 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:20 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:20 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:20 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:51 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:29:51 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:22 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:22 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-11 12:30:53 0 [Warning] InnoDB: A long semaphore wait: Attempting backtrace. You can use the following information to find out ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1018B Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_parallel_apply 'innodb' w1 [ pass ] 8832 galera_sr.galera_sr_kill_all_nobootstrap 'innodb' w1 [ pass ] 40331 galera_sr.galera_sr_rollback 'innodb' w1 [ pass ] 9514 galera_sr.galera_sr_rollback_retry 'innodb' w1 [ pass ] 5113 galera_sr.galera_sr_rollback_savepoint 'innodb' w1 [ pass ] 7165 galera_sr.galera_sr_insert_select 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 12:34:27 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr_insert_select.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.228539' (4/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_insert_select-innodb/mysqld.1/data/core.228539 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 228539] [New LWP 228540] [New LWP 228565] [New LWP 228587] [New LWP 228581] [New LWP 228590] [New LWP 228591] [New LWP 228601] [New LWP 228592] [New LWP 228603] [New LWP 228593] [New LWP 228597] [New LWP 228594] [New LWP 228596] [New LWP 228600] [New LWP 228850] [New LWP 228602] [New LWP 228585] [New LWP 228584] [New LWP 228542] [New LWP 228854] [New LWP 228599] [New LWP 228626] [New LWP 228598] [New LWP 229710] [New LWP 228583] [New LWP 228586] [New LWP 228582] [New LWP 228566] [New LWP 228564] [New LWP 228563] [New LWP 228541] [New LWP 228588] [New LWP 228595] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fdea6f8c9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fdea6f8c9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056531b9d3368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056531b211503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdea533b20d in poll () from /lib64/libc.so.6 #5 0x000056531adb1a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056531adb0d25 in mysqld_main (argc=161, argv=0x56531d376d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056531ada4ec5 in main (argc=23, argv=0x7fffcc104708) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fde6d7fa700 (LWP 228595)): #0 0x00007fdea6f8ed1d in fsync () from /lib64/libpthread.so.0 #1 0x000056531b50facb in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056531b50fe21 in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056531b7583a2 in pfs_os_file_flush_func (file=..., src_file=0x56531bdd2a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056531b75b11d in fil_flush_low (space=0x56531d8473b0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056531b7669f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000056531b4eb1cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000056531b4ebc31 in log_write_up_to (lsn=5178425, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000056531b4ebd61 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x000056531b609430 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #10 0x000056531b6099f3 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #11 0x000056531b60a551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fde78ff9700 (LWP 228588)): #0 0x00007fdea6f8ed1d in fsync () from /lib64/libpthread.so.0 #1 0x000056531b50facb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056531b50fe21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056531b7583a2 in pfs_os_file_flush_func (file=..., src_file=0x56531bdd2a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056531b75b11d in fil_flush_low (space=0x56531d847010, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056531b7669f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000056531b6e3cbd in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x000056531b6ee3a5 in buf_flush_end (buf_pool=0x56531d723140, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x000056531b6ee563 in buf_flush_do_batch (buf_pool=0x56531d723140, type=BUF_FLUSH_LIST, min_n=7, lsn_limit=18446744073709551615, n=0x7fde78ff8b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x000056531b6f124d in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x000056531b6f219b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fde9de33700 (LWP 228541)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fde9ee95fd7 in gu::Lock::wait (cond=..., this=0x7fde9de32bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fde9ee96069 in galera::ServiceThd::thd_func (arg=0x56531d4f5998) at galera/src/galera_service_thd.cpp:30 #3 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fde96e7d700 (LWP 228563)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fde9ee70867 in gu::Lock::wait (cond=..., this=0x7fde96e7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fde9ee733ab in RecvBuf::front (this=this@entry=0x56531d5358a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fde9ee70f46 in gcomm_recv (backend=, msg=0x56531d4fd830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fde9ee64eec in core_msg_recv (backend=backend@entry=0x56531d4fd978, recv_msg=recv_msg@entry=0x56531d4fd830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fde9ee68164 in gcs_core_recv (conn=0x56531d4fd7d0, recv_act=recv_act@entry=0x7fde96e7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fde9ee6f3d8 in gcs_recv_thread (arg=0x56531d4fd550) at gcs/src/gcs.cpp:1398 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fde9c330700 (LWP 228564)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b9d8997 in safe_cond_wait (cond=0x7fde88015608, mp=0x7fde88015558, file=0x56531bb42b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056531b130088 in inline_mysql_cond_wait (that=0x7fde88015608, mutex=0x7fde88015558, src_file=0x56531bb42e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056531b132d20 in Wsrep_thd_queue::pop_front (this=0x7fde88015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056531b131740 in wsrep_rollback_process (rollbacker=0x7fde88000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000056531b122282 in start_wsrep_THD (arg=0x56531d545320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fde9c29a700 (LWP 228566)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fde9ed8a132 in fifo_lock_get (q=q@entry=0x56531d523bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007fde9ed8a7a1 in gu_fifo_get_head (q=0x56531d523bd0, err=0x7fde9c29906c) at galerautils/src/gu_fifo.c:330 #3 0x00007fde9ee6c0cc in gcs_recv (conn=0x56531d4fd550, action=0x7fde9c2990b0) at gcs/src/gcs.cpp:2024 #4 0x00007fde9ee9916c in galera::GcsActionSource::process (this=0x56531d4fcf30, recv_ctx=0x7fde9c2994d0, exit_loop=@0x7fde9c29929e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fde9eeb8033 in galera::ReplicatorSMM::async_recv (this=0x56531d4f5280, recv_ctx=0x7fde9c2994d0) at galera/src/replicator_smm.cpp:383 #6 0x000056531ba8d276 in wsrep::wsrep_provider_v26::run_applier (this=0x56531d47c360, applier_ctx=0x7fde9c2994d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000056531b130636 in wsrep_replication_process (thd=0x7fde8c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000056531b122282 in start_wsrep_THD (arg=0x56531d54e2a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fde7bfff700 (LWP 228582)): #0 0x00007fdea6b58644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056531b50e41c in LinuxAIOHandler::collect (this=0x7fde7bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056531b50e945 in LinuxAIOHandler::poll (this=0x7fde7bffeb00, m1=0x7fde7bffeba8, m2=0x7fde7bffebb0, request=0x7fde7bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056531b50eaa8 in os_aio_linux_handler (global_segment=0, m1=0x7fde7bffeba8, m2=0x7fde7bffebb0, request=0x7fde7bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056531b5135cc in os_aio_handler (segment=0, m1=0x7fde7bffeba8, m2=0x7fde7bffebb0, request=0x7fde7bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056531b7664a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056531b60c909 in io_handler_thread (arg=0x56531c793620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fde79ffb700 (LWP 228586)): #0 0x00007fdea6b58644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056531b50e41c in LinuxAIOHandler::collect (this=0x7fde79ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056531b50e945 in LinuxAIOHandler::poll (this=0x7fde79ffab00, m1=0x7fde79ffaba8, m2=0x7fde79ffabb0, request=0x7fde79ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056531b50eaa8 in os_aio_linux_handler (global_segment=4, m1=0x7fde79ffaba8, m2=0x7fde79ffabb0, request=0x7fde79ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056531b5135cc in os_aio_handler (segment=4, m1=0x7fde79ffaba8, m2=0x7fde79ffabb0, request=0x7fde79ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056531b7664a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056531b60c909 in io_handler_thread (arg=0x56531c793640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fde7b7fe700 (LWP 228583)): #0 0x00007fdea6b58644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056531b50e41c in LinuxAIOHandler::collect (this=0x7fde7b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056531b50e945 in LinuxAIOHandler::poll (this=0x7fde7b7fdb00, m1=0x7fde7b7fdba8, m2=0x7fde7b7fdbb0, request=0x7fde7b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056531b50eaa8 in os_aio_linux_handler (global_segment=1, m1=0x7fde7b7fdba8, m2=0x7fde7b7fdbb0, request=0x7fde7b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056531b5135cc in os_aio_handler (segment=1, m1=0x7fde7b7fdba8, m2=0x7fde7b7fdbb0, request=0x7fde7b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056531b7664a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056531b60c909 in io_handler_thread (arg=0x56531c793628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fde94608700 (LWP 229710)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51b0ff in os_event::wait (this=0x56531d7d5a80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056531b51ab85 in os_event::wait_low (this=0x56531d7d5a80, reset_sig_count=3238) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056531b51aef6 in os_event_wait_low (event=0x56531d7d5a80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056531b4eb761 in log_write_up_to (lsn=5181036, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000056531b656d91 in trx_flush_log_if_needed_low (lsn=5181036) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #6 0x000056531b656dda in trx_flush_log_if_needed (lsn=5181036, trx=0x7fde950f2390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #7 0x000056531b658c03 in trx_commit_complete_for_mysql (trx=0x7fde950f2390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #8 0x000056531b43d514 in innobase_commit (hton=0x56531d611b10, thd=0x7fde280cb290, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #9 0x000056531b217039 in commit_one_phase_2 (thd=0x7fde280cb290, all=true, trans=0x7fde280ce7e0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #10 0x000056531b216f3b in ha_commit_one_phase (thd=0x7fde280cb290, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #11 0x000056531b2162b7 in ha_commit_trans (thd=0x7fde280cb290, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #12 0x000056531b05be8e in trans_commit (thd=0x7fde280cb290) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #13 0x000056531b41b18c in Wsrep_storage_service::commit (this=0x7fde280c44c0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #14 0x000056531ba834d3 in wsrep::transaction::certify_fragment (this=0x7fde280069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #15 0x000056531ba82c37 in wsrep::transaction::streaming_step (this=0x7fde280069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #16 0x000056531ba7fff2 in wsrep::transaction::after_row (this=0x7fde280069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #17 0x000056531b2293a8 in wsrep::client_state::after_row (this=0x7fde28006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #18 0x000056531b212625 in wsrep_after_row (thd=0x7fde28000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #19 0x000056531b22452a in wsrep_after_row (thd=0x7fde28000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #20 0x000056531b22491b in handler::ha_write_row (this=0x7fde28072068, buf=0x7fde280690b0 "\375\037\021") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #21 0x000056531ae9a443 in write_record (thd=0x7fde28000b00, table=0x7fde28069590, info=0x7fde28017040) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #22 0x000056531ae9f742 in select_insert::send_data (this=0x7fde28016ff8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #23 0x000056531af57e07 in end_send (join=0x7fde280170a8, join_tab=0x7fde280b79e8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #24 0x000056531b094fa7 in JOIN_CACHE::generate_full_extensions (this=0x7fde280b7ff8, rec_ptr=0x7fde2818d56c "p\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #25 0x000056531b094cc2 in JOIN_CACHE::join_matching_records (this=0x7fde280b7ff8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #26 0x000056531b09468e in JOIN_CACHE::join_records (this=0x7fde280b7ff8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #27 0x000056531b094828 in JOIN_CACHE::join_records (this=0x7fde280b7ec8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #28 0x000056531b094828 in JOIN_CACHE::join_records (this=0x7fde280b7d98, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #29 0x000056531af54abb in sub_select_cache (join=0x7fde280170a8, join_tab=0x7fde280b6ed8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #30 0x000056531af54cc8 in sub_select (join=0x7fde280170a8, join_tab=0x7fde280b6b28, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #31 0x000056531af54493 in do_select (join=0x7fde280170a8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #32 0x000056531af2cfb1 in JOIN::exec_inner (this=0x7fde280170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #33 0x000056531af2c3fa in JOIN::exec (this=0x7fde280170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #34 0x000056531af2d6ec in mysql_select (thd=0x7fde28000b00, tables=0x7fde28014dd0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7fde28016ff8, unit=0x7fde280049c0, select_lex=0x7fde28014700) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #35 0x000056531af1efed in handle_select (thd=0x7fde28000b00, lex=0x7fde280048f8, result=0x7fde28016ff8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #36 0x000056531aee13fc in mysql_execute_command (thd=0x7fde28000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #37 0x000056531aeec935 in mysql_parse (thd=0x7fde28000b00, rawbuf=0x7fde28013e58 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fde946070d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #38 0x000056531aeebf68 in wsrep_mysql_parse (thd=0x7fde28000b00, rawbuf=0x7fde28013e58 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fde946070d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #39 0x000056531aed7e25 in dispatch_command (command=COM_QUERY, thd=0x7fde28000b00, packet=0x7fde2800a431 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #40 0x000056531aed680d in do_command (thd=0x7fde28000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #41 0x000056531b047069 in do_handle_one_connection (connect=0x56531d601b70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #42 0x000056531b046dad in handle_one_connection (arg=0x56531d601b70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #43 0x000056531b96b45f in pfs_spawn_thread (arg=0x56531d4b1a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #44 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #45 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fde5f7fe700 (LWP 228598)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51b0ff in os_event::wait (this=0x56531d63d740) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056531b51ab85 in os_event::wait_low (this=0x56531d63d740, reset_sig_count=8167) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056531b51aef6 in os_event_wait_low (event=0x56531d63d740, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056531b606df3 in srv_resume_thread (slot=0x56531c5557b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056531b60a9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fde9c072700 (LWP 228626)): #0 0x00007fdea527f40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000056531ada4f44 in my_sigwait (set=0x7fde9c071cf0, sig=0x7fde9c071cb4, code=0x7fde9c071cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000056531adab584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000056531b96b45f in pfs_spawn_thread (arg=0x56531d600f10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fde5effd700 (LWP 228599)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51b0ff in os_event::wait (this=0x56531d63d6c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056531b51ab85 in os_event::wait_low (this=0x56531d63d6c0, reset_sig_count=8689) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056531b51aef6 in os_event_wait_low (event=0x56531d63d6c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056531b606df3 in srv_resume_thread (slot=0x56531c555780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056531b60a9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fde94653700 (LWP 228854)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b9d8c99 in safe_cond_timedwait (cond=0x56531c7181a0 , mp=0x56531c71aba0 , abstime=0x7fde94652c60, file=0x56531ba944c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056531ada5845 in inline_mysql_cond_timedwait (that=0x56531c7181a0 , mutex=0x56531c71aba0 , abstime=0x7fde94652c60, src_file=0x56531ba96538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056531adaa61b in cache_thread (thd=0x7fde30000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000056531adaa8c9 in one_thread_per_connection_end (thd=0x7fde30000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000056531b0470f3 in do_handle_one_connection (connect=0x56531d9ac550) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000056531b046dad in handle_one_connection (arg=0x56531d9ac550) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000056531b96b45f in pfs_spawn_thread (arg=0x56531d9ce360) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fde9767e700 (LWP 228542)): #0 0x00007fdea5346483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fde9ee376f8 in asio::detail::epoll_reactor::run (this=0x56531d536040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fde9ee3bdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x56531d535ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fde9ee3c0a2 in asio::detail::task_io_service::run (this=0x56531d535ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fde9ee53afd in asio::io_service::run (this=0x56531d534fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fde9ee54251 in gcomm::AsioProtonet::event_loop (this=0x56531d534f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fde9ee722ed in GCommConn::run (this=0x56531d5356c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fde9ee75399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fde7affd700 (LWP 228584)): #0 0x00007fdea6b58644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056531b50e41c in LinuxAIOHandler::collect (this=0x7fde7affcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056531b50e945 in LinuxAIOHandler::poll (this=0x7fde7affcb00, m1=0x7fde7affcba8, m2=0x7fde7affcbb0, request=0x7fde7affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056531b50eaa8 in os_aio_linux_handler (global_segment=2, m1=0x7fde7affcba8, m2=0x7fde7affcbb0, request=0x7fde7affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056531b5135cc in os_aio_handler (segment=2, m1=0x7fde7affcba8, m2=0x7fde7affcbb0, request=0x7fde7affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056531b7664a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056531b60c909 in io_handler_thread (arg=0x56531c793630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fde7a7fc700 (LWP 228585)): #0 0x00007fdea6b58644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056531b50e41c in LinuxAIOHandler::collect (this=0x7fde7a7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056531b50e945 in LinuxAIOHandler::poll (this=0x7fde7a7fbb00, m1=0x7fde7a7fbba8, m2=0x7fde7a7fbbb0, request=0x7fde7a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056531b50eaa8 in os_aio_linux_handler (global_segment=3, m1=0x7fde7a7fbba8, m2=0x7fde7a7fbbb0, request=0x7fde7a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056531b5135cc in os_aio_handler (segment=3, m1=0x7fde7a7fbba8, m2=0x7fde7a7fbbb0, request=0x7fde7a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056531b7664a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056531b60c909 in io_handler_thread (arg=0x56531c793638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fde5d7fa700 (LWP 228602)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51b0ff in os_event::wait (this=0x56531d63da40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056531b51ab85 in os_event::wait_low (this=0x56531d63da40, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056531b51aef6 in os_event_wait_low (event=0x56531d63da40, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056531b6cfb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fde9469e700 (LWP 228850)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b9d8997 in safe_cond_wait (cond=0x56531c7181e0 , mp=0x56531c71b560 , file=0x56531baa3cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056531add64ee in inline_mysql_cond_wait (that=0x56531c7181e0 , mutex=0x56531c71b560 , src_file=0x56531baa4830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000056531add7d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056531b96b45f in pfs_spawn_thread (arg=0x56531d601400) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fde5e7fc700 (LWP 228600)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51b0ff in os_event::wait (this=0x56531d63d940) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056531b51ab85 in os_event::wait_low (this=0x56531d63d940, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056531b51aef6 in os_event_wait_low (event=0x56531d63d940, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056531b6e6eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fde6cff9700 (LWP 228596)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51aa1a in os_event::timed_wait (this=0x56531d63d640, abstime=0x7fde6cff8cb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056531b51ad31 in os_event::wait_time_low (this=0x56531d63d640, time_in_usec=10000, reset_sig_count=2155) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056531b51aed1 in os_event_wait_time_low (event=0x56531d63d640, time_in_usec=10000, reset_sig_count=2155) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056531b606dcc in srv_resume_thread (slot=0x56531c555748 , sig_count=2155, wait=true, timeout_usec=10000) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:898 #5 0x000056531b60ae04 in srv_purge_coordinator_suspend (slot=0x56531c555748 , rseg_history_len=17) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000056531b60b09a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fde6dffb700 (LWP 228594)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51aa1a in os_event::timed_wait (this=0x56531d7dd790, abstime=0x7fde6dffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056531b51ad31 in os_event::wait_time_low (this=0x56531d7dd790, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056531b51aed1 in os_event_wait_time_low (event=0x56531d7dd790, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056531b6692fb in ib_wqueue_timedwait (wq=0x56531d7dd660, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056531b7a9ea4 in fts_optimize_thread (arg=0x56531d7dd660) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fde5ffff700 (LWP 228597)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51b0ff in os_event::wait (this=0x56531d63d7c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056531b51ab85 in os_event::wait_low (this=0x56531d63d7c0, reset_sig_count=7785) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056531b51aef6 in os_event_wait_low (event=0x56531d63d7c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056531b606df3 in srv_resume_thread (slot=0x56531c5557f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056531b60a9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fde6e7fc700 (LWP 228593)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51aa1a in os_event::timed_wait (this=0x56531d8479f0, abstime=0x7fde6e7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056531b51ad31 in os_event::wait_time_low (this=0x56531d8479f0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056531b51aed1 in os_event_wait_time_low (event=0x56531d8479f0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056531b750142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fde5cff9700 (LWP 228603)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b9d8997 in safe_cond_wait (cond=0x7fde5cff8c70, mp=0x7fde5cff8cb0, file=0x56531bca6c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056531b432487 in inline_mysql_cond_wait (that=0x7fde5cff8c70, mutex=0x7fde5cff8cb0, src_file=0x56531bca85d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056531b4370c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000056531b96b45f in pfs_spawn_thread (arg=0x56531d87e600) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fde6ffff700 (LWP 228592)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51aa1a in os_event::timed_wait (this=0x56531d63d8c0, abstime=0x7fde6fffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056531b51ad31 in os_event::wait_time_low (this=0x56531d63d8c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056531b51aed1 in os_event_wait_time_low (event=0x56531d63d8c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056531b608ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fde5dffb700 (LWP 228601)): #0 0x00007fdea6f8ee3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056531b51b401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056531b6c1a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fde6effd700 (LWP 228591)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51aa1a in os_event::timed_wait (this=0x56531d63d840, abstime=0x7fde6effcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056531b51ad31 in os_event::wait_time_low (this=0x56531d63d840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056531b51aed1 in os_event_wait_time_low (event=0x56531d63d840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056531b608ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fde6f7fe700 (LWP 228590)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b51aa1a in os_event::timed_wait (this=0x56531d815930, abstime=0x7fde6f7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056531b51ad31 in os_event::wait_time_low (this=0x56531d815930, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056531b51aed1 in os_event_wait_time_low (event=0x56531d815930, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056531b4e788e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fde95cf0700 (LWP 228581)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b9d8c99 in safe_cond_timedwait (cond=0x56531cfa4ea0 , mp=0x56531cfa4de0 , abstime=0x7fde95cefcb0, file=0x56531be1c610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056531b842acb in inline_mysql_cond_timedwait (that=0x56531cfa4ea0 , mutex=0x56531cfa4de0 , abstime=0x7fde95cefcb0, src_file=0x56531be1c658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056531b842fef in my_service_thread_sleep (control=0x56531c559b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056531b837488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056531b96b45f in pfs_spawn_thread (arg=0x56531d5f57e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fde797fa700 (LWP 228587)): #0 0x00007fdea6b58644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056531b50e41c in LinuxAIOHandler::collect (this=0x7fde797f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056531b50e945 in LinuxAIOHandler::poll (this=0x7fde797f9b00, m1=0x7fde797f9ba8, m2=0x7fde797f9bb0, request=0x7fde797f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056531b50eaa8 in os_aio_linux_handler (global_segment=5, m1=0x7fde797f9ba8, m2=0x7fde797f9bb0, request=0x7fde797f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056531b5135cc in os_aio_handler (segment=5, m1=0x7fde797f9ba8, m2=0x7fde797f9bb0, request=0x7fde797f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056531b7664a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056531b60c909 in io_handler_thread (arg=0x56531c793648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fde9c2e5700 (LWP 228565)): #0 0x00007fdea6f8b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b9d8997 in safe_cond_wait (cond=0x7fde90017b68, mp=0x7fde90017ab8, file=0x56531bb42b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056531b130088 in inline_mysql_cond_wait (that=0x7fde90017b68, mutex=0x7fde90017ab8, src_file=0x56531bb42e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056531b132d20 in Wsrep_thd_queue::pop_front (this=0x7fde90017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056531b131b00 in wsrep_post_rollback_process (post_rollbacker=0x7fde90001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000056531b122282 in start_wsrep_THD (arg=0x56531d54dc00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fde9f9e0700 (LWP 228540)): #0 0x00007fdea6f8bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056531b9d8c99 in safe_cond_timedwait (cond=0x56531cfb7640 , mp=0x56531cfb7580 , abstime=0x7fde9f9dfd70, file=0x56531be60840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056531b9d9c53 in inline_mysql_cond_timedwait (that=0x56531cfb7640 , mutex=0x56531cfb7580 , abstime=0x7fde9f9dfd70, src_file=0x56531be60888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056531b9da7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056531b96b45f in pfs_spawn_thread (arg=0x56531d4be730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdea6f87dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdea5345ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fdea7398880 (LWP 228539)): #0 0x00007fdea6f8c9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056531b9d3368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056531b211503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdea533b20d in poll () from /lib64/libc.so.6 #5 0x000056531adb1a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056531adb0d25 in mysqld_main (argc=161, argv=0x56531d376d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056531ada4ec5 in main (argc=23, argv=0x7fffcc104708) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_insert_select-innodb/mysqld.1/data/core.228539 - found 'core.228908' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_insert_select-innodb/mysqld.2/data/core.228908 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 228908] [New LWP 229132] [New LWP 229035] [New LWP 229131] [New LWP 229130] [New LWP 229520] [New LWP 229519] [New LWP 229518] [New LWP 229522] [New LWP 229523] [New LWP 229571] [New LWP 229521] [New LWP 229568] [New LWP 229579] [New LWP 229524] [New LWP 229570] [New LWP 229621] [New LWP 229569] [New LWP 229572] [New LWP 229622] [New LWP 229578] [New LWP 229573] [New LWP 229574] [New LWP 233152] [New LWP 229575] [New LWP 229581] [New LWP 229576] [New LWP 229517] [New LWP 229577] [New LWP 229580] [New LWP 229133] [New LWP 229041] [New LWP 228946] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007ff7b92d39d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007ff7b92d39d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005606c5f00368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005606c573e503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff7b768220d in poll () from /lib64/libc.so.6 #5 0x00005606c52dea8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005606c52ddd25 in mysqld_main (argc=162, argv=0x5606c8127d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005606c52d1ec5 in main (argc=23, argv=0x7fffac887508) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7ff7b1d27700 (LWP 228946)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5f05c99 in safe_cond_timedwait (cond=0x5606c74e4640 , mp=0x5606c74e4580 , abstime=0x7ff7b1d26d70, file=0x5606c638d840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005606c5f06c53 in inline_mysql_cond_timedwait (that=0x5606c74e4640 , mutex=0x5606c74e4580 , abstime=0x7ff7b1d26d70, src_file=0x5606c638d888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005606c5f077b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005606c5e9845f in pfs_spawn_thread (arg=0x5606c826f7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff7a237b700 (LWP 229041)): #0 0x00007ff7b768d483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff7b117e6f8 in asio::detail::epoll_reactor::run (this=0x5606c82e1000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007ff7b1182df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5606c82e0ea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007ff7b11830a2 in asio::detail::task_io_service::run (this=0x5606c82e0ea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007ff7b119aafd in asio::io_service::run (this=0x5606c82dfbc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007ff7b119b251 in gcomm::AsioProtonet::event_loop (this=0x5606c82dfb20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007ff7b11b92ed in GCommConn::run (this=0x5606c82e0630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007ff7b11bc399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff7b02e4700 (LWP 229133)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff7b10d1132 in fifo_lock_get (q=q@entry=0x5606c82d4ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007ff7b10d17a1 in gu_fifo_get_head (q=0x5606c82d4ac0, err=0x7ff7b02e306c) at galerautils/src/gu_fifo.c:330 #3 0x00007ff7b11b30cc in gcs_recv (conn=0x5606c82ae440, action=0x7ff7b02e30b0) at gcs/src/gcs.cpp:2024 #4 0x00007ff7b11e016c in galera::GcsActionSource::process (this=0x5606c82d8e60, recv_ctx=0x7ff7b02e34d0, exit_loop=@0x7ff7b02e329e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007ff7b11ff033 in galera::ReplicatorSMM::async_recv (this=0x5606c82a61f0, recv_ctx=0x7ff7b02e34d0) at galera/src/replicator_smm.cpp:383 #6 0x00005606c5fba276 in wsrep::wsrep_provider_v26::run_applier (this=0x5606c821be50, applier_ctx=0x7ff7b02e34d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005606c565d636 in wsrep_replication_process (thd=0x7ff798000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005606c564f282 in start_wsrep_THD (arg=0x5606c82f8bf0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff7477fe700 (LWP 229580)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a480ff in os_event::wait (this=0x5606c83f16a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005606c5a47b85 in os_event::wait_low (this=0x5606c83f16a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005606c5a47ef6 in os_event_wait_low (event=0x5606c83f16a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005606c5bfcb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff7657fa700 (LWP 229577)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a480ff in os_event::wait (this=0x5606c83f1420) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005606c5a47b85 in os_event::wait_low (this=0x5606c83f1420, reset_sig_count=7778) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005606c5a47ef6 in os_event_wait_low (event=0x5606c83f1420, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005606c5b33df3 in srv_resume_thread (slot=0x5606c6a827f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005606c5b379f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff78385a700 (LWP 229517)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5f05c99 in safe_cond_timedwait (cond=0x5606c74d1ea0 , mp=0x5606c74d1de0 , abstime=0x7ff783859cb0, file=0x5606c6349610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005606c5d6facb in inline_mysql_cond_timedwait (that=0x5606c74d1ea0 , mutex=0x5606c74d1de0 , abstime=0x7ff783859cb0, src_file=0x5606c6349658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005606c5d6ffef in my_service_thread_sleep (control=0x5606c6a86b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005606c5d64488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005606c5e9845f in pfs_spawn_thread (arg=0x5606c83a96d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff765ffb700 (LWP 229576)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a480ff in os_event::wait (this=0x5606c83f13a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005606c5a47b85 in os_event::wait_low (this=0x5606c83f13a0, reset_sig_count=8529) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005606c5a47ef6 in os_event_wait_low (event=0x5606c83f13a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005606c5b33df3 in srv_resume_thread (slot=0x5606c6a827b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005606c5b379f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff746ffd700 (LWP 229581)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5f05997 in safe_cond_wait (cond=0x7ff746ffcc70, mp=0x7ff746ffccb0, file=0x5606c61d3c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005606c595f487 in inline_mysql_cond_wait (that=0x7ff746ffcc70, mutex=0x7ff746ffccb0, src_file=0x5606c61d55d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005606c59640c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005606c5e9845f in pfs_spawn_thread (arg=0x5606c85ac200) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff7667fc700 (LWP 229575)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a480ff in os_event::wait (this=0x5606c83f1320) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005606c5a47b85 in os_event::wait_low (this=0x5606c83f1320, reset_sig_count=8856) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005606c5a47ef6 in os_event_wait_low (event=0x5606c83f1320, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005606c5b33df3 in srv_resume_thread (slot=0x5606c6a82780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005606c5b379f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff7a0145700 (LWP 233152)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5f05c99 in safe_cond_timedwait (cond=0x5606c6c451a0 , mp=0x5606c6c47ba0 , abstime=0x7ff7a0144c60, file=0x5606c5fc14c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005606c52d2845 in inline_mysql_cond_timedwait (that=0x5606c6c451a0 , mutex=0x5606c6c47ba0 , abstime=0x7ff7a0144c60, src_file=0x5606c5fc3538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005606c52d761b in cache_thread (thd=0x7ff728000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005606c52d78c9 in one_thread_per_connection_end (thd=0x7ff728000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005606c55740f3 in do_handle_one_connection (connect=0x5606c83b8040) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005606c5573dad in handle_one_connection (arg=0x5606c83b8040) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005606c5e9845f in pfs_spawn_thread (arg=0x5606c8779eb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff766ffd700 (LWP 229574)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a480ff in os_event::wait (this=0x5606c83f12a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005606c5a47b85 in os_event::wait_low (this=0x5606c83f12a0, reset_sig_count=2296) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005606c5a47ef6 in os_event_wait_low (event=0x5606c83f12a0, reset_sig_count=2296) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005606c5b33df3 in srv_resume_thread (slot=0x5606c6a82748 , sig_count=2296, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005606c5b37e04 in srv_purge_coordinator_suspend (slot=0x5606c6a82748 , rseg_history_len=9) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005606c5b3809a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff7677fe700 (LWP 229573)): #0 0x00007ff7b92d5e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005606c5a48401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005606c5b3735b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005606c5b3748b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff764ff9700 (LWP 229578)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a480ff in os_event::wait (this=0x5606c83f15a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005606c5a47b85 in os_event::wait_low (this=0x5606c83f15a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005606c5a47ef6 in os_event_wait_low (event=0x5606c83f15a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005606c5c13eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff7b0071700 (LWP 229622)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5f05997 in safe_cond_wait (cond=0x5606c6c451e0 , mp=0x5606c6c48560 , file=0x5606c5fd0cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005606c53034ee in inline_mysql_cond_wait (that=0x5606c6c451e0 , mutex=0x5606c6c48560 , src_file=0x5606c5fd1830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005606c5304d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005606c5e9845f in pfs_spawn_thread (arg=0x5606c8765ad0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff767fff700 (LWP 229572)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a47a1a in os_event::timed_wait (this=0x5606c8599790, abstime=0x7ff767ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005606c5a47d31 in os_event::wait_time_low (this=0x5606c8599790, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005606c5a47ed1 in os_event_wait_time_low (event=0x5606c8599790, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005606c5b962fb in ib_wqueue_timedwait (wq=0x5606c8599660, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005606c5cd6ea4 in fts_optimize_thread (arg=0x5606c8599660) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff7755f5700 (LWP 229569)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a47a1a in os_event::timed_wait (this=0x5606c83f14a0, abstime=0x7ff7755f4ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005606c5a47d31 in os_event::wait_time_low (this=0x5606c83f14a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005606c5a47ed1 in os_event_wait_time_low (event=0x5606c83f14a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005606c5b35ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff7b00bc700 (LWP 229621)): #0 0x00007ff7b75c640a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005606c52d1f44 in my_sigwait (set=0x7ff7b00bbcf0, sig=0x7ff7b00bbcb4, code=0x7ff7b00bbcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005606c52d8584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005606c5e9845f in pfs_spawn_thread (arg=0x5606c83b3c40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff774df4700 (LWP 229570)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a47a1a in os_event::timed_wait (this=0x5606c83f1520, abstime=0x7ff774df3d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005606c5a47d31 in os_event::wait_time_low (this=0x5606c83f1520, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005606c5a47ed1 in os_event_wait_time_low (event=0x5606c83f1520, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005606c5b35ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff776ffd700 (LWP 229524)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a47a1a in os_event::timed_wait (this=0x5606c83f1620, abstime=0x7ff776ffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005606c5a47d31 in os_event::wait_time_low (this=0x5606c83f1620, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005606c5a47ed1 in os_event_wait_time_low (event=0x5606c83f1620, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005606c5c1dd1d in pc_sleep_if_needed (next_loop_time=1549881213197, sig_count=2, cur_time=1549881212197) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x00005606c5c1ec56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff747fff700 (LWP 229579)): #0 0x00007ff7b92d5e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005606c5a48401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005606c5beea3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff775df6700 (LWP 229568)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a47a1a in os_event::timed_wait (this=0x5606c85c99d0, abstime=0x7ff775df5d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005606c5a47d31 in os_event::wait_time_low (this=0x5606c85c99d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005606c5a47ed1 in os_event_wait_time_low (event=0x5606c85c99d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005606c5a1488e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff780805700 (LWP 229521)): #0 0x00007ff7b8e9f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005606c5a3b41c in LinuxAIOHandler::collect (this=0x7ff780804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005606c5a3b945 in LinuxAIOHandler::poll (this=0x7ff780804b00, m1=0x7ff780804ba8, m2=0x7ff780804bb0, request=0x7ff780804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005606c5a3baa8 in os_aio_linux_handler (global_segment=3, m1=0x7ff780804ba8, m2=0x7ff780804bb0, request=0x7ff780804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005606c5a405cc in os_aio_handler (segment=3, m1=0x7ff780804ba8, m2=0x7ff780804bb0, request=0x7ff780804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005606c5c934a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005606c5b39909 in io_handler_thread (arg=0x5606c6cc0638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff7765f7700 (LWP 229571)): #0 0x00007ff7b92d2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5a47a1a in os_event::timed_wait (this=0x5606c85fba90, abstime=0x7ff7765f6d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005606c5a47d31 in os_event::wait_time_low (this=0x5606c85fba90, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005606c5a47ed1 in os_event_wait_time_low (event=0x5606c85fba90, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005606c5c7d142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff7777fe700 (LWP 229523)): #0 0x00007ff7b8e9f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005606c5a3b41c in LinuxAIOHandler::collect (this=0x7ff7777fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005606c5a3b945 in LinuxAIOHandler::poll (this=0x7ff7777fdb00, m1=0x7ff7777fdba8, m2=0x7ff7777fdbb0, request=0x7ff7777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005606c5a3baa8 in os_aio_linux_handler (global_segment=5, m1=0x7ff7777fdba8, m2=0x7ff7777fdbb0, request=0x7ff7777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005606c5a405cc in os_aio_handler (segment=5, m1=0x7ff7777fdba8, m2=0x7ff7777fdbb0, request=0x7ff7777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005606c5c934a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005606c5b39909 in io_handler_thread (arg=0x5606c6cc0648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff777fff700 (LWP 229522)): #0 0x00007ff7b8e9f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005606c5a3b41c in LinuxAIOHandler::collect (this=0x7ff777ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005606c5a3b945 in LinuxAIOHandler::poll (this=0x7ff777ffeb00, m1=0x7ff777ffeba8, m2=0x7ff777ffebb0, request=0x7ff777ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005606c5a3baa8 in os_aio_linux_handler (global_segment=4, m1=0x7ff777ffeba8, m2=0x7ff777ffebb0, request=0x7ff777ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005606c5a405cc in os_aio_handler (segment=4, m1=0x7ff777ffeba8, m2=0x7ff777ffebb0, request=0x7ff777ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005606c5c934a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005606c5b39909 in io_handler_thread (arg=0x5606c6cc0640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff782008700 (LWP 229518)): #0 0x00007ff7b8e9f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005606c5a3b41c in LinuxAIOHandler::collect (this=0x7ff782007b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005606c5a3b945 in LinuxAIOHandler::poll (this=0x7ff782007b00, m1=0x7ff782007ba8, m2=0x7ff782007bb0, request=0x7ff782007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005606c5a3baa8 in os_aio_linux_handler (global_segment=0, m1=0x7ff782007ba8, m2=0x7ff782007bb0, request=0x7ff782007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005606c5a405cc in os_aio_handler (segment=0, m1=0x7ff782007ba8, m2=0x7ff782007bb0, request=0x7ff782007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005606c5c934a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005606c5b39909 in io_handler_thread (arg=0x5606c6cc0620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff781807700 (LWP 229519)): #0 0x00007ff7b8e9f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005606c5a3b41c in LinuxAIOHandler::collect (this=0x7ff781806b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005606c5a3b945 in LinuxAIOHandler::poll (this=0x7ff781806b00, m1=0x7ff781806ba8, m2=0x7ff781806bb0, request=0x7ff781806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005606c5a3baa8 in os_aio_linux_handler (global_segment=1, m1=0x7ff781806ba8, m2=0x7ff781806bb0, request=0x7ff781806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005606c5a405cc in os_aio_handler (segment=1, m1=0x7ff781806ba8, m2=0x7ff781806bb0, request=0x7ff781806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005606c5c934a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005606c5b39909 in io_handler_thread (arg=0x5606c6cc0628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff781006700 (LWP 229520)): #0 0x00007ff7b8e9f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005606c5a3b41c in LinuxAIOHandler::collect (this=0x7ff781005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005606c5a3b945 in LinuxAIOHandler::poll (this=0x7ff781005b00, m1=0x7ff781005ba8, m2=0x7ff781005bb0, request=0x7ff781005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005606c5a3baa8 in os_aio_linux_handler (global_segment=2, m1=0x7ff781005ba8, m2=0x7ff781005bb0, request=0x7ff781005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005606c5a405cc in os_aio_handler (segment=2, m1=0x7ff781005ba8, m2=0x7ff781005bb0, request=0x7ff781005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005606c5c934a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005606c5b39909 in io_handler_thread (arg=0x5606c6cc0630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff7a1b7a700 (LWP 229130)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff7b11b7867 in gu::Lock::wait (cond=..., this=0x7ff7a1b79850) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff7b11ba3ab in RecvBuf::front (this=this@entry=0x5606c82e0818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007ff7b11b7f46 in gcomm_recv (backend=, msg=0x5606c82ae720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007ff7b11abeec in core_msg_recv (backend=backend@entry=0x5606c82ae868, recv_msg=recv_msg@entry=0x5606c82ae720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007ff7b11af164 in gcs_core_recv (conn=0x5606c82ae6c0, recv_act=recv_act@entry=0x7ff7a1b79d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007ff7b11b63d8 in gcs_recv_thread (arg=0x5606c82ae440) at gcs/src/gcs.cpp:1398 #7 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff7b037a700 (LWP 229131)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5f05997 in safe_cond_wait (cond=0x7ff78c015608, mp=0x7ff78c015558, file=0x5606c606fb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005606c565d088 in inline_mysql_cond_wait (that=0x7ff78c015608, mutex=0x7ff78c015558, src_file=0x5606c606fe80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005606c565fd20 in Wsrep_thd_queue::pop_front (this=0x7ff78c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005606c565e740 in wsrep_rollback_process (rollbacker=0x7ff78c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005606c564f282 in start_wsrep_THD (arg=0x5606c8302b90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff7b0b7b700 (LWP 229035)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff7b11dcfd7 in gu::Lock::wait (cond=..., this=0x7ff7b0b7abf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff7b11dd069 in galera::ServiceThd::thd_func (arg=0x5606c82a6908) at galera/src/galera_service_thd.cpp:30 #3 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff7b032f700 (LWP 229132)): #0 0x00007ff7b92d2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005606c5f05997 in safe_cond_wait (cond=0x7ff794017b68, mp=0x7ff794017ab8, file=0x5606c606fb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005606c565d088 in inline_mysql_cond_wait (that=0x7ff794017b68, mutex=0x7ff794017ab8, src_file=0x5606c606fe80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005606c565fd20 in Wsrep_thd_queue::pop_front (this=0x7ff794017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005606c565eb00 in wsrep_post_rollback_process (post_rollbacker=0x7ff794001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005606c564f282 in start_wsrep_THD (arg=0x5606c8302460) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ff7b92cedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff7b768cead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff7b96df880 (LWP 228908)): #0 0x00007ff7b92d39d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005606c5f00368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005606c573e503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff7b768220d in poll () from /lib64/libc.so.6 #5 0x00005606c52dea8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005606c52ddd25 in mysqld_main (argc=162, argv=0x5606c8127d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005606c52d1ec5 in main (argc=23, argv=0x7fffac887508) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_insert_select-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_insert_select-innodb/' galera_sr.galera-features#56 'innodb' w4 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 12:34:45 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera-features#56.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); CREATE TABLE t1 (f1 INTEGER AUTO_INCREMENT PRIMARY KEY, f2 INTEGER) Engine=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; connection node_2; SET GLOBAL wsrep_slave_threads = 4; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1a; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_2; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) - found 'core.228350' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera-features#56-innodb/mysqld.1/data/core.228350 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 228350] [New LWP 228383] [New LWP 228391] [New LWP 228400] [New LWP 228436] [New LWP 228439] [New LWP 228440] [New LWP 228446] [New LWP 228453] [New LWP 228455] [New LWP 228451] [New LWP 228398] [New LWP 228438] [New LWP 228441] [New LWP 228442] [New LWP 228448] [New LWP 231267] [New LWP 231278] [New LWP 228390] [New LWP 228396] [New LWP 228449] [New LWP 228452] [New LWP 228454] [New LWP 228457] [New LWP 228458] [New LWP 228604] [New LWP 228459] [New LWP 228456] [New LWP 228437] [New LWP 228445] [New LWP 228447] [New LWP 228450] [New LWP 228443] [New LWP 231297] [New LWP 228401] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f44dfe7b9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f44dfe7b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005623b77ff368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005623b703d503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f44de22a20d in poll () from /lib64/libc.so.6 #5 0x00005623b6bdda8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005623b6bdcd25 in mysqld_main (argc=161, argv=0x5623b99dad20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005623b6bd0ec5 in main (argc=23, argv=0x7ffead100a98) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f44cf7ee700 (LWP 228401)): #0 0x00007f44dfe7dd1d in fsync () from /lib64/libpthread.so.0 #1 0x00005623b733bacb in os_file_fsync_posix (file=84) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005623b733be21 in os_file_flush_func (file=84) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005623b75843a2 in pfs_os_file_flush_func (file=..., src_file=0x5623b7bfea70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=935) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005623b7587560 in fil_space_extend_must_retry (space=0x7f4474081b90, node=0x7f447402e9c0, size=15, success=0x7f44cf7e97ff) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:935 #5 0x00005623b7587cf0 in fil_space_extend (space=0x7f4474081b90, size=15) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:1103 #6 0x00005623b75a0ac9 in fsp_try_extend_data_file_with_pages (space=0x7f4474081b90, page_no=14, header=0x7f44cdb90026 "", mtr=0x7f44cf7e9ee0) at /home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc:794 #7 0x00005623b75a5f10 in fsp_reserve_free_pages (space=0x7f4474081b90, space_header=0x7f44cdb90026 "", size=14, mtr=0x7f44cf7e9ee0, n_pages=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc:2578 #8 0x00005623b75a60d9 in fsp_reserve_free_extents (n_reserved=0x7f44cf7e9990, space=0x7f4474081b90, n_ext=3, alloc_type=FSP_NORMAL, mtr=0x7f44cf7e9ee0, n_pages=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc:2652 #9 0x00005623b74c749a in btr_cur_pessimistic_insert (flags=0, cursor=0x7f44cf7e9ac0, offsets=0x7f44cf7e9a58, heap=0x7f44cf7e9a50, entry=0x7f44b8068a98, rec=0x7f44cf7e9a60, big_rec=0x7f44cf7e9a48, n_ext=0, thr=0x7f44b8066d50, mtr=0x7f44cf7e9ee0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3689 #10 0x00005623b73b0ad3 in row_ins_clust_index_entry_low (flags=0, mode=33, index=0x7f4474030498, n_uniq=1, entry=0x7f44b8068a98, n_ext=0, thr=0x7f44b8066d50, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:2777 #11 0x00005623b73b249e in row_ins_clust_index_entry (index=0x7f4474030498, entry=0x7f44b8068a98, thr=0x7f44b8066d50, n_ext=0, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3297 #12 0x00005623b73b28aa in row_ins_index_entry (index=0x7f4474030498, entry=0x7f44b8068a98, thr=0x7f44b8066d50) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3405 #13 0x00005623b73b2fe9 in row_ins_index_entry_step (node=0x7f44b8066ad8, thr=0x7f44b8066d50) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3574 #14 0x00005623b73b341a in row_ins (node=0x7f44b8066ad8, thr=0x7f44b8066d50) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3717 #15 0x00005623b73b3c19 in row_ins_step (thr=0x7f44b8066d50) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3962 #16 0x00005623b73d3786 in row_insert_for_mysql (mysql_rec=0x7f44b8065980 "\375n\020", prebuilt=0x7f44b8066588, ins_mode=ROW_INS_NORMAL) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:1460 #17 0x00005623b7271358 in ha_innobase::write_row (this=0x7f44b8065dd8, record=0x7f44b8065980 "\375n\020") at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:8050 #18 0x00005623b70507cb in handler::ha_write_row (this=0x7f44b8065dd8, buf=0x7f44b8065980 "\375n\020") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6463 #19 0x00005623b71949d3 in Rows_log_event::write_row (this=0x7f44b80a8ee0, rgi=0x7f44b804ac00, overwrite=false) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:13208 #20 0x00005623b7194cf6 in Write_rows_log_event::do_exec_row (this=0x7f44b80a8ee0, rgi=0x7f44b804ac00) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:13418 #21 0x00005623b718fdd4 in Rows_log_event::do_apply_event (this=0x7f44b80a8ee0, rgi=0x7f44b804ac00) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:11541 #22 0x00005623b6c1b789 in Log_event::apply_event (this=0x7f44b80a8ee0, rgi=0x7f44b804ac00) at /home/stepan/galera/git/10.4/server/sql/log_event.h:1481 #23 0x00005623b6f5b96d in wsrep_apply_events (thd=0x7f44b80556f0, rli=0x7f44b805b7d0, events_buf=0x7f44d2e8be20, buf_len=0) at /home/stepan/galera/git/10.4/server/sql/wsrep_applier.cc:206 #24 0x00005623b6f3c691 in Wsrep_applier_service::apply_write_set (this=0x7f44b8036480, ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:492 #25 0x00005623b78a19e5 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:62 #26 0x00005623b78a24b1 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:241 #27 0x00005623b78a5807 in wsrep::server_state::on_apply (this=0x5623b9b236a0, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:944 #28 0x00005623b78baed1 in wsrep::high_priority_service::apply (this=0x7f44cf7ed4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #29 0x00005623b78b85b8 in (anonymous namespace)::apply_cb (ctx=0x7f44cf7ed4d0, wsh=0x7f44cf7ec3e0, flags=0, buf=0x7f44cf7ec420, meta=0x7f44cf7ec6b0, exit_loop=0x7f44cf7ec5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #30 0x00007f44d3c9f3b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7f444407b190, recv_ctx=recv_ctx@entry=0x7f44cf7ed4d0, apply_cb=0x5623b78b8395 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f44cf7ec5ef: false) at galera/src/trx_handle.cpp:418 #31 0x00007f44d3cd1d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5623b9b59280, recv_ctx=recv_ctx@entry=0x7f44cf7ed4d0, ts=...) at galera/src/replicator_smm.cpp:489 #32 0x00007f44d3cd7a40 in galera::ReplicatorSMM::process_trx (this=0x5623b9b59280, recv_ctx=0x7f44cf7ed4d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #33 0x00007f44d3cb8c6b in galera::GcsActionSource::process_writeset (this=this@entry=0x5623b9b60f30, recv_ctx=0x7f44cf7ed4d0, act=..., exit_loop=@0x7f44cf7ed29e: false) at galera/src/gcs_action_source.cpp:62 #34 0x00007f44d3cb8e45 in galera::GcsActionSource::dispatch (this=this@entry=0x5623b9b60f30, recv_ctx=recv_ctx@entry=0x7f44cf7ed4d0, act=..., exit_loop=@0x7f44cf7ed29e: false) at galera/src/gcs_action_source.cpp:109 #35 0x00007f44d3cb921d in galera::GcsActionSource::process (this=0x5623b9b60f30, recv_ctx=0x7f44cf7ed4d0, exit_loop=@0x7f44cf7ed29e: false) at galera/src/gcs_action_source.cpp:182 #36 0x00007f44d3cd8033 in galera::ReplicatorSMM::async_recv (this=0x5623b9b59280, recv_ctx=0x7f44cf7ed4d0) at galera/src/replicator_smm.cpp:383 #37 0x00005623b78b9276 in wsrep::wsrep_provider_v26::run_applier (this=0x5623b9ae0360, applier_ctx=0x7f44cf7ed4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #38 0x00005623b6f5c636 in wsrep_replication_process (thd=0x7f44b8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #39 0x00005623b6f4e282 in start_wsrep_THD (arg=0x5623b9b93810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #40 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #41 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f44a4348700 (LWP 231297)): #0 0x00007f44dfe7dd1d in fsync () from /lib64/libpthread.so.0 #1 0x00005623b733bacb in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005623b733be21 in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005623b75843a2 in pfs_os_file_flush_func (file=..., src_file=0x5623b7bfea70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005623b758711d in fil_flush_low (space=0x5623b9eab7f0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x00005623b75929f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x00005623b73171cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005623b7317c31 in log_write_up_to (lsn=5618350, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005623b7482d91 in trx_flush_log_if_needed_low (lsn=5618350) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005623b7482dda in trx_flush_log_if_needed (lsn=5618350, trx=0x7f44ce03d4b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005623b7484c03 in trx_commit_complete_for_mysql (trx=0x7f44ce03d4b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005623b7269514 in innobase_commit (hton=0x5623b9c75b10, thd=0x7f447400b680, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x00005623b7043039 in commit_one_phase_2 (thd=0x7f447400b680, all=true, trans=0x7f447400ebd0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x00005623b7042f3b in ha_commit_one_phase (thd=0x7f447400b680, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x00005623b70422b7 in ha_commit_trans (thd=0x7f447400b680, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x00005623b6e87e8e in trans_commit (thd=0x7f447400b680) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005623b724718c in Wsrep_storage_service::commit (this=0x7f4474075f00, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x00005623b78af4d3 in wsrep::transaction::certify_fragment (this=0x7f447402bd10, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x00005623b78aec37 in wsrep::transaction::streaming_step (this=0x7f447402bd10, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x00005623b78abff2 in wsrep::transaction::after_row (this=0x7f447402bd10) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x00005623b70553a8 in wsrep::client_state::after_row (this=0x7f447402bca0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x00005623b703e625 in wsrep_after_row (thd=0x7f4474025e40) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x00005623b705052a in wsrep_after_row (thd=0x7f4474025e40) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x00005623b705091b in handler::ha_write_row (this=0x7f447402f1d8, buf=0x7f4474081440 "\375\061\022") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x00005623b6cc6443 in write_record (thd=0x7f4474025e40, table=0x7f447407e5a0, info=0x7f447403bc00) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x00005623b6ccb742 in select_insert::send_data (this=0x7f447403bbb8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x00005623b6d83e07 in end_send (join=0x7f447403bc68, join_tab=0x7f4474086b88, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x00005623b6ec0fa7 in JOIN_CACHE::generate_full_extensions (this=0x7f4474086f38, rec_ptr=0x7f447410d6fc "\240") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x00005623b6ec0cc2 in JOIN_CACHE::join_matching_records (this=0x7f4474086f38, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x00005623b6ec068e in JOIN_CACHE::join_records (this=0x7f4474086f38, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x00005623b6ec0828 in JOIN_CACHE::join_records (this=0x7f447403ea08, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x00005623b6ec0828 in JOIN_CACHE::join_records (this=0x7f447403e8d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x00005623b6d80abb in sub_select_cache (join=0x7f447403bc68, join_tab=0x7f4474086078, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x00005623b6d80cc8 in sub_select (join=0x7f447403bc68, join_tab=0x7f4474085cc8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x00005623b6d80493 in do_select (join=0x7f447403bc68, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x00005623b6d58fb1 in JOIN::exec_inner (this=0x7f447403bc68) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x00005623b6d583fa in JOIN::exec (this=0x7f447403bc68) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x00005623b6d596ec in mysql_select (thd=0x7f4474025e40, tables=0x7f4474039990, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f447403bbb8, unit=0x7f4474029d00, select_lex=0x7f4474039470) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x00005623b6d4afed in handle_select (thd=0x7f4474025e40, lex=0x7f4474029c38, result=0x7f447403bbb8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x00005623b6d0d3fc in mysql_execute_command (thd=0x7f4474025e40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #40 0x00005623b6d18935 in mysql_parse (thd=0x7f4474025e40, rawbuf=0x7f4474038be8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f44a43470d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #41 0x00005623b6d17f68 in wsrep_mysql_parse (thd=0x7f4474025e40, rawbuf=0x7f4474038be8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f44a43470d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #42 0x00005623b6d03e25 in dispatch_command (command=COM_QUERY, thd=0x7f4474025e40, packet=0x7f44740150d1 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #43 0x00005623b6d0280d in do_command (thd=0x7f4474025e40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x00005623b6e73069 in do_handle_one_connection (connect=0x5623ba02eab0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #45 0x00005623b6e72dad in handle_one_connection (arg=0x5623ba02eab0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #46 0x00005623b779745f in pfs_spawn_thread (arg=0x5623b9ff1940) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f44adffb700 (LWP 228443)): #0 0x00007f44dfe7dd1d in fsync () from /lib64/libpthread.so.0 #1 0x00005623b733bacb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005623b733be21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005623b75843a2 in pfs_os_file_flush_func (file=..., src_file=0x5623b7bfea70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005623b758711d in fil_flush_low (space=0x5623b9eab450, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x00005623b75929f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x00005623b750fcbd in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x00005623b751a3a5 in buf_flush_end (buf_pool=0x5623b9d87140, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x00005623b751a563 in buf_flush_do_batch (buf_pool=0x5623b9d87140, type=BUF_FLUSH_LIST, min_n=4, lsn_limit=18446744073709551615, n=0x7f44adffab70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x00005623b751d24d in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x00005623b751e19b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f44a67fc700 (LWP 228450)): #0 0x00007f44dfe7de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005623b7347401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005623b743635b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005623b743648b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f44a77fe700 (LWP 228447)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7346a1a in os_event::timed_wait (this=0x5623b9ca18c0, abstime=0x7f44a77fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005623b7346d31 in os_event::wait_time_low (this=0x5623b9ca18c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005623b7346ed1 in os_event_wait_time_low (event=0x5623b9ca18c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005623b7434ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f44acff9700 (LWP 228445)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7346a1a in os_event::timed_wait (this=0x5623b9e79d70, abstime=0x7f44acff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005623b7346d31 in os_event::wait_time_low (this=0x5623b9e79d70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005623b7346ed1 in os_event_wait_time_low (event=0x5623b9e79d70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005623b731388e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f44cd3e8700 (LWP 228437)): #0 0x00007f44dfa47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005623b733a41c in LinuxAIOHandler::collect (this=0x7f44cd3e7b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005623b733a945 in LinuxAIOHandler::poll (this=0x7f44cd3e7b00, m1=0x7f44cd3e7ba8, m2=0x7f44cd3e7bb0, request=0x7f44cd3e7be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005623b733aaa8 in os_aio_linux_handler (global_segment=0, m1=0x7f44cd3e7ba8, m2=0x7f44cd3e7bb0, request=0x7f44cd3e7be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005623b733f5cc in os_aio_handler (segment=0, m1=0x7f44cd3e7ba8, m2=0x7f44cd3e7bb0, request=0x7f44cd3e7be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005623b75924a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005623b7438909 in io_handler_thread (arg=0x5623b85bf620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f448affd700 (LWP 228456)): #0 0x00007f44dfe7de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005623b7347401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005623b74eda3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f44cd5e8700 (LWP 228459)): #0 0x00007f44de16e40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005623b6bd0f44 in my_sigwait (set=0x7f44cd5e7cf0, sig=0x7f44cd5e7cb4, code=0x7f44cd5e7cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005623b6bd7584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005623b779745f in pfs_spawn_thread (arg=0x5623b9c64f10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f44cd59d700 (LWP 228604)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7804997 in safe_cond_wait (cond=0x5623b85441e0 , mp=0x5623b8547560 , file=0x5623b78cfcc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005623b6c024ee in inline_mysql_cond_wait (that=0x5623b85441e0 , mutex=0x5623b8547560 , src_file=0x5623b78d0830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005623b6c03d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005623b779745f in pfs_spawn_thread (arg=0x5623b9c65400) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4489ffb700 (LWP 228458)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7804997 in safe_cond_wait (cond=0x7f4489ffac70, mp=0x7f4489ffacb0, file=0x5623b7ad2c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005623b725e487 in inline_mysql_cond_wait (that=0x7f4489ffac70, mutex=0x7f4489ffacb0, src_file=0x5623b7ad45d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005623b72630c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005623b779745f in pfs_spawn_thread (arg=0x5623b9ee2940) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f448a7fc700 (LWP 228457)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b73470ff in os_event::wait (this=0x5623b9ca1a40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005623b7346b85 in os_event::wait_low (this=0x5623b9ca1a40, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005623b7346ef6 in os_event_wait_low (event=0x5623b9ca1a40, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005623b74fbb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f448bfff700 (LWP 228454)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b73470ff in os_event::wait (this=0x5623b9db3260) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005623b7346b85 in os_event::wait_low (this=0x5623b9db3260, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005623b7346ef6 in os_event_wait_low (event=0x5623b9db3260, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005623b7440bfc in sync_array_wait_event (arr=0x5623b9c789b0, cell=@0x7f448bffc848: 0x5623b9c79960) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005623b74441c8 in rw_lock_x_lock_func (lock=0x7f44cd80a428, pass=0, file_name=0x5623b7bb5f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:735 #6 0x00005623b74ef589 in pfs_rw_lock_x_lock_func (lock=0x7f44cd80a428, pass=0, file_name=0x5623b7bb5f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:558 #7 0x00005623b75012f9 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x5623b7bb5f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333, mtr=0x7f448bffe1a0, err=0x7f448bffcb74) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4824 #8 0x00005623b74bb253 in btr_block_get_func (page_id=..., zip_size=0, mode=2, file=0x5623b7bb5f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333, index=0x5623ba03d248, mtr=0x7f448bffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0btr.ic:57 #9 0x00005623b74bd51c in btr_cur_latch_leaves (block=0x7f44cd8179e0, page_id=..., zip_size=0, latch_mode=33, cursor=0x5623b9ec3178, mtr=0x7f448bffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:333 #10 0x00005623b74c1927 in btr_cur_search_to_nth_level_func (index=0x5623ba03d248, level=0, tuple=0x7f4484081ef8, mode=PAGE_CUR_LE, latch_mode=33, cursor=0x5623b9ec3178, ahi_latch=0x0, file=0x5623b7b66d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7f448bffe1a0, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1792 #11 0x00005623b74d9036 in btr_pcur_open_with_no_init_func (index=0x5623ba03d248, tuple=0x7f4484081ef8, mode=PAGE_CUR_LE, latch_mode=65569, cursor=0x5623b9ec3178, ahi_latch=0x0, file=0x5623b7b66d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7f448bffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:527 #12 0x00005623b74da33f in btr_pcur_restore_position_func (latch_mode=65569, cursor=0x5623b9ec3178, file=0x5623b7b66d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7f448bffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0pcur.cc:391 #13 0x00005623b73f3be6 in row_purge_reposition_pcur (mode=65569, node=0x5623b9ec30e0, mtr=0x7f448bffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:100 #14 0x00005623b73f3db3 in row_purge_remove_clust_if_poss_low (node=0x5623b9ec30e0, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:140 #15 0x00005623b73f4209 in row_purge_remove_clust_if_poss (node=0x5623b9ec30e0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #16 0x00005623b73f5a36 in row_purge_del_mark (node=0x5623b9ec30e0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #17 0x00005623b73f6fbe in row_purge_record_func (node=0x5623b9ec30e0, undo_rec=0x5623b9ec35b8 "", thr=0x5623b9ec3028, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #18 0x00005623b73f72fb in row_purge (node=0x5623b9ec30e0, undo_rec=0x5623b9ec35b8 "", thr=0x5623b9ec3028) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #19 0x00005623b73f76a6 in row_purge_step (thr=0x5623b9ec3028) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #20 0x00005623b7379a9a in que_thr_step (thr=0x5623b9ec3028) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #21 0x00005623b7379cca in que_run_threads_low (thr=0x5623b9ec3028) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #22 0x00005623b7379eba in que_run_threads (thr=0x5623b9ec3028) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #23 0x00005623b7436855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #24 0x00005623b74369fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #25 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #26 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f44a57fa700 (LWP 228452)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b73470ff in os_event::wait (this=0x5623b9ca16c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005623b7346b85 in os_event::wait_low (this=0x5623b9ca16c0, reset_sig_count=12349) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005623b7346ef6 in os_event_wait_low (event=0x5623b9ca16c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005623b7432df3 in srv_resume_thread (slot=0x5623b8381780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005623b74369f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f44ad7fa700 (LWP 228449)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7346a1a in os_event::timed_wait (this=0x5623b9e41d40, abstime=0x7f44ad7f9b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005623b7346d31 in os_event::wait_time_low (this=0x5623b9e41d40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005623b7346ed1 in os_event_wait_time_low (event=0x5623b9e41d40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005623b74952fb in ib_wqueue_timedwait (wq=0x5623b9e41c10, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005623b75d5ea4 in fts_optimize_thread (arg=0x5623b9e41c10) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f44cffef700 (LWP 228396)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f44d3c90867 in gu::Lock::wait (cond=..., this=0x7f44cffee850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f44d3c933ab in RecvBuf::front (this=this@entry=0x5623b9b998a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f44d3c90f46 in gcomm_recv (backend=, msg=0x5623b9b61830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f44d3c84eec in core_msg_recv (backend=backend@entry=0x5623b9b61978, recv_msg=recv_msg@entry=0x5623b9b61830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f44d3c88164 in gcs_core_recv (conn=0x5623b9b617d0, recv_act=recv_act@entry=0x7f44cffeed70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f44d3c8f3d8 in gcs_recv_thread (arg=0x5623b9b61550) at gcs/src/gcs.cpp:1398 #7 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f44d2c74700 (LWP 228390)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f44d3cb5fd7 in gu::Lock::wait (cond=..., this=0x7f44d2c73bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f44d3cb6069 in galera::ServiceThd::thd_func (arg=0x5623b9b59998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f44ac0f0700 (LWP 231278)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b73470ff in os_event::wait (this=0x7f44740818a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005623b7346b85 in os_event::wait_low (this=0x7f44740818a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005623b7346ef6 in os_event_wait_low (event=0x7f44740818a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005623b7440bfc in sync_array_wait_event (arr=0x5623b9c789b0, cell=@0x7f44ac0eaf40: 0x5623b9c78f20) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005623b744447b in rw_lock_sx_lock_func (lock=0x7f4474030680, pass=0, file_name=0x5623b7bb5f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=1418) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x00005623b72de9a8 in pfs_rw_lock_sx_lock_func (lock=0x7f4474030680, pass=0, file_name=0x5623b7bb5f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=1418) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:680 #7 0x00005623b72f1df1 in mtr_t::sx_lock (this=0x7f44ac0ec660, lock=0x7f4474030680, file=0x5623b7bb5f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=1418) at /home/stepan/galera/git/10.4/server/storage/innobase/include/mtr0mtr.ic:260 #8 0x00005623b74c0675 in btr_cur_search_to_nth_level_func (index=0x7f4474030498, level=0, tuple=0x7f44bc01d378, mode=PAGE_CUR_LE, latch_mode=33, cursor=0x7f44ac0ec240, ahi_latch=0x0, file=0x5623b7b4b508 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, mtr=0x7f44ac0ec660, autoinc=6713) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1418 #9 0x00005623b73a9420 in btr_pcur_open_low (index=0x7f4474030498, level=0, tuple=0x7f44bc01d378, mode=PAGE_CUR_LE, latch_mode=33, cursor=0x7f44ac0ec240, file=0x5623b7b4b508 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, autoinc=6713, mtr=0x7f44ac0ec660) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:464 #10 0x00005623b73b02ca in row_ins_clust_index_entry_low (flags=0, mode=33, index=0x7f4474030498, n_uniq=1, entry=0x7f44bc01d378, n_ext=0, thr=0x7f44bc0380d0, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:2639 #11 0x00005623b73b249e in row_ins_clust_index_entry (index=0x7f4474030498, entry=0x7f44bc01d378, thr=0x7f44bc0380d0, n_ext=0, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3297 #12 0x00005623b73b28aa in row_ins_index_entry (index=0x7f4474030498, entry=0x7f44bc01d378, thr=0x7f44bc0380d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3405 #13 0x00005623b73b2fe9 in row_ins_index_entry_step (node=0x7f44bc037e58, thr=0x7f44bc0380d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3574 #14 0x00005623b73b341a in row_ins (node=0x7f44bc037e58, thr=0x7f44bc0380d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3717 #15 0x00005623b73b3c19 in row_ins_step (thr=0x7f44bc0380d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3962 #16 0x00005623b73d3786 in row_insert_for_mysql (mysql_rec=0x7f44bc007060 "\375\071\032", prebuilt=0x7f44bc037908, ins_mode=ROW_INS_NORMAL) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:1460 #17 0x00005623b7271358 in ha_innobase::write_row (this=0x7f44bc037158, record=0x7f44bc007060 "\375\071\032") at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:8050 #18 0x00005623b7050779 in handler::ha_write_row (this=0x7f44bc037158, buf=0x7f44bc007060 "\375\071\032") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6463 #19 0x00005623b6cc6443 in write_record (thd=0x7f44bc000d70, table=0x7f44bc0090c0, info=0x7f44bc016e00) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #20 0x00005623b6ccb742 in select_insert::send_data (this=0x7f44bc016db8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #21 0x00005623b6d83e07 in end_send (join=0x7f44bc016e68, join_tab=0x7f44bc024468, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #22 0x00005623b6ec0fa7 in JOIN_CACHE::generate_full_extensions (this=0x7f44bc019d38, rec_ptr=0x7f44bc0c247c "\260") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #23 0x00005623b6ec0cc2 in JOIN_CACHE::join_matching_records (this=0x7f44bc019d38, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #24 0x00005623b6ec068e in JOIN_CACHE::join_records (this=0x7f44bc019d38, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #25 0x00005623b6ec0828 in JOIN_CACHE::join_records (this=0x7f44bc019c08, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #26 0x00005623b6ec0828 in JOIN_CACHE::join_records (this=0x7f44bc019ad8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #27 0x00005623b6d80abb in sub_select_cache (join=0x7f44bc016e68, join_tab=0x7f44bc023958, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #28 0x00005623b6d80cc8 in sub_select (join=0x7f44bc016e68, join_tab=0x7f44bc0235a8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #29 0x00005623b6d80493 in do_select (join=0x7f44bc016e68, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #30 0x00005623b6d58fb1 in JOIN::exec_inner (this=0x7f44bc016e68) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #31 0x00005623b6d583fa in JOIN::exec (this=0x7f44bc016e68) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #32 0x00005623b6d596ec in mysql_select (thd=0x7f44bc000d70, tables=0x7f44bc014c10, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f44bc016db8, unit=0x7f44bc004c30, select_lex=0x7f44bc0146f0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #33 0x00005623b6d4afed in handle_select (thd=0x7f44bc000d70, lex=0x7f44bc004b68, result=0x7f44bc016db8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #34 0x00005623b6d0d3fc in mysql_execute_command (thd=0x7f44bc000d70) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #35 0x00005623b6d18935 in mysql_parse (thd=0x7f44bc000d70, rawbuf=0x7f44bc013e68 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f44ac0ef0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #36 0x00005623b6d17f68 in wsrep_mysql_parse (thd=0x7f44bc000d70, rawbuf=0x7f44bc013e68 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f44ac0ef0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #37 0x00005623b6d03e25 in dispatch_command (command=COM_QUERY, thd=0x7f44bc000d70, packet=0x7f44bc00a441 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #38 0x00005623b6d0280d in do_command (thd=0x7f44bc000d70) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #39 0x00005623b6e73069 in do_handle_one_connection (connect=0x5623b9fcd300) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #40 0x00005623b6e72dad in handle_one_connection (arg=0x5623b9fcd300) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #41 0x00005623b779745f in pfs_spawn_thread (arg=0x5623b9ee49a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #42 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #43 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f44ac429700 (LWP 231267)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7804c99 in safe_cond_timedwait (cond=0x5623b85441a0 , mp=0x5623b8546ba0 , abstime=0x7f44ac428c60, file=0x5623b78c04c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005623b6bd1845 in inline_mysql_cond_timedwait (that=0x5623b85441a0 , mutex=0x5623b8546ba0 , abstime=0x7f44ac428c60, src_file=0x5623b78c2538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005623b6bd661b in cache_thread (thd=0x7f447c000d90) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005623b6bd68c9 in one_thread_per_connection_end (thd=0x7f447c000d90, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005623b6e730f3 in do_handle_one_connection (connect=0x5623b9c65b70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005623b6e72dad in handle_one_connection (arg=0x5623b9c65b70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005623b779745f in pfs_spawn_thread (arg=0x5623ba02c8b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f44a6ffd700 (LWP 228448)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7346a1a in os_event::timed_wait (this=0x5623b9eabe30, abstime=0x7f44a6ffcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005623b7346d31 in os_event::wait_time_low (this=0x5623b9eabe30, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005623b7346ed1 in os_event_wait_time_low (event=0x5623b9eabe30, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005623b757c142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f44ae7fc700 (LWP 228442)): #0 0x00007f44dfa47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005623b733a41c in LinuxAIOHandler::collect (this=0x7f44ae7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005623b733a945 in LinuxAIOHandler::poll (this=0x7f44ae7fbb00, m1=0x7f44ae7fbba8, m2=0x7f44ae7fbbb0, request=0x7f44ae7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005623b733aaa8 in os_aio_linux_handler (global_segment=5, m1=0x7f44ae7fbba8, m2=0x7f44ae7fbbb0, request=0x7f44ae7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005623b733f5cc in os_aio_handler (segment=5, m1=0x7f44ae7fbba8, m2=0x7f44ae7fbbb0, request=0x7f44ae7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005623b75924a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005623b7438909 in io_handler_thread (arg=0x5623b85bf648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f44aeffd700 (LWP 228441)): #0 0x00007f44dfa47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005623b733a41c in LinuxAIOHandler::collect (this=0x7f44aeffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005623b733a945 in LinuxAIOHandler::poll (this=0x7f44aeffcb00, m1=0x7f44aeffcba8, m2=0x7f44aeffcbb0, request=0x7f44aeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005623b733aaa8 in os_aio_linux_handler (global_segment=4, m1=0x7f44aeffcba8, m2=0x7f44aeffcbb0, request=0x7f44aeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005623b733f5cc in os_aio_handler (segment=4, m1=0x7f44aeffcba8, m2=0x7f44aeffcbb0, request=0x7f44aeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005623b75924a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005623b7438909 in io_handler_thread (arg=0x5623b85bf640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f44ccbe7700 (LWP 228438)): #0 0x00007f44dfa47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005623b733a41c in LinuxAIOHandler::collect (this=0x7f44ccbe6b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005623b733a945 in LinuxAIOHandler::poll (this=0x7f44ccbe6b00, m1=0x7f44ccbe6ba8, m2=0x7f44ccbe6bb0, request=0x7f44ccbe6be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005623b733aaa8 in os_aio_linux_handler (global_segment=1, m1=0x7f44ccbe6ba8, m2=0x7f44ccbe6bb0, request=0x7f44ccbe6be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005623b733f5cc in os_aio_handler (segment=1, m1=0x7f44ccbe6ba8, m2=0x7f44ccbe6bb0, request=0x7f44ccbe6be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005623b75924a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005623b7438909 in io_handler_thread (arg=0x5623b85bf628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f44d80ad700 (LWP 228398)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7804997 in safe_cond_wait (cond=0x7f44c4017b68, mp=0x7f44c4017ab8, file=0x5623b796eb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005623b6f5c088 in inline_mysql_cond_wait (that=0x7f44c4017b68, mutex=0x7f44c4017ab8, src_file=0x5623b796ee80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005623b6f5ed20 in Wsrep_thd_queue::pop_front (this=0x7f44c4017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005623b6f5d740 in wsrep_rollback_process (rollbacker=0x7f44c4001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005623b6f4e282 in start_wsrep_THD (arg=0x5623b9ba9320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f44a5ffb700 (LWP 228451)): #0 0x00007f44de219d47 in sched_yield () from /lib64/libc.so.6 #1 0x00005623b7347393 in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x00005623b7460bd9 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x00005623b7460de1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x00005623b7436cca in srv_do_purge (n_total_purged=0x7f44a5ffad98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x00005623b74370f7 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f448b7fe700 (LWP 228455)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b73470ff in os_event::wait (this=0x5623b9ca1940) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005623b7346b85 in os_event::wait_low (this=0x5623b9ca1940, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005623b7346ef6 in os_event_wait_low (event=0x5623b9ca1940, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005623b7512eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f44a4ff9700 (LWP 228453)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b73470ff in os_event::wait (this=0x5623b9ca1740) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005623b7346b85 in os_event::wait_low (this=0x5623b9ca1740, reset_sig_count=11514) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005623b7346ef6 in os_event_wait_low (event=0x5623b9ca1740, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005623b7432df3 in srv_resume_thread (slot=0x5623b83817b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005623b74369f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f44a7fff700 (LWP 228446)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7346a1a in os_event::timed_wait (this=0x5623b9ca1840, abstime=0x7f44a7ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005623b7346d31 in os_event::wait_time_low (this=0x5623b9ca1840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005623b7346ed1 in os_event_wait_time_low (event=0x5623b9ca1840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005623b7434ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f44af7fe700 (LWP 228440)): #0 0x00007f44dfa47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005623b733a41c in LinuxAIOHandler::collect (this=0x7f44af7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005623b733a945 in LinuxAIOHandler::poll (this=0x7f44af7fdb00, m1=0x7f44af7fdba8, m2=0x7f44af7fdbb0, request=0x7f44af7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005623b733aaa8 in os_aio_linux_handler (global_segment=3, m1=0x7f44af7fdba8, m2=0x7f44af7fdbb0, request=0x7f44af7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005623b733f5cc in os_aio_handler (segment=3, m1=0x7f44af7fdba8, m2=0x7f44af7fdbb0, request=0x7f44af7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005623b75924a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005623b7438909 in io_handler_thread (arg=0x5623b85bf638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f44affff700 (LWP 228439)): #0 0x00007f44dfa47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005623b733a41c in LinuxAIOHandler::collect (this=0x7f44afffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005623b733a945 in LinuxAIOHandler::poll (this=0x7f44afffeb00, m1=0x7f44afffeba8, m2=0x7f44afffebb0, request=0x7f44afffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005623b733aaa8 in os_aio_linux_handler (global_segment=2, m1=0x7f44afffeba8, m2=0x7f44afffebb0, request=0x7f44afffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005623b733f5cc in os_aio_handler (segment=2, m1=0x7f44afffeba8, m2=0x7f44afffebb0, request=0x7f44afffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005623b75924a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005623b7438909 in io_handler_thread (arg=0x5623b85bf630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f44cec3a700 (LWP 228436)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7804c99 in safe_cond_timedwait (cond=0x5623b8dd0ea0 , mp=0x5623b8dd0de0 , abstime=0x7f44cec39cb0, file=0x5623b7c48610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005623b766eacb in inline_mysql_cond_timedwait (that=0x5623b8dd0ea0 , mutex=0x5623b8dd0de0 , abstime=0x7f44cec39cb0, src_file=0x5623b7c48658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005623b766efef in my_service_thread_sleep (control=0x5623b8385b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005623b7663488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005623b779745f in pfs_spawn_thread (arg=0x5623b9c597e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f44d8062700 (LWP 228400)): #0 0x00007f44dfe7a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7804997 in safe_cond_wait (cond=0x7f44b0015608, mp=0x7f44b0015558, file=0x5623b796eb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005623b6f5c088 in inline_mysql_cond_wait (that=0x7f44b0015608, mutex=0x7f44b0015558, src_file=0x5623b796ee80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005623b6f5ed20 in Wsrep_thd_queue::pop_front (this=0x7f44b0015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005623b6f5db00 in wsrep_post_rollback_process (post_rollbacker=0x7f44b0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005623b6f4e282 in start_wsrep_THD (arg=0x5623b9bb1c00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f44d07f0700 (LWP 228391)): #0 0x00007f44de235483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f44d3c576f8 in asio::detail::epoll_reactor::run (this=0x5623b9b9a040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f44d3c5bdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5623b9b99ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f44d3c5c0a2 in asio::detail::task_io_service::run (this=0x5623b9b99ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f44d3c73afd in asio::io_service::run (this=0x5623b9b98fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f44d3c74251 in gcomm::AsioProtonet::event_loop (this=0x5623b9b98f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f44d3c922ed in GCommConn::run (this=0x5623b9b996c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f44d3c95399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f44d88cf700 (LWP 228383)): #0 0x00007f44dfe7ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005623b7804c99 in safe_cond_timedwait (cond=0x5623b8de3640 , mp=0x5623b8de3580 , abstime=0x7f44d88ced70, file=0x5623b7c8c840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005623b7805c53 in inline_mysql_cond_timedwait (that=0x5623b8de3640 , mutex=0x5623b8de3580 , abstime=0x7f44d88ced70, src_file=0x5623b7c8c888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005623b78067b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005623b779745f in pfs_spawn_thread (arg=0x5623b9b22730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f44dfe76dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f44de234ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f44e0287880 (LWP 228350)): #0 0x00007f44dfe7b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005623b77ff368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005623b703d503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f44de22a20d in poll () from /lib64/libc.so.6 #5 0x00005623b6bdda8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005623b6bdcd25 in mysqld_main (argc=161, argv=0x5623b99dad20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005623b6bd0ec5 in main (argc=23, argv=0x7ffead100a98) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.228670' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera-features#56-innodb/mysqld.2/data/core.228670 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 228670] [New LWP 228708] [New LWP 228710] [New LWP 228746] [New LWP 228748] [New LWP 228747] [New LWP 229330] [New LWP 229228] [New LWP 229331] [New LWP 229230] [New LWP 229372] [New LWP 229318] [New LWP 229377] [New LWP 229320] [New LWP 229321] [New LWP 234086] [New LWP 229326] [New LWP 234085] [New LWP 229328] [New LWP 229373] [New LWP 229329] [New LWP 229333] [New LWP 234084] [New LWP 229317] [New LWP 229332] [New LWP 229327] [New LWP 229325] [New LWP 229319] [New LWP 229232] [New LWP 229233] [New LWP 229229] [New LWP 229226] [New LWP 229231] [New LWP 229227] [New LWP 228709] [New LWP 228745] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fd74becf9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fd74becf9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055d4b239a368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055d4b1bd8503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd74a27e20d in poll () from /lib64/libc.so.6 #5 0x000055d4b1778a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055d4b1777d25 in mysqld_main (argc=162, argv=0x55d4b54e3d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055d4b176bec5 in main (argc=23, argv=0x7ffd7ad5e238) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 36 (Thread 0x7fd734af0700 (LWP 228745)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd73fc90867 in gu::Lock::wait (cond=..., this=0x7fd734aef850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd73fc933ab in RecvBuf::front (this=this@entry=0x55d4b569e818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fd73fc90f46 in gcomm_recv (backend=, msg=0x55d4b566a720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fd73fc84eec in core_msg_recv (backend=backend@entry=0x55d4b566a868, recv_msg=recv_msg@entry=0x55d4b566a720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fd73fc88164 in gcs_core_recv (conn=0x55d4b566a6c0, recv_act=recv_act@entry=0x7fd734aefd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fd73fc8f3d8 in gcs_recv_thread (arg=0x55d4b566a440) at gcs/src/gcs.cpp:1398 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fd737775700 (LWP 228709)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd73fcb5fd7 in gu::Lock::wait (cond=..., this=0x7fd737774bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd73fcb6069 in galera::ServiceThd::thd_func (arg=0x55d4b5662908) at galera/src/galera_service_thd.cpp:30 #3 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fd720e1f700 (LWP 229227)): #0 0x00007fd74ba9b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d4b1ed541c in LinuxAIOHandler::collect (this=0x7fd720e1eb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d4b1ed5945 in LinuxAIOHandler::poll (this=0x7fd720e1eb00, m1=0x7fd720e1eba8, m2=0x7fd720e1ebb0, request=0x7fd720e1ebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d4b1ed5aa8 in os_aio_linux_handler (global_segment=0, m1=0x7fd720e1eba8, m2=0x7fd720e1ebb0, request=0x7fd720e1ebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d4b1eda5cc in os_aio_handler (segment=0, m1=0x7fd720e1eba8, m2=0x7fd720e1ebb0, request=0x7fd720e1ebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d4b212d4a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d4b1fd3909 in io_handler_thread (arg=0x55d4b315a620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fd70e7fc700 (LWP 229231)): #0 0x00007fd74ba9b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d4b1ed541c in LinuxAIOHandler::collect (this=0x7fd70e7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d4b1ed5945 in LinuxAIOHandler::poll (this=0x7fd70e7fbb00, m1=0x7fd70e7fbba8, m2=0x7fd70e7fbbb0, request=0x7fd70e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d4b1ed5aa8 in os_aio_linux_handler (global_segment=4, m1=0x7fd70e7fbba8, m2=0x7fd70e7fbbb0, request=0x7fd70e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d4b1eda5cc in os_aio_handler (segment=4, m1=0x7fd70e7fbba8, m2=0x7fd70e7fbbb0, request=0x7fd70e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d4b212d4a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d4b1fd3909 in io_handler_thread (arg=0x55d4b315a640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fd722671700 (LWP 229226)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b239fc99 in safe_cond_timedwait (cond=0x55d4b396bea0 , mp=0x55d4b396bde0 , abstime=0x7fd722670cb0, file=0x55d4b27e3610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d4b2209acb in inline_mysql_cond_timedwait (that=0x55d4b396bea0 , mutex=0x55d4b396bde0 , abstime=0x7fd722670cb0, src_file=0x55d4b27e3658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d4b2209fef in my_service_thread_sleep (control=0x55d4b2f20b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055d4b21fe488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055d4b233245f in pfs_spawn_thread (arg=0x55d4b57676b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fd70f7fe700 (LWP 229229)): #0 0x00007fd74ba9b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d4b1ed541c in LinuxAIOHandler::collect (this=0x7fd70f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d4b1ed5945 in LinuxAIOHandler::poll (this=0x7fd70f7fdb00, m1=0x7fd70f7fdba8, m2=0x7fd70f7fdbb0, request=0x7fd70f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d4b1ed5aa8 in os_aio_linux_handler (global_segment=2, m1=0x7fd70f7fdba8, m2=0x7fd70f7fdbb0, request=0x7fd70f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d4b1eda5cc in os_aio_handler (segment=2, m1=0x7fd70f7fdba8, m2=0x7fd70f7fdbb0, request=0x7fd70f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d4b212d4a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d4b1fd3909 in io_handler_thread (arg=0x55d4b315a630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fd70d7fa700 (LWP 229233)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee1a1a in os_event::timed_wait (this=0x55d4b57af630, abstime=0x7fd70d7f9af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d4b1ee1d31 in os_event::wait_time_low (this=0x55d4b57af630, time_in_usec=726000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d4b1ee1ed1 in os_event_wait_time_low (event=0x55d4b57af630, time_in_usec=726000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d4b20b7d1d in pc_sleep_if_needed (next_loop_time=1549881227935, sig_count=2, cur_time=1549881227209) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055d4b20b8c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fd70dffb700 (LWP 229232)): #0 0x00007fd74ba9b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d4b1ed541c in LinuxAIOHandler::collect (this=0x7fd70dffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d4b1ed5945 in LinuxAIOHandler::poll (this=0x7fd70dffab00, m1=0x7fd70dffaba8, m2=0x7fd70dffabb0, request=0x7fd70dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d4b1ed5aa8 in os_aio_linux_handler (global_segment=5, m1=0x7fd70dffaba8, m2=0x7fd70dffabb0, request=0x7fd70dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d4b1eda5cc in os_aio_handler (segment=5, m1=0x7fd70dffaba8, m2=0x7fd70dffabb0, request=0x7fd70dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d4b212d4a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d4b1fd3909 in io_handler_thread (arg=0x55d4b315a648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fd6fb7fe700 (LWP 229319)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee1a1a in os_event::timed_wait (this=0x55d4b57af530, abstime=0x7fd6fb7fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d4b1ee1d31 in os_event::wait_time_low (this=0x55d4b57af530, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d4b1ee1ed1 in os_event_wait_time_low (event=0x55d4b57af530, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d4b1fcfada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fd6f9ffb700 (LWP 229325)): #0 0x00007fd74bed1e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055d4b1ee2401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055d4b1fd135b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055d4b1fd148b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fd6f8ff9700 (LWP 229327)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee20ff in os_event::wait (this=0x55d4b57af330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d4b1ee1b85 in os_event::wait_low (this=0x55d4b57af330, reset_sig_count=13055) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d4b1ee1ef6 in os_event_wait_low (event=0x55d4b57af330, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d4b1fcddf3 in srv_resume_thread (slot=0x55d4b2f1c780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d4b1fd19f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fd6f1ffb700 (LWP 229332)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee20ff in os_event::wait (this=0x55d4b57af6b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d4b1ee1b85 in os_event::wait_low (this=0x55d4b57af6b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d4b1ee1ef6 in os_event_wait_low (event=0x55d4b57af6b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d4b2096b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fd6fbfff700 (LWP 229317)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee1a1a in os_event::timed_wait (this=0x55d4b59879e0, abstime=0x7fd6fbffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d4b1ee1d31 in os_event::wait_time_low (this=0x55d4b59879e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d4b1ee1ed1 in os_event_wait_time_low (event=0x55d4b59879e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d4b1eae88e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fd720f3e700 (LWP 234084)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd73fbaa132 in fifo_lock_get (q=q@entry=0x55d4b5690ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007fd73fbaa7a1 in gu_fifo_get_head (q=0x55d4b5690ac0, err=0x7fd720f3d06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd73fc8c0cc in gcs_recv (conn=0x55d4b566a440, action=0x7fd720f3d0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd73fcb916c in galera::GcsActionSource::process (this=0x55d4b5696e60, recv_ctx=0x7fd720f3d4d0, exit_loop=@0x7fd720f3d29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd73fcd8033 in galera::ReplicatorSMM::async_recv (this=0x55d4b56621f0, recv_ctx=0x7fd720f3d4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055d4b2454276 in wsrep::wsrep_provider_v26::run_applier (this=0x55d4b55d7e50, applier_ctx=0x7fd720f3d4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055d4b1af7636 in wsrep_replication_process (thd=0x7fd718000c40, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055d4b1ae9282 in start_wsrep_THD (arg=0x7fd6c00f2d20) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fd6f17fa700 (LWP 229333)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b239f997 in safe_cond_wait (cond=0x7fd6f17f9c70, mp=0x7fd6f17f9cb0, file=0x55d4b266dc40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d4b1df9487 in inline_mysql_cond_wait (that=0x7fd6f17f9c70, mutex=0x7fd6f17f9cb0, src_file=0x55d4b266f5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d4b1dfe0c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055d4b233245f in pfs_spawn_thread (arg=0x55d4b596a670) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fd6f37fe700 (LWP 229329)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee20ff in os_event::wait (this=0x55d4b57af430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d4b1ee1b85 in os_event::wait_low (this=0x55d4b57af430, reset_sig_count=11471) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d4b1ee1ef6 in os_event_wait_low (event=0x55d4b57af430, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d4b1fcddf3 in srv_resume_thread (slot=0x55d4b2f1c7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d4b1fd19f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fd720fd4700 (LWP 229373)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b239f997 in safe_cond_wait (cond=0x55d4b30df1e0 , mp=0x55d4b30e2560 , file=0x55d4b246acc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d4b179d4ee in inline_mysql_cond_wait (that=0x55d4b30df1e0 , mutex=0x55d4b30e2560 , src_file=0x55d4b246b830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055d4b179ed80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055d4b233245f in pfs_spawn_thread (arg=0x55d4b5b11db0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fd6f3fff700 (LWP 229328)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee20ff in os_event::wait (this=0x55d4b57af3b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d4b1ee1b85 in os_event::wait_low (this=0x55d4b57af3b0, reset_sig_count=12559) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d4b1ee1ef6 in os_event_wait_low (event=0x55d4b57af3b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d4b1fcddf3 in srv_resume_thread (slot=0x55d4b2f1c7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d4b1fd19f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fd720ef3700 (LWP 234085)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd73fbaa132 in fifo_lock_get (q=q@entry=0x55d4b5690ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007fd73fbaa7a1 in gu_fifo_get_head (q=0x55d4b5690ac0, err=0x7fd720ef206c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd73fc8c0cc in gcs_recv (conn=0x55d4b566a440, action=0x7fd720ef20b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd73fcb916c in galera::GcsActionSource::process (this=0x55d4b5696e60, recv_ctx=0x7fd720ef24d0, exit_loop=@0x7fd720ef229e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd73fcd8033 in galera::ReplicatorSMM::async_recv (this=0x55d4b56621f0, recv_ctx=0x7fd720ef24d0) at galera/src/replicator_smm.cpp:383 #6 0x000055d4b2454276 in wsrep::wsrep_provider_v26::run_applier (this=0x55d4b55d7e50, applier_ctx=0x7fd720ef24d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055d4b1af7636 in wsrep_replication_process (thd=0x7fd6bc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055d4b1ae9282 in start_wsrep_THD (arg=0x7fd6c0094500) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fd6f97fa700 (LWP 229326)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee20ff in os_event::wait (this=0x55d4b57af2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d4b1ee1b85 in os_event::wait_low (this=0x55d4b57af2b0, reset_sig_count=3306) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d4b1ee1ef6 in os_event_wait_low (event=0x55d4b57af2b0, reset_sig_count=3306) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d4b1fcddf3 in srv_resume_thread (slot=0x55d4b2f1c748 , sig_count=3306, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d4b1fd1e04 in srv_purge_coordinator_suspend (slot=0x55d4b2f1c748 , rseg_history_len=34) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055d4b1fd209a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fd720ea8700 (LWP 234086)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd73fbaa132 in fifo_lock_get (q=q@entry=0x55d4b5690ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007fd73fbaa7a1 in gu_fifo_get_head (q=0x55d4b5690ac0, err=0x7fd720ea706c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd73fc8c0cc in gcs_recv (conn=0x55d4b566a440, action=0x7fd720ea70b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd73fcb916c in galera::GcsActionSource::process (this=0x55d4b5696e60, recv_ctx=0x7fd720ea74d0, exit_loop=@0x7fd720ea729e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd73fcd8033 in galera::ReplicatorSMM::async_recv (this=0x55d4b56621f0, recv_ctx=0x7fd720ea74d0) at galera/src/replicator_smm.cpp:383 #6 0x000055d4b2454276 in wsrep::wsrep_provider_v26::run_applier (this=0x55d4b55d7e50, applier_ctx=0x7fd720ea74d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055d4b1af7636 in wsrep_replication_process (thd=0x7fd6b8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055d4b1ae9282 in start_wsrep_THD (arg=0x7fd6c01c3930) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fd6fa7fc700 (LWP 229321)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee1a1a in os_event::timed_wait (this=0x55d4b5957aa0, abstime=0x7fd6fa7fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d4b1ee1d31 in os_event::wait_time_low (this=0x55d4b5957aa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d4b1ee1ed1 in os_event_wait_time_low (event=0x55d4b5957aa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d4b20302fb in ib_wqueue_timedwait (wq=0x55d4b5957970, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055d4b2170ea4 in fts_optimize_thread (arg=0x55d4b5957970) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fd6faffd700 (LWP 229320)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee1a1a in os_event::timed_wait (this=0x55d4b59b9aa0, abstime=0x7fd6faffcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d4b1ee1d31 in os_event::wait_time_low (this=0x55d4b59b9aa0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d4b1ee1ed1 in os_event_wait_time_low (event=0x55d4b59b9aa0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d4b2117142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fd720f89700 (LWP 229377)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd73fc8d7f5 in gcs_replv (conn=0x55d4b566a440, act_in=0x7fd720f85d60, act=act@entry=0x7fd720f85d40, scheduled=) at gcs/src/gcs.cpp:1843 #2 0x00007fd73fcdfb7e in galera::Gcs::replv (this=this@entry=0x55d4b56628f8, actv=..., act=..., scheduled=scheduled@entry=true) at galera/src/galera_gcs.hpp:138 #3 0x00007fd73fcd51f1 in galera::ReplicatorSMM::replicate (this=0x55d4b56621f0, trx=..., meta=0x7fd720f863e8) at galera/src/replicator_smm.cpp:661 #4 0x00007fd73fcf1932 in galera_certify (gh=, conn_id=, trx_handle=, flags=, meta=0x7fd720f863e8) at galera/src/wsrep_provider.cpp:535 #5 0x000055d4b24545e8 in wsrep::wsrep_provider_v26::certify (this=0x55d4b55d7e50, client_id=..., ws_handle=..., flags=0, ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:737 #6 0x000055d4b244a34e in wsrep::transaction::certify_fragment (this=0x7fd6c00069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1140 #7 0x000055d4b2449c37 in wsrep::transaction::streaming_step (this=0x7fd6c00069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #8 0x000055d4b2446ff2 in wsrep::transaction::after_row (this=0x7fd6c00069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #9 0x000055d4b1bf03a8 in wsrep::client_state::after_row (this=0x7fd6c0006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #10 0x000055d4b1bd9625 in wsrep_after_row (thd=0x7fd6c0000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #11 0x000055d4b1beb52a in wsrep_after_row (thd=0x7fd6c0000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #12 0x000055d4b1beb91b in handler::ha_write_row (this=0x7fd6c021ebf8, buf=0x7fd6c01f4740 "\375p\020") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #13 0x000055d4b1861443 in write_record (thd=0x7fd6c0000b00, table=0x7fd6c021dfc0, info=0x7fd6c0018180) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #14 0x000055d4b1866742 in select_insert::send_data (this=0x7fd6c0018138, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #15 0x000055d4b191ee07 in end_send (join=0x7fd6c00181e8, join_tab=0x7fd6c023b228, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #16 0x000055d4b1a5bfa7 in JOIN_CACHE::generate_full_extensions (this=0x7fd6c023b5d8, rec_ptr=0x7fd6c02c11b0 "T\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #17 0x000055d4b1a5bcc2 in JOIN_CACHE::join_matching_records (this=0x7fd6c023b5d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #18 0x000055d4b1a5b68e in JOIN_CACHE::join_records (this=0x7fd6c023b5d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #19 0x000055d4b1a5b828 in JOIN_CACHE::join_records (this=0x7fd6c001af88, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #20 0x000055d4b1a5b828 in JOIN_CACHE::join_records (this=0x7fd6c001ae58, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #21 0x000055d4b191babb in sub_select_cache (join=0x7fd6c00181e8, join_tab=0x7fd6c023a718, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #22 0x000055d4b191bcc8 in sub_select (join=0x7fd6c00181e8, join_tab=0x7fd6c023a368, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #23 0x000055d4b191b493 in do_select (join=0x7fd6c00181e8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #24 0x000055d4b18f3fb1 in JOIN::exec_inner (this=0x7fd6c00181e8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #25 0x000055d4b18f33fa in JOIN::exec (this=0x7fd6c00181e8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #26 0x000055d4b18f46ec in mysql_select (thd=0x7fd6c0000b00, tables=0x7fd6c0015f10, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7fd6c0018138, unit=0x7fd6c00049c0, select_lex=0x7fd6c00159f0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #27 0x000055d4b18e5fed in handle_select (thd=0x7fd6c0000b00, lex=0x7fd6c00048f8, result=0x7fd6c0018138, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #28 0x000055d4b18a83fc in mysql_execute_command (thd=0x7fd6c0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #29 0x000055d4b18b3935 in mysql_parse (thd=0x7fd6c0000b00, rawbuf=0x7fd6c0015168 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fd720f880d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #30 0x000055d4b18b2f68 in wsrep_mysql_parse (thd=0x7fd6c0000b00, rawbuf=0x7fd6c0015168 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fd720f880d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #31 0x000055d4b189ee25 in dispatch_command (command=COM_QUERY, thd=0x7fd6c0000b00, packet=0x7fd6c0028201 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #32 0x000055d4b189d80d in do_command (thd=0x7fd6c0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #33 0x000055d4b1a0e069 in do_handle_one_connection (connect=0x55d4b5b10a90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #34 0x000055d4b1a0ddad in handle_one_connection (arg=0x55d4b5b10a90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #35 0x000055d4b233245f in pfs_spawn_thread (arg=0x55d4b5b36560) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #36 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #37 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fd70cff9700 (LWP 229318)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee1a1a in os_event::timed_wait (this=0x55d4b57af4b0, abstime=0x7fd70cff8ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d4b1ee1d31 in os_event::wait_time_low (this=0x55d4b57af4b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d4b1ee1ed1 in os_event_wait_time_low (event=0x55d4b57af4b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d4b1fcfffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fd72101f700 (LWP 229372)): #0 0x00007fd74a1c240a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055d4b176bf44 in my_sigwait (set=0x7fd72101ecf0, sig=0x7fd72101ecb4, code=0x7fd72101ecb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055d4b1772584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055d4b233245f in pfs_spawn_thread (arg=0x55d4b5772d20) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fd70effd700 (LWP 229230)): #0 0x00007fd74ba9b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d4b1ed541c in LinuxAIOHandler::collect (this=0x7fd70effcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d4b1ed5945 in LinuxAIOHandler::poll (this=0x7fd70effcb00, m1=0x7fd70effcba8, m2=0x7fd70effcbb0, request=0x7fd70effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d4b1ed5aa8 in os_aio_linux_handler (global_segment=3, m1=0x7fd70effcba8, m2=0x7fd70effcbb0, request=0x7fd70effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d4b1eda5cc in os_aio_handler (segment=3, m1=0x7fd70effcba8, m2=0x7fd70effcbb0, request=0x7fd70effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d4b212d4a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d4b1fd3909 in io_handler_thread (arg=0x55d4b315a638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fd6f27fc700 (LWP 229331)): #0 0x00007fd74bed1e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055d4b1ee2401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055d4b2088a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fd70ffff700 (LWP 229228)): #0 0x00007fd74ba9b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d4b1ed541c in LinuxAIOHandler::collect (this=0x7fd70fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d4b1ed5945 in LinuxAIOHandler::poll (this=0x7fd70fffeb00, m1=0x7fd70fffeba8, m2=0x7fd70fffebb0, request=0x7fd70fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d4b1ed5aa8 in os_aio_linux_handler (global_segment=1, m1=0x7fd70fffeba8, m2=0x7fd70fffebb0, request=0x7fd70fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d4b1eda5cc in os_aio_handler (segment=1, m1=0x7fd70fffeba8, m2=0x7fd70fffebb0, request=0x7fd70fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d4b212d4a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d4b1fd3909 in io_handler_thread (arg=0x55d4b315a628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fd6f2ffd700 (LWP 229330)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b1ee20ff in os_event::wait (this=0x55d4b57af5b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d4b1ee1b85 in os_event::wait_low (this=0x55d4b57af5b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d4b1ee1ef6 in os_event_wait_low (event=0x55d4b57af5b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d4b20adeb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fd7342a4700 (LWP 228747)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b239f997 in safe_cond_wait (cond=0x7fd724015608, mp=0x7fd724015558, file=0x55d4b2509b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d4b1af7088 in inline_mysql_cond_wait (that=0x7fd724015608, mutex=0x7fd724015558, src_file=0x55d4b2509e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d4b1af9d20 in Wsrep_thd_queue::pop_front (this=0x7fd724015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055d4b1af8b00 in wsrep_post_rollback_process (post_rollbacker=0x7fd724000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055d4b1ae9282 in start_wsrep_THD (arg=0x55d4b56bf700) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fd734259700 (LWP 228748)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd73fbaa132 in fifo_lock_get (q=q@entry=0x55d4b5690ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007fd73fbaa7a1 in gu_fifo_get_head (q=0x55d4b5690ac0, err=0x7fd73425806c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd73fc8c0cc in gcs_recv (conn=0x55d4b566a440, action=0x7fd7342580b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd73fcb916c in galera::GcsActionSource::process (this=0x55d4b5696e60, recv_ctx=0x7fd7342584d0, exit_loop=@0x7fd73425829e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd73fcd8033 in galera::ReplicatorSMM::async_recv (this=0x55d4b56621f0, recv_ctx=0x7fd7342584d0) at galera/src/replicator_smm.cpp:383 #6 0x000055worker[2] mysql-test-run: WARNING: Test reserved for w4 picked up by w2 d4b2454276 in wsrep::wsrep_provider_v26::run_applier (this=0x55d4b55d7e50, applier_ctx=0x7fd7342584d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055d4b1af7636 in wsrep_replication_process (thd=0x7fd71c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055d4b1ae9282 in start_wsrep_THD (arg=0x55d4b56c0b90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fd7342ef700 (LWP 228746)): #0 0x00007fd74bece965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b239f997 in safe_cond_wait (cond=0x7fd72c017b68, mp=0x7fd72c017ab8, file=0x55d4b2509b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d4b1af7088 in inline_mysql_cond_wait (that=0x7fd72c017b68, mutex=0x7fd72c017ab8, src_file=0x55d4b2509e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d4b1af9d20 in Wsrep_thd_queue::pop_front (this=0x7fd72c017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055d4b1af8740 in wsrep_rollback_process (rollbacker=0x7fd72c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055d4b1ae9282 in start_wsrep_THD (arg=0x55d4b56c0980) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fd7352f1700 (LWP 228710)): #0 0x00007fd74a289483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd73fc576f8 in asio::detail::epoll_reactor::run (this=0x55d4b569f000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fd73fc5bdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55d4b569eea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fd73fc5c0a2 in asio::detail::task_io_service::run (this=0x55d4b569eea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fd73fc73afd in asio::io_service::run (this=0x55d4b569dbc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fd73fc74251 in gcomm::AsioProtonet::event_loop (this=0x55d4b569db20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fd73fc922ed in GCommConn::run (this=0x55d4b569e630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fd73fc95399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fd744923700 (LWP 228708)): #0 0x00007fd74beced12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d4b239fc99 in safe_cond_timedwait (cond=0x55d4b397e640 , mp=0x55d4b397e580 , abstime=0x7fd744922d70, file=0x55d4b2827840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d4b23a0c53 in inline_mysql_cond_timedwait (that=0x55d4b397e640 , mutex=0x55d4b397e580 , abstime=0x7fd744922d70, src_file=0x55d4b2827888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d4b23a17b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055d4b233245f in pfs_spawn_thread (arg=0x55d4b562b7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd74becadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd74a288ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fd74c2db880 (LWP 228670)): #0 0x00007fd74becf9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055d4b239a368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055d4b1bd8503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd74a27e20d in poll () from /lib64/libc.so.6 #5 0x000055d4b1778a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055d4b1777d25 in mysqld_main (argc=162, argv=0x55d4b54e3d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055d4b176bec5 in main (argc=23, argv=0x7ffd7ad5e238) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera-features#56-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera-features#56-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_conflict_with_rollback_master galera_sr.galera_sr_ddl_master galera_sr.galera_sr_insert_select Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.GCF-623 galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera-features#56 galera_sr.GCF-845 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_kill_connection 'innodb' w2 [ pass ] 7826 galera_sr.galera_sr_blob 'innodb' w2 [ pass ] 10194 galera_sr.galera_sr_kill_query 'innodb' w2 [ pass ] 18941 galera_sr.galera_sr_bf_abort 'innodb' w4 [ pass ] 57753 galera_sr.galera_sr_cc_master 'innodb' w4 [ fail ] Test ended at 2019-02-11 12:36:58 CURRENT_TEST: galera_sr.galera_sr_cc_master Server [mysqld.2 - pid: 23912, winpid: 23912, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2019-02-11 12:36:24 36 [Note] WSREP: Stop replication 2019-02-11 12:36:24 36 [Note] WSREP: Server status change synced -> disconnecting 2019-02-11 12:36:24 36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 12:36:24 36 [Note] WSREP: Closing send monitor... 2019-02-11 12:36:24 36 [Note] WSREP: Closed send monitor. 2019-02-11 12:36:24 36 [Note] WSREP: gcomm: terminating thread 2019-02-11 12:36:24 36 [Note] WSREP: gcomm: joining thread 2019-02-11 12:36:24 36 [Note] WSREP: gcomm: closing backend 2019-02-11 12:36:27 36 [Note] WSREP: (b59f330d, 'tcp://0.0.0.0:16085') connection to peer b1b1afa8 with addr tcp://127.0.0.1:16082 timed out, no messages seen in PT3S 2019-02-11 12:36:27 36 [Note] WSREP: (b59f330d, 'tcp://0.0.0.0:16085') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16082 2019-02-11 12:36:29 36 [Note] WSREP: (b59f330d, 'tcp://0.0.0.0:16085') reconnecting to b1b1afa8 (tcp://127.0.0.1:16082), attempt 0 2019-02-11 12:36:32 36 [Note] WSREP: (b59f330d, 'tcp://0.0.0.0:16085') connection to peer b1b1afa8 with addr tcp://127.0.0.1:16082 timed out, no messages seen in PT3S 2019-02-11 12:36:34 36 [Note] WSREP: (b59f330d, 'tcp://0.0.0.0:16085') connection established to b1b1afa8 tcp://127.0.0.1:16082 2019-02-11 12:36:34 36 [Note] WSREP: evs::proto(b59f330d, LEAVING, view_id(REG,b1b1afa8,2)) suspecting node: b1b1afa8 2019-02-11 12:36:34 36 [Note] WSREP: evs::proto(b59f330d, LEAVING, view_id(REG,b1b1afa8,2)) suspected node without join message, declaring inactive 2019-02-11 12:36:34 36 [Note] WSREP: view(view_id(NON_PRIM,b1b1afa8,2) memb { b59f330d,0 } joined { } left { } partitioned { b1b1afa8,0 }) 2019-02-11 12:36:34 36 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-11 12:36:34 36 [Note] WSREP: view((empty)) 2019-02-11 12:36:34 36 [Note] WSREP: gcomm: closed 2019-02-11 12:36:34 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-11 12:36:34 0 [Note] WSREP: Writing down CC checksum: b027917b 4ac745b6 d3b1b4d7 71c668ea at offset 144 2019-02-11 12:36:34 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-11 12:36:34 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 12:36:34 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-11 12:36:34 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 60) 2019-02-11 12:36:34 0 [Note] WSREP: Received self-leave message. 2019-02-11 12:36:34 0 [Note] WSREP: Writing down CC checksum: 149d4c0e 6c87c8aa 59651505 f1555a37 at offset 64 2019-02-11 12:36:34 3 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-11 12:36:34 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-11 12:36:34 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 12:36:34 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-11 12:36:34 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-11 12:36:34 3 [Note] WSREP: ####### drain monitors upto 60 2019-02-11 12:36:34 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-11 12:36:34 3 [Note] WSREP: ####### My UUID: b59f330d-2de8-11e9-b76a-2e1ec2356c07 2019-02-11 12:36:34 3 [Note] WSREP: ####### ST not required 2019-02-11 12:36:34 36 [Note] WSREP: recv_thread() joined. 2019-02-11 12:36:34 36 [Note] WSREP: Closing replication queue. 2019-02-11 12:36:34 36 [Note] WSREP: Closing slave action queue. 2019-02-11 12:36:34 3 [Note] WSREP: ================================================ View: id: b1d0472a-2de8-11e9-842f-dfe27b46ae07:-1 seqno: -1 status: 1 prococol_version: 4 own_index: 0 final: 0 members 2019-02-11 12:36:34 3 [Note] WSREP: id: b59f330d-2de8-11e9-b76a-2e1ec2356c07 name: cnts7gl1.localdomain 2019-02-11 12:36:34 3 [Note] WSREP: ================================================= 2019-02-11 12:36:34 3 [Note] WSREP: Non-primary view 2019-02-11 12:36:34 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 12:36:34 3 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-11 12:36:34 3 [Note] WSREP: ####### drain monitors upto 60 2019-02-11 12:36:34 3 [Note] WSREP: ####### My UUID: b59f330d-2de8-11e9-b76a-2e1ec2356c07 2019-02-11 12:36:34 3 [Note] WSREP: ####### ST not required 2019-02-11 12:36:34 3 [Note] WSREP: ================================================ View: id: b1d0472a-2de8-11e9-842f-dfe27b46ae07:-1 seqno: -1 status: 1 prococol_version: 4 own_index: -1 final: 1 members 2019-02-11 12:36:34 3 [Note] WSREP: ================================================= 2019-02-11 12:36:34 3 [Note] WSREP: Non-primary view 2019-02-11 12:36:34 3 [Note] WSREP: Server status change disconnecting -> disconnected 2019-02-11 12:36:34 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 12:36:34 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 12:36:35 3 [Note] WSREP: Applier thread exiting 0 2019-02-11 12:36:35 30 [Warning] WSREP: Failed to replicate rollback fragment for 190: 3 2019-02-11 12:36:37 2 [Note] WSREP: rollbacker thread exiting 2019-02-11 12:36:37 36 [Note] WSREP: Stop replication 2019-02-11 12:36:39 37 [Note] WSREP: rollbacker thread exiting 2019-02-11 12:36:39 36 [Note] WSREP: Start replication 2019-02-11 12:36:39 36 [Note] WSREP: Connecting with bootstrap option: 0 2019-02-11 12:36:39 36 [Note] WSREP: Setting GCS initial position to b1d0472a-2de8-11e9-842f-dfe27b46ae07:60 2019-02-11 12:36:39 36 [Note] WSREP: protonet asio version 0 2019-02-11 12:36:39 36 [Note] WSREP: Using CRC-32C for message checksums. 2019-02-11 12:36:39 36 [Note] WSREP: backend: asio 2019-02-11 12:36:39 36 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-02-11 12:36:39 36 [Warning] WSREP: access file(/home/stepan/galera/git/10.4/server/mysql-test/var/4/mysqld.2/data//gvwstate.dat) failed(No such file or directory) 2019-02-11 12:36:39 36 [Note] WSREP: restore pc from disk failed 2019-02-11 12:36:39 36 [Note] WSREP: GMCast version 0 2019-02-11 12:36:39 36 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') listening at tcp://0.0.0.0:16085 2019-02-11 12:36:39 36 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') multicast: , ttl: 1 2019-02-11 12:36:39 36 [Note] WSREP: EVS version 1 2019-02-11 12:36:39 36 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '127.0.0.1:16082' 2019-02-11 12:36:39 36 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') connection established to b1b1afa8 tcp://127.0.0.1:16082 2019-02-11 12:36:39 36 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') turning message relay requesting on, nonlive peers: 2019-02-11 12:36:39 36 [Note] WSREP: EVS version upgrade 0 -> 1 2019-02-11 12:36:39 36 [Note] WSREP: declaring b1b1afa8 at tcp://127.0.0.1:16082 stable 2019-02-11 12:36:39 36 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-02-11 12:36:39 36 [Note] WSREP: Node b1b1afa8 state prim 2019-02-11 12:36:39 36 [Note] WSREP: view(view_id(PRIM,b1b1afa8,4) memb { b1b1afa8,0 e9f582bc,0 } joined { } left { } partitioned { }) 2019-02-11 12:36:39 36 [Note] WSREP: save pc into disk 2019-02-11 12:36:40 36 [Note] WSREP: gcomm: connected 2019-02-11 12:36:40 36 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-02-11 12:36:40 36 [Note] WSREP: Shifting CLOSED -> OPEN (TO: -1) 2019-02-11 12:36:40 36 [Note] WSREP: Opened channel 'my_wsrep_cluster' 2019-02-11 12:36:40 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2019-02-11 12:36:40 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-02-11 12:36:40 0 [Note] WSREP: STATE EXCHANGE: sent state msg: ea68999f-2de8-11e9-bbf4-ef34ec5f7306 2019-02-11 12:36:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: ea68999f-2de8-11e9-bbf4-ef34ec5f7306 from 0 (cnts7gl1.localdomain) 2019-02-11 12:36:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: ea68999f-2de8-11e9-bbf4-ef34ec5f7306 from 1 (cnts7gl1.localdomain) 2019-02-11 12:36:40 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 3, members = 1/2 (joined/total), act_id = 61, last_appl. = 0, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = b1d0472a-2de8-11e9-842f-dfe27b46ae07 2019-02-11 12:36:40 0 [Note] WSREP: Writing down CC checksum: 7bb89728 c818d1ae 6c98cf2f a86db21b at offset 224 2019-02-11 12:36:40 0 [Note] WSREP: Flow-control interval: [23, 23] 2019-02-11 12:36:40 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 12:36:40 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 62) 2019-02-11 12:36:40 41 [Note] WSREP: Starting applier thread 41 2019-02-11 12:36:40 41 [Note] WSREP: ####### processing CC 62, local, ordered 2019-02-11 12:36:40 41 [Note] WSREP: ####### drain monitors upto 60 2019-02-11 12:36:40 41 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-02-11 12:36:40 41 [Note] WSREP: ####### My UUID: e9f582bc-2de8-11e9-b842-5a8d0218a4f3 2019-02-11 12:36:40 41 [Note] WSREP: Server cnts7gl1.localdomain connected to cluster at position b1d0472a-2de8-11e9-842f-dfe27b46ae07:62 with ID e9f582bc-2de8-11e9-b842-5a8d0218a4f3 2019-02-11 12:36:40 41 [Note] WSREP: Server status change disconnected -> connected 2019-02-11 12:36:40 41 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 12:36:40 41 [Note] WSREP: State transfer required: Group state: b1d0472a-2de8-11e9-842f-dfe27b46ae07:62 Local state: b1d0472a-2de8-11e9-842f-dfe27b46ae07:60 2019-02-11 12:36:40 41 [Note] WSREP: Server status change connected -> joiner 2019-02-11 12:36:40 41 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 12:36:40 41 [Note] WSREP: You have configured 'rsync' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2019-02-11 12:36:40 41 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-02-11 12:36:40 0 [Note] WSREP: Service thread queue flushed. 2019-02-11 12:36:40 41 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-02-11 12:36:40 41 [Note] WSREP: ####### IST uuid:b1d0472a-2de8-11e9-842f-dfe27b46ae07 f: 61, l: 62, p: 10 2019-02-11 12:36:40 41 [Note] WSREP: IST receiver addr using tcp://127.0.0.1:16086 2019-02-11 12:36:40 41 [Note] WSREP: Prepared IST receiver for 61-62, listening at: tcp://127.0.0.1:16086 2019-02-11 12:36:40 0 [Note] WSREP: Member 1.0 (cnts7gl1.localdomain) requested state transfer from '*any*'. Selected 0.0 (cnts7gl1.localdomain)(SYNCED) as donor. 2019-02-11 12:36:40 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 62) 2019-02-11 12:36:40 41 [Note] WSREP: Requesting state transfer: success, donor: 0 2019-02-11 12:36:40 41 [Note] WSREP: Receiving IST: 2 writesets, seqnos 61-62 2019-02-11 12:36:41 0 [Note] WSREP: ####### IST applying starts with 61 2019-02-11 12:36:41 0 [Note] WSREP: ####### IST current seqno initialized to 3 2019-02-11 12:36:41 0 [Note] WSREP: Receiving IST... 0.0% ( 0/60 events) complete. 2019-02-11 12:36:41 0 [Note] WSREP: IST preload starting at 3 2019-02-11 12:36:41 0 [Note] WSREP: Service thread queue flushed. 2019-02-11 12:36:41 0 [Note] WSREP: 0.0 (cnts7gl1.localdomain): State transfer to 1.0 (cnts7gl1.localdomain) complete. 2019-02-11 12:36:41 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:2, protocol version: 5 2019-02-11 12:36:41 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-11 12:36:41 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 62 last received: 5 2019-02-11 12:36:41 41 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) at galera/src/replicator_smm.hpp:pop_front():320. Null event. 2019-02-11 12:36:41 0 [Note] WSREP: Member 0.0 (cnts7gl1.localdomain) synced with group. 2019-02-11 12:36:41 41 [Note] WSREP: Closing send monitor... 2019-02-11 12:36:41 41 [Note] WSREP: Closed send monitor. 2019-02-11 12:36:41 41 [Note] WSREP: gcomm: terminating thread 2019-02-11 12:36:41 41 [Note] WSREP: gcomm: joining thread 2019-02-11 12:36:41 41 [Note] WSREP: gcomm: closing backend 2019-02-11 12:36:42 41 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') turning message relay requesting off 2019-02-11 12:36:44 41 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') connection to peer b1b1afa8 with addr tcp://127.0.0.1:16082 timed out, no messages seen in PT3S 2019-02-11 12:36:44 41 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16082 2019-02-11 12:36:46 41 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') reconnecting to b1b1afa8 (tcp://127.0.0.1:16082), attempt 0 2019-02-11 12:36:49 41 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') connection to peer b1b1afa8 with addr tcp://127.0.0.1:16082 timed out, no messages seen in PT3S 2019-02-11 12:36:51 41 [Note] WSREP: (e9f582bc, 'tcp://0.0.0.0:16085') connection established to b1b1afa8 tcp://127.0.0.1:16082 2019-02-11 12:36:51 41 [Note] WSREP: evs::proto(e9f582bc, LEAVING, view_id(REG,b1b1afa8,4)) suspecting node: b1b1afa8 2019-02-11 12:36:51 41 [Note] WSREP: evs::proto(e9f582bc, LEAVING, view_id(REG,b1b1afa8,4)) suspected node without join message, declaring inactive 2019-02-11 12:36:51 41 [Note] WSREP: view(view_id(NON_PRIM,b1b1afa8,4) memb { e9f582bc,0 } joined { } left { } partitioned { b1b1afa8,0 }) 2019-02-11 12:36:51 41 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-11 12:36:51 41 [Note] WSREP: view((empty)) 2019-02-11 12:36:51 41 [Note] WSREP: gcomm: closed 2019-02-11 12:36:51 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-11 12:36:51 0 [Note] WSREP: Writing down CC checksum: 1d1599a2 46647c0d e2fa3463 b89414d3 at offset 144 2019-02-11 12:36:51 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-11 12:36:51 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 12:36:51 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-11 12:36:51 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 62) 2019-02-11 12:36:51 0 [Note] WSREP: Received self-leave message. 2019-02-11 12:36:51 0 [Note] WSREP: Writing down CC checksum: 149d4c0e 6c87c8aa 59651505 f1555a37 at offset 64 2019-02-11 12:36:51 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-11 12:36:51 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 12:36:51 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-11 12:36:51 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-11 12:36:51 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-11 12:36:51 41 [Note] WSREP: recv_thread() joined. 2019-02-11 12:36:51 41 [Note] WSREP: Closing replication queue. 2019-02-11 12:36:51 41 [Note] WSREP: Closing slave action queue. mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. 190211 12:36:51 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.3-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63454 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f191000fb80 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f192c2419a0 thread_stack 0x49000 /home/stepan/galera/git/10.4/server/sql/mysqld(my_print_stacktrace+0x40)[0x557b0b9ef27a] mysys/stacktrace.c:269(my_print_stacktrace)[0x557b0b22d0d7] sigaction.c:0(__restore_rt)[0x7f1935a9f5d0] :0(__GI_raise)[0x7f1933d8e207] :0(__GI_abort)[0x7f1933d8f8f8] :0(__assert_fail_base)[0x7f1933d87026] :0(__GI___assert_fail)[0x7f1933d870d2] src/replicator_str.cpp:945(galera::ReplicatorSMM::request_state_transfer(void*, wsrep_uuid const&, long, void const*, long))[0x7f192d9d9dc7] src/replicator_smm.cpp:2733(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7f192d9c8e78] src/gcs_action_source.cpp:154(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f192d9a8ec9] src/gcs_action_source.cpp:188(galera::GcsActionSource::process(void*, bool&))[0x7f192d9a921d] src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7f192d9c8033] /home/stepan/galera/git/10.4/server/sql/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x30)[0x557b0baa9276] src/wsrep_provider_v26.cpp:690(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x557b0b14c636] sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x557b0b13e282] pthread_create.c:0(start_thread)[0x7f1935a97dd5] /lib64/libc.so.6(clone+0x6d)[0x7f1933e55ead] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 41 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file at /home/stepan/galera/git/10.4/server/mysql-test/var/4/mysqld.2/data/ ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > connection node_1; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2a; SET SESSION wsrep_sync_wait=0; SET GLOBAL wsrep_cluster_address = ''; SET SESSION wsrep_sync_wait = DEFAULT; connection node_2; INSERT INTO t1 VALUES (6); ERROR HY000: Lost connection to MySQL server during query connection node_1; SELECT COUNT(*) = 0 FROM mysql.wsrep_streaming_log; COUNT(*) = 0 1 connection node_2a; connection node_1; connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2b; - found 'core.23914' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.23914 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 27596] [New LWP 27633] [New LWP 24342] [New LWP 23951] [New LWP 23914] [New LWP 24347] [New LWP 24345] [New LWP 24395] [New LWP 23930] [New LWP 24396] [New LWP 24348] [New LWP 24344] [New LWP 24397] [New LWP 24391] [New LWP 24346] [New LWP 24399] [New LWP 24392] [New LWP 24349] [New LWP 24405] [New LWP 24393] [New LWP 27594] [New LWP 24390] [New LWP 24394] [New LWP 24398] [New LWP 24400] [New LWP 24401] [New LWP 24402] [New LWP 24403] [New LWP 27595] [New LWP 27027] [New LWP 24404] [New LWP 24343] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f1935a9c9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f1935a9c9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557b0b9ef368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557b0b22d503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f1933d8e207 in raise () from /lib64/libc.so.6 #5 0x00007f1933d8f8f8 in abort () from /lib64/libc.so.6 #6 0x00007f1933d87026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f1933d870d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007f192d9d9dc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x557b0ea491f0, recv_ctx=recv_ctx@entry=0x7f192c2414d0, group_uuid=..., cc_seqno=cc_seqno@entry=62, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007f192d9c8e78 in galera::ReplicatorSMM::process_conf_change (this=0x557b0ea491f0, recv_ctx=0x7f192c2414d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007f192d9a8ec9 in galera::GcsActionSource::dispatch (this=this@entry=0x557b0ea81e60, recv_ctx=recv_ctx@entry=0x7f192c2414d0, act=..., exit_loop=@0x7f192c24129e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007f192d9a921d in galera::GcsActionSource::process (this=0x557b0ea81e60, recv_ctx=0x7f192c2414d0, exit_loop=@0x7f192c24129e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007f192d9c8033 in galera::ReplicatorSMM::async_recv (this=0x557b0ea491f0, recv_ctx=0x7f192c2414d0) at galera/src/replicator_smm.cpp:383 #13 0x0000557b0baa9276 in wsrep::wsrep_provider_v26::run_applier (this=0x557b0e9bee50, applier_ctx=0x7f192c2414d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #14 0x0000557b0b14c636 in wsrep_replication_process (thd=0x7f191000fb80, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x0000557b0b13e282 in start_wsrep_THD (arg=0x7f18a4040980) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #16 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f19171ae700 (LWP 24343)): #0 0x00007f1935668644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557b0b52a41c in LinuxAIOHandler::collect (this=0x7f19171adb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557b0b52a945 in LinuxAIOHandler::poll (this=0x7f19171adb00, m1=0x7f19171adba8, m2=0x7f19171adbb0, request=0x7f19171adbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557b0b52aaa8 in os_aio_linux_handler (global_segment=0, m1=0x7f19171adba8, m2=0x7f19171adbb0, request=0x7f19171adbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557b0b52f5cc in os_aio_handler (segment=0, m1=0x7f19171adba8, m2=0x7f19171adbb0, request=0x7f19171adbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557b0b7824a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557b0b628909 in io_handler_thread (arg=0x557b0c7af620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f191c06b700 (LWP 24404)): #0 0x00007f1933d8f40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000557b0adc0f44 in my_sigwait (set=0x7f191c06acf0, sig=0x7f191c06acb4, code=0x7f191c06acb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000557b0adc7584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000557b0b98745f in pfs_spawn_thread (arg=0x557b0eb5dbc0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f1917318700 (LWP 27027)): #0 0x00007f1933e4b20d in poll () from /lib64/libc.so.6 #1 0x0000557b0ba6060c in vio_io_wait (vio=0x557b0ee97e00, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:976 #2 0x0000557b0ba5ec2c in vio_socket_io_wait (vio=0x557b0ee97e00, event=VIO_IO_EVENT_READ) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:118 #3 0x0000557b0ba5ee12 in vio_read (vio=0x557b0ee97e00, buf=0x7f18a400a430 "\a", size=4) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:195 #4 0x0000557b0b3960c0 in my_real_read (net=0x7f18a4000df8, complen=0x7f1917317810, header=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:962 #5 0x0000557b0b396947 in my_net_read_packet_reallen (net=0x7f18a4000df8, read_from_server=1 '\001', reallen=0x7f1917317878) at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1243 #6 0x0000557b0b3968fa in my_net_read_packet (net=0x7f18a4000df8, read_from_server=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1227 #7 0x0000557b0aef2146 in do_command (thd=0x7f18a4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1229 #8 0x0000557b0b063069 in do_handle_one_connection (connect=0x557b0ee9a050) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #9 0x0000557b0b062dad in handle_one_connection (arg=0x557b0ee9a050) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #10 0x0000557b0b98745f in pfs_spawn_thread (arg=0x557b0ee986b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #11 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f1917363700 (LWP 27595)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b9f4997 in safe_cond_wait (cond=0x7f190c0169a8, mp=0x7f190c0168f8, file=0x557b0bb5eb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557b0b14c088 in inline_mysql_cond_wait (that=0x7f190c0169a8, mutex=0x7f190c0168f8, src_file=0x557b0bb5ee80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557b0b14ed20 in Wsrep_thd_queue::pop_front (this=0x7f190c0168a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000557b0b14db00 in wsrep_post_rollback_process (post_rollbacker=0x7f190c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000557b0b13e282 in start_wsrep_THD (arg=0x7f18a4040c30) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f18daffd700 (LWP 24403)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b9f4997 in safe_cond_wait (cond=0x7f18daffcc70, mp=0x7f18daffccb0, file=0x557b0bcc2c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557b0b44e487 in inline_mysql_cond_wait (that=0x7f18daffcc70, mutex=0x7f18daffccb0, src_file=0x557b0bcc45d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557b0b4530c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000557b0b98745f in pfs_spawn_thread (arg=0x557b0ed54ee0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f18db7fe700 (LWP 24402)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b5370ff in os_event::wait (this=0x557b0eb9a630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557b0b536b85 in os_event::wait_low (this=0x557b0eb9a630, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557b0b536ef6 in os_event_wait_low (event=0x557b0eb9a630, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557b0b6ebb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f18dbfff700 (LWP 24401)): #0 0x00007f1935a9ee3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000557b0b537401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557b0b6dda3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f18e4ff9700 (LWP 24400)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b5370ff in os_event::wait (this=0x557b0eb9a530) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557b0b536b85 in os_event::wait_low (this=0x557b0eb9a530, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557b0b536ef6 in os_event_wait_low (event=0x557b0eb9a530, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557b0b702eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f18e5ffb700 (LWP 24398)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b5370ff in os_event::wait (this=0x557b0eb9a2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557b0b536b85 in os_event::wait_low (this=0x557b0eb9a2b0, reset_sig_count=313) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557b0b536ef6 in os_event_wait_low (event=0x557b0eb9a2b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557b0b622df3 in srv_resume_thread (slot=0x557b0c571780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557b0b6269f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f18e7fff700 (LWP 24394)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b536a1a in os_event::timed_wait (this=0x557b0ed420a0, abstime=0x7f18e7ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557b0b536d31 in os_event::wait_time_low (this=0x557b0ed420a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557b0b536ed1 in os_event_wait_time_low (event=0x557b0ed420a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557b0b6852fb in ib_wqueue_timedwait (wq=0x557b0ed41f70, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000557b0b7c5ea4 in fts_optimize_thread (arg=0x557b0ed41f70) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f1905ffb700 (LWP 24390)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b536a1a in os_event::timed_wait (this=0x557b0ed72520, abstime=0x7f1905ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557b0b536d31 in os_event::wait_time_low (this=0x557b0ed72520, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557b0b536ed1 in os_event_wait_time_low (event=0x557b0ed72520, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557b0b50388e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f192c1ac700 (LWP 27594)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b9f4997 in safe_cond_wait (cond=0x7f19080168e8, mp=0x7f1908016838, file=0x557b0bb5eb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557b0b14c088 in inline_mysql_cond_wait (that=0x7f19080168e8, mutex=0x7f1908016838, src_file=0x557b0bb5ee80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557b0b14ed20 in Wsrep_thd_queue::pop_front (this=0x7f19080167e0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000557b0b14d740 in wsrep_rollback_process (rollbacker=0x7f1908001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000557b0b13e282 in start_wsrep_THD (arg=0x7f18a402cf10) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f1904ff9700 (LWP 24393)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b536a1a in os_event::timed_wait (this=0x557b0eda45e0, abstime=0x7f1904ff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557b0b536d31 in os_event::wait_time_low (this=0x557b0eda45e0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557b0b536ed1 in os_event_wait_time_low (event=0x557b0eda45e0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557b0b76c142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f19173ae700 (LWP 24405)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b9f4997 in safe_cond_wait (cond=0x557b0c7341e0 , mp=0x557b0c737560 , file=0x557b0babfcc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557b0adf24ee in inline_mysql_cond_wait (that=0x557b0c7341e0 , mutex=0x557b0c737560 , src_file=0x557b0bac0830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000557b0adf3d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000557b0b98745f in pfs_spawn_thread (arg=0x557b0eb5e0b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f1906ffd700 (LWP 24349)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b536a1a in os_event::timed_wait (this=0x557b0eb9a5b0, abstime=0x7f1906ffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557b0b536d31 in os_event::wait_time_low (this=0x557b0eb9a5b0, time_in_usec=800000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557b0b536ed1 in os_event_wait_time_low (event=0x557b0eb9a5b0, time_in_usec=800000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557b0b70cd1d in pc_sleep_if_needed (next_loop_time=1549881412105, sig_count=2, cur_time=1549881411305) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000557b0b70dc56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f19067fc700 (LWP 24392)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b536a1a in os_event::timed_wait (this=0x557b0eb9a4b0, abstime=0x7f19067fbd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557b0b536d31 in os_event::wait_time_low (this=0x557b0eb9a4b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557b0b536ed1 in os_event_wait_time_low (event=0x557b0eb9a4b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557b0b624ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f18e57fa700 (LWP 24399)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b5370ff in os_event::wait (this=0x557b0eb9a3b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557b0b536b85 in os_event::wait_low (this=0x557b0eb9a3b0, reset_sig_count=269) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557b0b536ef6 in os_event_wait_low (event=0x557b0eb9a3b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557b0b622df3 in srv_resume_thread (slot=0x557b0c5717f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557b0b6269f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f1914c78700 (LWP 24346)): #0 0x00007f1935668644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557b0b52a41c in LinuxAIOHandler::collect (this=0x7f1914c77b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557b0b52a945 in LinuxAIOHandler::poll (this=0x7f1914c77b00, m1=0x7f1914c77ba8, m2=0x7f1914c77bb0, request=0x7f1914c77be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557b0b52aaa8 in os_aio_linux_handler (global_segment=3, m1=0x7f1914c77ba8, m2=0x7f1914c77bb0, request=0x7f1914c77be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557b0b52f5cc in os_aio_handler (segment=3, m1=0x7f1914c77ba8, m2=0x7f1914c77bb0, request=0x7f1914c77be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557b0b7824a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557b0b628909 in io_handler_thread (arg=0x557b0c7af638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f19057fa700 (LWP 24391)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b536a1a in os_event::timed_wait (this=0x557b0eb9a430, abstime=0x7f19057f9ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557b0b536d31 in os_event::wait_time_low (this=0x557b0eb9a430, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557b0b536ed1 in os_event_wait_time_low (event=0x557b0eb9a430, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557b0b624ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f18e67fc700 (LWP 24397)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b5370ff in os_event::wait (this=0x557b0eb9a330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557b0b536b85 in os_event::wait_low (this=0x557b0eb9a330, reset_sig_count=263) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557b0b536ef6 in os_event_wait_low (event=0x557b0eb9a330, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557b0b622df3 in srv_resume_thread (slot=0x557b0c5717b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557b0b6269f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f1915c7a700 (LWP 24344)): #0 0x00007f1935668644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557b0b52a41c in LinuxAIOHandler::collect (this=0x7f1915c79b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557b0b52a945 in LinuxAIOHandler::poll (this=0x7f1915c79b00, m1=0x7f1915c79ba8, m2=0x7f1915c79bb0, request=0x7f1915c79be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557b0b52aaa8 in os_aio_linux_handler (global_segment=1, m1=0x7f1915c79ba8, m2=0x7f1915c79bb0, request=0x7f1915c79be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557b0b52f5cc in os_aio_handler (segment=1, m1=0x7f1915c79ba8, m2=0x7f1915c79bb0, request=0x7f1915c79be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557b0b7824a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557b0b628909 in io_handler_thread (arg=0x557b0c7af628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f19077fe700 (LWP 24348)): #0 0x00007f1935668644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557b0b52a41c in LinuxAIOHandler::collect (this=0x7f19077fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557b0b52a945 in LinuxAIOHandler::poll (this=0x7f19077fdb00, m1=0x7f19077fdba8, m2=0x7f19077fdbb0, request=0x7f19077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557b0b52aaa8 in os_aio_linux_handler (global_segment=5, m1=0x7f19077fdba8, m2=0x7f19077fdbb0, request=0x7f19077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557b0b52f5cc in os_aio_handler (segment=5, m1=0x7f19077fdba8, m2=0x7f19077fdbb0, request=0x7f19077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557b0b7824a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557b0b628909 in io_handler_thread (arg=0x557b0c7af648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f18e6ffd700 (LWP 24396)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b5370ff in os_event::wait (this=0x557b0eb9a230) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557b0b536b85 in os_event::wait_low (this=0x557b0eb9a230, reset_sig_count=99) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557b0b536ef6 in os_event_wait_low (event=0x557b0eb9a230, reset_sig_count=99) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557b0b622df3 in srv_resume_thread (slot=0x557b0c571748 , sig_count=99, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557b0b626e04 in srv_purge_coordinator_suspend (slot=0x557b0c571748 , rseg_history_len=27) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000557b0b62709a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f192e4f0700 (LWP 23930)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b9f4c99 in safe_cond_timedwait (cond=0x557b0cfd3640 , mp=0x557b0cfd3580 , abstime=0x7f192e4efd70, file=0x557b0be7c840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557b0b9f5c53 in inline_mysql_cond_timedwait (that=0x557b0cfd3640 , mutex=0x557b0cfd3580 , abstime=0x7f192e4efd70, src_file=0x557b0be7c888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557b0b9f67b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000557b0b98745f in pfs_spawn_thread (arg=0x557b0ea127d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f18e77fe700 (LWP 24395)): #0 0x00007f1935a9ee3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000557b0b537401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557b0b62635b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000557b0b62648b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f1915479700 (LWP 24345)): #0 0x00007f1935668644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557b0b52a41c in LinuxAIOHandler::collect (this=0x7f1915478b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557b0b52a945 in LinuxAIOHandler::poll (this=0x7f1915478b00, m1=0x7f1915478ba8, m2=0x7f1915478bb0, request=0x7f1915478be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557b0b52aaa8 in os_aio_linux_handler (global_segment=2, m1=0x7f1915478ba8, m2=0x7f1915478bb0, request=0x7f1915478be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557b0b52f5cc in os_aio_handler (segment=2, m1=0x7f1915478ba8, m2=0x7f1915478bb0, request=0x7f1915478be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557b0b7824a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557b0b628909 in io_handler_thread (arg=0x557b0c7af630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f1907fff700 (LWP 24347)): #0 0x00007f1935668644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557b0b52a41c in LinuxAIOHandler::collect (this=0x7f1907ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557b0b52a945 in LinuxAIOHandler::poll (this=0x7f1907ffeb00, m1=0x7f1907ffeba8, m2=0x7f1907ffebb0, request=0x7f1907ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557b0b52aaa8 in os_aio_linux_handler (global_segment=4, m1=0x7f1907ffeba8, m2=0x7f1907ffebb0, request=0x7f1907ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557b0b52f5cc in os_aio_handler (segment=4, m1=0x7f1907ffeba8, m2=0x7f1907ffebb0, request=0x7f1907ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557b0b7824a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557b0b628909 in io_handler_thread (arg=0x557b0c7af640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f1935ea8880 (LWP 23914)): #0 0x00007f1933e4b20d in poll () from /lib64/libc.so.6 #1 0x0000557b0adcda8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #2 0x0000557b0adccd25 in mysqld_main (argc=162, argv=0x557b0e8cad20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #3 0x0000557b0adc0ec5 in main (argc=23, argv=0x7ffe52900b78) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 4 (Thread 0x7f192d344700 (LWP 23951)): #0 0x00007f1935a9b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f192d9a5fd7 in gu::Lock::wait (cond=..., this=0x7f192d343bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f192d9a6069 in galera::ServiceThd::thd_func (arg=0x557b0ea49908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f191ca6c700 (LWP 24342)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557b0b9f4c99 in safe_cond_timedwait (cond=0x557b0cfc0ea0 , mp=0x557b0cfc0de0 , abstime=0x7f191ca6bcb0, file=0x557b0be38610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557b0b85eacb in inline_mysql_cond_timedwait (that=0x557b0cfc0ea0 , mutex=0x557b0cfc0de0 , abstime=0x7f191ca6bcb0, src_file=0x557b0be38658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557b0b85efef in my_service_thread_sleep (control=0x557b0c575b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000557b0b853488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000557b0b98745f in pfs_spawn_thread (arg=0x557b0eb52550) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f192c1f7700 (LWP 27633)): #0 0x00007f1935a9bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f192d982bdb in gu::Lock::wait (this=this@entry=0x7f192c1f46f0, cond=..., date=...) at galerautils/src/gu_lock.hpp:62 #2 0x00007f192d9cd976 in galera::Monitor::wait (this=this@entry=0x557b0ea4a190, gtid=..., wait_until=...) at galera/src/monitor.hpp:355 #3 0x00007f192d9bf57f in galera::ReplicatorSMM::sync_wait (this=0x557b0ea491f0, upto=, tout=, gtid=0x7f192c1f4b30) at galera/src/replicator_smm.cpp:1661 #4 0x00007f192d9de643 in galera_sync_wait (wsrep=, upto=, tout=, gtid=) at galera/src/wsrep_provider.cpp:998 #5 0x0000557b0baa9f2d in wsrep::wsrep_provider_v26::causal_read (this=0x557b0e9bee50, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:845 #6 0x0000557b0ba94481 in wsrep::server_state::causal_read (this=0x557b0ea13950, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:648 #7 0x0000557b0ba8ed0b in wsrep::client_state::sync_wait (this=0x7f18a8006960, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/client_state.cpp:417 #8 0x0000557b0b136b2a in wsrep_sync_wait (thd=0x7f18a8000b00, mask=1) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:1022 #9 0x0000557b0aef99f4 in mysql_execute_command (thd=0x7f18a8000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:3779 #10 0x0000557b0af08935 in mysql_parse (thd=0x7f18a8000b00, rawbuf=0x7f18a8013e58 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7f192c1f60d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #11 0x0000557b0af07f68 in wsrep_mysql_parse (thd=0x7f18a8000b00, rawbuf=0x7f18a8013e58 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7f192c1f60d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #12 0x0000557b0aef3e25 in dispatch_command (command=COM_QUERY, thd=0x7f18a8000b00, packet=0x7f18a800a431 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", packet_length=102, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #13 0x0000557b0aef280d in do_command (thd=0x7f18a8000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #14 0x0000557b0b063069 in do_handle_one_connection (connect=0x557b0ea990b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #15 0x0000557b0b062dad in handle_one_connection (arg=0x557b0ea990b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #16 0x0000557b0b98745f in pfs_spawn_thread (arg=0x557b0ea8bf60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #17 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007f1933e55ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f192c242700 (LWP 27596)): #0 0x00007f1935a9c9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557b0b9ef368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557b0b22d503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f1933d8e207 in raise () from /lib64/libc.so.6 #5 0x00007f1933d8f8f8 in abort () from /lib64/libc.so.6 #6 0x00007f1933d87026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f1933d870d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007f192d9d9dc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x557b0ea491f0, recv_ctx=recv_ctx@entry=0x7f192c2414d0, group_uuid=..., cc_seqno=cc_seqno@entry=62, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007f192d9c8e78 in galera::ReplicatorSMM::process_conf_change (this=0x557b0ea491f0, recv_ctx=0x7f192c2414d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007f192d9a8ec9 in galera::GcsActionSource::dispatch (this=this@entry=0x557b0ea81e60, recv_ctx=recv_ctx@entry=0x7f192c2414d0, act=..., exit_loop=@0x7f192c24129e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007f192d9a921d in galera::GcsActionSource::process (this=0x557b0ea81e60, recv_ctx=0x7f192c2414d0, exit_loop=@0x7f192c24129e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007f192d9c8033 in galera::ReplicatorSMM::async_recv (this=0x557b0ea491f0, recv_ctx=0x7f192c2414d0) at galera/src/replicator_smm.cpp:383 #13 0x0000557b0baa9276 in wsrep::wsrep_provider_v26::run_applier (this=0x557b0e9bee50, applier_ctx=0x7f192c2414d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #14 0x0000557b0b14c636 in wsrep_replication_process (thd=0x7f191000fb80, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x0000557b0b13e282 in start_wsrep_THD (arg=0x7f18a4040980) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #16 0x00007f1935a97dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f1933e55ead in clone () from /lib64/libc.so.6 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_cc_master-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_cc_master galera_sr.galera_sr_bf_abort 2019-02-11 12:36:35 30 [Warning] WSREP: Failed to replicate rollback fragment for 190: 3 2019-02-11 12:36:41 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-11 12:36:41 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 62 last received: 5 2019-02-11 12:36:41 41 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. Attempting backtrace. You can use the following information to find out galera_sr.GCF-437 'innodb' w3 [ fail ] Test ended at 2019-02-11 12:37:09 CURRENT_TEST: galera_sr.GCF-1018B mysqltest got signal 6 read_command_buf (0x55a3594cbc60): } conn->name (0x55a35955bae0): node_1 Attempting backtrace... stack_bottom = 0x0 thread_stack 0x3c000 mysys/stacktrace.c:269(my_print_stacktrace)[0x55a356fea1e0] client/mysqltest.cc:9058(dump_backtrace())[0x55a356f9dae2] client/mysqltest.cc:9074(signal_handler)[0x55a356f9db15] sigaction.c:0(__restore_rt)[0x7f108d3895d0] :0(__poll_nocancel)[0x7f108bd941f0] pvio/pvio_socket.c:526(pvio_socket_wait_io_or_timeout)[0x55a356fcc32b] pvio/pvio_socket.c:312(pvio_socket_read)[0x55a356fcbfa8] libmariadb/ma_pvio.c:255(ma_pvio_read)[0x55a356fafbf2] libmariadb/ma_pvio.c:297(ma_pvio_cache_read)[0x55a356fafda4] libmariadb/ma_net.c:373(ma_real_read)[0x55a356fd0bb2] libmariadb/ma_net.c:427(ma_net_read)[0x55a356fd0de2] libmariadb/mariadb_lib.c:191(ma_net_safe_read)[0x55a356fa43aa] libmariadb/mariadb_lib.c:2115(mthd_my_read_query_result)[0x55a356fa9888] libmariadb/mariadb_lib.c:2173(mysql_read_query_result)[0x55a356fa9b98] tests/nonblock-wrappers.h:324(wrap_mysql_read_query_result(st_mysql*))[0x55a356f8acea] client/mysqltest.cc:8006(run_query_normal(st_connection*, st_command*, int, char*, unsigned long, st_dynamic_string*, st_dynamic_string*))[0x55a356f9b78d] client/mysqltest.cc:8786(run_query(st_connection*, st_command*, int))[0x55a356f9d13d] client/mysqltest.cc:9569(main)[0x55a356f9ef61] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f108bcc33d5] /home/stepan/galera/git/10.4/server/client/mysqltest(+0x59169)[0x55a356f8a169] Writing a core file... CURRENT_TEST: galera_sr.GCF-437 mysqltest: At line 13: query 'INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5' succeeded - should have failed with errno 1534... The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=MyISAM; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 2 * 1024 * 1024; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-437-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-437-innodb/' galera_sr.galera_sr_kill_all_pcrecovery 'innodb' w3 [ pass ] 25023 galera_sr.galera_sr_cc_slave 'innodb' w4 [ pass ] 31681 MTR's internal check of the test case 'galera_sr.galera_sr_cc_slave' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16081 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result 2019-02-11 12:37:15.695178000 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.reject 2019-02-11 12:37:47.771178000 +0200 @@ -77,7 +77,7 @@ ARIA_USED_FOR_TEMP_TABLES ON AUTOCOMMIT ON AUTOMATIC_SP_PRIVILEGES ON -AUTO_INCREMENT_OFFSET 2 +AUTO_INCREMENT_OFFSET 1 BACK_LOG 80 BASEDIR /home/stepan/galera/git/10.4/server BIG_TABLES OFF mysqltest: Result content mismatch not ok galera_sr.galera_sr_concurrent 'innodb' w4 [ pass ] 12887 galera_sr.galera_sr_conflict 'innodb' w4 [ pass ] 8103 galera_sr.galera_sr_conflict_on_commit 'innodb' w4 [ pass ] 8707 galera_sr.galera_sr_conflict_on_commit2 'innodb' w4 [ pass ] 8711 galera_sr.galera_sr_unit_statements 'innodb' w4 [ pass ] 9147 galera_sr.galera_sr_ws_size 'innodb' w4 [ pass ] 31640 galera_sr.galera_sr_ws_size2 'innodb' w4 [ pass ] 10964 galera_sr.galera_var_ignore_apply_errors_sr 'innodb' w4 [ pass ] 7844 galera_sr.mysql-wsrep#215 'innodb' w4 [ pass ] 17507 galera_sr.mysql-wsrep-features#138 'innodb' w4 [ pass ] 8290 galera_sr.mysql-wsrep-features#14 'innodb' w4 [ pass ] 6183 galera_sr.mysql-wsrep-features#148 'innodb' w4 [ pass ] 21416 MTR's internal check of the test case 'galera_sr.mysql-wsrep-features#148' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16081 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result 2019-02-11 12:40:30.985178000 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.reject 2019-02-11 12:40:52.687178000 +0200 @@ -1099,4 +1099,4 @@ user_variables 1.0 DISABLED INFORMATION SCHEMA 100403.0 NULL NULL Sergey Vojtovich User-defined variables GPL OFF Gamma 1.0 partition 1.0 DISABLED STORAGE ENGINE 100403.0 NULL NULL Mikael Ronstrom, MySQL AB Partition Storage Engine Helper GPL OFF Stable 1.0 VARIABLE_NAME VARIABLE_VALUE -DEBUG_SYNC ON - current signal: '' +DEBUG_SYNC ON - current signal: 'signal.wsrep_apply_cb' mysqltest: Result length mismatch not ok galera_sr.galera_sr_rollback_statement 'innodb' w1 [ fail ] Test ended at 2019-02-11 12:41:08 CURRENT_TEST: galera_sr.galera_sr_rollback_statement mysqltest: At line 34: query 'INSERT INTO t2 SELECT * FROM t1' succeeded - should have failed with errno 1062... The result from queries just before the failure was: connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 CHAR(255)) ENGINE=InnoDB; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3; ALTER TABLE t1 CHANGE f1 f1 INTEGER; ALTER TABLE t1 DROP PRIMARY KEY; INSERT INTO t1 VALUES (1, 'abc'); SET SESSION wsrep_trx_fragment_size = 1; INSERT INTO t2 SELECT * FROM t1; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_rollback_statement-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_rollback_statement-innodb/' galera_sr.mysql-wsrep-features#15 'innodb' w4 [ pass ] 5613 galera_sr.mysql-wsrep-features#165 'innodb' w4 [ fail ] Test ended at 2019-02-11 12:41:40 CURRENT_TEST: galera_sr.mysql-wsrep-features#165 mysqltest: In included file "/home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.test at line 27: At line 101: query 'SELECT * FROM t1' failed: 1205: Lock wait timeout exceeded; try restarting transaction The result from queries just before the failure was: < snip > connection node_1; SET DEBworker[2] Trying to dump core for [mysqltest - pid: 25335, winpid: 25335] worker[2] Trying to dump core for [mysqld.1 - pid: 22654, winpid: 22654] worker[2] Trying to dump core for [mysqld.2 - pid: 22986, winpid: 22986] UG_SYNC = 'now WAIT_FOR wait'; connection node_1; SET DEBUG_SYNC = 'now SIGNAL continue'; connection node_1c; UPDATE t1 SET f2 = 'x' WHERE f1 = 3; connection node_1a; COMMIT; connection node_1b; ERROR 40001: Deadlock found when trying to get lock; try restarting transaction connection node_1; SELECT * FROM t1; f1 f2 1 a 2 a 3 x 4 a 5 a connection node_2; SELECT * FROM t1; More results from queries before failure can be found in /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/mysql-wsrep-features#165.log - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.mysql-wsrep-features#165-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#165-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 2019-02-11 12:41:30 2 [Warning] WSREP: Event 3 Write_rows_v1 apply failed: 121, seqno 12 galera_sr.galera_sr_shutdown_master 'innodb' w1 [ pass ] 31297 galera_sr.mysql-wsrep-features#22 'innodb' w4 [ pass ] 10152 galera_sr.mysql-wsrep-features#27 'innodb' w4 [ pass ] 9623 galera_sr.mysql-wsrep-features#35 'innodb' w4 [ pass ] 9717 galera_sr.galera_sr_shutdown_slave 'innodb' w1 [ pass ] 35199 galera_sr.mysql-wsrep-features#93 'innodb' w4 [ pass ] 7227 galera_sr.galera_sr_table_contents 'innodb' w1 [ fail ] Test ended at 2019-02-11 12:42:39 CURRENT_TEST: galera_sr.galera_sr_table_contents cp: cannot stat 'std_data/binlog-header.log': No such file or directory mysqltest: In included file "./suite/galera/include/galera_dump_sr_table.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/galera_sr_table_contents.test at line 21: At line 20: exec of 'cp std_data/binlog-header.log /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/sr_binlog.log' failed, error: 256, status: 1, errno: 11 Output from before failure: Warnings: Warning 1287 ' INTO FROM...' instead The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t1 VALUES (1),(2),(3); CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t2 VALUES (1),(2),(3); SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; Start of Simple Insert INSERT INTO t1 VALUES (4); Warnings: Warning 1287 ' INTO FROM...' instead - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_table_contents-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_table_contents-innodb/' galera_sr.mysql-wsrep-features#96 'innodb' w4 [ pass ] 12116 galera_sr.galera_sr_transaction_replay 'innodb' w1 [ pass ] 14767 galera_sr.galera_sr_kill_all_norecovery 'innodb' w4 [ pass ] 32144 MTR's internal check of the test case 'galera_sr.galera_sr_kill_all_norecovery' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16081 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result 2019-02-11 12:43:10.986178000 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.reject 2019-02-11 12:43:43.465178000 +0200 @@ -77,7 +77,7 @@ ARIA_USED_FOR_TEMP_TABLES ON AUTOCOMMIT ON AUTOMATIC_SP_PRIVILEGES ON -AUTO_INCREMENT_OFFSET 2 +AUTO_INCREMENT_OFFSET 1 BACK_LOG 80 BASEDIR /home/stepan/galera/git/10.4/server BIG_TABLES OFF mysqltest: Result content mismatch not ok galera_sr.galera_sr_kill_slave 'innodb' w4 [ pass ] 39367 galera_sr.galera_sr_load_data 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 12:51:44 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr_load_data.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 512; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; connection node_2; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.22655' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_load_data-innodb/mysqld.1/data/core.22655 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 22655] [New LWP 22685] [New LWP 22660] [New LWP 22699] [New LWP 22700] [New LWP 22701] [New LWP 22714] [New LWP 22709] [New LWP 22717] [New LWP 22712] [New LWP 22721] [New LWP 22713] [New LWP 22836] [New LWP 22719] [New LWP 22837] [New LWP 22722] [New LWP 22703] [New LWP 22704] [New LWP 22706] [New LWP 22711] [New LWP 22718] [New LWP 22720] [New LWP 23819] [New LWP 22659] [New LWP 22698] [New LWP 22702] [New LWP 22705] [New LWP 22708] [New LWP 22715] [New LWP 22723] [New LWP 22724] [New LWP 22745] [New LWP 22707] [New LWP 22716] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f638407f9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f638407f9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557a20342368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557a1fb80503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f638242e20d in poll () from /lib64/libc.so.6 #5 0x0000557a1f720a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000557a1f71fd25 in mysqld_main (argc=161, argv=0x557a22471d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000557a1f713ec5 in main (argc=23, argv=0x7fff07d93828) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f634a7fc700 (LWP 22716)): #0 0x00007f6384081d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000557a1fe7eacb in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000557a1fe7ee21 in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000557a200c73a2 in pfs_os_file_flush_func (file=..., src_file=0x557a20741a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000557a200ca11d in fil_flush_low (space=0x557a229427c0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000557a200d59f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000557a1fe5a1cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000557a1fe5ac31 in log_write_up_to (lsn=6048743, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000557a1fe5ad61 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x0000557a1ff78430 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #10 0x0000557a1ff789f3 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #11 0x0000557a1ff79551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f6356ffd700 (LWP 22707)): #0 0x00007f6384081d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000557a1fe7eacb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000557a1fe7ee21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000557a200c73a2 in pfs_os_file_flush_func (file=..., src_file=0x557a20741a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000557a200ca11d in fil_flush_low (space=0x557a22942420, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000557a200d59f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000557a200d5c7e in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x0000557a20052002 in buf_dblwr_update (bpage=0x7f63719fb350, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x0000557a200599ef in buf_flush_write_complete (bpage=0x7f63719fb350, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x0000557a200486a9 in buf_page_io_complete (bpage=0x7f63719fb350, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x0000557a200d5751 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x0000557a1ff7b909 in io_handler_thread (arg=0x557a21102640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f63717f0700 (LWP 22745)): #0 0x00007f638237240a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000557a1f713f44 in my_sigwait (set=0x7f63717efcf0, sig=0x7f63717efcb4, code=0x7f63717efcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000557a1f71a584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000557a202da45f in pfs_spawn_thread (arg=0x557a226fbfb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f6335ffb700 (LWP 22724)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a20347997 in safe_cond_wait (cond=0x7f6335ffac70, mp=0x7f6335ffacb0, file=0x557a20615c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557a1fda1487 in inline_mysql_cond_wait (that=0x7f6335ffac70, mutex=0x7f6335ffacb0, src_file=0x557a206175d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557a1fda60c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000557a202da45f in pfs_spawn_thread (arg=0x557a22979910) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f63367fc700 (LWP 22723)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe8a0ff in os_event::wait (this=0x557a22738a10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557a1fe89b85 in os_event::wait_low (this=0x557a22738a10, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557a1fe89ef6 in os_event_wait_low (event=0x557a22738a10, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557a2003eb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f634affd700 (LWP 22715)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe89a1a in os_event::timed_wait (this=0x557a228d8a90, abstime=0x7f634affcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557a1fe89d31 in os_event::wait_time_low (this=0x557a228d8a90, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557a1fe89ed1 in os_event_wait_time_low (event=0x557a228d8a90, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557a1ffd82fb in ib_wqueue_timedwait (wq=0x557a228d8960, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000557a20118ea4 in fts_optimize_thread (arg=0x557a228d8960) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f63567fc700 (LWP 22708)): #0 0x00007f6383c4b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557a1fe7d41c in LinuxAIOHandler::collect (this=0x7f63567fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557a1fe7d945 in LinuxAIOHandler::poll (this=0x7f63567fbb00, m1=0x7f63567fbba8, m2=0x7f63567fbbb0, request=0x7f63567fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557a1fe7daa8 in os_aio_linux_handler (global_segment=5, m1=0x7f63567fbba8, m2=0x7f63567fbbb0, request=0x7f63567fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557a1fe825cc in os_aio_handler (segment=5, m1=0x7f63567fbba8, m2=0x7f63567fbbb0, request=0x7f63567fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557a200d54a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557a1ff7b909 in io_handler_thread (arg=0x557a21102648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f6357fff700 (LWP 22705)): #0 0x00007f6383c4b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557a1fe7d41c in LinuxAIOHandler::collect (this=0x7f6357ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557a1fe7d945 in LinuxAIOHandler::poll (this=0x7f6357ffeb00, m1=0x7f6357ffeba8, m2=0x7f6357ffebb0, request=0x7f6357ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557a1fe7daa8 in os_aio_linux_handler (global_segment=2, m1=0x7f6357ffeba8, m2=0x7f6357ffebb0, request=0x7f6357ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557a1fe825cc in os_aio_handler (segment=2, m1=0x7f6357ffeba8, m2=0x7f6357ffebb0, request=0x7f6357ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557a200d54a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557a1ff7b909 in io_handler_thread (arg=0x557a21102630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f6372e42700 (LWP 22702)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a20347c99 in safe_cond_timedwait (cond=0x557a21913ea0 , mp=0x557a21913de0 , abstime=0x7f6372e41cb0, file=0x557a2078b610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557a201b1acb in inline_mysql_cond_timedwait (that=0x557a21913ea0 , mutex=0x557a21913de0 , abstime=0x7f6372e41cb0, src_file=0x557a2078b658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557a201b1fef in my_service_thread_sleep (control=0x557a20ec8b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000557a201a6488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000557a202da45f in pfs_spawn_thread (arg=0x557a226f0880) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f63741f7700 (LWP 22698)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6377c90867 in gu::Lock::wait (cond=..., this=0x7f63741f6850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6377c933ab in RecvBuf::front (this=this@entry=0x557a226308a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f6377c90f46 in gcomm_recv (backend=, msg=0x557a225f8830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f6377c84eec in core_msg_recv (backend=backend@entry=0x557a225f8978, recv_msg=recv_msg@entry=0x557a225f8830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f6377c88164 in gcs_core_recv (conn=0x557a225f87d0, recv_act=recv_act@entry=0x7f63741f6d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f6377c8f3d8 in gcs_recv_thread (arg=0x557a225f8550) at gcs/src/gcs.cpp:1398 #7 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f637cad3700 (LWP 22659)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a20347c99 in safe_cond_timedwait (cond=0x557a21926640 , mp=0x557a21926580 , abstime=0x7f637cad2d70, file=0x557a207cf840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557a20348c53 in inline_mysql_cond_timedwait (that=0x557a21926640 , mutex=0x557a21926580 , abstime=0x7f637cad2d70, src_file=0x557a207cf888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557a203497b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000557a202da45f in pfs_spawn_thread (arg=0x557a225b9730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f63716c4700 (LWP 23819)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a20347c99 in safe_cond_timedwait (cond=0x557a210871a0 , mp=0x557a21089ba0 , abstime=0x7f63716c3c60, file=0x557a204034c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557a1f714845 in inline_mysql_cond_timedwait (that=0x557a210871a0 , mutex=0x557a21089ba0 , abstime=0x7f63716c3c60, src_file=0x557a20405538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557a1f71961b in cache_thread (thd=0x7f6300000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000557a1f7198c9 in one_thread_per_connection_end (thd=0x7f6300000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000557a1f9b60f3 in do_handle_one_connection (connect=0x557a229cfc90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000557a1f9b5dad in handle_one_connection (arg=0x557a229cfc90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000557a202da45f in pfs_spawn_thread (arg=0x557a22a2f030) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f6337fff700 (LWP 22720)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe8a0ff in os_event::wait (this=0x557a22738690) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557a1fe89b85 in os_event::wait_low (this=0x557a22738690, reset_sig_count=9535) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557a1fe89ef6 in os_event_wait_low (event=0x557a22738690, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557a1ff75df3 in srv_resume_thread (slot=0x557a20ec4780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557a1ff799f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f63497fa700 (LWP 22718)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe8a0ff in os_event::wait (this=0x557a22738790) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557a1fe89b85 in os_event::wait_low (this=0x557a22738790, reset_sig_count=8361) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557a1fe89ef6 in os_event_wait_low (event=0x557a22738790, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557a1ff75df3 in srv_resume_thread (slot=0x557a20ec47f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557a1ff799f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f6354ff9700 (LWP 22711)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe89a1a in os_event::timed_wait (this=0x557a22910d40, abstime=0x7f6354ff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557a1fe89d31 in os_event::wait_time_low (this=0x557a22910d40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557a1fe89ed1 in os_event_wait_time_low (event=0x557a22910d40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557a1fe5688e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f63577fe700 (LWP 22706)): #0 0x00007f6383c4b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557a1fe7d41c in LinuxAIOHandler::collect (this=0x7f63577fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557a1fe7d945 in LinuxAIOHandler::poll (this=0x7f63577fdb00, m1=0x7f63577fdba8, m2=0x7f63577fdbb0, request=0x7f63577fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557a1fe7daa8 in os_aio_linux_handler (global_segment=3, m1=0x7f63577fdba8, m2=0x7f63577fdbb0, request=0x7f63577fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557a1fe825cc in os_aio_handler (segment=3, m1=0x7f63577fdba8, m2=0x7f63577fdbb0, request=0x7f63577fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557a200d54a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557a1ff7b909 in io_handler_thread (arg=0x557a21102638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f6370def700 (LWP 22704)): #0 0x00007f6383c4b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557a1fe7d41c in LinuxAIOHandler::collect (this=0x7f6370deeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557a1fe7d945 in LinuxAIOHandler::poll (this=0x7f6370deeb00, m1=0x7f6370deeba8, m2=0x7f6370deebb0, request=0x7f6370deebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557a1fe7daa8 in os_aio_linux_handler (global_segment=1, m1=0x7f6370deeba8, m2=0x7f6370deebb0, request=0x7f6370deebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557a1fe825cc in os_aio_handler (segment=1, m1=0x7f6370deeba8, m2=0x7f6370deebb0, request=0x7f6370deebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557a200d54a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557a1ff7b909 in io_handler_thread (arg=0x557a21102628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f63715f0700 (LWP 22703)): #0 0x00007f6383c4b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557a1fe7d41c in LinuxAIOHandler::collect (this=0x7f63715efb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557a1fe7d945 in LinuxAIOHandler::poll (this=0x7f63715efb00, m1=0x7f63715efba8, m2=0x7f63715efbb0, request=0x7f63715efbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557a1fe7daa8 in os_aio_linux_handler (global_segment=0, m1=0x7f63715efba8, m2=0x7f63715efbb0, request=0x7f63715efbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557a1fe825cc in os_aio_handler (segment=0, m1=0x7f63715efba8, m2=0x7f63715efbb0, request=0x7f63715efbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557a200d54a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000557a1ff7b909 in io_handler_thread (arg=0x557a21102620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f6336ffd700 (LWP 22722)): #0 0x00007f6384081e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000557a1fe8a401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557a20030a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f637175a700 (LWP 22837)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6377c8cb1f in _gcs_sm_enqueue_common (sm=sm@entry=0x7f6376e7d010, cond=cond@entry=0x7f6371756b58, block=block@entry=true, tail=tail@entry=2377) at gcs/src/gcs_sm.hpp:222 #2 0x00007f6377c8ce86 in gcs_sm_enter (sm=0x7f6376e7d010, cond=cond@entry=0x7f6371756b58, scheduled=scheduled@entry=true, block=block@entry=true) at gcs/src/gcs_sm.hpp:363 #3 0x00007f6377c8d6f6 in gcs_replv (conn=0x557a225f8550, act_in=0x7f6371756c80, act=act@entry=0x7f6371756c60, scheduled=) at gcs/src/gcs.cpp:1797 #4 0x00007f6377cdfb7e in galera::Gcs::replv (this=this@entry=0x557a225f0988, actv=..., act=..., scheduled=scheduled@entry=true) at galera/src/galera_gcs.hpp:138 #5 0x00007f6377cd51f1 in galera::ReplicatorSMM::replicate (this=0x557a225f0280, trx=..., meta=0x7f6371757308) at galera/src/replicator_smm.cpp:661 #6 0x00007f6377cf1932 in galera_certify (gh=, conn_id=, trx_handle=, flags=, meta=0x7f6371757308) at galera/src/wsrep_provider.cpp:535 #7 0x0000557a203fc5e8 in wsrep::wsrep_provider_v26::certify (this=0x557a22577360, client_id=..., ws_handle=..., flags=0, ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:737 #8 0x0000557a203f234e in wsrep::transaction::certify_fragment (this=0x7f63040069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1140 #9 0x0000557a203f1c37 in wsrep::transaction::streaming_step (this=0x7f63040069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #10 0x0000557a203eeff2 in wsrep::transaction::after_row (this=0x7f63040069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #11 0x0000557a1fb983a8 in wsrep::client_state::after_row (this=0x7f6304006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #12 0x0000557a1fb81625 in wsrep_after_row (thd=0x7f6304000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #13 0x0000557a1fb9352a in wsrep_after_row (thd=0x7f6304000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #14 0x0000557a1fb9391b in handler::ha_write_row (this=0x7f63040a4518, buf=0x7f63040a5be0 "\377E\033") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #15 0x0000557a1f809443 in write_record (thd=0x7f6304000b00, table=0x7f630426c2e0, info=0x7f6371757b10) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #16 0x0000557a1f83c3c3 in read_sep_field (thd=0x7f6304000b00, info=..., table_list=0x7f63040153a0, fields_vars=..., set_fields=..., set_values=..., read_info=..., enclosed=..., skip_lines=0, ignore_check_option_errors=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:1152 #17 0x0000557a1f83a97e in mysql_load (thd=0x7f6304000b00, ex=0x7f6304015318, table_list=0x7f63040153a0, fields_vars=..., set_fields=..., set_values=..., handle_duplicates=DUP_ERROR, ignore=false, read_file_from_client=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:660 #18 0x0000557a1f85175d in mysql_execute_command (thd=0x7f6304000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5148 #19 0x0000557a1f85b935 in mysql_parse (thd=0x7f6304000b00, rawbuf=0x7f6304015168 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7f63717590d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #20 0x0000557a1f85af68 in wsrep_mysql_parse (thd=0x7f6304000b00, rawbuf=0x7f6304015168 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7f63717590d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #21 0x0000557a1f846e25 in dispatch_command (command=COM_QUERY, thd=0x7f6304000b00, packet=0x7f6304028201 "", packet_length=114, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #22 0x0000557a1f84580d in do_command (thd=0x7f6304000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #23 0x0000557a1f9b6069 in do_handle_one_connection (connect=0x557a22aa7860) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #24 0x0000557a1f9b5dad in handle_one_connection (arg=0x557a22aa7860) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #25 0x0000557a202da45f in pfs_spawn_thread (arg=0x557a22ac9670) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #26 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #27 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f6348ff9700 (LWP 22719)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe8a0ff in os_event::wait (this=0x557a22738710) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557a1fe89b85 in os_event::wait_low (this=0x557a22738710, reset_sig_count=8523) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557a1fe89ef6 in os_event_wait_low (event=0x557a22738710, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557a1ff75df3 in srv_resume_thread (slot=0x557a20ec47b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557a1ff799f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f63717a5700 (LWP 22836)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a20347997 in safe_cond_wait (cond=0x557a210871e0 , mp=0x557a2108a560 , file=0x557a20412cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557a1f7454ee in inline_mysql_cond_wait (that=0x557a210871e0 , mutex=0x557a2108a560 , src_file=0x557a20413830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000557a1f746d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000557a202da45f in pfs_spawn_thread (arg=0x557a226fc4a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f634bfff700 (LWP 22713)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe89a1a in os_event::timed_wait (this=0x557a22738890, abstime=0x7f634bffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557a1fe89d31 in os_event::wait_time_low (this=0x557a22738890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557a1fe89ed1 in os_event_wait_time_low (event=0x557a22738890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557a1ff77ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f63377fe700 (LWP 22721)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe8a0ff in os_event::wait (this=0x557a22738910) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557a1fe89b85 in os_event::wait_low (this=0x557a22738910, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557a1fe89ef6 in os_event_wait_low (event=0x557a22738910, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557a20055eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f63557fa700 (LWP 22712)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe89a1a in os_event::timed_wait (this=0x557a22738810, abstime=0x7f63557f9ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557a1fe89d31 in os_event::wait_time_low (this=0x557a22738810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557a1fe89ed1 in os_event_wait_time_low (event=0x557a22738810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557a1ff77ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f6349ffb700 (LWP 22717)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe8a0ff in os_event::wait (this=0x557a22738610) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557a1fe89b85 in os_event::wait_low (this=0x557a22738610, reset_sig_count=2268) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557a1fe89ef6 in os_event_wait_low (event=0x557a22738610, reset_sig_count=2268) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557a1ff75df3 in srv_resume_thread (slot=0x557a20ec4748 , sig_count=2268, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557a1ff79e04 in srv_purge_coordinator_suspend (slot=0x557a20ec4748 , rseg_history_len=31) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000557a1ff7a09a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f6355ffb700 (LWP 22709)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe89a1a in os_event::timed_wait (this=0x557a22738990, abstime=0x7f6355ffaaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557a1fe89d31 in os_event::wait_time_low (this=0x557a22738990, time_in_usec=249000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557a1fe89ed1 in os_event_wait_time_low (event=0x557a22738990, time_in_usec=249000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557a2005fd1d in pc_sleep_if_needed (next_loop_time=1549882242942, sig_count=2, cur_time=1549882242693) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000557a20060c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f634b7fe700 (LWP 22714)): #0 0x00007f638407ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a1fe89a1a in os_event::timed_wait (this=0x557a22942e00, abstime=0x7f634b7fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557a1fe89d31 in os_event::wait_time_low (this=0x557a22942e00, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557a1fe89ed1 in os_event_wait_time_low (event=0x557a22942e00, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557a200bf142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f63739f6700 (LWP 22701)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6377baa132 in fifo_lock_get (q=q@entry=0x557a2261ebd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f6377baa7a1 in gu_fifo_get_head (q=0x557a2261ebd0, err=0x7f63739f506c) at galerautils/src/gu_fifo.c:330 #3 0x00007f6377c8c0cc in gcs_recv (conn=0x557a225f8550, action=0x7f63739f50b0) at gcs/src/gcs.cpp:2024 #4 0x00007f6377cb916c in galera::GcsActionSource::process (this=0x557a225f7f30, recv_ctx=0x7f63739f54d0, exit_loop=@0x7f63739f529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f6377cd8033 in galera::ReplicatorSMM::async_recv (this=0x557a225f0280, recv_ctx=0x7f63739f54d0) at galera/src/replicator_smm.cpp:383 #6 0x0000557a203fc276 in wsrep::wsrep_provider_v26::run_applier (this=0x557a22577360, applier_ctx=0x7f63739f54d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000557a1fa9f636 in wsrep_replication_process (thd=0x7f6360000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000557a1fa91282 in start_wsrep_THD (arg=0x557a226492a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f637c05e700 (LWP 22700)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a20347997 in safe_cond_wait (cond=0x7f6358015608, mp=0x7f6358015558, file=0x557a204b1b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557a1fa9f088 in inline_mysql_cond_wait (that=0x7f6358015608, mutex=0x7f6358015558, src_file=0x557a204b1e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557a1faa1d20 in Wsrep_thd_queue::pop_front (this=0x7f6358015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000557a1faa0b00 in wsrep_post_rollback_process (post_rollbacker=0x7f6358000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000557a1fa91282 in start_wsrep_THD (arg=0x557a22640320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f637c0a9700 (LWP 22699)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557a20347997 in safe_cond_wait (cond=0x7f6368017b68, mp=0x7f6368017ab8, file=0x557a204b1b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557a1fa9f088 in inline_mysql_cond_wait (that=0x7f6368017b68, mutex=0x7f6368017ab8, src_file=0x557a204b1e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557a1faa1d20 in Wsrep_thd_queue::pop_front (this=0x7f6368017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000557a1faa0740 in wsrep_rollback_process (rollbacker=0x7f6368001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000557a1fa91282 in start_wsrep_THD (arg=0x557a22649e90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f6376e7c700 (LWP 22660)): #0 0x00007f638407e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6377cb5fd7 in gu::Lock::wait (cond=..., this=0x7f6376e7bbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6377cb6069 in galera::ServiceThd::thd_func (arg=0x557a225f0998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f63749f8700 (LWP 22685)): #0 0x00007f6382439483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6377c576f8 in asio::detail::epoll_reactor::run (this=0x557a22631040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f6377c5bdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x557a22630ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f6377c5c0a2 in asio::detail::task_io_service::run (this=0x557a22630ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f6377c73afd in asio::io_service::run (this=0x557a2262ffe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f6377c74251 in gcomm::AsioProtonet::event_loop (this=0x557a2262ff40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f6377c922ed in GCommConn::run (this=0x557a226306c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f6377c95399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f638407add5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6382438ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f638448b880 (LWP 22655)): #0 0x00007f638407f9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557a20342368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557a1fb80503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f638242e20d in poll () from /lib64/libc.so.6 #5 0x0000557a1f720a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000557a1f71fd25 in mysqld_main (argc=161, argv=0x557a22471d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000557a1f713ec5 in main (argc=23, argv=0x7fff07d93828) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.22988' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_load_data-innodb/mysqld.2/data/core.22988 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 22988] [New LWP 23028] [New LWP 23537] [New LWP 23480] [New LWP 23482] [New LWP 23533] [New LWP 23534] [New LWP 23544] [New LWP 23545] [New LWP 23547] [New LWP 23550] [New LWP 23539] [New LWP 23541] [New LWP 23007] [New LWP 23540] [New LWP 23538] [New LWP 23546] [New LWP 23543] [New LWP 23487] [New LWP 23486] [New LWP 23485] [New LWP 23027] [New LWP 23063] [New LWP 23481] [New LWP 23483] [New LWP 23484] [New LWP 23542] [New LWP 23536] [New LWP 23535] [New LWP 23532] [New LWP 23065] [New LWP 23064] [New LWP 23066] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f06eee039d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f06eee039d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559da7931368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559da716f503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f06ed1b220d in poll () from /lib64/libc.so.6 #5 0x0000559da6d0fa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559da6d0ed25 in mysqld_main (argc=162, argv=0x559da9a9ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559da6d02ec5 in main (argc=23, argv=0x7ffde54670d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f06e4191700 (LWP 23066)): #0 0x00007f06eee05d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000559da746dacb in os_file_fsync_posix (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000559da746de21 in os_file_flush_func (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000559da76b63a2 in pfs_os_file_flush_func (file=..., src_file=0x559da7d30a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559da76b911d in fil_flush_low (space=0x559da9f723c0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000559da76c49f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000559da74491cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000559da7449c31 in log_write_up_to (lsn=3458078, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000559da75b4d91 in trx_flush_log_if_needed_low (lsn=3458078) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000559da75b4dda in trx_flush_log_if_needed (lsn=3458078, trx=0x7f06d4992140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000559da75b6c03 in trx_commit_complete_for_mysql (trx=0x7f06d4992140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000559da739b514 in innobase_commit (hton=0x559da9d3c920, thd=0x7f06c0000b00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x0000559da7175039 in commit_one_phase_2 (thd=0x7f06c0000b00, all=true, trans=0x7f06c0004050, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000559da7174f3b in ha_commit_one_phase (thd=0x7f06c0000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000559da71742b7 in ha_commit_trans (thd=0x7f06c0000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000559da6fb9e8e in trans_commit (thd=0x7f06c0000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000559da706d3d6 in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7f06e41904d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:264 #17 0x0000559da79d3a51 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:67 #18 0x0000559da79d44b1 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:241 #19 0x0000559da79d7807 in wsrep::server_state::on_apply (this=0x559da9be7740, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:944 #20 0x0000559da79eced1 in wsrep::high_priority_service::apply (this=0x7f06e41904d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x0000559da79ea5b8 in (anonymous namespace)::apply_cb (ctx=0x7f06e41904d0, wsh=0x7f06e418f3e0, flags=0, buf=0x7f06e418f420, meta=0x7f06e418f6b0, exit_loop=0x7f06e418f5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007f06e6cf63b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7f06c004b0c0, recv_ctx=recv_ctx@entry=0x7f06e41904d0, apply_cb=0x559da79ea395 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f06e418f5ef: false) at galera/src/trx_handle.cpp:418 #23 0x00007f06e6d28d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x559da9c1d1f0, recv_ctx=recv_ctx@entry=0x7f06e41904d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007f06e6d2ea40 in galera::ReplicatorSMM::process_trx (this=0x559da9c1d1f0, recv_ctx=0x7f06e41904d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #25 0x00007f06e6d0fc6b in galera::GcsActionSource::process_writeset (this=this@entry=0x559da9c4fe60, recv_ctx=0x7f06e41904d0, act=..., exit_loop=@0x7f06e419029e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007f06e6d0fe45 in galera::GcsActionSource::dispatch (this=this@entry=0x559da9c4fe60, recv_ctx=recv_ctx@entry=0x7f06e41904d0, act=..., exit_loop=@0x7f06e419029e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007f06e6d1021d in galera::GcsActionSource::process (this=0x559da9c4fe60, recv_ctx=0x7f06e41904d0, exit_loop=@0x7f06e419029e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007f06e6d2f033 in galera::ReplicatorSMM::async_recv (this=0x559da9c1d1f0, recv_ctx=0x7f06e41904d0) at galera/src/replicator_smm.cpp:383 #29 0x0000559da79eb276 in wsrep::wsrep_provider_v26::run_applier (this=0x559da9b92e50, applier_ctx=0x7f06e41904d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x0000559da708e636 in wsrep_replication_process (thd=0x7f06c0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x0000559da7080282 in start_wsrep_THD (arg=0x559da9c3dd70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #32 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f06e4227700 (LWP 23064)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7936997 in safe_cond_wait (cond=0x7f06d0017b68, mp=0x7f06d0017ab8, file=0x559da7aa0b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559da708e088 in inline_mysql_cond_wait (that=0x7f06d0017b68, mutex=0x7f06d0017ab8, src_file=0x559da7aa0e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559da7090d20 in Wsrep_thd_queue::pop_front (this=0x7f06d0017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559da708f740 in wsrep_rollback_process (rollbacker=0x7f06d0001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000559da7080282 in start_wsrep_THD (arg=0x559da9c79b20) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f06e41dc700 (LWP 23065)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7936997 in safe_cond_wait (cond=0x7f06c8015608, mp=0x7f06c8015558, file=0x559da7aa0b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559da708e088 in inline_mysql_cond_wait (that=0x7f06c8015608, mutex=0x7f06c8015558, src_file=0x559da7aa0e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559da7090d20 in Wsrep_thd_queue::pop_front (this=0x7f06c8015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559da708fb00 in wsrep_post_rollback_process (post_rollbacker=0x7f06c8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000559da7080282 in start_wsrep_THD (arg=0x559da9c79510) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f06a77fe700 (LWP 23532)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7478a1a in os_event::timed_wait (this=0x559da9f40940, abstime=0x7f06a77fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559da7478d31 in os_event::wait_time_low (this=0x559da9f40940, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559da7478ed1 in os_event_wait_time_low (event=0x559da9f40940, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559da744588e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f06a67fc700 (LWP 23535)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7478a1a in os_event::timed_wait (this=0x559da9f72a00, abstime=0x7f06a67fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559da7478d31 in os_event::wait_time_low (this=0x559da9f72a00, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559da7478ed1 in os_event_wait_time_low (event=0x559da9f72a00, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559da76ae142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f06a5ffb700 (LWP 23536)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7478a1a in os_event::timed_wait (this=0x559da9f10cc0, abstime=0x7f06a5ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559da7478d31 in os_event::wait_time_low (this=0x559da9f10cc0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559da7478ed1 in os_event_wait_time_low (event=0x559da9f10cc0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559da75c72fb in ib_wqueue_timedwait (wq=0x559da9f10b90, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559da7707ea4 in fts_optimize_thread (arg=0x559da9f10b90) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f06967fc700 (LWP 23542)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da74790ff in os_event::wait (this=0x559da9d68510) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559da7478b85 in os_event::wait_low (this=0x559da9d68510, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559da7478ef6 in os_event_wait_low (event=0x559da9d68510, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559da7644eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f06ae3fc700 (LWP 23484)): #0 0x00007f06ee9cf644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559da746c41c in LinuxAIOHandler::collect (this=0x7f06ae3fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559da746c945 in LinuxAIOHandler::poll (this=0x7f06ae3fbb00, m1=0x7f06ae3fbba8, m2=0x7f06ae3fbbb0, request=0x7f06ae3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559da746caa8 in os_aio_linux_handler (global_segment=3, m1=0x7f06ae3fbba8, m2=0x7f06ae3fbbb0, request=0x7f06ae3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559da74715cc in os_aio_handler (segment=3, m1=0x7f06ae3fbba8, m2=0x7f06ae3fbbb0, request=0x7f06ae3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559da76c44a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559da756a909 in io_handler_thread (arg=0x559da86f1638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f06aebfd700 (LWP 23483)): #0 0x00007f06ee9cf644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559da746c41c in LinuxAIOHandler::collect (this=0x7f06aebfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559da746c945 in LinuxAIOHandler::poll (this=0x7f06aebfcb00, m1=0x7f06aebfcba8, m2=0x7f06aebfcbb0, request=0x7f06aebfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559da746caa8 in os_aio_linux_handler (global_segment=2, m1=0x7f06aebfcba8, m2=0x7f06aebfcbb0, request=0x7f06aebfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559da74715cc in os_aio_handler (segment=2, m1=0x7f06aebfcba8, m2=0x7f06aebfcbb0, request=0x7f06aebfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559da76c44a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559da756a909 in io_handler_thread (arg=0x559da86f1630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f06afbff700 (LWP 23481)): #0 0x00007f06ee9cf644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559da746c41c in LinuxAIOHandler::collect (this=0x7f06afbfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559da746c945 in LinuxAIOHandler::poll (this=0x7f06afbfeb00, m1=0x7f06afbfeba8, m2=0x7f06afbfebb0, request=0x7f06afbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559da746caa8 in os_aio_linux_handler (global_segment=0, m1=0x7f06afbfeba8, m2=0x7f06afbfebb0, request=0x7f06afbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559da74715cc in os_aio_handler (segment=0, m1=0x7f06afbfeba8, m2=0x7f06afbfebb0, request=0x7f06afbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559da76c44a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559da756a909 in io_handler_thread (arg=0x559da86f1620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f06d77fd700 (LWP 23063)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f06e6ce7867 in gu::Lock::wait (cond=..., this=0x7f06d77fc850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f06e6cea3ab in RecvBuf::front (this=this@entry=0x559da9c57818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f06e6ce7f46 in gcomm_recv (backend=, msg=0x559da9c25720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f06e6cdbeec in core_msg_recv (backend=backend@entry=0x559da9c25868, recv_msg=recv_msg@entry=0x559da9c25720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f06e6cdf164 in gcs_core_recv (conn=0x559da9c256c0, recv_act=recv_act@entry=0x7f06d77fcd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f06e6ce63d8 in gcs_recv_thread (arg=0x559da9c25440) at gcs/src/gcs.cpp:1398 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f06e66ab700 (LWP 23027)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f06e6d0cfd7 in gu::Lock::wait (cond=..., this=0x7f06e66aabf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f06e6d0d069 in galera::ServiceThd::thd_func (arg=0x559da9c1d908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f06adbfb700 (LWP 23485)): #0 0x00007f06ee9cf644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559da746c41c in LinuxAIOHandler::collect (this=0x7f06adbfab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559da746c945 in LinuxAIOHandler::poll (this=0x7f06adbfab00, m1=0x7f06adbfaba8, m2=0x7f06adbfabb0, request=0x7f06adbfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559da746caa8 in os_aio_linux_handler (global_segment=4, m1=0x7f06adbfaba8, m2=0x7f06adbfabb0, request=0x7f06adbfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559da74715cc in os_aio_handler (segment=4, m1=0x7f06adbfaba8, m2=0x7f06adbfabb0, request=0x7f06adbfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559da76c44a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559da756a909 in io_handler_thread (arg=0x559da86f1640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f06ad3fa700 (LWP 23486)): #0 0x00007f06ee9cf644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559da746c41c in LinuxAIOHandler::collect (this=0x7f06ad3f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559da746c945 in LinuxAIOHandler::poll (this=0x7f06ad3f9b00, m1=0x7f06ad3f9ba8, m2=0x7f06ad3f9bb0, request=0x7f06ad3f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559da746caa8 in os_aio_linux_handler (global_segment=5, m1=0x7f06ad3f9ba8, m2=0x7f06ad3f9bb0, request=0x7f06ad3f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559da74715cc in os_aio_handler (segment=5, m1=0x7f06ad3f9ba8, m2=0x7f06ad3f9bb0, request=0x7f06ad3f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559da76c44a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559da756a909 in io_handler_thread (arg=0x559da86f1648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f06acbf9700 (LWP 23487)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7478a1a in os_event::timed_wait (this=0x559da9d68590, abstime=0x7f06acbf8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559da7478d31 in os_event::wait_time_low (this=0x559da9d68590, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559da7478ed1 in os_event_wait_time_low (event=0x559da9d68590, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559da764ed1d in pc_sleep_if_needed (next_loop_time=1549882250534, sig_count=2, cur_time=1549882249534) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000559da764fc56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f0695ffb700 (LWP 23543)): #0 0x00007f06eee05e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559da7479401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559da761fa3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f06d4140700 (LWP 23546)): #0 0x00007f06ed0f640a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000559da6d02f44 in my_sigwait (set=0x7f06d413fcf0, sig=0x7f06d413fcb4, code=0x7f06d413fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000559da6d09584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000559da78c945f in pfs_spawn_thread (arg=0x559da9d2bca0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f06a4ff9700 (LWP 23538)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da74790ff in os_event::wait (this=0x559da9d68210) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559da7478b85 in os_event::wait_low (this=0x559da9d68210, reset_sig_count=2348) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559da7478ef6 in os_event_wait_low (event=0x559da9d68210, reset_sig_count=2348) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559da7564df3 in srv_resume_thread (slot=0x559da84b3748 , sig_count=2348, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559da7568e04 in srv_purge_coordinator_suspend (slot=0x559da84b3748 , rseg_history_len=25) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000559da756909a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f06977fe700 (LWP 23540)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da74790ff in os_event::wait (this=0x559da9d68310) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559da7478b85 in os_event::wait_low (this=0x559da9d68310, reset_sig_count=8789) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559da7478ef6 in os_event_wait_low (event=0x559da9d68310, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559da7564df3 in srv_resume_thread (slot=0x559da84b37b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559da75689f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f06e7857700 (LWP 23007)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7936c99 in safe_cond_timedwait (cond=0x559da8f15640 , mp=0x559da8f15580 , abstime=0x7f06e7856d70, file=0x559da7dbe840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559da7937c53 in inline_mysql_cond_timedwait (that=0x559da8f15640 , mutex=0x559da8f15580 , abstime=0x7f06e7856d70, src_file=0x559da7dbe888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559da79387b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559da78c945f in pfs_spawn_thread (arg=0x559da9be67d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f0696ffd700 (LWP 23541)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da74790ff in os_event::wait (this=0x559da9d68390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559da7478b85 in os_event::wait_low (this=0x559da9d68390, reset_sig_count=8163) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559da7478ef6 in os_event_wait_low (event=0x559da9d68390, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559da7564df3 in srv_resume_thread (slot=0x559da84b37f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559da75689f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f0697fff700 (LWP 23539)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da74790ff in os_event::wait (this=0x559da9d68290) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559da7478b85 in os_event::wait_low (this=0x559da9d68290, reset_sig_count=9724) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559da7478ef6 in os_event_wait_low (event=0x559da9d68290, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559da7564df3 in srv_resume_thread (slot=0x559da84b3780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559da75689f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f06d40aa700 (LWP 23550)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7936c99 in safe_cond_timedwait (cond=0x559da86761a0 , mp=0x559da8678ba0 , abstime=0x7f06d40a9c60, file=0x559da79f24c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559da6d03845 in inline_mysql_cond_timedwait (that=0x559da86761a0 , mutex=0x559da8678ba0 , abstime=0x7f06d40a9c60, src_file=0x559da79f4538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559da6d0861b in cache_thread (thd=0x7f06b8000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000559da6d088c9 in one_thread_per_connection_end (thd=0x7f06b8000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000559da6fa50f3 in do_handle_one_connection (connect=0x559daa0683d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000559da6fa4dad in handle_one_connection (arg=0x559daa0683d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000559da78c945f in pfs_spawn_thread (arg=0x559daa096560) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f06d40f5700 (LWP 23547)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7936997 in safe_cond_wait (cond=0x559da86761e0 , mp=0x559da8679560 , file=0x559da7a01cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559da6d344ee in inline_mysql_cond_wait (that=0x559da86761e0 , mutex=0x559da8679560 , src_file=0x559da7a02830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000559da6d35d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559da78c945f in pfs_spawn_thread (arg=0x559daa0cad30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f0694ff9700 (LWP 23545)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7936997 in safe_cond_wait (cond=0x7f0694ff8c70, mp=0x7f0694ff8cb0, file=0x559da7c04c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559da7390487 in inline_mysql_cond_wait (that=0x7f0694ff8c70, mutex=0x7f0694ff8cb0, src_file=0x559da7c065d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559da73950c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000559da78c945f in pfs_spawn_thread (arg=0x559da9f235d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f06957fa700 (LWP 23544)): #0 0x00007f06eee02965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da74790ff in os_event::wait (this=0x559da9d68610) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559da7478b85 in os_event::wait_low (this=0x559da9d68610, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559da7478ef6 in os_event_wait_low (event=0x559da9d68610, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559da762db26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f06a6ffd700 (LWP 23534)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7478a1a in os_event::timed_wait (this=0x559da9d68490, abstime=0x7f06a6ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559da7478d31 in os_event::wait_time_low (this=0x559da9d68490, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559da7478ed1 in os_event_wait_time_low (event=0x559da9d68490, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559da7566ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f06a7fff700 (LWP 23533)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7478a1a in os_event::timed_wait (this=0x559da9d68410, abstime=0x7f06a7ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559da7478d31 in os_event::wait_time_low (this=0x559da9d68410, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559da7478ed1 in os_event_wait_time_low (event=0x559da9d68410, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559da7566ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f06af3fe700 (LWP 23482)): #0 0x00007f06ee9cf644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559da746c41c in LinuxAIOHandler::collect (this=0x7f06af3fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559da746c945 in LinuxAIOHandler::poll (this=0x7f06af3fdb00, m1=0x7f06af3fdba8, m2=0x7f06af3fdbb0, request=0x7f06af3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559da746caa8 in os_aio_linux_handler (global_segment=1, m1=0x7f06af3fdba8, m2=0x7f06af3fdbb0, request=0x7f06af3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559da74715cc in os_aio_handler (segment=1, m1=0x7f06af3fdba8, m2=0x7f06af3fdbb0, request=0x7f06af3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559da76c44a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559da756a909 in io_handler_thread (arg=0x559da86f1628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f06d5592700 (LWP 23480)): #0 0x00007f06eee02d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559da7936c99 in safe_cond_timedwait (cond=0x559da8f02ea0 , mp=0x559da8f02de0 , abstime=0x7f06d5591cb0, file=0x559da7d7a610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559da77a0acb in inline_mysql_cond_timedwait (that=0x559da8f02ea0 , mutex=0x559da8f02de0 , abstime=0x7f06d5591cb0, src_file=0x559da7d7a658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559da77a0fef in my_service_thread_sleep (control=0x559da84b7b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559da7795488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559da78c945f in pfs_spawn_thread (arg=0x559da9d205f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f06a57fa700 (LWP 23537)): #0 0x00007f06eee05e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559da7479401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559da756835b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000559da756848b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f06d7ffe700 (LWP 23028)): #0 0x00007f06ed1bd483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f06e6cae6f8 in asio::detail::epoll_reactor::run (this=0x559da9c58000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f06e6cb2df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559da9c57ea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f06e6cb30a2 in asio::detail::task_io_service::run (this=0x559da9c57ea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f06e6ccaafd in asio::io_service::run (this=0x559da9c56bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f06e6ccb251 in gcomm::AsioProtonet::event_loop (this=0x559da9c56b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f06e6ce92ed in GCommConn::run (this=0x559da9c57630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f06e6cec399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f06eedfedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f06ed1bcead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f06ef20f880 (LWP 22988)): #0 0x00007f06eee039d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559da7931368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559da716f503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f06ed1b220d in poll () from /lib64/libc.so.6 #5 0x0000559da6d0fa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:64worker[3] Trying to dump core for [mysqltest - pid: 31114, winpid: 31114] worker[3] Trying to dump core for [mysqld.1 - pid: 30295, winpid: 30295] worker[3] Trying to dump core for [mysqld.2 - pid: 30539, winpid: 30539] 17 #6 0x0000559da6d0ed25 in mysqld_main (argc=162, argv=0x559da9a9ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559da6d02ec5 in main (argc=23, argv=0x7ffde54670d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_load_data-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_load_data galera_sr.galera_sr_blob galera_sr.galera_sr_kill_query galera_sr.galera_sr_kill_connection Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_myisam 'innodb' w2 [ pass ] 4269 galera_sr.galera_sr_load_data_splitting 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 12:53:58 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr_load_data_splitting.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 512; SET GLOBAL wsrep_load_data_splitting = TRUE; Warnings: Warning 1287 '@@wsrep_load_data_splitting' is deprecated and will be removed in a future release CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; connection node_2; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.30296' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.1/data/core.30296 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 30296] [New LWP 30342] [New LWP 30297] [New LWP 30339] [New LWP 30345] [New LWP 30341] [New LWP 30340] [New LWP 30403] [New LWP 30349] [New LWP 30343] [New LWP 30406] [New LWP 30346] [New LWP 30410] [New LWP 30405] [New LWP 30411] [New LWP 30408] [New LWP 30415] [New LWP 30413] [New LWP 30416] [New LWP 30414] [New LWP 30417] [New LWP 30443] [New LWP 30442] [New LWP 30409] [New LWP 30412] [New LWP 30407] [New LWP 30404] [New LWP 30348] [New LWP 30298] [New LWP 30300] [New LWP 30344] [New LWP 30347] [New LWP 31176] [New LWP 30350] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f9e253359d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f9e253359d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000560d64493368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000560d63cd1503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9e236e420d in poll () from /lib64/libc.so.6 #5 0x0000560d63871a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000560d63870d25 in mysqld_main (argc=161, argv=0x560d67304d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000560d63864ec5 in main (argc=23, argv=0x7fffffccd148) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f9df2ffd700 (LWP 30350)): #0 0x00007f9e25337d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000560d63fcfacb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000560d63fcfe21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000560d642183a2 in pfs_os_file_flush_func (file=..., src_file=0x560d64892a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000560d6421b11d in fil_flush_low (space=0x560d677d5560, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000560d642269f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000560d641a3cbd in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x0000560d641ae3a5 in buf_flush_end (buf_pool=0x560d676b1250, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x0000560d641ae563 in buf_flush_do_batch (buf_pool=0x560d676b1250, type=BUF_FLUSH_LIST, min_n=2, lsn_limit=18446744073709551615, n=0x7f9df2ffcb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x0000560d641b124d in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x0000560d641b219b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f9dfe9ad700 (LWP 31176)): #0 0x00007f9e25337d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000560d63fcfacb in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000560d63fcfe21 in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000560d642183a2 in pfs_os_file_flush_func (file=..., src_file=0x560d64892a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000560d6421b11d in fil_flush_low (space=0x560d677d5900, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000560d642269f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000560d63fab1cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000560d63fabc31 in log_write_up_to (lsn=5595957, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000560d64116d91 in trx_flush_log_if_needed_low (lsn=5595957) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000560d64116dda in trx_flush_log_if_needed (lsn=5595957, trx=0x7f9dff401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000560d64118c03 in trx_commit_complete_for_mysql (trx=0x7f9dff401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000560d63efd514 in innobase_commit (hton=0x560d6759fb30, thd=0x7f9da405ab40, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x0000560d63cd7039 in commit_one_phase_2 (thd=0x7f9da405ab40, all=true, trans=0x7f9da405e090, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000560d63cd6f3b in ha_commit_one_phase (thd=0x7f9da405ab40, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000560d63cd62b7 in ha_commit_trans (thd=0x7f9da405ab40, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000560d63b1be8e in trans_commit (thd=0x7f9da405ab40) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000560d63edb18c in Wsrep_storage_service::commit (this=0x7f9da40a08f0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x0000560d645434d3 in wsrep::transaction::certify_fragment (this=0x7f9da40069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x0000560d64542c37 in wsrep::transaction::streaming_step (this=0x7f9da40069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x0000560d6453fff2 in wsrep::transaction::after_row (this=0x7f9da40069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x0000560d63ce93a8 in wsrep::client_state::after_row (this=0x7f9da4006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x0000560d63cd2625 in wsrep_after_row (thd=0x7f9da4000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x0000560d63ce452a in wsrep_after_row (thd=0x7f9da4000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x0000560d63ce491b in handler::ha_write_row (this=0x7f9da4026b68, buf=0x7f9da401f830 "\377C\032") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x0000560d6395a443 in write_record (thd=0x7f9da4000b00, table=0x7f9da4025f30, info=0x7f9dfe9aab10) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x0000560d6398d3c3 in read_sep_field (thd=0x7f9da4000b00, info=..., table_list=0x7f9da4014090, fields_vars=..., set_fields=..., set_values=..., read_info=..., enclosed=..., skip_lines=0, ignore_check_option_errors=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:1152 #26 0x0000560d6398b97e in mysql_load (thd=0x7f9da4000b00, ex=0x7f9da4014008, table_list=0x7f9da4014090, fields_vars=..., set_fields=..., set_values=..., handle_duplicates=DUP_ERROR, ignore=false, read_file_from_client=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:660 #27 0x0000560d639a275d in mysql_execute_command (thd=0x7f9da4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5148 #28 0x0000560d639ac935 in mysql_parse (thd=0x7f9da4000b00, rawbuf=0x7f9da4013e58 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7f9dfe9ac0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #29 0x0000560d639abf68 in wsrep_mysql_parse (thd=0x7f9da4000b00, rawbuf=0x7f9da4013e58 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7f9dfe9ac0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #30 0x0000560d63997e25 in dispatch_command (command=COM_QUERY, thd=0x7f9da4000b00, packet=0x7f9da400a431 "", packet_length=114, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #31 0x0000560d6399680d in do_command (thd=0x7f9da4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #32 0x0000560d63b07069 in do_handle_one_connection (connect=0x560d6758f630) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #33 0x0000560d63b06dad in handle_one_connection (arg=0x560d6758f630) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #34 0x0000560d6442b45f in pfs_spawn_thread (arg=0x560d678f4340) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #35 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #36 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f9dfcfaa700 (LWP 30347)): #0 0x00007f9e24f01644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000560d63fce41c in LinuxAIOHandler::collect (this=0x7f9dfcfa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000560d63fce945 in LinuxAIOHandler::poll (this=0x7f9dfcfa9b00, m1=0x7f9dfcfa9ba8, m2=0x7f9dfcfa9bb0, request=0x7f9dfcfa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000560d63fceaa8 in os_aio_linux_handler (global_segment=3, m1=0x7f9dfcfa9ba8, m2=0x7f9dfcfa9bb0, request=0x7f9dfcfa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000560d63fd35cc in os_aio_handler (segment=3, m1=0x7f9dfcfa9ba8, m2=0x7f9dfcfa9bb0, request=0x7f9dfcfa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000560d642264a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000560d640cc909 in io_handler_thread (arg=0x560d65253638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f9dfe7ad700 (LWP 30344)): #0 0x00007f9e24f01644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000560d63fce41c in LinuxAIOHandler::collect (this=0x7f9dfe7acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000560d63fce945 in LinuxAIOHandler::poll (this=0x7f9dfe7acb00, m1=0x7f9dfe7acba8, m2=0x7f9dfe7acbb0, request=0x7f9dfe7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000560d63fceaa8 in os_aio_linux_handler (global_segment=0, m1=0x7f9dfe7acba8, m2=0x7f9dfe7acbb0, request=0x7f9dfe7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000560d63fd35cc in os_aio_handler (segment=0, m1=0x7f9dfe7acba8, m2=0x7f9dfe7acbb0, request=0x7f9dfe7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000560d642264a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000560d640cc909 in io_handler_thread (arg=0x560d65253620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f9e15b7b700 (LWP 30300)): #0 0x00007f9e236ef483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9e1d1e06f8 in asio::detail::epoll_reactor::run (this=0x560d674c4040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f9e1d1e4df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x560d674c3ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f9e1d1e50a2 in asio::detail::task_io_service::run (this=0x560d674c3ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f9e1d1fcafd in asio::io_service::run (this=0x560d674c2fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f9e1d1fd251 in gcomm::AsioProtonet::event_loop (this=0x560d674c2f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f9e1d21b2ed in GCommConn::run (this=0x560d674c36c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f9e1d21e399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f9e17fff700 (LWP 30298)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9e1d23efd7 in gu::Lock::wait (cond=..., this=0x7f9e17ffebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f9e1d23f069 in galera::ServiceThd::thd_func (arg=0x560d67483998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f9df3fff700 (LWP 30348)): #0 0x00007f9e24f01644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000560d63fce41c in LinuxAIOHandler::collect (this=0x7f9df3ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000560d63fce945 in LinuxAIOHandler::poll (this=0x7f9df3ffeb00, m1=0x7f9df3ffeba8, m2=0x7f9df3ffebb0, request=0x7f9df3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000560d63fceaa8 in os_aio_linux_handler (global_segment=4, m1=0x7f9df3ffeba8, m2=0x7f9df3ffebb0, request=0x7f9df3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000560d63fd35cc in os_aio_handler (segment=4, m1=0x7f9df3ffeba8, m2=0x7f9df3ffebb0, request=0x7f9df3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000560d642264a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000560d640cc909 in io_handler_thread (arg=0x560d65253640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f9df17fa700 (LWP 30404)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdaa1a in os_event::timed_wait (this=0x560d675cb950, abstime=0x7f9df17f9ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000560d63fdad31 in os_event::wait_time_low (this=0x560d675cb950, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000560d63fdaed1 in os_event_wait_time_low (event=0x560d675cb950, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000560d640c8ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f9de37fe700 (LWP 30407)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdaa1a in os_event::timed_wait (this=0x560d6781cc80, abstime=0x7f9de37fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000560d63fdad31 in os_event::wait_time_low (this=0x560d6781cc80, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000560d63fdaed1 in os_event_wait_time_low (event=0x560d6781cc80, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000560d641292fb in ib_wqueue_timedwait (wq=0x560d6781cb50, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000560d64269ea4 in fts_optimize_thread (arg=0x560d6781cb50) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f9de0ff9700 (LWP 30412)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdb0ff in os_event::wait (this=0x560d675cb850) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000560d63fdab85 in os_event::wait_low (this=0x560d675cb850, reset_sig_count=8050) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000560d63fdaef6 in os_event_wait_low (event=0x560d675cb850, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000560d640c6df3 in srv_resume_thread (slot=0x560d650157b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000560d640ca9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f9de27fc700 (LWP 30409)): #0 0x00007f9e236d3d47 in sched_yield () from /lib64/libc.so.6 #1 0x0000560d63fdb393 in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x0000560d640f4bd9 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x0000560d640f4de1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x0000560d640cacca in srv_do_purge (n_total_purged=0x7f9de27fbd98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x0000560d640cb0f7 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f9e14095700 (LWP 30442)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d64498997 in safe_cond_wait (cond=0x560d651d81e0 , mp=0x560d651db560 , file=0x560d64563cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000560d638964ee in inline_mysql_cond_wait (that=0x560d651d81e0 , mutex=0x560d651db560 , src_file=0x560d64564830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000560d63897d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000560d6442b45f in pfs_spawn_thread (arg=0x560d679305c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f9e1404a700 (LWP 30443)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d64498c99 in safe_cond_timedwait (cond=0x560d651d81a0 , mp=0x560d651daba0 , abstime=0x7f9e14049c60, file=0x560d645544c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000560d63865845 in inline_mysql_cond_timedwait (that=0x560d651d81a0 , mutex=0x560d651daba0 , abstime=0x7f9e14049c60, src_file=0x560d64556538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000560d6386a61b in cache_thread (thd=0x7f9dac000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000560d6386a8c9 in one_thread_per_connection_end (thd=0x7f9dac000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000560d63b070f3 in do_handle_one_connection (connect=0x560d6758f630) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000560d63b06dad in handle_one_connection (arg=0x560d6758f630) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000560d6442b45f in pfs_spawn_thread (arg=0x560d67954be0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f9e140e0700 (LWP 30417)): #0 0x00007f9e2362840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000560d63864f44 in my_sigwait (set=0x7f9e140dfcf0, sig=0x7f9e140dfcb4, code=0x7f9e140dfcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000560d6386b584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000560d6442b45f in pfs_spawn_thread (arg=0x560d6758ed70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f9dcf7fe700 (LWP 30414)): #0 0x00007f9e25337e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000560d63fdb401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000560d64181a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f9dce7fc700 (LWP 30416)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d64498997 in safe_cond_wait (cond=0x7f9dce7fbc70, mp=0x7f9dce7fbcb0, file=0x560d64766c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000560d63ef2487 in inline_mysql_cond_wait (that=0x7f9dce7fbc70, mutex=0x7f9dce7fbcb0, src_file=0x560d647685d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000560d63ef70c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000560d6442b45f in pfs_spawn_thread (arg=0x560d67771710) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f9dcffff700 (LWP 30413)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdb0ff in os_event::wait (this=0x560d675cba50) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000560d63fdab85 in os_event::wait_low (this=0x560d675cba50, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000560d63fdaef6 in os_event_wait_low (event=0x560d675cba50, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000560d641a6eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f9dceffd700 (LWP 30415)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdb0ff in os_event::wait (this=0x560d675cbb50) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000560d63fdab85 in os_event::wait_low (this=0x560d675cbb50, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000560d63fdaef6 in os_event_wait_low (event=0x560d675cbb50, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000560d6418fb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f9de2ffd700 (LWP 30408)): #0 0x00007f9e25337e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000560d63fdb401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000560d640ca35b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000560d640ca48b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f9de17fa700 (LWP 30411)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdb0ff in os_event::wait (this=0x560d676d2db0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000560d63fdab85 in os_event::wait_low (this=0x560d676d2db0, reset_sig_count=288) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000560d63fdaef6 in os_event_wait_low (event=0x560d676d2db0, reset_sig_count=288) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000560d640d4bfc in sync_array_wait_event (arr=0x560d675a2af0, cell=@0x7f9de17f7f08: 0x560d675a2c20) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000560d640d7911 in rw_lock_s_lock_spin (lock=0x7f9dfebc7568, pass=0, file_name=0x560d647fe350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x0000560d64182cfe in rw_lock_s_lock_func (lock=0x7f9dfebc7568, pass=0, file_name=0x560d647fe350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x0000560d64183742 in pfs_rw_lock_s_lock_func (lock=0x7f9dfebc7568, pass=0, file_name=0x560d647fe350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:651 #8 0x0000560d6419525c in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=1, guess=0x0, mode=10, file=0x560d647fe350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7f9de17f9750, err=0x7f9de17f82a0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4811 #9 0x0000560d64154c34 in btr_cur_search_to_nth_level_func (index=0x7f9dc4013f28, level=0, tuple=0x7f9dc400a548, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x560d677f01c0, ahi_latch=0x0, file=0x560d647fe350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7f9de17f9750, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1548 #10 0x0000560d640905d2 in btr_pcur_open_low (index=0x7f9dc4013f28, level=0, tuple=0x7f9dc400a548, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x560d677f01c0, file=0x560d647fe350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, autoinc=0, mtr=0x7f9de17f9750) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:464 #11 0x0000560d6409377a in row_search_on_row_ref (pcur=0x560d677f01c0, mode=2, table=0x7f9dc4012888, ref=0x7f9dc400a548, mtr=0x7f9de17f9750) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc:1219 #12 0x0000560d64087c21 in row_purge_reposition_pcur (mode=2, node=0x560d677f0128, mtr=0x7f9de17f9750) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:104 #13 0x0000560d64089b24 in row_purge_reset_trx_id (node=0x560d677f0128, mtr=0x7f9de17f9750) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:823 #14 0x0000560d6408b124 in row_purge_record_func (node=0x560d677f0128, undo_rec=0x560d677f0cf8 "", thr=0x560d677efc80, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1212 #15 0x0000560d6408b2fb in row_purge (node=0x560d677f0128, undo_rec=0x560d677f0cf8 "", thr=0x560d677efc80) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #16 0x0000560d6408b6a6 in row_purge_step (thr=0x560d677efc80) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #17 0x0000560d6400da9a in que_thr_step (thr=0x560d677efc80) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #18 0x0000560d6400dcca in que_run_threads_low (thr=0x560d677efc80) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #19 0x0000560d6400deba in que_run_threads (thr=0x560d677efc80) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #20 0x0000560d640ca855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #21 0x0000560d640ca9fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #22 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f9df0ff9700 (LWP 30405)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdaa1a in os_event::timed_wait (this=0x560d675cb9d0, abstime=0x7f9df0ff8d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000560d63fdad31 in os_event::wait_time_low (this=0x560d675cb9d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000560d63fdaed1 in os_event_wait_time_low (event=0x560d675cb9d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000560d640c8ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f9de1ffb700 (LWP 30410)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdb0ff in os_event::wait (this=0x560d676d4400) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000560d63fdab85 in os_event::wait_low (this=0x560d676d4400, reset_sig_count=31) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000560d63fdaef6 in os_event_wait_low (event=0x560d676d4400, reset_sig_count=31) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000560d640d4bfc in sync_array_wait_event (arr=0x560d675a2af0, cell=@0x7f9de1ff97c0: 0x560d675a2ca0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000560d640d847b in rw_lock_sx_lock_func (lock=0x7f9dfebc8678, pass=0, file_name=0x560d6489bf00 "/home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc", line=154) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x0000560d641837f3 in pfs_rw_lock_sx_lock_func (lock=0x7f9dfebc8678, pass=0, file_name=0x560d6489bf00 "/home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc", line=154) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x0000560d64195290 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=4, guess=0x0, mode=10, file=0x560d6489bf00 "/home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc", line=154, mtr=0x7f9de1ffa1a0, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4817 #8 0x0000560d6423c460 in fsp_get_space_header (space=0x560d677d8780, mtr=0x7f9de1ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc:154 #9 0x0000560d64239fe2 in fsp_reserve_free_extents (n_reserved=0x7f9de1ff9bf8, space=0x560d677d8780, n_ext=1, alloc_type=FSP_CLEANING, mtr=0x7f9de1ffa1a0, n_pages=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fsp/fsp0fsp.cc:2643 #10 0x0000560d64162b83 in btr_cur_pessimistic_delete (err=0x7f9de1ff9e5c, has_reserved_extents=0, cursor=0x560d677efb68, flags=0, rollback=false, mtr=0x7f9de1ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5884 #11 0x0000560d64088084 in row_purge_remove_clust_if_poss_low (node=0x560d677efad0, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:175 #12 0x0000560d64088209 in row_purge_remove_clust_if_poss (node=0x560d677efad0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #13 0x0000560d64089a36 in row_purge_del_mark (node=0x560d677efad0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #14 0x0000560d6408afbe in row_purge_record_func (node=0x560d677efad0, undo_rec=0x560d677effa8 "", thr=0x560d677efa18, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #15 0x0000560d6408b2fb in row_purge (node=0x560d677efad0, undo_rec=0x560d677effa8 "", thr=0x560d677efa18) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #16 0x0000560d6408b6a6 in row_purge_step (thr=0x560d677efa18) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #17 0x0000560d6400da9a in que_thr_step (thr=0x560d677efa18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #18 0x0000560d6400dcca in que_run_threads_low (thr=0x560d677efa18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #19 0x0000560d6400deba in que_run_threads (thr=0x560d677efa18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #20 0x0000560d640ca855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #21 0x0000560d640ca9fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #22 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f9dfd7ab700 (LWP 30346)): #0 0x00007f9e24f01644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000560d63fce41c in LinuxAIOHandler::collect (this=0x7f9dfd7aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000560d63fce945 in LinuxAIOHandler::poll (this=0x7f9dfd7aab00, m1=0x7f9dfd7aaba8, m2=0x7f9dfd7aabb0, request=0x7f9dfd7aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000560d63fceaa8 in os_aio_linux_handler (global_segment=2, m1=0x7f9dfd7aaba8, m2=0x7f9dfd7aabb0, request=0x7f9dfd7aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000560d63fd35cc in os_aio_handler (segment=2, m1=0x7f9dfd7aaba8, m2=0x7f9dfd7aabb0, request=0x7f9dfd7aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000560d642264a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000560d640cc909 in io_handler_thread (arg=0x560d65253630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f9de3fff700 (LWP 30406)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdaa1a in os_event::timed_wait (this=0x560d677d5f40, abstime=0x7f9de3ffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000560d63fdad31 in os_event::wait_time_low (this=0x560d677d5f40, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000560d63fdaed1 in os_event_wait_time_low (event=0x560d677d5f40, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000560d64210142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f9dfffff700 (LWP 30343)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d64498c99 in safe_cond_timedwait (cond=0x560d65a64ea0 , mp=0x560d65a64de0 , abstime=0x7f9dffffecb0, file=0x560d648dc610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000560d64302acb in inline_mysql_cond_timedwait (that=0x560d65a64ea0 , mutex=0x560d65a64de0 , abstime=0x7f9dffffecb0, src_file=0x560d648dc658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000560d64302fef in my_service_thread_sleep (control=0x560d65019b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000560d642f7488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000560d6442b45f in pfs_spawn_thread (arg=0x560d67583800) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f9df37fe700 (LWP 30349)): #0 0x00007f9e24f01644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000560d63fce41c in LinuxAIOHandler::collect (this=0x7f9df37fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000560d63fce945 in LinuxAIOHandler::poll (this=0x7f9df37fdb00, m1=0x7f9df37fdba8, m2=0x7f9df37fdbb0, request=0x7f9df37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000560d63fceaa8 in os_aio_linux_handler (global_segment=5, m1=0x7f9df37fdba8, m2=0x7f9df37fdbb0, request=0x7f9df37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000560d63fd35cc in os_aio_handler (segment=5, m1=0x7f9df37fdba8, m2=0x7f9df37fdbb0, request=0x7f9df37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000560d642264a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000560d640cc909 in io_handler_thread (arg=0x560d65253648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f9df1ffb700 (LWP 30403)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d63fdaa1a in os_event::timed_wait (this=0x560d677a3e80, abstime=0x7f9df1ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000560d63fdad31 in os_event::wait_time_low (this=0x560d677a3e80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000560d63fdaed1 in os_event_wait_time_low (event=0x560d677a3e80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000560d63fa788e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f9e1c1dc700 (LWP 30340)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d64498997 in safe_cond_wait (cond=0x7f9e0c017b68, mp=0x7f9e0c017ab8, file=0x560d64602b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000560d63bf0088 in inline_mysql_cond_wait (that=0x7f9e0c017b68, mutex=0x7f9e0c017ab8, src_file=0x560d64602e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000560d63bf2d20 in Wsrep_thd_queue::pop_front (this=0x7f9e0c017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000560d63bf1740 in wsrep_rollback_process (rollbacker=0x7f9e0c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000560d63be2282 in start_wsrep_THD (arg=0x560d674d34c0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f9e1c191700 (LWP 30341)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d64498997 in safe_cond_wait (cond=0x7f9e00015608, mp=0x7f9e00015558, file=0x560d64602b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000560d63bf0088 in inline_mysql_cond_wait (that=0x7f9e00015608, mutex=0x7f9e00015558, src_file=0x560d64602e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000560d63bf2d20 in Wsrep_thd_queue::pop_front (this=0x7f9e00015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000560d63bf1b00 in wsrep_post_rollback_process (post_rollbacker=0x7f9e00000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000560d63be2282 in start_wsrep_THD (arg=0x560d674dbda0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f9dfdfac700 (LWP 30345)): #0 0x00007f9e24f01644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000560d63fce41c in LinuxAIOHandler::collect (this=0x7f9dfdfabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000560d63fce945 in LinuxAIOHandler::poll (this=0x7f9dfdfabb00, m1=0x7f9dfdfabba8, m2=0x7f9dfdfabbb0, request=0x7f9dfdfabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000560d63fceaa8 in os_aio_linux_handler (global_segment=1, m1=0x7f9dfdfabba8, m2=0x7f9dfdfabbb0, request=0x7f9dfdfabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000560d63fd35cc in os_aio_handler (segment=1, m1=0x7f9dfdfabba8, m2=0x7f9dfdfabbb0, request=0x7f9dfdfabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000560d642264a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000560d640cc909 in io_handler_thread (arg=0x560d65253628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f9e1537a700 (LWP 30339)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9e1d219867 in gu::Lock::wait (cond=..., this=0x7f9e15379850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f9e1d21c3ab in RecvBuf::front (this=this@entry=0x560d674c38a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f9e1d219f46 in gcomm_recv (backend=, msg=0x560d6748b830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f9e1d20deec in core_msg_recv (backend=backend@entry=0x560d6748b978, recv_msg=recv_msg@entry=0x560d6748b830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f9e1d211164 in gcs_core_recv (conn=0x560d6748b7d0, recv_act=recv_act@entry=0x7f9e15379d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f9e1d2183d8 in gcs_recv_thread (arg=0x560d6748b550) at gcs/src/gcs.cpp:1398 #7 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f9e1dd89700 (LWP 30297)): #0 0x00007f9e25334d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000560d64498c99 in safe_cond_timedwait (cond=0x560d65a77640 , mp=0x560d65a77580 , abstime=0x7f9e1dd88d70, file=0x560d64920840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000560d64499c53 in inline_mysql_cond_timedwait (that=0x560d65a77640 , mutex=0x560d65a77580 , abstime=0x7f9e1dd88d70, src_file=0x560d64920888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000560d6449a7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000560d6442b45f in pfs_spawn_thread (arg=0x560d6744c730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f9e1c146700 (LWP 30342)): #0 0x00007f9e25334965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9e1d133132 in fifo_lock_get (q=q@entry=0x560d674b1bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f9e1d1337a1 in gu_fifo_get_head (q=0x560d674b1bd0, err=0x7f9e1c14506c) at galerautils/src/gu_fifo.c:330 #3 0x00007f9e1d2150cc in gcs_recv (conn=0x560d6748b550, action=0x7f9e1c1450b0) at gcs/src/gcs.cpp:2024 #4 0x00007f9e1d24216c in galera::GcsActionSource::process (this=0x560d6748af30, recv_ctx=0x7f9e1c1454d0, exit_loop=@0x7f9e1c14529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f9e1d261033 in galera::ReplicatorSMM::async_recv (this=0x560d67483280, recv_ctx=0x7f9e1c1454d0) at galera/src/replicator_smm.cpp:383 #6 0x0000560d6454d276 in wsrep::wsrep_provider_v26::run_applier (this=0x560d6740a360, applier_ctx=0x7f9e1c1454d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000560d63bf0636 in wsrep_replication_process (thd=0x7f9e04000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000560d63be2282 in start_wsrep_THD (arg=0x560d674dc440) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f9e25330dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f9e236eeead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f9e25741880 (LWP 30296)): #0 0x00007f9e253359d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000560d64493368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000560d63cd1503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9e236e420d in poll () from /lib64/libc.so.6 #5 0x0000560d63871a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000560d63870d25 in mysqld_main (argc=161, argv=0x560d67304d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000560d63864ec5 in main (argc=23, argv=0x7fffffccd148) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.30540' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.2/data/core.30540 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 30540] [New LWP 30582] [New LWP 30628] [New LWP 30629] [New LWP 30630] [New LWP 30541] [New LWP 30976] [New LWP 30973] [New LWP 30631] [New LWP 30975] [New LWP 30974] [New LWP 31003] [New LWP 30977] [New LWP 31004] [New LWP 31006] [New LWP 30979] [New LWP 31008] [New LWP 31009] [New LWP 31011] [New LWP 31010] [New LWP 31013] [New LWP 31014] [New LWP 31018] [New LWP 31017] [New LWP 31019] [New LWP 31016] [New LWP 31015] [New LWP 31007] [New LWP 31012] [New LWP 31005] [New LWP 30980] [New LWP 30978] [New LWP 30583] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fdc032939d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fdc032939d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557989b56368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557989394503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdc0164220d in poll () from /lib64/libc.so.6 #5 0x0000557988f34a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000557988f33d25 in mysqld_main (argc=162, argv=0x55798c3e9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000557988f27ec5 in main (argc=23, argv=0x7ffc8d641208) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fdbebffe700 (LWP 30583)): #0 0x00007fdc0164d483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fdbfb13e6f8 in asio::detail::epoll_reactor::run (this=0x55798c5a5010, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fdbfb142df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55798c5a3f70, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fdbfb1430a2 in asio::detail::task_io_service::run (this=0x55798c5a3f70, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fdbfb15aafd in asio::io_service::run (this=0x55798c5a1f70) at asio/asio/impl/io_service.ipp:58 #5 0x00007fdbfb15b251 in gcomm::AsioProtonet::event_loop (this=0x55798c5a1ed0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fdbfb1792ed in GCommConn::run (this=0x55798c5a4770) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fdbfb17c399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fdbe1ffb700 (LWP 30978)): #0 0x00007fdc02e5f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055798969141c in LinuxAIOHandler::collect (this=0x7fdbe1ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557989691945 in LinuxAIOHandler::poll (this=0x7fdbe1ffab00, m1=0x7fdbe1ffaba8, m2=0x7fdbe1ffabb0, request=0x7fdbe1ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557989691aa8 in os_aio_linux_handler (global_segment=4, m1=0x7fdbe1ffaba8, m2=0x7fdbe1ffabb0, request=0x7fdbe1ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579896965cc in os_aio_handler (segment=4, m1=0x7fdbe1ffaba8, m2=0x7fdbe1ffabb0, request=0x7fdbe1ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579898e94a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055798978f909 in io_handler_thread (arg=0x55798a916640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fdbe0ff9700 (LWP 30980)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969da1a in os_event::timed_wait (this=0x55798c6b5740, abstime=0x7fdbe0ff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055798969dd31 in os_event::wait_time_low (this=0x55798c6b5740, time_in_usec=630000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055798969ded1 in os_event_wait_time_low (event=0x55798c6b5740, time_in_usec=630000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557989873d1d in pc_sleep_if_needed (next_loop_time=1549882382097, sig_count=2, cur_time=1549882381467) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000557989874c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fdbbaffd700 (LWP 31005)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969da1a in os_event::timed_wait (this=0x55798c6b5640, abstime=0x7fdbbaffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055798969dd31 in os_event::wait_time_low (this=0x55798c6b5640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055798969ded1 in os_event_wait_time_low (event=0x55798c6b5640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055798978bada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fdbab7fe700 (LWP 31012)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969e0ff in os_event::wait (this=0x55798c6b5440) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055798969db85 in os_event::wait_low (this=0x55798c6b5440, reset_sig_count=9181) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055798969def6 in os_event_wait_low (event=0x55798c6b5440, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557989789df3 in srv_resume_thread (slot=0x55798a6d8780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055798978d9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fdbb9ffb700 (LWP 31007)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969da1a in os_event::timed_wait (this=0x55798c8575d0, abstime=0x7fdbb9ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055798969dd31 in os_event::wait_time_low (this=0x55798c8575d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055798969ded1 in os_event_wait_time_low (event=0x55798c8575d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005579897ec2fb in ib_wqueue_timedwait (wq=0x55798c85dfb0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055798992cea4 in fts_optimize_thread (arg=0x55798c85dfb0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fdba9ffb700 (LWP 31015)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969e0ff in os_event::wait (this=0x55798c6b57c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055798969db85 in os_event::wait_low (this=0x55798c6b57c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055798969def6 in os_event_wait_low (event=0x55798c6b57c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557989852b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fdba97fa700 (LWP 31016)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557989b5b997 in safe_cond_wait (cond=0x7fdba97f9c70, mp=0x7fdba97f9cb0, file=0x557989e29c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005579895b5487 in inline_mysql_cond_wait (that=0x7fdba97f9c70, mutex=0x7fdba97f9cb0, src_file=0x557989e2b5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005579895ba0c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000557989aee45f in pfs_spawn_thread (arg=0x55798c8f81d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fdbe8b6e700 (LWP 31019)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557989b5bc99 in safe_cond_timedwait (cond=0x55798a89b1a0 , mp=0x55798a89dba0 , abstime=0x7fdbe8b6dc60, file=0x557989c174c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557988f28845 in inline_mysql_cond_timedwait (that=0x55798a89b1a0 , mutex=0x55798a89dba0 , abstime=0x7fdbe8b6dc60, src_file=0x557989c19538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557988f2d61b in cache_thread (thd=0x7fdbd001ad00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000557988f2d8c9 in one_thread_per_connection_end (thd=0x7fdbd001ad00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005579891ca0f3 in do_handle_one_connection (connect=0x55798ca2a230) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005579891c9dad in handle_one_connection (arg=0x55798ca2a230) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000557989aee45f in pfs_spawn_thread (arg=0x55798c9b3bb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fdbe8c04700 (LWP 31017)): #0 0x00007fdc0158640a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000557988f27f44 in my_sigwait (set=0x7fdbe8c03cf0, sig=0x7fdbe8c03cb4, code=0x7fdbe8c03cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000557988f2e584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000557989aee45f in pfs_spawn_thread (arg=0x55798c678d00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fdbe8bb9700 (LWP 31018)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557989b5b997 in safe_cond_wait (cond=0x55798a89b1e0 , mp=0x55798a89e560 , file=0x557989c26cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557988f594ee in inline_mysql_cond_wait (that=0x55798a89b1e0 , mutex=0x55798a89e560 , src_file=0x557989c27830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000557988f5ad80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000557989aee45f in pfs_spawn_thread (arg=0x55798ca18370) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fdbaa7fc700 (LWP 31014)): #0 0x00007fdc03295e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055798969e401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557989844a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fdbaaffd700 (LWP 31013)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969e0ff in os_event::wait (this=0x55798c6b56c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055798969db85 in os_event::wait_low (this=0x55798c6b56c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055798969def6 in os_event_wait_low (event=0x55798c6b56c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557989869eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fdbabfff700 (LWP 31010)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969e0ff in os_event::wait (this=0x55798c6b5540) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055798969db85 in os_event::wait_low (this=0x55798c6b5540, reset_sig_count=7880) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055798969def6 in os_event_wait_low (event=0x55798c6b5540, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557989789df3 in srv_resume_thread (slot=0x55798a6d87f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055798978d9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fdba37fe700 (LWP 31011)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969e0ff in os_event::wait (this=0x55798c6b54c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055798969db85 in os_event::wait_low (this=0x55798c6b54c0, reset_sig_count=8354) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055798969def6 in os_event_wait_low (event=0x55798c6b54c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557989789df3 in srv_resume_thread (slot=0x55798a6d87b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055798978d9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fdbb8ff9700 (LWP 31009)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969e0ff in os_event::wait (this=0x55798c6b53c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055798969db85 in os_event::wait_low (this=0x55798c6b53c0, reset_sig_count=2226) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055798969def6 in os_event_wait_low (event=0x55798c6b53c0, reset_sig_count=2226) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557989789df3 in srv_resume_thread (slot=0x55798a6d8748 , sig_count=2226, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055798978de04 in srv_purge_coordinator_suspend (slot=0x55798a6d8748 , rseg_history_len=3) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055798978e09a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fdbb97fa700 (LWP 31008)): #0 0x00007fdc03295e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055798969e401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055798978d35b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055798978d48b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fdbe17fa700 (LWP 30979)): #0 0x00007fdc02e5f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055798969141c in LinuxAIOHandler::collect (this=0x7fdbe17f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557989691945 in LinuxAIOHandler::poll (this=0x7fdbe17f9b00, m1=0x7fdbe17f9ba8, m2=0x7fdbe17f9bb0, request=0x7fdbe17f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557989691aa8 in os_aio_linux_handler (global_segment=5, m1=0x7fdbe17f9ba8, m2=0x7fdbe17f9bb0, request=0x7fdbe17f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579896965cc in os_aio_handler (segment=5, m1=0x7fdbe17f9ba8, m2=0x7fdbe17f9bb0, request=0x7fdbe17f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579898e94a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055798978f909 in io_handler_thread (arg=0x55798a916648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fdbba7fc700 (LWP 31006)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969da1a in os_event::timed_wait (this=0x55798c8bfbb0, abstime=0x7fdbba7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055798969dd31 in os_event::wait_time_low (this=0x55798c8bfbb0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055798969ded1 in os_event_wait_time_low (event=0x55798c8bfbb0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005579898d3142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fdbbbfff700 (LWP 31004)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969da1a in os_event::timed_wait (this=0x55798c6b55c0, abstime=0x7fdbbbffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055798969dd31 in os_event::wait_time_low (this=0x55798c6b55c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055798969ded1 in os_event_wait_time_low (event=0x55798c6b55c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055798978bffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fdbe27fc700 (LWP 30977)): #0 0x00007fdc02e5f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055798969141c in LinuxAIOHandler::collect (this=0x7fdbe27fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557989691945 in LinuxAIOHandler::poll (this=0x7fdbe27fbb00, m1=0x7fdbe27fbba8, m2=0x7fdbe27fbbb0, request=0x7fdbe27fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557989691aa8 in os_aio_linux_handler (global_segment=3, m1=0x7fdbe27fbba8, m2=0x7fdbe27fbbb0, request=0x7fdbe27fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579896965cc in os_aio_handler (segment=3, m1=0x7fdbe27fbba8, m2=0x7fdbe27fbbb0, request=0x7fdbe27fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579898e94a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055798978f909 in io_handler_thread (arg=0x55798a916638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fdbbb7fe700 (LWP 31003)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055798969da1a in os_event::timed_wait (this=0x55798c88daf0, abstime=0x7fdbbb7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055798969dd31 in os_event::wait_time_low (this=0x55798c88daf0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055798969ded1 in os_event_wait_time_low (event=0x55798c88daf0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055798966a88e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fdbe8a04700 (LWP 30974)): #0 0x00007fdc02e5f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055798969141c in LinuxAIOHandler::collect (this=0x7fdbe8a03b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557989691945 in LinuxAIOHandler::poll (this=0x7fdbe8a03b00, m1=0x7fdbe8a03ba8, m2=0x7fdbe8a03bb0, request=0x7fdbe8a03be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557989691aa8 in os_aio_linux_handler (global_segment=0, m1=0x7fdbe8a03ba8, m2=0x7fdbe8a03bb0, request=0x7fdbe8a03be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579896965cc in os_aio_handler (segment=0, m1=0x7fdbe8a03ba8, m2=0x7fdbe8a03bb0, request=0x7fdbe8a03be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579898e94a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055798978f909 in io_handler_thread (arg=0x55798a916620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fdbe37fe700 (LWP 30975)): #0 0x00007fdc02e5f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055798969141c in LinuxAIOHandler::collect (this=0x7fdbe37fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557989691945 in LinuxAIOHandler::poll (this=0x7fdbe37fdb00, m1=0x7fdbe37fdba8, m2=0x7fdbe37fdbb0, request=0x7fdbe37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557989691aa8 in os_aio_linux_handler (global_segment=1, m1=0x7fdbe37fdba8, m2=0x7fdbe37fdbb0, request=0x7fdbe37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579896965cc in os_aio_handler (segment=1, m1=0x7fdbe37fdba8, m2=0x7fdbe37fdbb0, request=0x7fdbe37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579898e94a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055798978f909 in io_handler_thread (arg=0x55798a916628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fdbf8621700 (LWP 30631)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdbfb091132 in fifo_lock_get (q=q@entry=0x55798c59a100) at galerautils/src/gu_fifo.c:262 #2 0x00007fdbfb0917a1 in gu_fifo_get_head (q=0x55798c59a100, err=0x7fdbf862006c) at galerautils/src/gu_fifo.c:330 #3 0x00007fdbfb1730cc in gcs_recv (conn=0x55798c573b60, action=0x7fdbf86200b0) at gcs/src/gcs.cpp:2024 #4 0x00007fdbfb1a016c in galera::GcsActionSource::process (this=0x55798c573840, recv_ctx=0x7fdbf86204d0, exit_loop=@0x7fdbf862029e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fdbfb1bf033 in galera::ReplicatorSMM::async_recv (this=0x55798c5681f0, recv_ctx=0x7fdbf86204d0) at galera/src/replicator_smm.cpp:383 #6 0x0000557989c10276 in wsrep::wsrep_provider_v26::run_applier (this=0x55798c4dde50, applier_ctx=0x7fdbf86204d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005579892b3636 in wsrep_replication_process (thd=0x7fdbd8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005579892a5282 in start_wsrep_THD (arg=0x55798c58c710) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fdbea056700 (LWP 30973)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557989b5bc99 in safe_cond_timedwait (cond=0x55798b127ea0 , mp=0x55798b127de0 , abstime=0x7fdbea055cb0, file=0x557989f9f610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005579899c5acb in inline_mysql_cond_timedwait (that=0x55798b127ea0 , mutex=0x55798b127de0 , abstime=0x7fdbea055cb0, src_file=0x557989f9f658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005579899c5fef in my_service_thread_sleep (control=0x55798a6dcb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005579899ba488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000557989aee45f in pfs_spawn_thread (arg=0x55798c66d690) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fdbe2ffd700 (LWP 30976)): #0 0x00007fdc02e5f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055798969141c in LinuxAIOHandler::collect (this=0x7fdbe2ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557989691945 in LinuxAIOHandler::poll (this=0x7fdbe2ffcb00, m1=0x7fdbe2ffcba8, m2=0x7fdbe2ffcbb0, request=0x7fdbe2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557989691aa8 in os_aio_linux_handler (global_segment=2, m1=0x7fdbe2ffcba8, m2=0x7fdbe2ffcbb0, request=0x7fdbe2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579896965cc in os_aio_handler (segment=2, m1=0x7fdbe2ffcba8, m2=0x7fdbe2ffcbb0, request=0x7fdbe2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579898e94a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055798978f909 in io_handler_thread (arg=0x55798a916630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fdbfbce7700 (LWP 30541)): #0 0x00007fdc03292d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557989b5bc99 in safe_cond_timedwait (cond=0x55798b13a640 , mp=0x55798b13a580 , abstime=0x7fdbfbce6d70, file=0x557989fe3840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557989b5cc53 in inline_mysql_cond_timedwait (that=0x55798b13a640 , mutex=0x55798b13a580 , abstime=0x7fdbfbce6d70, src_file=0x557989fe3888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557989b5d7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000557989aee45f in pfs_spawn_thread (arg=0x55798c5317d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fdbf866c700 (LWP 30630)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557989b5b997 in safe_cond_wait (cond=0x7fdbcc015608, mp=0x7fdbcc015558, file=0x557989cc5b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005579892b3088 in inline_mysql_cond_wait (that=0x7fdbcc015608, mutex=0x7fdbcc015558, src_file=0x557989cc5e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005579892b5d20 in Wsrep_tworker[1] Trying to dump core for [mysqltest - pid: 46859, winpid: 46859] worker[1] Trying to dump core for [mysqld.1 - pid: 45781, winpid: 45781] worker[1] Trying to dump core for [mysqld.2 - pid: 46105, winpid: 46105] hd_queue::pop_front (this=0x7fdbcc015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005579892b4b00 in wsrep_post_rollback_process (post_rollbacker=0x7fdbcc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005579892a5282 in start_wsrep_THD (arg=0x55798c5bcd20) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fdbf86b7700 (LWP 30629)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557989b5b997 in safe_cond_wait (cond=0x7fdbdc016858, mp=0x7fdbdc0167a8, file=0x557989cc5b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005579892b3088 in inline_mysql_cond_wait (that=0x7fdbdc016858, mutex=0x7fdbdc0167a8, src_file=0x557989cc5e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005579892b5d20 in Wsrep_thd_queue::pop_front (this=0x7fdbdc016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005579892b4740 in wsrep_rollback_process (rollbacker=0x7fdbdc001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005579892a5282 in start_wsrep_THD (arg=0x55798c5c5450) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fdbe3fff700 (LWP 30628)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdbfb177867 in gu::Lock::wait (cond=..., this=0x7fdbe3ffe850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fdbfb17a3ab in RecvBuf::front (this=this@entry=0x55798c5a4958, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fdbfb177f46 in gcomm_recv (backend=, msg=0x55798c573e40, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fdbfb16beec in core_msg_recv (backend=backend@entry=0x55798c573f88, recv_msg=recv_msg@entry=0x55798c573e40, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fdbfb16f164 in gcs_core_recv (conn=0x55798c573de0, recv_act=recv_act@entry=0x7fdbe3ffed70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fdbfb1763d8 in gcs_recv_thread (arg=0x55798c573b60) at gcs/src/gcs.cpp:1398 #7 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fdbfab3b700 (LWP 30582)): #0 0x00007fdc03292965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdbfb19cfd7 in gu::Lock::wait (cond=..., this=0x7fdbfab3abf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fdbfb19d069 in galera::ServiceThd::thd_func (arg=0x55798c568908) at galera/src/galera_service_thd.cpp:30 #3 0x00007fdc0328edd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdc0164cead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fdc0369f880 (LWP 30540)): #0 0x00007fdc032939d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557989b56368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557989394503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdc0164220d in poll () from /lib64/libc.so.6 #5 0x0000557988f34a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000557988f33d25 in mysqld_main (argc=162, argv=0x55798c3e9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000557988f27ec5 in main (argc=23, argv=0x7ffc8d641208) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_load_data_splitting-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_mysqldump_sst 'innodb' w4 [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-11 12:55:21 line 2019-02-11 12:51:52 13 [Warning] WSREP: no THD for trx: 51 2019-02-11 12:51:52 13 [Warning] WSREP: no THD for trx: 51 ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/mysqld.2.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_mysqldump_sst-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' galera_sr.galera_sr_small_gcache 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 12:59:41 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr_small_gcache.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE t1 (f1 INTEGER) ENGINE=InnoDB; INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.45782' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.1/data/core.45782 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 45782] [New LWP 45871] [New LWP 45864] [New LWP 45869] [New LWP 45825] [New LWP 45826] [New LWP 45829] [New LWP 45828] [New LWP 45873] [New LWP 45875] [New LWP 45866] [New LWP 45874] [New LWP 45876] [New LWP 45883] [New LWP 45884] [New LWP 45877] [New LWP 45901] [New LWP 45885] [New LWP 45881] [New LWP 45827] [New LWP 46036] [New LWP 45824] [New LWP 45886] [New LWP 45879] [New LWP 45870] [New LWP 46037] [New LWP 45868] [New LWP 45865] [New LWP 46984] [New LWP 45882] [New LWP 45867] [New LWP 45880] [New LWP 45805] [New LWP 45878] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f77ccaff9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f77ccaff9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f1ad8da368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f1ad118503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f77caeae20d in poll () from /lib64/libc.so.6 #5 0x000055f1accb8a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f1accb7d25 in mysqld_main (argc=161, argv=0x55f1af254d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f1accabec5 in main (argc=23, argv=0x7fff7cfd2f98) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f779a7fc700 (LWP 45878)): #0 0x00007f77ccb01d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055f1ad416acb in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055f1ad416e21 in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055f1ad65f3a2 in pfs_os_file_flush_func (file=..., src_file=0x55f1adcd9a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055f1ad66211d in fil_flush_low (space=0x55f1af7257f0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055f1ad66d9f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055f1ad3f21cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055f1ad3f2c31 in log_write_up_to (lsn=3156428, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055f1ad3f2d61 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x000055f1ad510430 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #10 0x000055f1ad5109f3 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #11 0x000055f1ad511551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f77c5553700 (LWP 45805)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad8dfc99 in safe_cond_timedwait (cond=0x55f1aeebe640 , mp=0x55f1aeebe580 , abstime=0x7f77c5552d70, file=0x55f1add67840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f1ad8e0c53 in inline_mysql_cond_timedwait (that=0x55f1aeebe640 , mutex=0x55f1aeebe580 , abstime=0x7f77c5552d70, src_file=0x55f1add67888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f1ad8e17b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055f1ad87245f in pfs_spawn_thread (arg=0x55f1af39c730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f77927fc700 (LWP 45880)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad4220ff in os_event::wait (this=0x55f1af641ab0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f1ad421b85 in os_event::wait_low (this=0x55f1af641ab0, reset_sig_count=281) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f1ad421ef6 in os_event_wait_low (event=0x55f1af641ab0, reset_sig_count=281) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f1ad51bbfc in sync_array_wait_event (arr=0x55f1af4f2aa0, cell=@0x7f77927f9f08: 0x55f1af4f2c10) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055f1ad51e911 in rw_lock_s_lock_spin (lock=0x7f77a6bdef38, pass=0, file_name=0x55f1adc45350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000055f1ad5c9cfe in rw_lock_s_lock_func (lock=0x7f77a6bdef38, pass=0, file_name=0x55f1adc45350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000055f1ad5ca742 in pfs_rw_lock_s_lock_func (lock=0x7f77a6bdef38, pass=0, file_name=0x55f1adc45350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:651 #8 0x000055f1ad5dc25c in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=1, guess=0x0, mode=10, file=0x55f1adc45350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7f77927fb750, err=0x7f77927fa2a0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4811 #9 0x000055f1ad59bc34 in btr_cur_search_to_nth_level_func (index=0x55f1af8b72d8, level=0, tuple=0x7f777800a548, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x55f1af73d7d0, ahi_latch=0x0, file=0x55f1adc45350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7f77927fb750, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1548 #10 0x000055f1ad4d75d2 in btr_pcur_open_low (index=0x55f1af8b72d8, level=0, tuple=0x7f777800a548, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x55f1af73d7d0, file=0x55f1adc45350 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, autoinc=0, mtr=0x7f77927fb750) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:464 #11 0x000055f1ad4da77a in row_search_on_row_ref (pcur=0x55f1af73d7d0, mode=2, table=0x55f1af8b3be8, ref=0x7f777800a548, mtr=0x7f77927fb750) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc:1219 #12 0x000055f1ad4cec21 in row_purge_reposition_pcur (mode=2, node=0x55f1af73d738, mtr=0x7f77927fb750) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:104 #13 0x000055f1ad4d0b24 in row_purge_reset_trx_id (node=0x55f1af73d738, mtr=0x7f77927fb750) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:823 #14 0x000055f1ad4d2124 in row_purge_record_func (node=0x55f1af73d738, undo_rec=0x55f1af73e308 "", thr=0x55f1af73d290, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1212 #15 0x000055f1ad4d22fb in row_purge (node=0x55f1af73d738, undo_rec=0x55f1af73e308 "", thr=0x55f1af73d290) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #16 0x000055f1ad4d26a6 in row_purge_step (thr=0x55f1af73d290) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #17 0x000055f1ad454a9a in que_thr_step (thr=0x55f1af73d290) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #18 0x000055f1ad454cca in que_run_threads_low (thr=0x55f1af73d290) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #19 0x000055f1ad454eba in que_run_threads (thr=0x55f1af73d290) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #20 0x000055f1ad511855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #21 0x000055f1ad5119fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #22 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f77a57ab700 (LWP 45867)): #0 0x00007f77cc6cb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f1ad41541c in LinuxAIOHandler::collect (this=0x7f77a57aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f1ad415945 in LinuxAIOHandler::poll (this=0x7f77a57aab00, m1=0x7f77a57aaba8, m2=0x7f77a57aabb0, request=0x7f77a57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f1ad415aa8 in os_aio_linux_handler (global_segment=2, m1=0x7f77a57aaba8, m2=0x7f77a57aabb0, request=0x7f77a57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f1ad41a5cc in os_aio_handler (segment=2, m1=0x7f77a57aaba8, m2=0x7f77a57aabb0, request=0x7f77a57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f1ad66d4a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f1ad513909 in io_handler_thread (arg=0x55f1ae69a630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f77917fa700 (LWP 45882)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad4220ff in os_event::wait (this=0x55f1af51b7c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f1ad421b85 in os_event::wait_low (this=0x55f1af51b7c0, reset_sig_count=8012) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f1ad421ef6 in os_event_wait_low (event=0x55f1af51b7c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f1ad50ddf3 in srv_resume_thread (slot=0x55f1ae45c7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f1ad5119f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f77bc1ed700 (LWP 46984)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad4220ff in os_event::wait (this=0x55f1af6b3db0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f1ad421b85 in os_event::wait_low (this=0x55f1af6b3db0, reset_sig_count=3051) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f1ad421ef6 in os_event_wait_low (event=0x55f1af6b3db0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f1ad3f2761 in log_write_up_to (lsn=3157591, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000055f1ad55dd91 in trx_flush_log_if_needed_low (lsn=3157591) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #6 0x000055f1ad55ddda in trx_flush_log_if_needed (lsn=3157591, trx=0x7f77a7401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #7 0x000055f1ad55fc03 in trx_commit_complete_for_mysql (trx=0x7f77a7401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #8 0x000055f1ad344514 in innobase_commit (hton=0x55f1af4efb60, thd=0x7f774c16b470, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #9 0x000055f1ad11e039 in commit_one_phase_2 (thd=0x7f774c16b470, all=true, trans=0x7f774c16e9c0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #10 0x000055f1ad11df3b in ha_commit_one_phase (thd=0x7f774c16b470, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #11 0x000055f1ad11d2b7 in ha_commit_trans (thd=0x7f774c16b470, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #12 0x000055f1acf62e8e in trans_commit (thd=0x7f774c16b470) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #13 0x000055f1ad32218c in Wsrep_storage_service::commit (this=0x7f774c163680, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #14 0x000055f1ad98a4d3 in wsrep::transaction::certify_fragment (this=0x7f774c0069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #15 0x000055f1ad989c37 in wsrep::transaction::streaming_step (this=0x7f774c0069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #16 0x000055f1ad986ff2 in wsrep::transaction::after_row (this=0x7f774c0069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #17 0x000055f1ad1303a8 in wsrep::client_state::after_row (this=0x7f774c006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #18 0x000055f1ad119625 in wsrep_after_row (thd=0x7f774c000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #19 0x000055f1ad12b52a in wsrep_after_row (thd=0x7f774c000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #20 0x000055f1ad12b91b in handler::ha_write_row (this=0x7f774c076a18, buf=0x7f774c07aa80 "\375\001") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #21 0x000055f1acda1443 in write_record (thd=0x7f774c000b00, table=0x7f774c07c4c0, info=0x7f774c016d38) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #22 0x000055f1acda6742 in select_insert::send_data (this=0x7f774c016cf0, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #23 0x000055f1ace5ee07 in end_send (join=0x7f774c016da0, join_tab=0x7f774c087868, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #24 0x000055f1acf9bfa7 in JOIN_CACHE::generate_full_extensions (this=0x7f774c019c70, rec_ptr=0x7f774c10aef0 "\344") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #25 0x000055f1acf9bcc2 in JOIN_CACHE::join_matching_records (this=0x7f774c019c70, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #26 0x000055f1acf9b68e in JOIN_CACHE::join_records (this=0x7f774c019c70, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #27 0x000055f1acf9b828 in JOIN_CACHE::join_records (this=0x7f774c019b40, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #28 0x000055f1acf9b828 in JOIN_CACHE::join_records (this=0x7f774c019a10, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #29 0x000055f1ace5babb in sub_select_cache (join=0x7f774c016da0, join_tab=0x7f774c086d58, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #30 0x000055f1ace5bcc8 in sub_select (join=0x7f774c016da0, join_tab=0x7f774c0869a8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #31 0x000055f1ace5b493 in do_select (join=0x7f774c016da0, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #32 0x000055f1ace33fb1 in JOIN::exec_inner (this=0x7f774c016da0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #33 0x000055f1ace333fa in JOIN::exec (this=0x7f774c016da0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #34 0x000055f1ace346ec in mysql_select (thd=0x7f774c000b00, tables=0x7f774c014ac8, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f774c016cf0, unit=0x7f774c0049c0, select_lex=0x7f774c0145a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #35 0x000055f1ace25fed in handle_select (thd=0x7f774c000b00, lex=0x7f774c0048f8, result=0x7f774c016cf0, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #36 0x000055f1acde83fc in mysql_execute_command (thd=0x7f774c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #37 0x000055f1acdf3935 in mysql_parse (thd=0x7f774c000b00, rawbuf=0x7f774c013e58 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7f77bc1ec0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #38 0x000055f1acdf2f68 in wsrep_mysql_parse (thd=0x7f774c000b00, rawbuf=0x7f774c013e58 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7f77bc1ec0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #39 0x000055f1acddee25 in dispatch_command (command=COM_QUERY, thd=0x7f774c000b00, packet=0x7f774c00a431 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=71, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #40 0x000055f1acddd80d in do_command (thd=0x7f774c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #41 0x000055f1acf4e069 in do_handle_one_connection (connect=0x55f1af870240) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #42 0x000055f1acf4ddad in handle_one_connection (arg=0x55f1af870240) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #43 0x000055f1ad87245f in pfs_spawn_thread (arg=0x55f1af38fa70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #44 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #45 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f77a67ad700 (LWP 45865)): #0 0x00007f77cc6cb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f1ad41541c in LinuxAIOHandler::collect (this=0x7f77a67acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f1ad415945 in LinuxAIOHandler::poll (this=0x7f77a67acb00, m1=0x7f77a67acba8, m2=0x7f77a67acbb0, request=0x7f77a67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f1ad415aa8 in os_aio_linux_handler (global_segment=0, m1=0x7f77a67acba8, m2=0x7f77a67acbb0, request=0x7f77a67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f1ad41a5cc in os_aio_handler (segment=0, m1=0x7f77a67acba8, m2=0x7f77a67acbb0, request=0x7f77a67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f1ad66d4a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f1ad513909 in io_handler_thread (arg=0x55f1ae69a620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f77a4faa700 (LWP 45868)): #0 0x00007f77cc6cb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f1ad41541c in LinuxAIOHandler::collect (this=0x7f77a4fa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f1ad415945 in LinuxAIOHandler::poll (this=0x7f77a4fa9b00, m1=0x7f77a4fa9ba8, m2=0x7f77a4fa9bb0, request=0x7f77a4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f1ad415aa8 in os_aio_linux_handler (global_segment=3, m1=0x7f77a4fa9ba8, m2=0x7f77a4fa9bb0, request=0x7f77a4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f1ad41a5cc in os_aio_handler (segment=3, m1=0x7f77a4fa9ba8, m2=0x7f77a4fa9bb0, request=0x7f77a4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f1ad66d4a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f1ad513909 in io_handler_thread (arg=0x55f1ae69a638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f77c4053700 (LWP 46037)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad8dfc99 in safe_cond_timedwait (cond=0x55f1ae61f1a0 , mp=0x55f1ae621ba0 , abstime=0x7f77c4052c60, file=0x55f1ad99b4c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f1accac845 in inline_mysql_cond_timedwait (that=0x55f1ae61f1a0 , mutex=0x55f1ae621ba0 , abstime=0x7f77c4052c60, src_file=0x55f1ad99d538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f1accb161b in cache_thread (thd=0x7f7754000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055f1accb18c9 in one_thread_per_connection_end (thd=0x7f7754000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055f1acf4e0f3 in do_handle_one_connection (connect=0x55f1af88a870) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055f1acf4ddad in handle_one_connection (arg=0x55f1af88a870) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055f1ad87245f in pfs_spawn_thread (arg=0x55f1af8ac540) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f779b7fe700 (LWP 45870)): #0 0x00007f77cc6cb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f1ad41541c in LinuxAIOHandler::collect (this=0x7f779b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f1ad415945 in LinuxAIOHandler::poll (this=0x7f779b7fdb00, m1=0x7f779b7fdba8, m2=0x7f779b7fdbb0, request=0x7f779b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f1ad415aa8 in os_aio_linux_handler (global_segment=5, m1=0x7f779b7fdba8, m2=0x7f779b7fdbb0, request=0x7f779b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f1ad41a5cc in os_aio_handler (segment=5, m1=0x7f779b7fdba8, m2=0x7f779b7fdbb0, request=0x7f779b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f1ad66d4a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f1ad513909 in io_handler_thread (arg=0x55f1ae69a648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f7792ffd700 (LWP 45879)): #0 0x00007f77cae9dd47 in sched_yield () from /lib64/libc.so.6 #1 0x000055f1ad422393 in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x000055f1ad53bbd9 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x000055f1ad53bde1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x000055f1ad511cca in srv_do_purge (n_total_purged=0x7f7792ffcd98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x000055f1ad5120f7 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f7767fff700 (LWP 45886)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad8df997 in safe_cond_wait (cond=0x7f7767ffec70, mp=0x7f7767ffecb0, file=0x55f1adbadc40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f1ad339487 in inline_mysql_cond_wait (that=0x7f7767ffec70, mutex=0x7f7767ffecb0, src_file=0x55f1adbaf5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f1ad33e0c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055f1ad87245f in pfs_spawn_thread (arg=0x55f1af75c920) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f77bffff700 (LWP 45824)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f77c4a08fd7 in gu::Lock::wait (cond=..., this=0x7f77bfffebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f77c4a09069 in galera::ServiceThd::thd_func (arg=0x55f1af3d3888) at galera/src/galera_service_thd.cpp:30 #3 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f77c409e700 (LWP 46036)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad8df997 in safe_cond_wait (cond=0x55f1ae61f1e0 , mp=0x55f1ae622560 , file=0x55f1ad9aacc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f1accdd4ee in inline_mysql_cond_wait (that=0x55f1ae61f1e0 , mutex=0x55f1ae622560 , src_file=0x55f1ad9ab830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055f1accded80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055f1ad87245f in pfs_spawn_thread (arg=0x55f1af4df450) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f77c43a7700 (LWP 45827)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad8df997 in safe_cond_wait (cond=0x7f77a8017b68, mp=0x7f77a8017ab8, file=0x55f1ada49b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f1ad037088 in inline_mysql_cond_wait (that=0x7f77a8017b68, mutex=0x7f77a8017ab8, src_file=0x55f1ada49e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f1ad039d20 in Wsrep_thd_queue::pop_front (this=0x7f77a8017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f1ad038740 in wsrep_rollback_process (rollbacker=0x7f77a8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055f1ad029282 in start_wsrep_THD (arg=0x55f1af42ce40) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f7791ffb700 (LWP 45881)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad4220ff in os_event::wait (this=0x55f1af650cf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f1ad421b85 in os_event::wait_low (this=0x55f1af650cf0, reset_sig_count=53) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f1ad421ef6 in os_event_wait_low (event=0x55f1af650cf0, reset_sig_count=53) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f1ad51bbfc in sync_array_wait_event (arr=0x55f1af4f2aa0, cell=@0x7f7791ff8848: 0x55f1af4f2b90) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055f1ad51f1c8 in rw_lock_x_lock_func (lock=0x7f77a6bea878, pass=0, file_name=0x55f1adc90f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:735 #6 0x000055f1ad5ca589 in pfs_rw_lock_x_lock_func (lock=0x7f77a6bea878, pass=0, file_name=0x55f1adc90f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:558 #7 0x000055f1ad5dc2f9 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x55f1adc90f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333, mtr=0x7f7791ffa1a0, err=0x7f7791ff8b74) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4824 #8 0x000055f1ad596253 in btr_block_get_func (page_id=..., zip_size=0, mode=2, file=0x55f1adc90f88 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=333, index=0x55f1af8b72d8, mtr=0x7f7791ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0btr.ic:57 #9 0x000055f1ad59851c in btr_cur_latch_leaves (block=0x7f77a6bea2d0, page_id=..., zip_size=0, latch_mode=33, cursor=0x55f1af73d178, mtr=0x7f7791ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:333 #10 0x000055f1ad59c927 in btr_cur_search_to_nth_level_func (index=0x55f1af8b72d8, level=0, tuple=0x7f777c013368, mode=PAGE_CUR_LE, latch_mode=33, cursor=0x55f1af73d178, ahi_latch=0x0, file=0x55f1adc41d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7f7791ffa1a0, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1792 #11 0x000055f1ad5b4036 in btr_pcur_open_with_no_init_func (index=0x55f1af8b72d8, tuple=0x7f777c013368, mode=PAGE_CUR_LE, latch_mode=65569, cursor=0x55f1af73d178, ahi_latch=0x0, file=0x55f1adc41d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7f7791ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:527 #12 0x000055f1ad5b533f in btr_pcur_restore_position_func (latch_mode=65569, cursor=0x55f1af73d178, file=0x55f1adc41d80 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7f7791ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0pcur.cc:391 #13 0x000055f1ad4cebe6 in row_purge_reposition_pcur (mode=65569, node=0x55f1af73d0e0, mtr=0x7f7791ffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:100 #14 0x000055f1ad4cedb3 in row_purge_remove_clust_if_poss_low (node=0x55f1af73d0e0, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:140 #15 0x000055f1ad4cf209 in row_purge_remove_clust_if_poss (node=0x55f1af73d0e0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #16 0x000055f1ad4d0a36 in row_purge_del_mark (node=0x55f1af73d0e0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #17 0x000055f1ad4d1fbe in row_purge_record_func (node=0x55f1af73d0e0, undo_rec=0x55f1af73d5b8 "", thr=0x55f1af73d028, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #18 0x000055f1ad4d22fb in row_purge (node=0x55f1af73d0e0, undo_rec=0x55f1af73d5b8 "", thr=0x55f1af73d028) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #19 0x000055f1ad4d26a6 in row_purge_step (thr=0x55f1af73d028) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #20 0x000055f1ad454a9a in que_thr_step (thr=0x55f1af73d028) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #21 0x000055f1ad454cca in que_run_threads_low (thr=0x55f1af73d028) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #22 0x000055f1ad454eba in que_run_threads (thr=0x55f1af73d028) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #23 0x000055f1ad511855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #24 0x000055f1ad5119fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #25 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #26 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f776f7fe700 (LWP 45885)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad4220ff in os_event::wait (this=0x55f1af51ba40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f1ad421b85 in os_event::wait_low (this=0x55f1af51ba40, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f1ad421ef6 in os_event_wait_low (event=0x55f1af51ba40, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f1ad5d6b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f77c40e9700 (LWP 45901)): #0 0x00007f77cadf240a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055f1accabf44 in my_sigwait (set=0x7f77c40e8cf0, sig=0x7f77c40e8cb4, code=0x7f77c40e8cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055f1accb2584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055f1ad87245f in pfs_spawn_thread (arg=0x55f1af4def60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f77937fe700 (LWP 45877)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad421a1a in os_event::timed_wait (this=0x55f1af6bbd40, abstime=0x7f77937fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f1ad421d31 in os_event::wait_time_low (this=0x55f1af6bbd40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f1ad421ed1 in os_event_wait_time_low (event=0x55f1af6bbd40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f1ad5702fb in ib_wqueue_timedwait (wq=0x55f1af6bbc10, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055f1ad6b0ea4 in fts_optimize_thread (arg=0x55f1af6bbc10) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f776ffff700 (LWP 45884)): #0 0x00007f77ccb01e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055f1ad422401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055f1ad5c8a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f7790ff9700 (LWP 45883)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad4220ff in os_event::wait (this=0x55f1af51b940) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f1ad421b85 in os_event::wait_low (this=0x55f1af51b940, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f1ad421ef6 in os_event_wait_low (event=0x55f1af51b940, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f1ad5edeb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f7793fff700 (LWP 45876)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad421a1a in os_event::timed_wait (this=0x55f1af725e30, abstime=0x7f7793ffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f1ad421d31 in os_event::wait_time_low (this=0x55f1af725e30, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f1ad421ed1 in os_event_wait_time_low (event=0x55f1af725e30, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f1ad657142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f77997fa700 (LWP 45874)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad421a1a in os_event::timed_wait (this=0x55f1af51b840, abstime=0x7f77997f9ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f1ad421d31 in os_event::wait_time_low (this=0x55f1af51b840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f1ad421ed1 in os_event_wait_time_low (event=0x55f1af51b840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f1ad50fffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f77a5fac700 (LWP 45866)): #0 0x00007f77cc6cb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f1ad41541c in LinuxAIOHandler::collect (this=0x7f77a5fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f1ad415945 in LinuxAIOHandler::poll (this=0x7f77a5fabb00, m1=0x7f77a5fabba8, m2=0x7f77a5fabbb0, request=0x7f77a5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f1ad415aa8 in os_aio_linux_handler (global_segment=1, m1=0x7f77a5fabba8, m2=0x7f77a5fabbb0, request=0x7f77a5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f1ad41a5cc in os_aio_handler (segment=1, m1=0x7f77a5fabba8, m2=0x7f77a5fabbb0, request=0x7f77a5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f1ad66d4a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f1ad513909 in io_handler_thread (arg=0x55f1ae69a628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f7798ff9700 (LWP 45875)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad421a1a in os_event::timed_wait (this=0x55f1af51b8c0, abstime=0x7f7798ff8d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f1ad421d31 in os_event::wait_time_low (this=0x55f1af51b8c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f1ad421ed1 in os_event_wait_time_low (event=0x55f1af51b8c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f1ad50fada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f7799ffb700 (LWP 45873)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad421a1a in os_event::timed_wait (this=0x55f1af6f3d70, abstime=0x7f7799ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f1ad421d31 in os_event::wait_time_low (this=0x55f1af6f3d70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f1ad421ed1 in os_event_wait_time_low (event=0x55f1af6f3d70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f1ad3ee88e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f77c435c700 (LWP 45828)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad8df997 in safe_cond_wait (cond=0x7f77b4015608, mp=0x7f77b4015558, file=0x55f1ada49b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f1ad037088 in inline_mysql_cond_wait (that=0x7f77b4015608, mutex=0x7f77b4015558, src_file=0x55f1ada49e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f1ad039d20 in Wsrep_thd_queue::pop_front (this=0x7f77b4015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f1ad038b00 in wsrep_post_rollback_process (post_rollbacker=0x7f77b4000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055f1ad029282 in start_wsrep_THD (arg=0x55f1af4232d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f77c4311700 (LWP 45829)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f77c48fd132 in fifo_lock_get (q=q@entry=0x55f1af401a90) at galerautils/src/gu_fifo.c:262 #2 0x00007f77c48fd7a1 in gu_fifo_get_head (q=0x55f1af401a90, err=0x7f77c431006c) at galerautils/src/gu_fifo.c:330 #3 0x00007f77c49df0cc in gcs_recv (conn=0x55f1af3db4f0, action=0x7f77c43100b0) at gcs/src/gcs.cpp:2024 #4 0x00007f77c4a0c16c in galera::GcsActionSource::process (this=0x55f1af3db140, recv_ctx=0x7f77c43104d0, exit_loop=@0x7f77c431029e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f77c4a2b033 in galera::ReplicatorSMM::async_recv (this=0x55f1af3d3170, recv_ctx=0x7f77c43104d0) at galera/src/replicator_smm.cpp:383 #6 0x000055f1ad994276 in wsrep::wsrep_provider_v26::run_applier (this=0x55f1af35a360, applier_ctx=0x7f77c43104d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055f1ad037636 in wsrep_replication_process (thd=0x7f77ac000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055f1ad029282 in start_wsrep_THD (arg=0x55f1af42c250) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f77bd37a700 (LWP 45826)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f77c49e3867 in gu::Lock::wait (cond=..., this=0x7f77bd379850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f77c49e63ab in RecvBuf::front (this=this@entry=0x55f1af413718, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f77c49e3f46 in gcomm_recv (backend=, msg=0x55f1af3db7d0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f77c49d7eec in core_msg_recv (backend=backend@entry=0x55f1af3db918, recv_msg=recv_msg@entry=0x55f1af3db7d0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f77c49db164 in gcs_core_recv (conn=0x55f1af3db770, recv_act=recv_act@entry=0x7f77bd379d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f77c49e23d8 in gcs_recv_thread (arg=0x55f1af3db4f0) at gcs/src/gcs.cpp:1398 #7 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f77bdb7b700 (LWP 45825)): #0 0x00007f77caeb9483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f77c49aa6f8 in asio::detail::epoll_reactor::run (this=0x55f1af413f00, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f77c49aedf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55f1af412c10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f77c49af0a2 in asio::detail::task_io_service::run (this=0x55f1af412c10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f77c49c6afd in asio::io_service::run (this=0x55f1af413be0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f77c49c7251 in gcomm::AsioProtonet::event_loop (this=0x55f1af413b40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f77c49e52ed in GCommConn::run (this=0x55f1af413530) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f77c49e8399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f779bfff700 (LWP 45869)): #0 0x00007f77cc6cb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f1ad41541c in LinuxAIOHandler::collect (this=0x7f779bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f1ad415945 in LinuxAIOHandler::poll (this=0x7f779bffeb00, m1=0x7f779bffeba8, m2=0x7f779bffebb0, request=0x7f779bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f1ad415aa8 in os_aio_linux_handler (global_segment=4, m1=0x7f779bffeba8, m2=0x7f779bffebb0, request=0x7f779bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f1ad41a5cc in os_aio_handler (segment=4, m1=0x7f779bffeba8, m2=0x7f779bffebb0, request=0x7f779bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f1ad66d4a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f1ad513909 in io_handler_thread (arg=0x55f1ae69a640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f77a7fff700 (LWP 45864)): #0 0x00007f77ccafed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad8dfc99 in safe_cond_timedwait (cond=0x55f1aeeabea0 , mp=0x55f1aeeabde0 , abstime=0x7f77a7ffecb0, file=0x55f1add23610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f1ad749acb in inline_mysql_cond_timedwait (that=0x55f1aeeabea0 , mutex=0x55f1aeeabde0 , abstime=0x7f77a7ffecb0, src_file=0x55f1add23658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f1ad749fef in my_service_thread_sleep (control=0x55f1ae460b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055f1ad73e488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055f1ad87245f in pfs_spawn_thread (arg=0x55f1af4d3830) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f779affd700 (LWP 45871)): #0 0x00007f77ccafe965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f1ad4220ff in os_event::wait (this=0x55f1af6b3db0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f1ad421b85 in os_event::wait_low (this=0x55f1af6b3db0, reset_sig_count=3051) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f1ad421ef6 in os_event_wait_low (event=0x55f1af6b3db0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f1ad3f2761 in log_write_up_to (lsn=3156408, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000055f1ad5f25f2 in buf_flush_write_block_low (bpage=0x7f77a6bea7b0, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:962 #6 0x000055f1ad5f2fb6 in buf_flush_page (buf_pool=0x55f1af601140, bpage=0x7f77a6bea7b0, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1165 #7 0x000055f1ad5f38fe in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=1, n_to_flush=4) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1389 #8 0x000055f1ad5f3c81 in buf_flush_page_and_try_neighbors (bpage=0x7f77a6bea7b0, flush_type=BUF_FLUSH_LIST, n_to_flush=4, count=0x7f779affc8c8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1461 #9 0x000055f1ad5f489a in buf_do_flush_list_batch (buf_pool=0x55f1af601140, min_n=4, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1719 #10 0x000055f1ad5f4e92 in buf_flush_batch (buf_pool=0x55f1af601140, flush_type=BUF_FLUSH_LIST, min_n=4, lsn_limit=18446744073709551615, n=0x7f779affcb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1789 #11 0x000055f1ad5f5552 in buf_flush_do_batch (buf_pool=0x55f1af601140, type=BUF_FLUSH_LIST, min_n=4, lsn_limit=18446744073709551615, n=0x7f779affcb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1947 #12 0x000055f1ad5f824d in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #13 0x000055f1ad5f919b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #14 0x00007f77ccafadd5 in start_thread () from /lib64/libpthread.so.0 #15 0x00007f77caeb8ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f77ccf09880 (LWP 45782)): #0 0x00007f77ccaff9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f1ad8da368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f1ad118503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f77caeae20d in poll () from /lib64/libc.so.6 #5 0x000055f1accb8a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f1accb7d25 in mysqld_main (argc=161, argv=0x55f1af254d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f1accabec5 in main (argc=23, argv=0x7fff7cfd2f98) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.46107' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.2/data/core.46107 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 46107] [New LWP 46209] [New LWP 46121] [New LWP 46109] [New LWP 46206] [New LWP 46690] [New LWP 46152] [New LWP 46207] [New LWP 46738] [New LWP 46688] [New LWP 46695] [New LWP 46740] [New LWP 46691] [New LWP 46742] [New LWP 46693] [New LWP 46747] [New LWP 46739] [New LWP 46750] [New LWP 46741] [New LWP 46689] [New LWP 46749] [New LWP 46694] [New LWP 46754] [New LWP 46692] [New LWP 46755] [New LWP 46793] [New LWP 46757] [New LWP 46799] [New LWP 46756] [New LWP 46752] [New LWP 46794] [New LWP 46748] [New LWP 46208] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007ff04bac79d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007ff04bac79d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000562ba1634368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000562ba0e72503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff049e7620d in poll () from /lib64/libc.so.6 #5 0x0000562ba0a12a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000562ba0a11d25 in mysqld_main (argc=162, argv=0x562ba3fcbd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000562ba0a05ec5 in main (argc=23, argv=0x7fff1017d878) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7ff04069f700 (LWP 46208)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba1639997 in safe_cond_wait (cond=0x7ff01c015608, mp=0x7ff01c015558, file=0x562ba17a3b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ba0d91088 in inline_mysql_cond_wait (that=0x7ff01c015608, mutex=0x7ff01c015558, src_file=0x562ba17a3e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562ba0d93d20 in Wsrep_thd_queue::pop_front (this=0x7ff01c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000562ba0d92b00 in wsrep_post_rollback_process (post_rollbacker=0x7ff01c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000562ba0d83282 in start_wsrep_THD (arg=0x562ba41a7320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff0017fa700 (LWP 46748)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117c0ff in os_event::wait (this=0x562ba42972f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ba117bb85 in os_event::wait_low (this=0x562ba42972f0, reset_sig_count=2243) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ba117bef6 in os_event_wait_low (event=0x562ba42972f0, reset_sig_count=2243) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ba1267df3 in srv_resume_thread (slot=0x562ba21b6748 , sig_count=2243, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ba126be04 in srv_purge_coordinator_suspend (slot=0x562ba21b6748 , rseg_history_len=18) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000562ba126c09a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff038bba700 (LWP 46794)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba1639997 in safe_cond_wait (cond=0x562ba23791e0 , mp=0x562ba237c560 , file=0x562ba1704cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ba0a374ee in inline_mysql_cond_wait (that=0x562ba23791e0 , mutex=0x562ba237c560 , src_file=0x562ba1705830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000562ba0a38d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000562ba15cc45f in pfs_spawn_thread (arg=0x562ba425b1f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fefeb7fe700 (LWP 46752)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117c0ff in os_event::wait (this=0x562ba42973f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ba117bb85 in os_event::wait_low (this=0x562ba42973f0, reset_sig_count=8423) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ba117bef6 in os_event_wait_low (event=0x562ba42973f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ba1267df3 in srv_resume_thread (slot=0x562ba21b67b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ba126b9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7feff27fc700 (LWP 46756)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117c0ff in os_event::wait (this=0x562ba42976f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ba117bb85 in os_event::wait_low (this=0x562ba42976f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ba117bef6 in os_event_wait_low (event=0x562ba42976f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ba1330b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff038b6f700 (LWP 46799)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba1639c99 in safe_cond_timedwait (cond=0x562ba23791a0 , mp=0x562ba237bba0 , abstime=0x7ff038b6ec60, file=0x562ba16f54c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562ba0a06845 in inline_mysql_cond_timedwait (that=0x562ba23791a0 , mutex=0x562ba237bba0 , abstime=0x7ff038b6ec60, src_file=0x562ba16f7538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562ba0a0b61b in cache_thread (thd=0x7ff020000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000562ba0a0b8c9 in one_thread_per_connection_end (thd=0x7ff020000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000562ba0ca80f3 in do_handle_one_connection (connect=0x562ba4597460) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000562ba0ca7dad in handle_one_connection (arg=0x562ba4597460) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000562ba15cc45f in pfs_spawn_thread (arg=0x562ba45955d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7feff1ffb700 (LWP 46757)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba1639997 in safe_cond_wait (cond=0x7feff1ffac70, mp=0x7feff1ffacb0, file=0x562ba1907c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ba1093487 in inline_mysql_cond_wait (that=0x7feff1ffac70, mutex=0x7feff1ffacb0, src_file=0x562ba19095d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562ba10980c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000562ba15cc45f in pfs_spawn_thread (arg=0x562ba44522b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff038c05700 (LWP 46793)): #0 0x00007ff049dba40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000562ba0a05f44 in my_sigwait (set=0x7ff038c04cf0, sig=0x7ff038c04cb4, code=0x7ff038c04cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000562ba0a0c584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000562ba15cc45f in pfs_spawn_thread (arg=0x562ba425ad00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7feff2ffd700 (LWP 46755)): #0 0x00007ff04bac9e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000562ba117c401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000562ba1322a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff02affd700 (LWP 46692)): #0 0x00007ff04b693644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ba116f41c in LinuxAIOHandler::collect (this=0x7ff02affcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ba116f945 in LinuxAIOHandler::poll (this=0x7ff02affcb00, m1=0x7ff02affcba8, m2=0x7ff02affcbb0, request=0x7ff02affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ba116faa8 in os_aio_linux_handler (global_segment=3, m1=0x7ff02affcba8, m2=0x7ff02affcbb0, request=0x7ff02affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ba11745cc in os_aio_handler (segment=3, m1=0x7ff02affcba8, m2=0x7ff02affcbb0, request=0x7ff02affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ba13c74a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ba126d909 in io_handler_thread (arg=0x562ba23f4638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7feff37fe700 (LWP 46754)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117c0ff in os_event::wait (this=0x562ba42975f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ba117bb85 in os_event::wait_low (this=0x562ba42975f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ba117bef6 in os_event_wait_low (event=0x562ba42975f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ba1347eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff029ffb700 (LWP 46694)): #0 0x00007ff04b693644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ba116f41c in LinuxAIOHandler::collect (this=0x7ff029ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ba116f945 in LinuxAIOHandler::poll (this=0x7ff029ffab00, m1=0x7ff029ffaba8, m2=0x7ff029ffabb0, request=0x7ff029ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ba116faa8 in os_aio_linux_handler (global_segment=5, m1=0x7ff029ffaba8, m2=0x7ff029ffabb0, request=0x7ff029ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ba11745cc in os_aio_handler (segment=5, m1=0x7ff029ffaba8, m2=0x7ff029ffabb0, request=0x7ff029ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ba13c74a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ba126d909 in io_handler_thread (arg=0x562ba23f4648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff000ff9700 (LWP 46749)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117c0ff in os_event::wait (this=0x562ba4297370) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ba117bb85 in os_event::wait_low (this=0x562ba4297370, reset_sig_count=9031) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ba117bef6 in os_event_wait_low (event=0x562ba4297370, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ba1267df3 in srv_resume_thread (slot=0x562ba21b6780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ba126b9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff038a05700 (LWP 46689)): #0 0x00007ff04b693644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ba116f41c in LinuxAIOHandler::collect (this=0x7ff038a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ba116f945 in LinuxAIOHandler::poll (this=0x7ff038a04b00, m1=0x7ff038a04ba8, m2=0x7ff038a04bb0, request=0x7ff038a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ba116faa8 in os_aio_linux_handler (global_segment=0, m1=0x7ff038a04ba8, m2=0x7ff038a04bb0, request=0x7ff038a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ba11745cc in os_aio_handler (segment=0, m1=0x7ff038a04ba8, m2=0x7ff038a04bb0, request=0x7ff038a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ba13c74a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ba126d909 in io_handler_thread (arg=0x562ba23f4620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff002ffd700 (LWP 46741)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117ba1a in os_event::timed_wait (this=0x562ba44a1ac0, abstime=0x7ff002ffcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ba117bd31 in os_event::wait_time_low (this=0x562ba44a1ac0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ba117bed1 in os_event_wait_time_low (event=0x562ba44a1ac0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ba13b1142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7feff3fff700 (LWP 46750)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117c0ff in os_event::wait (this=0x562ba4297470) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ba117bb85 in os_event::wait_low (this=0x562ba4297470, reset_sig_count=8027) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ba117bef6 in os_event_wait_low (event=0x562ba4297470, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ba1267df3 in srv_resume_thread (slot=0x562ba21b67f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ba126b9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff028ff9700 (LWP 46739)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117ba1a in os_event::timed_wait (this=0x562ba42974f0, abstime=0x7ff028ff8ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ba117bd31 in os_event::wait_time_low (this=0x562ba42974f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ba117bed1 in os_event_wait_time_low (event=0x562ba42974f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ba1269ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff001ffb700 (LWP 46747)): #0 0x00007ff04bac9e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000562ba117c401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000562ba126b35b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000562ba126b48b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff02a7fc700 (LWP 46693)): #0 0x00007ff04b693644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ba116f41c in LinuxAIOHandler::collect (this=0x7ff02a7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ba116f945 in LinuxAIOHandler::poll (this=0x7ff02a7fbb00, m1=0x7ff02a7fbba8, m2=0x7ff02a7fbbb0, request=0x7ff02a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ba116faa8 in os_aio_linux_handler (global_segment=4, m1=0x7ff02a7fbba8, m2=0x7ff02a7fbbb0, request=0x7ff02a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ba11745cc in os_aio_handler (segment=4, m1=0x7ff02a7fbba8, m2=0x7ff02a7fbbb0, request=0x7ff02a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ba13c74a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ba126d909 in io_handler_thread (arg=0x562ba23f4640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff0027fc700 (LWP 46742)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117ba1a in os_event::timed_wait (this=0x562ba443f470, abstime=0x7ff0027fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ba117bd31 in os_event::wait_time_low (this=0x562ba443f470, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ba117bed1 in os_event_wait_time_low (event=0x562ba443f470, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ba12ca2fb in ib_wqueue_timedwait (wq=0x562ba443f340, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000562ba140aea4 in fts_optimize_thread (arg=0x562ba443f340) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff02b7fe700 (LWP 46691)): #0 0x00007ff04b693644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ba116f41c in LinuxAIOHandler::collect (this=0x7ff02b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ba116f945 in LinuxAIOHandler::poll (this=0x7ff02b7fdb00, m1=0x7ff02b7fdba8, m2=0x7ff02b7fdbb0, request=0x7ff02b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ba116faa8 in os_aio_linux_handler (global_segment=2, m1=0x7ff02b7fdba8, m2=0x7ff02b7fdbb0, request=0x7ff02b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ba11745cc in os_aio_handler (segment=2, m1=0x7ff02b7fdba8, m2=0x7ff02b7fdbb0, request=0x7ff02b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ba13c74a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ba126d909 in io_handler_thread (arg=0x562ba23f4630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff0037fe700 (LWP 46740)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117ba1a in os_event::timed_wait (this=0x562ba4297570, abstime=0x7ff0037fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ba117bd31 in os_event::wait_time_low (this=0x562ba4297570, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ba117bed1 in os_event_wait_time_low (event=0x562ba4297570, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ba1269ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff0297fa700 (LWP 46695)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117ba1a in os_event::timed_wait (this=0x562ba4297670, abstime=0x7ff0297f9af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ba117bd31 in os_event::wait_time_low (this=0x562ba4297670, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ba117bed1 in os_event_wait_time_low (event=0x562ba4297670, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ba1351d1d in pc_sleep_if_needed (next_loop_time=1549882726955, sig_count=2, cur_time=1549882725955) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000562ba1352c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff03a057700 (LWP 46688)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba1639c99 in safe_cond_timedwait (cond=0x562ba2c05ea0 , mp=0x562ba2c05de0 , abstime=0x7ff03a056cb0, file=0x562ba1a7d610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562ba14a3acb in inline_mysql_cond_timedwait (that=0x562ba2c05ea0 , mutex=0x562ba2c05de0 , abstime=0x7ff03a056cb0, src_file=0x562ba1a7d658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562ba14a3fef in my_service_thread_sleep (control=0x562ba21bab00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000562ba1498488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000562ba15cc45f in pfs_spawn_thread (arg=0x562ba424f690) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff003fff700 (LWP 46738)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba117ba1a in os_event::timed_wait (this=0x562ba446fa00, abstime=0x7ff003ffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ba117bd31 in os_event::wait_time_low (this=0x562ba446fa00, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ba117bed1 in os_event_wait_time_low (event=0x562ba446fa00, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ba114888e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff0406ea700 (LWP 46207)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba1639997 in safe_cond_wait (cond=0x7ff024016858, mp=0x7ff0240167a8, file=0x562ba17a3b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ba0d91088 in inline_mysql_cond_wait (that=0x7ff024016858, mutex=0x7ff0240167a8, src_file=0x562ba17a3e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562ba0d93d20 in Wsrep_thd_queue::pop_front (this=0x7ff024016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000562ba0d92740 in wsrep_rollback_process (rollbacker=0x7ff024001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000562ba0d83282 in start_wsrep_THD (arg=0x562ba41a8100) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff040eeb700 (LWP 46152)): #0 0x00007ff049e81483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff0439726f8 in asio::detail::epoll_reactor::run (this=0x562ba4186f50, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007ff043976df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x562ba4186df0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007ff0439770a2 in asio::detail::task_io_service::run (this=0x562ba4186df0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007ff04398eafd in asio::io_service::run (this=0x562ba4185b50) at asio/asio/impl/io_service.ipp:58 #5 0x00007ff04398f251 in gcomm::AsioProtonet::event_loop (this=0x562ba4185ab0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007ff0439ad2ed in GCommConn::run (this=0x562ba4186530) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007ff0439b0399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff02bfff700 (LWP 46690)): #0 0x00007ff04b693644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ba116f41c in LinuxAIOHandler::collect (this=0x7ff02bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ba116f945 in LinuxAIOHandler::poll (this=0x7ff02bffeb00, m1=0x7ff02bffeba8, m2=0x7ff02bffebb0, request=0x7ff02bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ba116faa8 in os_aio_linux_handler (global_segment=1, m1=0x7ff02bffeba8, m2=0x7ff02bffebb0, request=0x7ff02bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ba11745cc in os_aio_handler (segment=1, m1=0x7ff02bffeba8, m2=0x7ff02bffebb0, request=0x7ff02bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ba13c74a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ba126d909 in io_handler_thread (arg=0x562ba23f4628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff03bfff700 (LWP 46206)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff0439ab867 in gu::Lock::wait (cond=..., this=0x7ff03bffe850) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff0439ae3ab in RecvBuf::front (this=this@entry=0x562ba4186718, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007ff0439abf46 in gcomm_recv (backend=, msg=0x562ba41527d0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007ff04399feec in core_msg_recv (backend=backend@entry=0x562ba4152918, recv_msg=recv_msg@entry=0x562ba41527d0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007ff0439a3164 in gcs_core_recv (conn=0x562ba4152770, recv_act=recv_act@entry=0x7ff03bffed70, timeout=9223372035999worker[2] Trying to dump core for [mysqltest - pid: 67558, winpid: 67558] worker[2] Trying to dump core for [mysqld.1 - pid: 66320, winpid: 66320] worker[2] Trying to dump core for [mysqld.2 - pid: 66872, winpid: 66872] 999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007ff0439aa3d8 in gcs_recv_thread (arg=0x562ba41524f0) at gcs/src/gcs.cpp:1398 #7 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff04451b700 (LWP 46109)): #0 0x00007ff04bac6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ba1639c99 in safe_cond_timedwait (cond=0x562ba2c18640 , mp=0x562ba2c18580 , abstime=0x7ff04451ad70, file=0x562ba1ac1840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562ba163ac53 in inline_mysql_cond_timedwait (that=0x562ba2c18640 , mutex=0x562ba2c18580 , abstime=0x7ff04451ad70, src_file=0x562ba1ac1888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562ba163b7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000562ba15cc45f in pfs_spawn_thread (arg=0x562ba41137d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff04336f700 (LWP 46121)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff0439d0fd7 in gu::Lock::wait (cond=..., this=0x7ff04336ebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff0439d1069 in galera::ServiceThd::thd_func (arg=0x562ba414a888) at galera/src/galera_service_thd.cpp:30 #3 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff040654700 (LWP 46209)): #0 0x00007ff04bac6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff0438c5132 in fifo_lock_get (q=q@entry=0x562ba4178a90) at galerautils/src/gu_fifo.c:262 #2 0x00007ff0438c57a1 in gu_fifo_get_head (q=0x562ba4178a90, err=0x7ff04065306c) at galerautils/src/gu_fifo.c:330 #3 0x00007ff0439a70cc in gcs_recv (conn=0x562ba41524f0, action=0x7ff0406530b0) at gcs/src/gcs.cpp:2024 #4 0x00007ff0439d416c in galera::GcsActionSource::process (this=0x562ba4152140, recv_ctx=0x7ff0406534d0, exit_loop=@0x7ff04065329e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007ff0439f3033 in galera::ReplicatorSMM::async_recv (this=0x562ba414a170, recv_ctx=0x7ff0406534d0) at galera/src/replicator_smm.cpp:383 #6 0x0000562ba16ee276 in wsrep::wsrep_provider_v26::run_applier (this=0x562ba40d1360, applier_ctx=0x7ff0406534d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000562ba0d91636 in wsrep_replication_process (thd=0x7ff030000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000562ba0d83282 in start_wsrep_THD (arg=0x562ba41a76d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007ff04bac2dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff049e80ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff04bed3880 (LWP 46107)): #0 0x00007ff04bac79d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000562ba1634368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000562ba0e72503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff049e7620d in poll () from /lib64/libc.so.6 #5 0x0000562ba0a12a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000562ba0a11d25 in mysqld_main (argc=162, argv=0x562ba3fcbd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000562ba0a05ec5 in main (argc=23, argv=0x7fff1017d878) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_small_gcache-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_small_gcache-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_small_gcache Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.mysql-wsrep-features#9 'innodb' w1 [ pass ] 26536 galera_sr.GCF-572 'innodb' w1 [ pass ] 7146 galera_sr.GCF-580 'innodb' w1 [ pass ] 7649 galera_sr.GCF-585 'innodb' w1 [ pass ] 36198 galera_sr.GCF-597 'innodb' w1 [ pass ] 10629 galera_sr.GCF-620 'innodb' w1 [ pass ] 8603 galera_sr.GCF-1043A 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 13:08:08 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/GCF-1043A.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 REPLACE INTO t1 VALUES (1,'y'),(2,'x') REPLACE INTO t1 VALUES (1,'y'),(2,'y'),(3,'y') connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.66321' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.GCF-1043A-innodb/mysqld.1/data/core.66321 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 66321] [New LWP 66338] [New LWP 66382] [New LWP 66380] [New LWP 66439] [New LWP 66443] [New LWP 66440] [New LWP 66445] [New LWP 66470] [New LWP 66468] [New LWP 66474] [New LWP 66469] [New LWP 66476] [New LWP 66475] [New LWP 66488] [New LWP 66486] [New LWP 66489] [New LWP 66336] [New LWP 66753] [New LWP 66521] [New LWP 66441] [New LWP 67473] [New LWP 66442] [New LWP 66446] [New LWP 66471] [New LWP 66478] [New LWP 66487] [New LWP 66754] [New LWP 67644] [New LWP 66477] [New LWP 66467] [New LWP 66444] [New LWP 66381] [New LWP 66379] [New LWP 66334] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f385e8639d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f385e8639d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f6e5808368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f6e5046503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f385cc1220d in poll () from /lib64/libc.so.6 #5 0x000055f6e4be6a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f6e4be5d25 in mysqld_main (argc=161, argv=0x55f6e8ca4d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f6e4bd9ec5 in main (argc=23, argv=0x7fff18b447b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f38572b7700 (LWP 66334)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580dc99 in safe_cond_timedwait (cond=0x55f6e6dec640 , mp=0x55f6e6dec580 , abstime=0x7f38572b6d70, file=0x55f6e5c95840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f6e580ec53 in inline_mysql_cond_timedwait (that=0x55f6e6dec640 , mutex=0x55f6e6dec580 , abstime=0x7f38572b6d70, src_file=0x55f6e5c95888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f6e580f7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e8dec730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f384e47c700 (LWP 66379)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3856747867 in gu::Lock::wait (cond=..., this=0x7f384e47b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f385674a3ab in RecvBuf::front (this=this@entry=0x55f6e8e638a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f3856747f46 in gcomm_recv (backend=, msg=0x55f6e8e2b830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f385673beec in core_msg_recv (backend=backend@entry=0x55f6e8e2b978, recv_msg=recv_msg@entry=0x55f6e8e2b830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f385673f164 in gcs_core_recv (conn=0x55f6e8e2b7d0, recv_act=recv_act@entry=0x7f384e47bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f38567463d8 in gcs_recv_thread (arg=0x55f6e8e2b550) at gcs/src/gcs.cpp:1398 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f38545bd700 (LWP 66381)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580d997 in safe_cond_wait (cond=0x7f3844017b68, mp=0x7f3844017ab8, file=0x55f6e5977b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f6e4f65088 in inline_mysql_cond_wait (that=0x7f3844017b68, mutex=0x7f3844017ab8, src_file=0x55f6e5977e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f6e4f67d20 in Wsrep_thd_queue::pop_front (this=0x7f3844017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f6e4f66b00 in wsrep_post_rollback_process (post_rollbacker=0x7f3844001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055f6e4f57282 in start_wsrep_THD (arg=0x55f6e8e5d810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f383dbfb700 (LWP 66444)): #0 0x00007f385e42f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f6e534341c in LinuxAIOHandler::collect (this=0x7f383dbfab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f6e5343945 in LinuxAIOHandler::poll (this=0x7f383dbfab00, m1=0x7f383dbfaba8, m2=0x7f383dbfabb0, request=0x7f383dbfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f6e5343aa8 in os_aio_linux_handler (global_segment=4, m1=0x7f383dbfaba8, m2=0x7f383dbfabb0, request=0x7f383dbfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f6e53485cc in os_aio_handler (segment=4, m1=0x7f383dbfaba8, m2=0x7f383dbfabb0, request=0x7f383dbfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f6e559b4a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f6e5441909 in io_handler_thread (arg=0x55f6e65c8640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f38277fe700 (LWP 66467)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e534fa1a in os_event::timed_wait (this=0x55f6e9143fb0, abstime=0x7f38277fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f6e534fd31 in os_event::wait_time_low (this=0x55f6e9143fb0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f6e534fed1 in os_event_wait_time_low (event=0x55f6e9143fb0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f6e531c88e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f381f7fe700 (LWP 66477)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e53500ff in os_event::wait (this=0x55f6e8f6b900) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f6e534fb85 in os_event::wait_low (this=0x55f6e8f6b900, reset_sig_count=8952) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f6e534fef6 in os_event_wait_low (event=0x55f6e8f6b900, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f6e543bdf3 in srv_resume_thread (slot=0x55f6e638a780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f6e543f9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f384c084700 (LWP 67644)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580dc99 in safe_cond_timedwait (cond=0x55f6e654d1a0 , mp=0x55f6e654fba0 , abstime=0x7f384c083c60, file=0x55f6e58c94c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f6e4bda845 in inline_mysql_cond_timedwait (that=0x55f6e654d1a0 , mutex=0x55f6e654fba0 , abstime=0x7f384c083c60, src_file=0x55f6e58cb538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f6e4bdf61b in cache_thread (thd=0x7f37d8000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055f6e4bdf8c9 in one_thread_per_connection_end (thd=0x7f37d8000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055f6e4e7c0f3 in do_handle_one_connection (connect=0x55f6e9202c90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055f6e4e7bdad in handle_one_connection (arg=0x55f6e9202c90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e8e83a60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f38540cd700 (LWP 66754)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580dc99 in safe_cond_timedwait (cond=0x55f6e654d1a0 , mp=0x55f6e654fba0 , abstime=0x7f38540ccc60, file=0x55f6e58c94c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f6e4bda845 in inline_mysql_cond_timedwait (that=0x55f6e654d1a0 , mutex=0x55f6e654fba0 , abstime=0x7f38540ccc60, src_file=0x55f6e58cb538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f6e4bdf61b in cache_thread (thd=0x7f37e0000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055f6e4bdf8c9 in one_thread_per_connection_end (thd=0x7f37e0000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055f6e4e7c0f3 in do_handle_one_connection (connect=0x55f6e92da6c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055f6e4e7bdad in handle_one_connection (arg=0x55f6e92da6c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e91bbf60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f381dffb700 (LWP 66487)): #0 0x00007f385e865e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055f6e5350401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055f6e54f6a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f381effd700 (LWP 66478)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e53500ff in os_event::wait (this=0x55f6e8f6b980) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f6e534fb85 in os_event::wait_low (this=0x55f6e8f6b980, reset_sig_count=8090) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f6e534fef6 in os_event_wait_low (event=0x55f6e8f6b980, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f6e543bdf3 in srv_resume_thread (slot=0x55f6e638a7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f6e543f9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f3825ffb700 (LWP 66471)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e534fa1a in os_event::timed_wait (this=0x55f6e910bd00, abstime=0x7f3825ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f6e534fd31 in os_event::wait_time_low (this=0x55f6e910bd00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f6e534fed1 in os_event_wait_time_low (event=0x55f6e910bd00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f6e549e2fb in ib_wqueue_timedwait (wq=0x55f6e910bbd0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055f6e55deea4 in fts_optimize_thread (arg=0x55f6e910bbd0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f383cbf9700 (LWP 66446)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e534fa1a in os_event::timed_wait (this=0x55f6e8f6bc00, abstime=0x7f383cbf8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f6e534fd31 in os_event::wait_time_low (this=0x55f6e8f6bc00, time_in_usec=386000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f6e534fed1 in os_event_wait_time_low (event=0x55f6e8f6bc00, time_in_usec=386000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f6e5525d1d in pc_sleep_if_needed (next_loop_time=1549883226972, sig_count=2, cur_time=1549883226586) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055f6e5526c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f383ebfd700 (LWP 66442)): #0 0x00007f385e42f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f6e534341c in LinuxAIOHandler::collect (this=0x7f383ebfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f6e5343945 in LinuxAIOHandler::poll (this=0x7f383ebfcb00, m1=0x7f383ebfcba8, m2=0x7f383ebfcbb0, request=0x7f383ebfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f6e5343aa8 in os_aio_linux_handler (global_segment=2, m1=0x7f383ebfcba8, m2=0x7f383ebfcbb0, request=0x7f383ebfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f6e53485cc in os_aio_handler (segment=2, m1=0x7f383ebfcba8, m2=0x7f383ebfcbb0, request=0x7f383ebfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f6e559b4a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f6e5441909 in io_handler_thread (arg=0x55f6e65c8630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f3854082700 (LWP 67473)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580dc99 in safe_cond_timedwait (cond=0x55f6e654d1a0 , mp=0x55f6e654fba0 , abstime=0x7f3854081c60, file=0x55f6e58c94c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f6e4bda845 in inline_mysql_cond_timedwait (that=0x55f6e654d1a0 , mutex=0x55f6e654fba0 , abstime=0x7f3854081c60, src_file=0x55f6e58cb538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f6e4bdf61b in cache_thread (thd=0x7f37dc000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055f6e4bdf8c9 in one_thread_per_connection_end (thd=0x7f37dc000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055f6e4e7c0f3 in do_handle_one_connection (connect=0x55f6e9202c90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055f6e4e7bdad in handle_one_connection (arg=0x55f6e9202c90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e8ddfa70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f383f3fe700 (LWP 66441)): #0 0x00007f385e42f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f6e534341c in LinuxAIOHandler::collect (this=0x7f383f3fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f6e5343945 in LinuxAIOHandler::poll (this=0x7f383f3fdb00, m1=0x7f383f3fdba8, m2=0x7f383f3fdbb0, request=0x7f383f3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f6e5343aa8 in os_aio_linux_handler (global_segment=1, m1=0x7f383f3fdba8, m2=0x7f383f3fdbb0, request=0x7f383f3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f6e53485cc in os_aio_handler (segment=1, m1=0x7f383f3fdba8, m2=0x7f383f3fdbb0, request=0x7f383f3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f6e559b4a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f6e5441909 in io_handler_thread (arg=0x55f6e65c8628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f3854163700 (LWP 66521)): #0 0x00007f385cb5640a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055f6e4bd9f44 in my_sigwait (set=0x7f3854162cf0, sig=0x7f3854162cb4, code=0x7f3854162cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055f6e4be0584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e91e8af0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f3854118700 (LWP 66753)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580d997 in safe_cond_wait (cond=0x55f6e654d1e0 , mp=0x55f6e6550560 , file=0x55f6e58d8cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f6e4c0b4ee in inline_mysql_cond_wait (that=0x55f6e654d1e0 , mutex=0x55f6e6550560 , src_file=0x55f6e58d9830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055f6e4c0cd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e9269850) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f385570a700 (LWP 66336)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f385676cfd7 in gu::Lock::wait (cond=..., this=0x7f3855709bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f385676d069 in galera::ServiceThd::thd_func (arg=0x55f6e8e23998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f381cff9700 (LWP 66489)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580d997 in safe_cond_wait (cond=0x7f381cff8c70, mp=0x7f381cff8cb0, file=0x55f6e5adbc40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f6e5267487 in inline_mysql_cond_wait (that=0x7f381cff8c70, mutex=0x7f381cff8cb0, src_file=0x55f6e5add5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f6e526c0c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e91ac910) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f381e7fc700 (LWP 66486)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e53500ff in os_event::wait (this=0x55f6e8f6bb80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f6e534fb85 in os_event::wait_low (this=0x55f6e8f6bb80, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f6e534fef6 in os_event_wait_low (event=0x55f6e8f6bb80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f6e551beb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f381d7fa700 (LWP 66488)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e53500ff in os_event::wait (this=0x55f6e8f6bc80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f6e534fb85 in os_event::wait_low (this=0x55f6e8f6bc80, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f6e534fef6 in os_event_wait_low (event=0x55f6e8f6bc80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f6e5504b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f3824ff9700 (LWP 66475)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e53500ff in os_event::wait (this=0x55f6e8f6b880) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f6e534fb85 in os_event::wait_low (this=0x55f6e8f6b880, reset_sig_count=2157) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f6e534fef6 in os_event_wait_low (event=0x55f6e8f6b880, reset_sig_count=2157) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f6e543bdf3 in srv_resume_thread (slot=0x55f6e638a748 , sig_count=2157, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f6e543fe04 in srv_purge_coordinator_suspend (slot=0x55f6e638a748 , rseg_history_len=12) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055f6e544009a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f381ffff700 (LWP 66476)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e53500ff in os_event::wait (this=0x55f6e8f6ba00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f6e534fb85 in os_event::wait_low (this=0x55f6e8f6ba00, reset_sig_count=7942) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f6e534fef6 in os_event_wait_low (event=0x55f6e8f6ba00, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f6e543bdf3 in srv_resume_thread (slot=0x55f6e638a7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f6e543f9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f3826ffd700 (LWP 66469)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e534fa1a in os_event::timed_wait (this=0x55f6e8f6bb00, abstime=0x7f3826ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f6e534fd31 in os_event::wait_time_low (this=0x55f6e8f6bb00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f6e534fed1 in os_event_wait_time_low (event=0x55f6e8f6bb00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f6e543dada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f38257fa700 (LWP 66474)): #0 0x00007f385e865e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055f6e5350401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055f6e543f35b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055f6e543f48b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f3827fff700 (LWP 66468)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e534fa1a in os_event::timed_wait (this=0x55f6e8f6ba80, abstime=0x7f3827ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f6e534fd31 in os_event::wait_time_low (this=0x55f6e8f6ba80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f6e534fed1 in os_event_wait_time_low (event=0x55f6e8f6ba80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f6e543dffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f38267fc700 (LWP 66470)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e534fa1a in os_event::timed_wait (this=0x55f6e9176070, abstime=0x7f38267fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f6e534fd31 in os_event::wait_time_low (this=0x55f6e9176070, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f6e534fed1 in os_event_wait_time_low (event=0x55f6e9176070, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f6e5585142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f383d3fa700 (LWP 66445)): #0 0x00007f385e42f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f6e534341c in LinuxAIOHandler::collect (this=0x7f383d3f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f6e5343945 in LinuxAIOHandler::poll (this=0x7f383d3f9b00, m1=0x7f383d3f9ba8, m2=0x7f383d3f9bb0, request=0x7f383d3f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f6e5343aa8 in os_aio_linux_handler (global_segment=5, m1=0x7f383d3f9ba8, m2=0x7f383d3f9bb0, request=0x7f383d3f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f6e53485cc in os_aio_handler (segment=5, m1=0x7f383d3f9ba8, m2=0x7f383d3f9bb0, request=0x7f383d3f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f6e559b4a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f6e5441909 in io_handler_thread (arg=0x55f6e65c8648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f383fbff700 (LWP 66440)): #0 0x00007f385e42f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f6e534341c in LinuxAIOHandler::collect (this=0x7f383fbfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f6e5343945 in LinuxAIOHandler::poll (this=0x7f383fbfeb00, m1=0x7f383fbfeba8, m2=0x7f383fbfebb0, request=0x7f383fbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f6e5343aa8 in os_aio_linux_handler (global_segment=0, m1=0x7f383fbfeba8, m2=0x7f383fbfebb0, request=0x7f383fbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f6e53485cc in os_aio_handler (segment=0, m1=0x7f383fbfeba8, m2=0x7f383fbfebb0, request=0x7f383fbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f6e559b4a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f6e5441909 in io_handler_thread (arg=0x55f6e65c8620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f383e3fc700 (LWP 66443)): #0 0x00007f385e42f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f6e534341c in LinuxAIOHandler::collect (this=0x7f383e3fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f6e5343945 in LinuxAIOHandler::poll (this=0x7f383e3fbb00, m1=0x7f383e3fbba8, m2=0x7f383e3fbbb0, request=0x7f383e3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f6e5343aa8 in os_aio_linux_handler (global_segment=3, m1=0x7f383e3fbba8, m2=0x7f383e3fbbb0, request=0x7f383e3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f6e53485cc in os_aio_handler (segment=3, m1=0x7f383e3fbba8, m2=0x7f383e3fbbb0, request=0x7f383e3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f6e559b4a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f6e5441909 in io_handler_thread (arg=0x55f6e65c8638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f384d4d6700 (LWP 66439)): #0 0x00007f385e862d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580dc99 in safe_cond_timedwait (cond=0x55f6e6dd9ea0 , mp=0x55f6e6dd9de0 , abstime=0x7f384d4d5cb0, file=0x55f6e5c51610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f6e5677acb in inline_mysql_cond_timedwait (that=0x55f6e6dd9ea0 , mutex=0x55f6e6dd9de0 , abstime=0x7f384d4d5cb0, src_file=0x55f6e5c51658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f6e5677fef in my_service_thread_sleep (control=0x55f6e638eb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055f6e566c488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055f6e57a045f in pfs_spawn_thread (arg=0x55f6e8f237c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f3854608700 (LWP 66380)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f6e580d997 in safe_cond_wait (cond=0x7f3838015608, mp=0x7f3838015558, file=0x55f6e5977b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f6e4f65088 in inline_mysql_cond_wait (that=0x7f3838015608, mutex=0x7f3838015558, src_file=0x55f6e5977e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f6e4f67d20 in Wsrep_thd_queue::pop_front (this=0x7f3838015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f6e4f66740 in wsrep_rollback_process (rollbacker=0x7f3838000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055f6e4f57282 in start_wsrep_THD (arg=0x55f6e8e73320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f3854572700 (LWP 66382)): #0 0x00007f385e862965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3856661132 in fifo_lock_get (q=q@entry=0x55f6e8e51bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f38566617a1 in gu_fifo_get_head (q=0x55f6e8e51bd0, err=0x7f385457106c) at galerautils/src/gu_fifo.c:330 #3 0x00007f38567430cc in gcs_recv (conn=0x55f6e8e2b550, action=0x7f38545710b0) at gcs/src/gcs.cpp:2024 #4 0x00007f385677016c in galera::GcsActionSource::process (this=0x55f6e8e2af30, recv_ctx=0x7f38545714d0, exit_loop=@0x7f385457129e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f385678f033 in galera::ReplicatorSMM::async_recv (this=0x55f6e8e23280, recv_ctx=0x7f38545714d0) at galera/src/replicator_smm.cpp:383 #6 0x000055f6e58c2276 in wsrep::wsrep_provider_v26::run_applier (this=0x55f6e8daa360, applier_ctx=0x7f38545714d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055f6e4f65636 in wsrep_replication_process (thd=0x7f3830000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055f6e4f57282 in start_wsrep_THD (arg=0x55f6e8e7bc00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f384ec7d700 (LWP 66338)): #0 0x00007f385cc1d483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f385670e6f8 in asio::detail::epoll_reactor::run (this=0x55f6e8e64040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f3856712df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55f6e8e63ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f38567130a2 in asio::detail::task_io_service::run (this=0x55f6e8e63ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f385672aafd in asio::io_service::run (this=0x55f6e8e62fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f385672b251 in gcomm::AsioProtonet::event_loop (this=0x55f6e8e62f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f38567492ed in GCommConn::run (this=0x55f6e8e636c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f385674c399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f385e85edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f385cc1cead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f385ec6f880 (LWP 66321)): #0 0x00007f385e8639d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f6e5808368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f6e5046503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f385cc1220d in poll () from /lib64/libc.so.6 #5 0x000055f6e4be6a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f6e4be5d25 in mysqld_main (argc=161, argv=0x55f6e8ca4d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f6e4bd9ec5 in main (argc=23, argv=0x7fff18b447b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.66875' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.GCF-1043A-innodb/mysqld.2/data/core.66875 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 66875] [New LWP 66894] [New LWP 66927] [New LWP 66928] [New LWP 67280] [New LWP 67281] [New LWP 67284] [New LWP 67285] [New LWP 67330] [New LWP 67331] [New LWP 67336] [New LWP 67339] [New LWP 67341] [New LWP 67342] [New LWP 67343] [New LWP 67283] [New LWP 66892] [New LWP 66926] [New LWP 67286] [New LWP 67287] [New LWP 67334] [New LWP 67337] [New LWP 67344] [New LWP 67350] [New LWP 67340] [New LWP 67338] [New LWP 67335] [New LWP 67333] [New LWP 67332] [New LWP 66925] [New LWP 67345] [New LWP 66893] [New LWP 67282] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f4fa1bc79d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f4fa1bc79d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559b67079368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559b668b7503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4f9ff7620d in poll () from /lib64/libc.so.6 #5 0x0000559b66457a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559b66456d25 in mysqld_main (argc=162, argv=0x559b6a32bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559b6644aec5 in main (argc=23, argv=0x7ffe4ae97168) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f4f6a1ac700 (LWP 67282)): #0 0x00007f4fa1793644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559b66bb441c in LinuxAIOHandler::collect (this=0x7f4f6a1abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559b66bb4945 in LinuxAIOHandler::poll (this=0x7f4f6a1abb00, m1=0x7f4f6a1abba8, m2=0x7f4f6a1abbb0, request=0x7f4f6a1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559b66bb4aa8 in os_aio_linux_handler (global_segment=1, m1=0x7f4f6a1abba8, m2=0x7f4f6a1abbb0, request=0x7f4f6a1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559b66bb95cc in os_aio_handler (segment=1, m1=0x7f4f6a1abba8, m2=0x7f4f6a1abbb0, request=0x7f4f6a1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559b66e0c4a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559b66cb2909 in io_handler_thread (arg=0x559b67e39628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f4f9946f700 (LWP 66893)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4f99ad0fd7 in gu::Lock::wait (cond=..., this=0x7f4f9946ebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4f99ad1069 in galera::ServiceThd::thd_func (arg=0x559b6a4aa908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f4f98064700 (LWP 67345)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b6707e997 in safe_cond_wait (cond=0x559b67dbe1e0 , mp=0x559b67dc1560 , file=0x559b67149cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559b6647c4ee in inline_mysql_cond_wait (that=0x559b67dbe1e0 , mutex=0x559b67dc1560 , src_file=0x559b6714a830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000559b6647dd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559b6701145f in pfs_spawn_thread (arg=0x559b6a95dda0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f4f8a47b700 (LWP 66925)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4f99aab867 in gu::Lock::wait (cond=..., this=0x7f4f8a47a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4f99aae3ab in RecvBuf::front (this=this@entry=0x559b6a4ea818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f4f99aabf46 in gcomm_recv (backend=, msg=0x559b6a4b2720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f4f99a9feec in core_msg_recv (backend=backend@entry=0x559b6a4b2868, recv_msg=recv_msg@entry=0x559b6a4b2720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f4f99aa3164 in gcs_core_recv (conn=0x559b6a4b26c0, recv_act=recv_act@entry=0x7f4f8a47ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f4f99aaa3d8 in gcs_recv_thread (arg=0x559b6a4b2440) at gcs/src/gcs.cpp:1398 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f4f5ddf6700 (LWP 67332)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc0a1a in os_event::timed_wait (this=0x559b6a5fb500, abstime=0x7f4f5ddf5d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559b66bc0d31 in os_event::wait_time_low (this=0x559b6a5fb500, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559b66bc0ed1 in os_event_wait_time_low (event=0x559b6a5fb500, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559b66caeada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4f5d5f5700 (LWP 67333)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc0a1a in os_event::timed_wait (this=0x559b6a805a70, abstime=0x7f4f5d5f4d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559b66bc0d31 in os_event::wait_time_low (this=0x559b6a805a70, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559b66bc0ed1 in os_event_wait_time_low (event=0x559b6a805a70, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559b66df6142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f4f4ffff700 (LWP 67335)): #0 0x00007f4fa1bc9e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559b66bc1401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559b66cb035b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000559b66cb048b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f4f4e7fc700 (LWP 67338)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc10ff in os_event::wait (this=0x559b6a5fb400) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559b66bc0b85 in os_event::wait_low (this=0x559b6a5fb400, reset_sig_count=7491) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559b66bc0ef6 in os_event_wait_low (event=0x559b6a5fb400, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559b66cacdf3 in srv_resume_thread (slot=0x559b67bfb7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559b66cb09f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4f4d7fa700 (LWP 67340)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc10ff in os_event::wait (this=0x559b6a5fb580) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559b66bc0b85 in os_event::wait_low (this=0x559b6a5fb580, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559b66bc0ef6 in os_event_wait_low (event=0x559b6a5fb580, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559b66d8ceb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4f880ec700 (LWP 67350)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b6707ec99 in safe_cond_timedwait (cond=0x559b67dbe1a0 , mp=0x559b67dc0ba0 , abstime=0x7f4f880ebc60, file=0x559b6713a4c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559b6644b845 in inline_mysql_cond_timedwait (that=0x559b67dbe1a0 , mutex=0x559b67dc0ba0 , abstime=0x7f4f880ebc60, src_file=0x559b6713c538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559b6645061b in cache_thread (thd=0x7f4f6c000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000559b664508c9 in one_thread_per_connection_end (thd=0x7f4f6c000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000559b666ed0f3 in do_handle_one_connection (connect=0x559b6a8fb440) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000559b666ecdad in handle_one_connection (arg=0x559b6a8fb440) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000559b6701145f in pfs_spawn_thread (arg=0x559b6a9295d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f4f980af700 (LWP 67344)): #0 0x00007f4f9feba40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000559b6644af44 in my_sigwait (set=0x7f4f980aecf0, sig=0x7f4f980aecb4, code=0x7f4f980aecb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000559b66451584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000559b6701145f in pfs_spawn_thread (arg=0x559b6a5bed30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f4f4effd700 (LWP 67337)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc10ff in os_event::wait (this=0x559b6a5fb380) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559b66bc0b85 in os_event::wait_low (this=0x559b6a5fb380, reset_sig_count=7691) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559b66bc0ef6 in os_event_wait_low (event=0x559b6a5fb380, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559b66cacdf3 in srv_resume_thread (slot=0x559b67bfb7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559b66cb09f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f4f5cdf4700 (LWP 67334)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc0a1a in os_event::timed_wait (this=0x559b6a7a3d30, abstime=0x7f4f5cdf3b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559b66bc0d31 in os_event::wait_time_low (this=0x559b6a7a3d30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559b66bc0ed1 in os_event_wait_time_low (event=0x559b6a7a3d30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559b66d0f2fb in ib_wqueue_timedwait (wq=0x559b6a7a3c00, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559b66e4fea4 in fts_optimize_thread (arg=0x559b6a7a3c00) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f4f5f7fe700 (LWP 67287)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc0a1a in os_event::timed_wait (this=0x559b6a5fb600, abstime=0x7f4f5f7fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559b66bc0d31 in os_event::wait_time_low (this=0x559b6a5fb600, time_in_usec=508000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559b66bc0ed1 in os_event_wait_time_low (event=0x559b6a5fb600, time_in_usec=508000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559b66d96d1d in pc_sleep_if_needed (next_loop_time=1549883234247, sig_count=2, cur_time=1549883233739) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000559b66d97c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f4f5ffff700 (LWP 67286)): #0 0x00007f4fa1793644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559b66bb441c in LinuxAIOHandler::collect (this=0x7f4f5fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559b66bb4945 in LinuxAIOHandler::poll (this=0x7f4f5fffeb00, m1=0x7f4f5fffeba8, m2=0x7f4f5fffebb0, request=0x7f4f5fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559b66bb4aa8 in os_aio_linux_handler (global_segment=5, m1=0x7f4f5fffeba8, m2=0x7f4f5fffebb0, request=0x7f4f5fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559b66bb95cc in os_aio_handler (segment=5, m1=0x7f4f5fffeba8, m2=0x7f4f5fffebb0, request=0x7f4f5fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559b66e0c4a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559b66cb2909 in io_handler_thread (arg=0x559b67e39648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f4f9836d700 (LWP 66926)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b6707e997 in safe_cond_wait (cond=0x7f4f74015608, mp=0x7f4f74015558, file=0x559b671e8b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559b667d6088 in inline_mysql_cond_wait (that=0x7f4f74015608, mutex=0x7f4f74015558, src_file=0x559b671e8e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559b667d8d20 in Wsrep_thd_queue::pop_front (this=0x7f4f74015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559b667d7740 in wsrep_rollback_process (rollbacker=0x7f4f74000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000559b667c8282 in start_wsrep_THD (arg=0x559b6a50cb90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4f9a61b700 (LWP 66892)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b6707ec99 in safe_cond_timedwait (cond=0x559b6865d640 , mp=0x559b6865d580 , abstime=0x7f4f9a61ad70, file=0x559b67506840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559b6707fc53 in inline_mysql_cond_timedwait (that=0x559b6865d640 , mutex=0x559b6865d580 , abstime=0x7f4f9a61ad70, src_file=0x559b67506888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559b670807b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559b6701145f in pfs_spawn_thread (arg=0x559b6a4737d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4f699ab700 (LWP 67283)): #0 0x00007f4fa1793644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559b66bb441c in LinuxAIOHandler::collect (this=0x7f4f699aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559b66bb4945 in LinuxAIOHandler::poll (this=0x7f4f699aab00, m1=0x7f4f699aaba8, m2=0x7f4f699aabb0, request=0x7f4f699aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559b66bb4aa8 in os_aio_linux_handler (global_segment=2, m1=0x7f4f699aaba8, m2=0x7f4f699aabb0, request=0x7f4f699aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559b66bb95cc in os_aio_handler (segment=2, m1=0x7f4f699aaba8, m2=0x7f4f699aabb0, request=0x7f4f699aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559b66e0c4a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559b66cb2909 in io_handler_thread (arg=0x559b67e39630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f4f337fe700 (LWP 67343)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b6707e997 in safe_cond_wait (cond=0x7f4f337fdc70, mp=0x7f4f337fdcb0, file=0x559b6734cc40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559b66ad8487 in inline_mysql_cond_wait (that=0x7f4f337fdc70, mutex=0x7f4f337fdcb0, src_file=0x559b6734e5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559b66add0c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000559b6701145f in pfs_spawn_thread (arg=0x559b6a7b6640) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4f33fff700 (LWP 67342)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc10ff in os_event::wait (this=0x559b6a5fb680) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559b66bc0b85 in os_event::wait_low (this=0x559b6a5fb680, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559b66bc0ef6 in os_event_wait_low (event=0x559b6a5fb680, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559b66d75b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f4f4cff9700 (LWP 67341)): #0 0x00007f4fa1bc9e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559b66bc1401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559b66d67a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f4f4dffb700 (LWP 67339)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc10ff in os_event::wait (this=0x559b6a5fb300) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559b66bc0b85 in os_event::wait_low (this=0x559b6a5fb300, reset_sig_count=8652) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559b66bc0ef6 in os_event_wait_low (event=0x559b6a5fb300, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559b66cacdf3 in srv_resume_thread (slot=0x559b67bfb780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559b66cb09f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4f4f7fe700 (LWP 67336)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc10ff in os_event::wait (this=0x559b6a5fb280) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559b66bc0b85 in os_event::wait_low (this=0x559b6a5fb280, reset_sig_count=2157) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559b66bc0ef6 in os_event_wait_low (event=0x559b6a5fb280, reset_sig_count=2157) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559b66cacdf3 in srv_resume_thread (slot=0x559b67bfb748 , sig_count=2157, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559b66cb0e04 in srv_purge_coordinator_suspend (slot=0x559b67bfb748 , rseg_history_len=26) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000559b66cb109a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4f5edf8700 (LWP 67331)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc0a1a in os_event::timed_wait (this=0x559b6a5fb480, abstime=0x7f4f5edf7ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559b66bc0d31 in os_event::wait_time_low (this=0x559b6a5fb480, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559b66bc0ed1 in os_event_wait_time_low (event=0x559b6a5fb480, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559b66caeffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f4f5e5f7700 (LWP 67330)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b66bc0a1a in os_event::timed_wait (this=0x559b6a7d39b0, abstime=0x7f4f5e5f6d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559b66bc0d31 in os_event::wait_time_low (this=0x559b6a7d39b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559b66bc0ed1 in os_event_wait_time_low (event=0x559b6a7d39b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559b66b8d88e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f4f689a9700 (LWP 67285)): #0 0x00007f4fa1793644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559b66bb441c in LinuxAIOHandler::collect (this=0x7f4f689a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559b66bb4945 in LinuxAIOHandler::poll (this=0x7f4f689a8b00, m1=0x7f4f689a8ba8, m2=0x7f4f689a8bb0, request=0x7f4f689a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559b66bb4aa8 in os_aio_linux_handler (global_segment=4, m1=0x7f4f689a8ba8, m2=0x7f4f689a8bb0, request=0x7f4f689a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559b66bb95cc in os_aio_handler (segment=4, m1=0x7f4f689a8ba8, m2=0x7f4f689a8bb0, request=0x7f4f689a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559b66e0c4a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559b66cb2909 in io_handler_thread (arg=0x559b67e39640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f4f691aa700 (LWP 67284)): #0 0x00007f4fa1793644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559b66bb441c in LinuxAIOHandler::collect (this=0x7f4f691a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559b66bb4945 in LinuxAIOHandler::poll (this=0x7f4f691a9b00, m1=0x7f4f691a9ba8, m2=0x7f4f691a9bb0, request=0x7f4f691a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559b66bb4aa8 in os_aio_linux_handler (global_segment=3, m1=0x7f4f691a9ba8, m2=0x7f4f691a9bb0, request=0x7f4f691a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559b66bb95cc in os_aio_handler (segment=3, m1=0x7f4f691a9ba8, m2=0x7f4f691a9bb0, request=0x7f4f691a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559b66e0c4a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559b66cb2909 in io_handler_thread (arg=0x559b67e39638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4f6a9ad700 (LWP 67281)): #0 0x00007f4fa1793644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559b66bb441c in LinuxAIOHandler::collect (this=0x7f4f6a9acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559b66bb4945 in LinuxAIOHandler::poll (this=0x7f4f6a9acb00, m1=0x7f4f6a9acba8, m2=0x7f4f6a9acbb0, request=0x7f4f6a9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559b66bb4aa8 in os_aio_linux_handler (global_segment=0, m1=0x7f4f6a9acba8, m2=0x7f4f6a9acbb0, request=0x7f4f6a9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559b66bb95cc in os_aio_handler (segment=0, m1=0x7f4f6a9acba8, m2=0x7f4f6a9acbb0, request=0x7f4f6a9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559b66e0c4a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559b66cb2909 in io_handler_thread (arg=0x559b67e39620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4f6bfff700 (LWP 67280)): #0 0x00007f4fa1bc6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b6707ec99 in safe_cond_timedwait (cond=0x559b6864aea0 , mp=0x559b6864ade0 , abstime=0x7f4f6bffecb0, file=0x559b674c2610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559b66ee8acb in inline_mysql_cond_timedwait (that=0x559b6864aea0 , mutex=0x559b6864ade0 , abstime=0x7f4f6bffecb0, src_file=0x559b674c2658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559b66ee8fef in my_service_thread_sleep (control=0x559b67bffb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559b66edd488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559b6701145f in pfs_spawn_thread (arg=0x559b6a5b3680) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4f982d7700 (LWP 66928)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4f999c5132 in fifo_lock_get (q=q@entry=0x559b6a4d8ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007f4f999c57a1 in gu_fifo_get_head (q=0x559b6a4d8ac0, err=0x7f4f982d606c) at galerautils/src/gu_fifo.c:330 #3 0x00007f4f99aa70cc in gcs_recv (conn=0x559b6a4b2440, action=0x7f4f982d60b0) at gcs/src/gcs.cpp:2024 #4 0x0worker[3] Trying to dump core for [mysqltest - pid: 72766, winpid: 72766] worker[3] Trying to dump core for [mysqld.1 - pid: 71834, winpid: 71834] worker[3] Trying to dump core for [mysqld.2 - pid: 72132, winpid: 72132] 0007f4f99ad416c in galera::GcsActionSource::process (this=0x559b6a4e2e60, recv_ctx=0x7f4f982d64d0, exit_loop=@0x7f4f982d629e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f4f99af3033 in galera::ReplicatorSMM::async_recv (this=0x559b6a4aa1f0, recv_ctx=0x7f4f982d64d0) at galera/src/replicator_smm.cpp:383 #6 0x0000559b67133276 in wsrep::wsrep_provider_v26::run_applier (this=0x559b6a41fe50, applier_ctx=0x7f4f982d64d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000559b667d6636 in wsrep_replication_process (thd=0x7f4f78000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000559b667c8282 in start_wsrep_THD (arg=0x559b6a50ba30) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4f98322700 (LWP 66927)): #0 0x00007f4fa1bc6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559b6707e997 in safe_cond_wait (cond=0x7f4f80017b68, mp=0x7f4f80017ab8, file=0x559b671e8b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559b667d6088 in inline_mysql_cond_wait (that=0x7f4f80017b68, mutex=0x7f4f80017ab8, src_file=0x559b671e8e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559b667d8d20 in Wsrep_thd_queue::pop_front (this=0x7f4f80017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559b667d7b00 in wsrep_post_rollback_process (post_rollbacker=0x7f4f80001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000559b667c8282 in start_wsrep_THD (arg=0x559b6a50b700) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f4f8ac7c700 (LWP 66894)): #0 0x00007f4f9ff81483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f4f99a726f8 in asio::detail::epoll_reactor::run (this=0x559b6a4eb000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f4f99a76df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559b6a4eaea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f4f99a770a2 in asio::detail::task_io_service::run (this=0x559b6a4eaea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f4f99a8eafd in asio::io_service::run (this=0x559b6a4e9bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f4f99a8f251 in gcomm::AsioProtonet::event_loop (this=0x559b6a4e9b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f4f99aad2ed in GCommConn::run (this=0x559b6a4ea630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f4f99ab0399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f4fa1bc2dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f4f9ff80ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4fa1fd3880 (LWP 66875)): #0 0x00007f4fa1bc79d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559b67079368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559b668b7503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4f9ff7620d in poll () from /lib64/libc.so.6 #5 0x0000559b66457a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559b66456d25 in mysqld_main (argc=162, argv=0x559b6a32bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559b6644aec5 in main (argc=23, argv=0x7ffe4ae97168) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.GCF-1043A-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043A-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_myisam galera_sr.GCF-1043A Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_many_fragments 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 13:10:19 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr_many_fragments.log == connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.71835' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.1/data/core.71835 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 71835] [New LWP 71879] [New LWP 71839] [New LWP 71880] [New LWP 71905] [New LWP 71904] [New LWP 71907] [New LWP 71859] [New LWP 71922] [New LWP 71925] [New LWP 71882] [New LWP 71923] [New LWP 71883] [New LWP 71929] [New LWP 71903] [New LWP 72080] [New LWP 71920] [New LWP 72081] [New LWP 71924] [New LWP 71926] [New LWP 71927] [New LWP 71932] [New LWP 71933] [New LWP 71931] [New LWP 71930] [New LWP 71921] [New LWP 71928] [New LWP 71919] [New LWP 71901] [New LWP 71899] [New LWP 71881] [New LWP 71906] [New LWP 72842] [New LWP 71902] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f4e2ebe19d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f4e2ebe19d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055cd171b5368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055cd169f3503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4e2cf9020d in poll () from /lib64/libc.so.6 #5 0x000055cd16593a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055cd16592d25 in mysqld_main (argc=161, argv=0x55cd18c10d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055cd16586ec5 in main (argc=23, argv=0x7ffe1149a6b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f4e077fe700 (LWP 71902)): #0 0x00007f4e2ebe3d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055cd16cf1acb in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055cd16cf1e21 in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055cd16f3a3a2 in pfs_os_file_flush_func (file=..., src_file=0x55cd175b4a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055cd16f3d11d in fil_flush_low (space=0x55cd190e1850, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055cd16f489f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055cd16f486a7 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4390 #7 0x000055cd16dee909 in io_handler_thread (arg=0x55cd17f75628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #8 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f4e1c27f700 (LWP 72842)): #0 0x00007f4e2ebe3d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055cd16cf1acb in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055cd16cf1e21 in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055cd16f3a3a2 in pfs_os_file_flush_func (file=..., src_file=0x55cd175b4a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055cd16f3d11d in fil_flush_low (space=0x55cd190e1850, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055cd16f489f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055cd16ccd1cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055cd16ccdc31 in log_write_up_to (lsn=6574779, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055cd16e38d91 in trx_flush_log_if_needed_low (lsn=6574779) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000055cd16e38dda in trx_flush_log_if_needed (lsn=6574779, trx=0x7f4e1cdb4390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000055cd16e3ac03 in trx_commit_complete_for_mysql (trx=0x7f4e1cdb4390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000055cd16c1f514 in innobase_commit (hton=0x55cd18eabd10, thd=0x7f4da414c640, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x000055cd169f9039 in commit_one_phase_2 (thd=0x7f4da414c640, all=true, trans=0x7f4da414fb90, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000055cd169f8f3b in ha_commit_one_phase (thd=0x7f4da414c640, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000055cd169f82b7 in ha_commit_trans (thd=0x7f4da414c640, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000055cd1683de8e in trans_commit (thd=0x7f4da414c640) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000055cd16bfd18c in Wsrep_storage_service::commit (this=0x7f4da4163300, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x000055cd172654d3 in wsrep::transaction::certify_fragment (this=0x7f4da40069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x000055cd17264c37 in wsrep::transaction::streaming_step (this=0x7f4da40069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x000055cd17261ff2 in wsrep::transaction::after_row (this=0x7f4da40069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x000055cd16a0b3a8 in wsrep::client_state::after_row (this=0x7f4da4006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x000055cd169f4625 in wsrep_after_row (thd=0x7f4da4000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x000055cd16a0652a in wsrep_after_row (thd=0x7f4da4000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x000055cd16a0691b in handler::ha_write_row (this=0x7f4da407ce88, buf=0x7f4da407d638 "\376o\020") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x000055cd1667c443 in write_record (thd=0x7f4da4000b00, table=0x7f4da4076c80, info=0x7f4da4017040) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x000055cd16681742 in select_insert::send_data (this=0x7f4da4016ff8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x000055cd16739e07 in end_send (join=0x7f4da40170a8, join_tab=0x7f4da4088be8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x000055cd16876fa7 in JOIN_CACHE::generate_full_extensions (this=0x7f4da40891f8, rec_ptr=0x7f4da410c00c "\020") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x000055cd16876cc2 in JOIN_CACHE::join_matching_records (this=0x7f4da40891f8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x000055cd1687668e in JOIN_CACHE::join_records (this=0x7f4da40891f8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x000055cd16876828 in JOIN_CACHE::join_records (this=0x7f4da40890c8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x000055cd16876828 in JOIN_CACHE::join_records (this=0x7f4da4088f98, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x000055cd16736abb in sub_select_cache (join=0x7f4da40170a8, join_tab=0x7f4da40880d8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x000055cd16736cc8 in sub_select (join=0x7f4da40170a8, join_tab=0x7f4da4087d28, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x000055cd16736493 in do_select (join=0x7f4da40170a8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x000055cd1670efb1 in JOIN::exec_inner (this=0x7f4da40170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x000055cd1670e3fa in JOIN::exec (this=0x7f4da40170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x000055cd1670f6ec in mysql_select (thd=0x7f4da4000b00, tables=0x7f4da4014dd0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7f4da4016ff8, unit=0x7f4da40049c0, select_lex=0x7f4da4014700) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x000055cd16700fed in handle_select (thd=0x7f4da4000b00, lex=0x7f4da40048f8, result=0x7f4da4016ff8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x000055cd166c33fc in mysql_execute_command (thd=0x7f4da4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #40 0x000055cd166ce935 in mysql_parse (thd=0x7f4da4000b00, rawbuf=0x7f4da4013e58 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f4e1c27e0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #41 0x000055cd166cdf68 in wsrep_mysql_parse (thd=0x7f4da4000b00, rawbuf=0x7f4da4013e58 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f4e1c27e0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #42 0x000055cd166b9e25 in dispatch_command (command=COM_QUERY, thd=0x7f4da4000b00, packet=0x7f4da400a431 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #43 0x000055cd166b880d in do_command (thd=0x7f4da4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x000055cd16829069 in do_handle_one_connection (connect=0x55cd1916ecd0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #45 0x000055cd16828dad in handle_one_connection (arg=0x55cd1916ecd0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #46 0x000055cd1714d45f in pfs_spawn_thread (arg=0x55cd19128270) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f4e057fa700 (LWP 71906)): #0 0x00007f4e2ebe3d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055cd16cf1acb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055cd16cf1e21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055cd16f3a3a2 in pfs_os_file_flush_func (file=..., src_file=0x55cd175b4a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055cd16f3d11d in fil_flush_low (space=0x55cd190e14b0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055cd16f489f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055cd16f48c7e in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x000055cd16ec5002 in buf_dblwr_update (bpage=0x7f4e1c56df30, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x000055cd16ecc9ef in buf_flush_write_complete (bpage=0x7f4e1c56df30, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000055cd16ebb6a9 in buf_page_io_complete (bpage=0x7f4e1c56df30, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x000055cd16f48751 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x000055cd16dee909 in io_handler_thread (arg=0x55cd17f75648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f4e1e5fc700 (LWP 71881)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd171ba997 in safe_cond_wait (cond=0x7f4e14015608, mp=0x7f4e14015558, file=0x55cd17324b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cd16912088 in inline_mysql_cond_wait (that=0x7f4e14015608, mutex=0x7f4e14015558, src_file=0x55cd17324e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055cd16914d20 in Wsrep_thd_queue::pop_front (this=0x7f4e14015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055cd16913740 in wsrep_rollback_process (rollbacker=0x7f4e14000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055cd16904282 in start_wsrep_THD (arg=0x55cd18ddf320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f4e1d9b2700 (LWP 71899)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd171bac99 in safe_cond_timedwait (cond=0x55cd18786ea0 , mp=0x55cd18786de0 , abstime=0x7f4e1d9b1cb0, file=0x55cd175fe610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055cd17024acb in inline_mysql_cond_timedwait (that=0x55cd18786ea0 , mutex=0x55cd18786de0 , abstime=0x7f4e1d9b1cb0, src_file=0x55cd175fe658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055cd17024fef in my_service_thread_sleep (control=0x55cd17d3bb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055cd17019488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055cd1714d45f in pfs_spawn_thread (arg=0x55cd18e8f7e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f4e07fff700 (LWP 71901)): #0 0x00007f4e2e7ad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cd16cf041c in LinuxAIOHandler::collect (this=0x7f4e07ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cd16cf0945 in LinuxAIOHandler::poll (this=0x7f4e07ffeb00, m1=0x7f4e07ffeba8, m2=0x7f4e07ffebb0, request=0x7f4e07ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cd16cf0aa8 in os_aio_linux_handler (global_segment=0, m1=0x7f4e07ffeba8, m2=0x7f4e07ffebb0, request=0x7f4e07ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cd16cf55cc in os_aio_handler (segment=0, m1=0x7f4e07ffeba8, m2=0x7f4e07ffebb0, request=0x7f4e07ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cd16f484a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cd16dee909 in io_handler_thread (arg=0x55cd17f75620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4dfb7fe700 (LWP 71919)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfca1a in os_event::timed_wait (this=0x55cd190affe0, abstime=0x7f4dfb7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cd16cfcd31 in os_event::wait_time_low (this=0x55cd190affe0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cd16cfced1 in os_event_wait_time_low (event=0x55cd190affe0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cd16cc988e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f4deeffd700 (LWP 71928)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfd0ff in os_event::wait (this=0x55cd18fbda80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cd16cfcb85 in os_event::wait_low (this=0x55cd18fbda80, reset_sig_count=880) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cd16cfcef6 in os_event_wait_low (event=0x55cd18fbda80, reset_sig_count=880) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cd16df6bfc in sync_array_wait_event (arr=0x55cd18eaebb0, cell=@0x7f4deeffae70: 0x55cd18eaece0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055cd16c480f7 in TTASEventMutex::enter (this=0x55cd18fbd3b0, max_spins=60, max_delay=4, filename=0x55cd1758f2a8 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x000055cd16c46352 in PolicyMutex >::enter (this=0x55cd18fbd3b0, n_spins=30, n_delay=4, name=0x55cd1758f2a8 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x000055cd16edb4cb in buf_LRU_get_free_block (buf_pool=0x55cd18fbd3b0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1092 #8 0x000055cd16eb8d13 in buf_page_init_for_read (err=0x7f4deeffb884, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5332 #9 0x000055cd16ee20be in buf_read_page_low (err=0x7f4deeffb884, sync=false, type=192, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #10 0x000055cd16ee2de8 in buf_read_page_background (page_id=..., zip_size=0, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:444 #11 0x000055cd16e7c002 in btr_cur_prefetch_siblings (block=0x7f4e1c596040) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3304 #12 0x000055cd16e847fd in btr_cur_optimistic_delete_func (cursor=0x55cd190f9188, flags=0, mtr=0x7f4deeffc1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5806 #13 0x000055cd16da9ff4 in row_purge_remove_clust_if_poss_low (node=0x55cd190f90f0, mode=2) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:168 #14 0x000055cd16daa1e3 in row_purge_remove_clust_if_poss (node=0x55cd190f90f0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:219 #15 0x000055cd16daba36 in row_purge_del_mark (node=0x55cd190f90f0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #16 0x000055cd16dacfbe in row_purge_record_func (node=0x55cd190f90f0, undo_rec=0x55cd190f95c8 "", thr=0x55cd190f9038, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #17 0x000055cd16dad2fb in row_purge (node=0x55cd190f90f0, undo_rec=0x55cd190f95c8 "", thr=0x55cd190f9038) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #18 0x000055cd16dad6a6 in row_purge_step (thr=0x55cd190f9038) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #19 0x000055cd16d2fa9a in que_thr_step (thr=0x55cd190f9038) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #20 0x000055cd16d2fcca in que_run_threads_low (thr=0x55cd190f9038) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #21 0x000055cd16d2feba in que_run_threads (thr=0x55cd190f9038) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #22 0x000055cd16dec855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #23 0x000055cd16dec9fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #24 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f4dfa7fc700 (LWP 71921)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfca1a in os_event::timed_wait (this=0x55cd18ed7b30, abstime=0x7f4dfa7fbd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cd16cfcd31 in os_event::wait_time_low (this=0x55cd18ed7b30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cd16cfced1 in os_event_wait_time_low (event=0x55cd18ed7b30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cd16deaada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4dedffb700 (LWP 71930)): #0 0x00007f4e2ebe3e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055cd16cfd401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055cd16ea3a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4ded7fa700 (LWP 71931)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfd0ff in os_event::wait (this=0x55cd18ed7cb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cd16cfcb85 in os_event::wait_low (this=0x55cd18ed7cb0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cd16cfcef6 in os_event_wait_low (event=0x55cd18ed7cb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cd16eb1b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f4e1c360700 (LWP 71933)): #0 0x00007f4e2ced440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055cd16586f44 in my_sigwait (set=0x7f4e1c35fcf0, sig=0x7f4e1c35fcb4, code=0x7f4e1c35fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055cd1658d584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055cd1714d45f in pfs_spawn_thread (arg=0x55cd19154b30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f4decff9700 (LWP 71932)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd171ba997 in safe_cond_wait (cond=0x7f4decff8c70, mp=0x7f4decff8cb0, file=0x55cd17488c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cd16c14487 in inline_mysql_cond_wait (that=0x7f4decff8c70, mutex=0x7f4decff8cb0, src_file=0x55cd1748a5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055cd16c190c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055cd1714d45f in pfs_spawn_thread (arg=0x55cd19118950) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f4def7fe700 (LWP 71927)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfd0ff in os_event::wait (this=0x55cd18ed7930) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cd16cfcb85 in os_event::wait_low (this=0x55cd18ed7930, reset_sig_count=9019) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cd16cfcef6 in os_event_wait_low (event=0x55cd18ed7930, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cd16de8df3 in srv_resume_thread (slot=0x55cd17d37780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055cd16dec9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f4deffff700 (LWP 71926)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfd0ff in os_event::wait (this=0x55cd18ed79b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cd16cfcb85 in os_event::wait_low (this=0x55cd18ed79b0, reset_sig_count=7875) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cd16cfcef6 in os_event_wait_low (event=0x55cd18ed79b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cd16de8df3 in srv_resume_thread (slot=0x55cd17d377b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055cd16dec9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f4df97fa700 (LWP 71924)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfd0ff in os_event::wait (this=0x55cd190700a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cd16cfcb85 in os_event::wait_low (this=0x55cd190700a0, reset_sig_count=125) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cd16cfcef6 in os_event_wait_low (event=0x55cd190700a0, reset_sig_count=125) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cd16df6bfc in sync_array_wait_event (arr=0x55cd18eaebb0, cell=@0x7f4df97f9bc8: 0x55cd18eaeca0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055cd16df9911 in rw_lock_s_lock_spin (lock=0x55cd17d34540 , pass=0, file_name=0x55cd174bd7d8 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000055cd16cca25f in rw_lock_s_lock_func (lock=0x55cd17d34540 , pass=0, file_name=0x55cd174bd7d8 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000055cd16cca792 in pfs_rw_lock_s_lock_func (lock=0x55cd17d34540 , pass=0, file_name=0x55cd174bd7d8 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000055cd16cceca1 in log_write_checkpoint_info (sync=true, end_lsn=6571909) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1385 #9 0x000055cd16ccf1a9 in log_checkpoint (sync=true, write_always=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1534 #10 0x000055cd16debc8b in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2178 #11 0x000055cd16dec551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f4e1c2ca700 (LWP 72081)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd171bac99 in safe_cond_timedwait (cond=0x55cd17efa1a0 , mp=0x55cd17efcba0 , abstime=0x7f4e1c2c9c60, file=0x55cd172764c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055cd16587845 in inline_mysql_cond_timedwait (that=0x55cd17efa1a0 , mutex=0x55cd17efcba0 , abstime=0x7f4e1c2c9c60, src_file=0x55cd17278538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055cd1658c61b in cache_thread (thd=0x7f4dbc000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055cd1658c8c9 in one_thread_per_connection_end (thd=0x7f4dbc000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055cd168290f3 in do_handle_one_connection (connect=0x55cd19246700) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055cd16828dad in handle_one_connection (arg=0x55cd19246700) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055cd1714d45f in pfs_spawn_thread (arg=0x55cd1926b700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4dfaffd700 (LWP 71920)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfca1a in os_event::timed_wait (this=0x55cd18ed7ab0, abstime=0x7f4dfaffcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cd16cfcd31 in os_event::wait_time_low (this=0x55cd18ed7ab0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cd16cfced1 in os_event_wait_time_low (event=0x55cd18ed7ab0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cd16deaffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4e1c315700 (LWP 72080)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd171ba997 in safe_cond_wait (cond=0x55cd17efa1e0 , mp=0x55cd17efd560 , file=0x55cd17285cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cd165b84ee in inline_mysql_cond_wait (that=0x55cd17efa1e0 , mutex=0x55cd17efd560 , src_file=0x55cd17286830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055cd165b9d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055cd1714d45f in pfs_spawn_thread (arg=0x55cd191d5890) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f4e06ffd700 (LWP 71903)): #0 0x00007f4e2e7ad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cd16cf041c in LinuxAIOHandler::collect (this=0x7f4e06ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cd16cf0945 in LinuxAIOHandler::poll (this=0x7f4e06ffcb00, m1=0x7f4e06ffcba8, m2=0x7f4e06ffcbb0, request=0x7f4e06ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cd16cf0aa8 in os_aio_linux_handler (global_segment=2, m1=0x7f4e06ffcba8, m2=0x7f4e06ffcbb0, request=0x7f4e06ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cd16cf55cc in os_aio_handler (segment=2, m1=0x7f4e06ffcba8, m2=0x7f4e06ffcbb0, request=0x7f4e06ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cd16f484a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cd16dee909 in io_handler_thread (arg=0x55cd17f75630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4dee7fc700 (LWP 71929)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfd0ff in os_event::wait (this=0x55cd18ed7bb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cd16cfcb85 in os_event::wait_low (this=0x55cd18ed7bb0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cd16cfcef6 in os_event_wait_low (event=0x55cd18ed7bb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cd16ec8eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f4e1e566700 (LWP 71883)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4e269df132 in fifo_lock_get (q=q@entry=0x55cd18dbdbd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f4e269df7a1 in gu_fifo_get_head (q=0x55cd18dbdbd0, err=0x7f4e1e56506c) at galerautils/src/gu_fifo.c:330 #3 0x00007f4e26ac10cc in gcs_recv (conn=0x55cd18d97550, action=0x7f4e1e5650b0) at gcs/src/gcs.cpp:2024 #4 0x00007f4e26aee16c in galera::GcsActionSource::process (this=0x55cd18d96f30, recv_ctx=0x7f4e1e5654d0, exit_loop=@0x7f4e1e56529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f4e26b0d033 in galera::ReplicatorSMM::async_recv (this=0x55cd18d8f280, recv_ctx=0x7f4e1e5654d0) at galera/src/replicator_smm.cpp:383 #6 0x000055cd1726f276 in wsrep::wsrep_provider_v26::run_applier (this=0x55cd18d16360, applier_ctx=0x7f4e1e5654d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055cd16912636 in wsrep_replication_process (thd=0x7f4e10000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055cd16904282 in start_wsrep_THD (arg=0x55cd18de82a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f4dfbfff700 (LWP 71923)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfca1a in os_event::timed_wait (this=0x55cd19077fb0, abstime=0x7f4dfbffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cd16cfcd31 in os_event::wait_time_low (this=0x55cd19077fb0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cd16cfced1 in os_event_wait_time_low (event=0x55cd19077fb0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cd16e4b2fb in ib_wqueue_timedwait (wq=0x55cd19077e80, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055cd16f8bea4 in fts_optimize_thread (arg=0x55cd19077e80) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4e1e5b1700 (LWP 71882)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd171ba997 in safe_cond_wait (cond=0x7f4e18017b68, mp=0x7f4e18017ab8, file=0x55cd17324b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cd16912088 in inline_mysql_cond_wait (that=0x7f4e18017b68, mutex=0x7f4e18017ab8, src_file=0x55cd17324e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055cd16914d20 in Wsrep_thd_queue::pop_front (this=0x7f4e18017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055cd16913b00 in wsrep_post_rollback_process (post_rollbacker=0x7f4e18001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055cd16904282 in start_wsrep_THD (arg=0x55cd18de7c00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4df8ff9700 (LWP 71925)): #0 0x00007f4e2cfe9cf7 in __memcpy_ssse3 () from /lib64/libc.so.6 #1 0x000055cd1711d9c1 in copy_events_waits (dest=0x7f4e28ac1fe8, source=0x7f4e29d88990) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs_events_waits.cc:82 #2 0x000055cd1711dae7 in insert_events_waits_history_long (wait=0x7f4e29d88990) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs_events_waits.cc:132 #3 0x000055cd17150ed4 in end_mutex_wait_v1 (locker=0x7f4df8ff8c70, rc=0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:3525 #4 0x000055cd16c481c3 in PolicyMutex >::pfs_end (this=0x55cd17d375c0 , locker=0x7f4df8ff8c70, ret=0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:743 #5 0x000055cd16c4638c in PolicyMutex >::enter (this=0x55cd17d375c0 , n_spins=30, n_delay=4, name=0x55cd1753ab48 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=2775) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:599 #6 0x000055cd16ded393 in srv_get_task_queue_length () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2775 #7 0x000055cd16e16bbb in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1270 #8 0x000055cd16e16de1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #9 0x000055cd16deccca in srv_do_purge (n_total_purged=0x7f4df8ff8d98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #10 0x000055cd16ded0f7 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #11 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f4df9ffb700 (LWP 71922)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfca1a in os_event::timed_wait (this=0x55cd190e1e90, abstime=0x7f4df9ffad60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cd16cfcd31 in os_event::wait_time_low (this=0x55cd190e1e90, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cd16cfced1 in os_event_wait_time_low (event=0x55cd190e1e90, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cd16f32142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f4e25a88700 (LWP 71859)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4e26aeafd7 in gu::Lock::wait (cond=..., this=0x7f4e25a87bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4e26aeb069 in galera::ServiceThd::thd_func (arg=0x55cd18d8f998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f4e04ff9700 (LWP 71907)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd16cfca1a in os_event::timed_wait (this=0x55cd18ed7c30, abstime=0x7f4e04ff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cd16cfcd31 in os_event::wait_time_low (this=0x55cd18ed7c30, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cd16cfced1 in os_event_wait_time_low (event=0x55cd18ed7c30, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cd16ed2d1d in pc_sleep_if_needed (next_loop_time=1549883358580, sig_count=2, cur_time=1549883357580) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055cd16ed3c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4e067fc700 (LWP 71904)): #0 0x00007f4e2e7ad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cd16cf041c in LinuxAIOHandler::collect (this=0x7f4e067fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cd16cf0945 in LinuxAIOHandler::poll (this=0x7f4e067fbb00, m1=0x7f4e067fbba8, m2=0x7f4e067fbbb0, request=0x7f4e067fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cd16cf0aa8 in os_aio_linux_handler (global_segment=3, m1=0x7f4e067fbba8, m2=0x7f4e067fbbb0, request=0x7f4e067fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cd16cf55cc in os_aio_handler (segment=3, m1=0x7f4e067fbba8, m2=0x7f4e067fbbb0, request=0x7f4e067fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cd16f484a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cd16dee909 in io_handler_thread (arg=0x55cd17f75638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4e05ffb700 (LWP 71905)): #0 0x00007f4e2e7ad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cd16cf041c in LinuxAIOHandler::collect (this=0x7f4e05ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cd16cf0945 in LinuxAIOHandler::poll (this=0x7f4e05ffab00, m1=0x7f4e05ffaba8, m2=0x7f4e05ffabb0, request=0x7f4e05ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cd16cf0aa8 in os_aio_linux_handler (global_segment=4, m1=0x7f4e05ffaba8, m2=0x7f4e05ffabb0, request=0x7f4e05ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cd16cf55cc in os_aio_handler (segment=4, m1=0x7f4e05ffaba8, m2=0x7f4e05ffabb0, request=0x7f4e05ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cd16f484a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cd16dee909 in io_handler_thread (arg=0x55cd17f75640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4e1edfd700 (LWP 71880)): #0 0x00007f4e2ebe0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4e26ac5867 in gu::Lock::wait (cond=..., this=0x7f4e1edfc850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4e26ac83ab in RecvBuf::front (this=this@entry=0x55cd18dcf8a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f4e26ac5f46 in gcomm_recv (backend=, msg=0x55cd18d97830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f4e26ab9eec in core_msg_recv (backend=backend@entry=0x55cd18d97978, recv_msg=recv_msg@entry=0x55cd18d97830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f4e26abd164 in gcs_core_recv (conn=0x55cd18d977d0, recv_act=recv_act@entry=0x7f4e1edfcd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f4e26ac43d8 in gcs_recv_thread (arg=0x55cd18d97550) at gcs/src/gcs.cpp:1398 #7 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4e27635700 (LWP 71839)): #0 0x00007f4e2ebe0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cd171bac99 in safe_cond_timedwait (cond=0x55cd18799640 , mp=0x55cd18799580 , abstime=0x7f4e27634d70, file=0x55cd17642840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055cd171bbc53 in inline_mysql_cond_timedwait (that=0x55cd18799640 , mutex=0x55cd18799580 , abstime=0x7f4e27634d70, src_file=0x55cd17642888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055cd171bc7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055cd1714d45f in pfs_spawn_thread (arg=0x55cd18d58730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f4e1f5fe700 (LWP 71879)): #0 0x00007f4e2cf9b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f4e26a8c6f8 in asio::detail::epoll_reactor::run (this=0x55cd18dd0040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f4e26a90df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55cd18dcfee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f4e26a910a2 in asio::detail::task_io_service::run (this=0x55cd18dcfee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f4e26aa8afd in asio::io_service::run (this=0x55cd18dcefe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f4e26aa9251 in gcomm::AsioProtonet::event_loop (this=0x55cd18dcef40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f4e26ac72ed in GCommConn::run (this=0x55cd18dcf6c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f4e26aca399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f4e2ebdcdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f4e2cf9aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4e2efed880 (LWP 71835)): #0 0x00007f4e2ebe19d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055cd171b5368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055cd169f3503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4e2cf9020d in poll () from /lib64/libc.so.6 #5 0x000055cd16593a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055cd16592d25 in mysqld_main (argc=161, argv=0x55cd18c10d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055cd16586ec5 in main (argc=23, argv=0x7ffe1149a6b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.72134' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.2/data/core.72134 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 72134] [New LWP 72177] [New LWP 72136] [New LWP 72643] [New LWP 72178] [New LWP 72230] [New LWP 72644] [New LWP 72647] [New LWP 72645] [New LWP 72699] [New LWP 72646] [New LWP 72691] [New LWP 72694] [New LWP 72697] [New LWP 72720] [New LWP 72700] [New LWP 72704] [New LWP 72705] [New LWP 72703] [New LWP 72706] [New LWP 72702] [New LWP 72698] [New LWP 72649] [New LWP 72695] [New LWP 72701] [New LWP 72696] [New LWP 72693] [New LWP 72229] [New LWP 72692] [New LWP 72228] [New LWP 72648] [New LWP 72650] [New LWP 72231] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007ff41c01b9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007ff41c01b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055a12c68c368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055a12beca503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff41a3ca20d in poll () from /lib64/libc.so.6 #5 0x000055a12ba6aa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055a12ba69d25 in mysqld_main (argc=162, argv=0x55a12e879d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055a12ba5dec5 in main (argc=23, argv=0x7ffe427dc128) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7ff4103a7700 (LWP 72231)): #0 0x00007ff41c01dd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055a12c1c8acb in os_file_fsync_posix (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055a12c1c8e21 in os_file_flush_func (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055a12c4113a2 in pfs_os_file_flush_func (file=..., src_file=0x55a12ca8ba70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055a12c41411d in fil_flush_low (space=0x55a12ed4d450, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055a12c41f9f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055a12c1a41cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055a12c1a4c31 in log_write_up_to (lsn=4066470, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055a12c30fd91 in trx_flush_log_if_needed_low (lsn=4066470) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000055a12c30fdda in trx_flush_log_if_needed (lsn=4066470, trx=0x7ff401bd6140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000055a12c311c03 in trx_commit_complete_for_mysql (trx=0x7ff401bd6140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000055a12c0f6514 in innobase_commit (hton=0x55a12eafb6c0, thd=0x7ff3ec000b00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x000055a12bed0039 in commit_one_phase_2 (thd=0x7ff3ec000b00, all=true, trans=0x7ff3ec004050, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000055a12becff3b in ha_commit_one_phase (thd=0x7ff3ec000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000055a12becf2b7 in ha_commit_trans (thd=0x7ff3ec000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000055a12bd14e8e in trans_commit (thd=0x7ff3ec000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000055a12bdc83d6 in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7ff4103a64d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:264 #17 0x000055a12c72ea51 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:67 #18 0x000055a12c72f4b1 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:241 #19 0x000055a12c732807 in wsrep::server_state::on_apply (this=0x55a12e9c2950, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:944 #20 0x000055a12c747ed1 in wsrep::high_priority_service::apply (this=0x7ff4103a64d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x000055a12c7455b8 in (anonymous namespace)::apply_cb (ctx=0x7ff4103a64d0, wsh=0x7ff4103a53e0, flags=0, buf=0x7ff4103a5420, meta=0x7ff4103a56b0, exit_loop=0x7ff4103a55ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007ff413f0e3b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7ff3ec057e20, recv_ctx=recv_ctx@entry=0x7ff4103a64d0, apply_cb=0x55a12c745395 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7ff4103a55ef: false) at galera/src/trx_handle.cpp:418 #23 0x00007ff413f40d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55a12e9f81f0, recv_ctx=recv_ctx@entry=0x7ff4103a64d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007ff413f46a40 in galera::ReplicatorSMM::process_trx (this=0x55a12e9f81f0, recv_ctx=0x7ff4103a64d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #25 0x00007ff413f27c6b in galera::GcsActionSource::process_writeset (this=this@entry=0x55a12ea2ae60, recv_ctx=0x7ff4103a64d0, act=..., exit_loop=@0x7ff4103a629e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007ff413f27e45 in galera::GcsActionSource::dispatch (this=this@entry=0x55a12ea2ae60, recv_ctx=recv_ctx@entry=0x7ff4103a64d0, act=..., exit_loop=@0x7ff4103a629e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007ff413f2821d in galera::GcsActionSource::process (this=0x55a12ea2ae60, recv_ctx=0x7ff4103a64d0, exit_loop=@0x7ff4103a629e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007ff413f47033 in galera::ReplicatorSMM::async_recv (this=0x55a12e9f81f0, recv_ctx=0x7ff4103a64d0) at galera/src/replicator_smm.cpp:383 #29 0x000055a12c746276 in wsrep::wsrep_provider_v26::run_applier (this=0x55a12e96de50, applier_ctx=0x7ff4103a64d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x000055a12bde9636 in wsrep_replication_process (thd=0x7ff3ec000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x000055a12bddb282 in start_wsrep_THD (arg=0x55a12ea54e40) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #32 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff3d97fa700 (LWP 72650)): #0 0x00007ff41c01dd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055a12c1c8acb in os_file_fsync_posix (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055a12c1c8e21 in os_file_flush_func (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055a12c4113a2 in pfs_os_file_flush_func (file=..., src_file=0x55a12ca8ba70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055a12c41411d in fil_flush_low (space=0x55a12ed4d0b0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055a12c41f9f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055a12c39ccbd in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x000055a12c3a73a5 in buf_flush_end (buf_pool=0x55a12ec28da0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x000055a12c3a7563 in buf_flush_do_batch (buf_pool=0x55a12ec28da0, type=BUF_FLUSH_LIST, min_n=3, lsn_limit=18446744073709551615, n=0x7ff3d97f9b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x000055a12c3aa24d in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x000055a12c3ab19b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff3da7fc700 (LWP 72648)): #0 0x00007ff41bbe7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a12c1c741c in LinuxAIOHandler::collect (this=0x7ff3da7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a12c1c7945 in LinuxAIOHandler::poll (this=0x7ff3da7fbb00, m1=0x7ff3da7fbba8, m2=0x7ff3da7fbbb0, request=0x7ff3da7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a12c1c7aa8 in os_aio_linux_handler (global_segment=4, m1=0x7ff3da7fbba8, m2=0x7ff3da7fbbb0, request=0x7ff3da7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a12c1cc5cc in os_aio_handler (segment=4, m1=0x7ff3da7fbba8, m2=0x7ff3da7fbbb0, request=0x7ff3da7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a12c41f4a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055a12c2c5909 in io_handler_thread (arg=0x55a12d44c640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff410c3e700 (LWP 72228)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff413eff867 in gu::Lock::wait (cond=..., this=0x7ff410c3d850) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff413f023ab in RecvBuf::front (this=this@entry=0x55a12ea32818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007ff413efff46 in gcomm_recv (backend=, msg=0x55a12ea00720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007ff413ef3eec in core_msg_recv (backend=backend@entry=0x55a12ea00868, recv_msg=recv_msg@entry=0x55a12ea00720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007ff413ef7164 in gcs_core_recv (conn=0x55a12ea006c0, recv_act=recv_act@entry=0x7ff410c3dd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007ff413efe3d8 in gcs_recv_thread (arg=0x55a12ea00440) at gcs/src/gcs.cpp:1398 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff3d8ff9700 (LWP 72692)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d3a1a in os_event::timed_wait (this=0x55a12eb434a0, abstime=0x7ff3d8ff8ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a12c1d3d31 in os_event::wait_time_low (this=0x55a12eb434a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a12c1d3ed1 in os_event_wait_time_low (event=0x55a12eb434a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a12c2c1ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff41043d700 (LWP 72229)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c691997 in safe_cond_wait (cond=0x7ff3f8016858, mp=0x7ff3f80167a8, file=0x55a12c7fbb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a12bde9088 in inline_mysql_cond_wait (that=0x7ff3f8016858, mutex=0x7ff3f80167a8, src_file=0x55a12c7fbe80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055a12bdebd20 in Wsrep_thd_queue::pop_front (this=0x7ff3f8016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055a12bdea740 in wsrep_rollback_process (rollbacker=0x7ff3f8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055a12bddb282 in start_wsrep_THD (arg=0x55a12ea53350) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff3cb7fe700 (LWP 72693)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d3a1a in os_event::timed_wait (this=0x55a12eb43520, abstime=0x7ff3cb7fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a12c1d3d31 in os_event::wait_time_low (this=0x55a12eb43520, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a12c1d3ed1 in os_event_wait_time_low (event=0x55a12eb43520, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a12c2c1ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff3c9ffb700 (LWP 72696)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d40ff in os_event::wait (this=0x55a12ecdba10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a12c1d3b85 in os_event::wait_low (this=0x55a12ecdba10, reset_sig_count=2820) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a12c1d3ef6 in os_event_wait_low (event=0x55a12ecdba10, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a12c1a4761 in log_write_up_to (lsn=4066557, flush_to_disk=true, rotate_key=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000055a12c1a5fc4 in log_checkpoint (sync=true, write_always=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1494 #6 0x000055a12c2c2c8b in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2178 #7 0x000055a12c2c3551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #8 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff3c2ffd700 (LWP 72701)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d40ff in os_event::wait (this=0x55a12eb435a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a12c1d3b85 in os_event::wait_low (this=0x55a12eb435a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a12c1d3ef6 in os_event_wait_low (event=0x55a12eb435a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a12c39feb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff3ca7fc700 (LWP 72695)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d3a1a in os_event::timed_wait (this=0x55a12eceba20, abstime=0x7ff3ca7fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a12c1d3d31 in os_event::wait_time_low (this=0x55a12eceba20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a12c1d3ed1 in os_event_wait_time_low (event=0x55a12eceba20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a12c3222fb in ib_wqueue_timedwait (wq=0x55a12ed50520, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055a12c462ea4 in fts_optimize_thread (arg=0x55a12ed50520) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff3d9ffb700 (LWP 72649)): #0 0x00007ff41bbe7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a12c1c741c in LinuxAIOHandler::collect (this=0x7ff3d9ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a12c1c7945 in LinuxAIOHandler::poll (this=0x7ff3d9ffab00, m1=0x7ff3d9ffaba8, m2=0x7ff3d9ffabb0, request=0x7ff3d9ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a12c1c7aa8 in os_aio_linux_handler (global_segment=5, m1=0x7ff3d9ffaba8, m2=0x7ff3d9ffabb0, request=0x7ff3d9ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a12c1cc5cc in os_aio_handler (segment=5, m1=0x7ff3d9ffaba8, m2=0x7ff3d9ffabb0, request=0x7ff3d9ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a12c41f4a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055a12c2c5909 in io_handler_thread (arg=0x55a12d44c648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff3c8ff9700 (LWP 72698)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d40ff in os_event::wait (this=0x55a12eb43320) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a12c1d3b85 in os_event::wait_low (this=0x55a12eb43320, reset_sig_count=8766) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a12c1d3ef6 in os_event_wait_low (event=0x55a12eb43320, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a12c2bfdf3 in srv_resume_thread (slot=0x55a12d20e780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a12c2c39f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff3c27fc700 (LWP 72702)): #0 0x00007ff41c01de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055a12c1d4401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055a12c37aa3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff401139700 (LWP 72706)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c691997 in safe_cond_wait (cond=0x55a12d3d11e0 , mp=0x55a12d3d4560 , file=0x55a12c75ccc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a12ba8f4ee in inline_mysql_cond_wait (that=0x55a12d3d11e0 , mutex=0x55a12d3d4560 , src_file=0x55a12c75d830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055a12ba90d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055a12c62445f in pfs_spawn_thread (arg=0x55a12ee42590) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff3c1ffb700 (LWP 72703)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d40ff in os_event::wait (this=0x55a12eb436a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a12c1d3b85 in os_event::wait_low (this=0x55a12eb436a0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a12c1d3ef6 in os_event_wait_low (event=0x55a12eb436a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a12c388b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff401184700 (LWP 72705)): #0 0x00007ff41a30e40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055a12ba5df44 in my_sigwait (set=0x7ff401183cf0, sig=0x7ff401183cb4, code=0x7ff401183cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055a12ba64584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055a12c62445f in pfs_spawn_thread (arg=0x55a12edc1790) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff3c17fa700 (LWP 72704)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c691997 in safe_cond_wait (cond=0x7ff3c17f9c70, mp=0x7ff3c17f9cb0, file=0x55a12c95fc40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a12c0eb487 in inline_mysql_cond_wait (that=0x7ff3c17f9c70, mutex=0x7ff3c17f9cb0, src_file=0x55a12c9615d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055a12c0f00c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055a12c62445f in pfs_spawn_thread (arg=0x55a12ecfe6e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff3c37fe700 (LWP 72700)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d40ff in os_event::wait (this=0x55a12eb433a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a12c1d3b85 in os_event::wait_low (this=0x55a12eb433a0, reset_sig_count=7910) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a12c1d3ef6 in os_event_wait_low (event=0x55a12eb433a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a12c2bfdf3 in srv_resume_thread (slot=0x55a12d20e7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a12c2c39f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff4010ee700 (LWP 72720)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c691c99 in safe_cond_timedwait (cond=0x55a12d3d11a0 , mp=0x55a12d3d3ba0 , abstime=0x7ff4010edc60, file=0x55a12c74d4c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055a12ba5e845 in inline_mysql_cond_timedwait (that=0x55a12d3d11a0 , mutex=0x55a12d3d3ba0 , abstime=0x7ff4010edc60, src_file=0x55a12c74f538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055a12ba6361b in cache_thread (thd=0x7ff3e4000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055a12ba638c9 in one_thread_per_connection_end (thd=0x7ff3e4000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055a12bd000f3 in do_handle_one_connection (connect=0x55a12ed95080) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055a12bcffdad in handle_one_connection (arg=0x55a12ed95080) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055a12c62445f in pfs_spawn_thread (arg=0x55a12edc4b90) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff3c97fa700 (LWP 72697)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d40ff in os_event::wait (this=0x55a12eb432a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a12c1d3b85 in os_event::wait_low (this=0x55a12eb432a0, reset_sig_count=2078) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a12c1d3ef6 in os_event_wait_low (event=0x55a12eb432a0, reset_sig_count=2078) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a12c2bfdf3 in srv_resume_thread (slot=0x55a12d20e748 , sig_count=2078, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a12c2c3e04 in srv_purge_coordinator_suspend (slot=0x55a12d20e748 , rseg_history_len=34) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055a12c2c409a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff3caffd700 (LWP 72694)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d3a1a in os_event::timed_wait (this=0x55a12ed4da90, abstime=0x7ff3caffcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a12c1d3d31 in os_event::wait_time_low (this=0x55a12ed4da90, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a12c1d3ed1 in os_event_wait_time_low (event=0x55a12ed4da90, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a12c409142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff3cbfff700 (LWP 72691)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d3a1a in os_event::timed_wait (this=0x55a12ed1b9d0, abstime=0x7ff3cbffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a12c1d3d31 in os_event::wait_time_low (this=0x55a12ed1b9d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a12c1d3ed1 in os_event_wait_time_low (event=0x55a12ed1b9d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a12c1a088e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff3db7fe700 (LWP 72646)): #0 0x00007ff41bbe7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a12c1c741c in LinuxAIOHandler::collect (this=0x7ff3db7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a12c1c7945 in LinuxAIOHandler::poll (this=0x7ff3db7fdb00, m1=0x7ff3db7fdba8, m2=0x7ff3db7fdbb0, request=0x7ff3db7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a12c1c7aa8 in os_aio_linux_handler (global_segment=2, m1=0x7ff3db7fdba8, m2=0x7ff3db7fdbb0, request=0x7ff3db7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a12c1cc5cc in os_aio_handler (segment=2, m1=0x7ff3db7fdba8, m2=0x7ff3db7fdbb0, request=0x7ff3db7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a12c41f4a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055a12c2c5909 in io_handler_thread (arg=0x55a12d44c630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff3c3fff700 (LWP 72699)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c1d40ff in os_event::wait (this=0x55a12eb43420) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a12c1d3b85 in os_event::wait_low (this=0x55a12eb43420, reset_sig_count=7351) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a12c1d3ef6 in os_event_wait_low (event=0x55a12eb43420, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a12c2bfdf3 in srv_resume_thread (slot=0x55a12d20e7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a12c2c39f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff3dbfff700 (LWP 72645)): #0 0x00007ff41bbe7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a12c1c741c in LinuxAIOHandler::collect (this=0x7ff3dbffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a12c1c7945 in LinuxAIOHandler::poll (this=0x7ff3dbffeb00, m1=0x7ff3dbffeba8, m2=0x7ff3dbffebb0, request=0x7ff3dbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a12c1c7aa8 in os_aio_linux_handler (global_segment=1, m1=0x7ff3dbffeba8, m2=0x7ff3dbffebb0, request=0x7ff3dbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a12c1cc5cc in os_aio_handler (segment=1, m1=0x7ff3dbffeba8, m2=0x7ff3dbffebb0, request=0x7ff3dbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a12c41f4a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055a12c2c5909 in io_handler_thread (arg=0x55a12d44c628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff3daffd700 (LWP 72647)): #0 0x00007ff41bbe7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a12c1c741c in LinuxAIOHandler::collect (this=0x7ff3daffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a12c1c7945 in LinuxAIOHandler::poll (this=0x7ff3daffcb00, m1=0x7ff3daffcba8, m2=0x7ff3daffcbb0, request=0x7ff3daffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a12c1c7aa8 in os_aio_linux_handler (global_segment=3, m1=0x7ff3daffcba8, m2=0x7ff3daffcbb0, request=0x7ff3daffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a12c1cc5cc in os_aio_handler (segment=3, m1=0x7ff3daffcba8, m2=0x7ff3daffcbb0, request=0x7ff3daffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a12c41f4a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055a12c2c5909 in io_handler_thread (arg=0x55a12d44c638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff400f84700 (LWP 72644)): #0 0x00007ff41bbe7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a12c1c741c in LinuxAIOHandler::collect (this=0x7ff400f83b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a12c1c7945 in LinuxAIOHandler::poll (this=0x7ff400f83b00, m1=0x7ff400f83ba8, m2=0x7ff400f83bb0, request=0x7ff400f83be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a12c1c7aa8 in os_aio_linux_handler (global_segment=0, m1=0x7ff400f83ba8, m2=0x7ff400f83bb0, request=0x7ff400f83be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a12c1cc5cc in os_aio_handler (segment=0, m1=0x7ff400f83ba8, m2=0x7ff400f83bb0, request=0x7ff400f83be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a12c41f4a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055a12c2c5909 in io_handler_thread (arg=0x55a12d44c620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff4103f2700 (LWP 72230)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c691997 in safe_cond_wait (cond=0x7ff3f0015608, mp=0x7ff3f0015558, file=0x55a12c7fbb60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a12bde9088 in inline_mysql_cond_wait (that=0x7ff3f0015608, mutex=0x7ff3f0015558, src_file=0x55a12c7fbe80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055a12bdebd20 in Wsrep_thd_queue::pop_front (this=0x7ff3f0015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055a12bdeab00 in wsrep_post_rollback_process (post_rollbacker=0x7ff3f0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055a12bddb282 in start_wsrep_THD (arg=0x55a12ea4abf0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff41143f700 (LWP 72178)): #0 0x00007ff41a3d5483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff413ec66f8 in asio::detail::epoll_reactor::run (this=0x55a12ea33000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007ff413ecadf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55a12ea32ea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007ff413ecb0a2 in asio::detail::task_io_service::run (this=0x55a12ea32ea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007ff413ee2afd in asio::io_service::run (this=0x55a12ea31bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007ff413ee3251 in gcomm::AsioProtonet::event_loop (this=0x55a12ea31b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007ff413f012ed in GCommConn::run (this=0x55a12ea32630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007ff413f04399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff4027d6700 (LWP 72643)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c691c99 in safe_cond_timedwait (cond=0x55a12dc5dea0 , mp=0x55a12dc5dde0 , abstime=0x7ff4027d5cb0, file=0x55a12cad5610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055a12c4fbacb in inline_mysql_cond_timedwait (that=0x55a12dc5dea0 , mutex=0x55a12dc5dde0 , abstime=0x7ff4027d5cb0, src_file=0x55a12cad5658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055a12c4fbfef in my_service_thread_sleep (control=0x55a12d212b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055a12c4f0488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055a12c62445f in pfs_spawn_thread (arg=0x55a12eafb6c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff414a6f700 (LWP 72136)): #0 0x00007ff41c01ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a12c691c99 in safe_cond_timedwait (cond=0x55a12dc70640 , mp=0x55a12dc70580 , abstime=0x7ff414a6ed70, file=0x55a12cb19840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055a12c692c53 in inline_mysql_cond_timedwait (that=0x55a12dc70640 , mutex=0x55a12dc70580 , abstime=0x7ff414a6ed70, src_file=0x55a12cb19888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055a12c6937b6 in timer_handlerworker[4] Trying to dump core for [mysqltest - pid: 76187, winpid: 76187] worker[4] Trying to dump core for [mysqld.1 - pid: 75231, winpid: 75231] worker[4] Trying to dump core for [mysqld.2 - pid: 75614, winpid: 75614] (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055a12c62445f in pfs_spawn_thread (arg=0x55a12e9c17d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff4138c3700 (LWP 72177)): #0 0x00007ff41c01a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff413f24fd7 in gu::Lock::wait (cond=..., this=0x7ff4138c2bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff413f25069 in galera::ServiceThd::thd_func (arg=0x55a12e9f8908) at galera/src/galera_service_thd.cpp:30 #3 0x00007ff41c016dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff41a3d4ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff41c427880 (LWP 72134)): #0 0x00007ff41c01b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055a12c68c368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055a12beca503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff41a3ca20d in poll () from /lib64/libc.so.6 #5 0x000055a12ba6aa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055a12ba69d25 in mysqld_main (argc=162, argv=0x55a12e879d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055a12ba5dec5 in main (argc=23, argv=0x7ffe427dc128) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_many_fragments-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_many_fragments Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.mysql-wsrep-features#8 'innodb' w4 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 13:11:42 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/mysql-wsrep-features#8.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); connection node_1; CREATE TABLE t1 (f1 INT PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(100), FULLTEXT (f2)) ENGINE=InnoDB; connection node_2; SELECT COUNT(*) = 13 FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE name LIKE 'test/%'; COUNT(*) = 13 1 connection node_1; INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) - found 'core.75232' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.1/data/core.75232 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 75232] [New LWP 75271] [New LWP 75290] [New LWP 75269] [New LWP 75294] [New LWP 75293] [New LWP 75291] [New LWP 75328] [New LWP 75313] [New LWP 75329] [New LWP 75330] [New LWP 75315] [New LWP 75506] [New LWP 75316] [New LWP 75327] [New LWP 75323] [New LWP 75317] [New LWP 75322] [New LWP 75333] [New LWP 76261] [New LWP 75324] [New LWP 75325] [New LWP 75331] [New LWP 75508] [New LWP 75270] [New LWP 75312] [New LWP 75314] [New LWP 75319] [New LWP 75321] [New LWP 75332] [New LWP 75334] [New LWP 75350] [New LWP 75318] [New LWP 75326] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f46024a69d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f46024a69d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005571d8a43368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005571d8281503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f460085520d in poll () from /lib64/libc.so.6 #5 0x00005571d7e21a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005571d7e20d25 in mysqld_main (argc=161, argv=0x5571dbafbd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005571d7e14ec5 in main (argc=23, argv=0x7ffd0588d368) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f45cd7a3700 (LWP 75326)): #0 0x00007f46024a8d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005571d857facb in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005571d857fe21 in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005571d87c83a2 in pfs_os_file_flush_func (file=..., src_file=0x5571d8e42a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005571d87cb11d in fil_flush_low (space=0x5571dbfd47e0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x00005571d87d69f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x00005571d855b1cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005571d855bc31 in log_write_up_to (lsn=3567326, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005571d855bd61 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x00005571d8679430 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #10 0x00005571d86799f3 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #11 0x00005571d867a551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f45d0faa700 (LWP 75318)): #0 0x00007f46024a8d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005571d857facb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005571d857fe21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005571d87c83a2 in pfs_os_file_flush_func (file=..., src_file=0x5571d8e42a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005571d87cb11d in fil_flush_low (space=0x5571dbfd4440, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x00005571d87d69f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x00005571d87d6c7e in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x00005571d8753002 in buf_dblwr_update (bpage=0x7f45d37b0270, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x00005571d875a9ef in buf_flush_write_complete (bpage=0x7f45d37b0270, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x00005571d87496a9 in buf_page_io_complete (bpage=0x7f45d37b0270, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x00005571d87d6751 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x00005571d867c909 in io_handler_thread (arg=0x5571d9803648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f45f046c700 (LWP 75350)): #0 0x00007f460079940a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005571d7e14f44 in my_sigwait (set=0x7f45f046bcf0, sig=0x7f45f046bcb4, code=0x7f45f046bcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005571d7e1b584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005571d89db45f in pfs_spawn_thread (arg=0x5571dbd8de20) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f45bcff9700 (LWP 75334)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d8a48997 in safe_cond_wait (cond=0x7f45bcff8c70, mp=0x7f45bcff8cb0, file=0x5571d8d16c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005571d84a2487 in inline_mysql_cond_wait (that=0x7f45bcff8c70, mutex=0x7f45bcff8cb0, src_file=0x5571d8d185d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005571d84a70c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005571d89db45f in pfs_spawn_thread (arg=0x5571dc00b930) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f45bdffb700 (LWP 75332)): #0 0x00007f46024a8e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005571d858b401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005571d8731a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f45cf7a7700 (LWP 75321)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858aa1a in os_event::timed_wait (this=0x5571dbfa2d60, abstime=0x7f45cf7a6d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005571d858ad31 in os_event::wait_time_low (this=0x5571dbfa2d60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005571d858aed1 in os_event_wait_time_low (event=0x5571dbfa2d60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005571d855788e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f45d07a9700 (LWP 75319)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858b0ff in os_event::wait (this=0x5571dbeb0800) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005571d858ab85 in os_event::wait_low (this=0x5571dbeb0800, reset_sig_count=891) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005571d858aef6 in os_event_wait_low (event=0x5571dbeb0800, reset_sig_count=891) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005571d8684bfc in sync_array_wait_event (arr=0x5571dbda1960, cell=@0x7f45d07a8a30: 0x5571dbda1a50) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005571d84d60f7 in TTASEventMutex::enter (this=0x5571dbeb0130, max_spins=60, max_delay=4, filename=0x5571d8e19528 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=2189) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x00005571d84d4352 in PolicyMutex >::enter (this=0x5571dbeb0130, n_spins=30, n_delay=4, name=0x5571d8e19528 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=2189) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x00005571d875ede3 in buf_flush_LRU_list (buf_pool=0x5571dbeb0130) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2189 #8 0x00005571d87611bf in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2730 #9 0x00005571d876219b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #10 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f45d2fae700 (LWP 75314)): #0 0x00007f4602072644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005571d857e41c in LinuxAIOHandler::collect (this=0x7f45d2fadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005571d857e945 in LinuxAIOHandler::poll (this=0x7f45d2fadb00, m1=0x7f45d2fadba8, m2=0x7f45d2fadbb0, request=0x7f45d2fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005571d857eaa8 in os_aio_linux_handler (global_segment=1, m1=0x7f45d2fadba8, m2=0x7f45d2fadbb0, request=0x7f45d2fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005571d85835cc in os_aio_handler (segment=1, m1=0x7f45d2fadba8, m2=0x7f45d2fadbb0, request=0x7f45d2fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005571d87d64a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005571d867c909 in io_handler_thread (arg=0x5571d9803628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f45f126e700 (LWP 75312)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d8a48c99 in safe_cond_timedwait (cond=0x5571da014ea0 , mp=0x5571da014de0 , abstime=0x7f45f126dcb0, file=0x5571d8e8c610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005571d88b2acb in inline_mysql_cond_timedwait (that=0x5571da014ea0 , mutex=0x5571da014de0 , abstime=0x7f45f126dcb0, src_file=0x5571d8e8c658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005571d88b2fef in my_service_thread_sleep (control=0x5571d95c9b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005571d88a7488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005571d89db45f in pfs_spawn_thread (arg=0x5571dbd826f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f45f934d700 (LWP 75270)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45fa3affd7 in gu::Lock::wait (cond=..., this=0x7f45f934cbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f45fa3b0069 in galera::ServiceThd::thd_func (arg=0x5571dbc7a998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f45f03d6700 (LWP 75508)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d8a48c99 in safe_cond_timedwait (cond=0x5571d97881a0 , mp=0x5571d978aba0 , abstime=0x7f45f03d5c60, file=0x5571d8b044c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005571d7e15845 in inline_mysql_cond_timedwait (that=0x5571d97881a0 , mutex=0x5571d978aba0 , abstime=0x7f45f03d5c60, src_file=0x5571d8b06538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005571d7e1a61b in cache_thread (thd=0x7f4590000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005571d7e1a8c9 in one_thread_per_connection_end (thd=0x7f4590000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005571d80b70f3 in do_handle_one_connection (connect=0x5571dc139880) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005571d80b6dad in handle_one_connection (arg=0x5571dc139880) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005571d89db45f in pfs_spawn_thread (arg=0x5571dc15b550) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f45be7fc700 (LWP 75331)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858b0ff in os_event::wait (this=0x5571dbdca930) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005571d858ab85 in os_event::wait_low (this=0x5571dbdca930, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005571d858aef6 in os_event_wait_low (event=0x5571dbdca930, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005571d8756eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f45cdfa4700 (LWP 75325)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858aa1a in os_event::timed_wait (this=0x5571dbf6aab0, abstime=0x7f45cdfa3b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005571d858ad31 in os_event::wait_time_low (this=0x5571dbf6aab0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005571d858aed1 in os_event_wait_time_low (event=0x5571dbf6aab0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005571d86d92fb in ib_wqueue_timedwait (wq=0x5571dbf6a980, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005571d8819ea4 in fts_optimize_thread (arg=0x5571dbf6a980) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f45ce7a5700 (LWP 75324)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858aa1a in os_event::timed_wait (this=0x5571dbfd4e20, abstime=0x7f45ce7a4d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005571d858ad31 in os_event::wait_time_low (this=0x5571dbfd4e20, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005571d858aed1 in os_event_wait_time_low (event=0x5571dbfd4e20, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005571d87c0142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f45f038b700 (LWP 76261)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858b0ff in os_event::wait (this=0x5571dbeb3e70) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005571d858ab85 in os_event::wait_low (this=0x5571dbeb3e70, reset_sig_count=710) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005571d858aef6 in os_event_wait_low (event=0x5571dbeb3e70, reset_sig_count=710) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005571d8684bfc in sync_array_wait_event (arr=0x5571dbda1960, cell=@0x7f45f0384d88: 0x5571dbda1ad0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005571d86881c8 in rw_lock_x_lock_func (lock=0x7f45d37b27c8, pass=0, file_name=0x5571d8d8f508 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:735 #6 0x00005571d8733589 in pfs_rw_lock_x_lock_func (lock=0x7f45d37b27c8, pass=0, file_name=0x5571d8d8f508 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:558 #7 0x00005571d87452f9 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x5571d8d8f508 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, mtr=0x7f45f03866a0, err=0x7f45f03850f0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4824 #8 0x00005571d8704c34 in btr_cur_search_to_nth_level_func (index=0x5571dc1662e8, level=0, tuple=0x7f458007cf68, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x7f45f0386280, ahi_latch=0x0, file=0x5571d8d8f508 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, mtr=0x7f45f03866a0, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1548 #9 0x00005571d85ed420 in btr_pcur_open_low (index=0x5571dc1662e8, level=0, tuple=0x7f458007cf68, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x7f45f0386280, file=0x5571d8d8f508 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, autoinc=0, mtr=0x7f45f03866a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:464 #10 0x00005571d85f42ca in row_ins_clust_index_entry_low (flags=2, mode=2, index=0x5571dc1662e8, n_uniq=3, entry=0x7f458007cf68, n_ext=0, thr=0x7f458007bd38, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:2639 #11 0x00005571d85f637c in row_ins_clust_index_entry (index=0x5571dc1662e8, entry=0x7f458007cf68, thr=0x7f458007bd38, n_ext=0, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3280 #12 0x00005571d85f68aa in row_ins_index_entry (index=0x5571dc1662e8, entry=0x7f458007cf68, thr=0x7f458007bd38) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3405 #13 0x00005571d85f6fe9 in row_ins_index_entry_step (node=0x7f458007ba40, thr=0x7f458007bd38) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3574 #14 0x00005571d85f741a in row_ins (node=0x7f458007ba40, thr=0x7f458007bd38) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3717 #15 0x00005571d85f7c19 in row_ins_step (thr=0x7f458007bd38) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3962 #16 0x00005571d8617786 in row_insert_for_mysql (mysql_rec=0x7f458007a3c0 "87790d11-2deb-11e9-aef2-4a5c044241b0N", prebuilt=0x7f458007b418, ins_mode=ROW_INS_NORMAL) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:1460 #17 0x00005571d84b5358 in ha_innobase::write_row (this=0x7f458007a808, record=0x7f458007a3c0 "87790d11-2deb-11e9-aef2-4a5c044241b0N") at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:8050 #18 0x00005571d82947cb in handler::ha_write_row (this=0x7f458007a808, buf=0x7f458007a3c0 "87790d11-2deb-11e9-aef2-4a5c044241b0N") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6463 #19 0x00005571d81a5267 in Wsrep_schema_impl::insert (table=0x7f4580079770) at /home/stepan/galera/git/10.4/server/sql/wsrep_schema.cc:369 #20 0x00005571d81a7873 in Wsrep_schema::append_fragment (this=0x5571dc0adc20, thd=0x7f4580172cb0, server_id=..., transaction_id=..., seqno=..., flags=0, data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_schema.cc:915 #21 0x00005571d848ac1c in Wsrep_storage_service::append_fragment (this=0x7f45801725b0, server_id=..., transaction_id=..., flags=0, data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:114 #22 0x00005571d8af3275 in wsrep::transaction::certify_fragment (this=0x7f45800069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1125 #23 0x00005571d8af2c37 in wsrep::transaction::streaming_step (this=0x7f45800069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #24 0x00005571d8aefff2 in wsrep::transaction::after_row (this=0x7f45800069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #25 0x00005571d82993a8 in wsrep::client_state::after_row (this=0x7f4580006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #26 0x00005571d8282625 in wsrep_after_row (thd=0x7f4580000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #27 0x00005571d829452a in wsrep_after_row (thd=0x7f4580000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #28 0x00005571d829491b in handler::ha_write_row (this=0x7f458009f0e8, buf=0x7f458009ec90 "\376E\022") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #29 0x00005571d7f0a443 in write_record (thd=0x7f4580000b00, table=0x7f458009e050, info=0x7f4580016ea0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #30 0x00005571d7f0f742 in select_insert::send_data (this=0x7f4580016e58, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #31 0x00005571d7fc7e07 in end_send (join=0x7f4580016f08, join_tab=0x7f45800aaa38, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #32 0x00005571d8104fa7 in JOIN_CACHE::generate_full_extensions (this=0x7f45800aade8, rec_ptr=0x7f458012e208 "\234") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #33 0x00005571d8104cc2 in JOIN_CACHE::join_matching_records (this=0x7f45800aade8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #34 0x00005571d810468e in JOIN_CACHE::join_records (this=0x7f45800aade8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #35 0x00005571d8104828 in JOIN_CACHE::join_records (this=0x7f4580019ca8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #36 0x00005571d8104828 in JOIN_CACHE::join_records (this=0x7f4580019b78, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #37 0x00005571d7fc4abb in sub_select_cache (join=0x7f4580016f08, join_tab=0x7f45800a9f28, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #38 0x00005571d7fc4cc8 in sub_select (join=0x7f4580016f08, join_tab=0x7f45800a9b78, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #39 0x00005571d7fc4493 in do_select (join=0x7f4580016f08, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #40 0x00005571d7f9cfb1 in JOIN::exec_inner (this=0x7f4580016f08) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #41 0x00005571d7f9c3fa in JOIN::exec (this=0x7f4580016f08) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #42 0x00005571d7f9d6ec in mysql_select (thd=0x7f4580000b00, tables=0x7f4580014c30, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f4580016e58, unit=0x7f45800049c0, select_lex=0x7f45800146f0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #43 0x00005571d7f8efed in handle_select (thd=0x7f4580000b00, lex=0x7f45800048f8, result=0x7f4580016e58, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #44 0x00005571d7f513fc in mysql_execute_command (thd=0x7f4580000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #45 0x00005571d7f5c935 in mysql_parse (thd=0x7f4580000b00, rawbuf=0x7f4580013e58 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7f45f038a0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #46 0x00005571d7f5bf68 in wsrep_mysql_parse (thd=0x7f4580000b00, rawbuf=0x7f4580013e58 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7f45f038a0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #47 0x00005571d7f47e25 in dispatch_command (command=COM_QUERY, thd=0x7f4580000b00, packet=0x7f458000a431 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=86, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #48 0x00005571d7f4680d in do_command (thd=0x7f4580000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #49 0x00005571d80b7069 in do_handle_one_connection (connect=0x5571dbd8ea80) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #50 0x00005571d80b6dad in handle_one_connection (arg=0x5571dbd8ea80) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #51 0x00005571d89db45f in pfs_spawn_thread (arg=0x5571dbc36a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #52 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #53 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f45bd7fa700 (LWP 75333)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858b0ff in os_event::wait (this=0x5571dbdcaa30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005571d858ab85 in os_event::wait_low (this=0x5571dbdcaa30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005571d858aef6 in os_event_wait_low (event=0x5571dbdcaa30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005571d873fb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f45cffa8700 (LWP 75322)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858aa1a in os_event::timed_wait (this=0x5571dbdca830, abstime=0x7f45cffa7ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005571d858ad31 in os_event::wait_time_low (this=0x5571dbdca830, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005571d858aed1 in os_event_wait_time_low (event=0x5571dbdca830, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005571d8678ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f45d17ab700 (LWP 75317)): #0 0x00007f4602072644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005571d857e41c in LinuxAIOHandler::collect (this=0x7f45d17aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005571d857e945 in LinuxAIOHandler::poll (this=0x7f45d17aab00, m1=0x7f45d17aaba8, m2=0x7f45d17aabb0, request=0x7f45d17aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005571d857eaa8 in os_aio_linux_handler (global_segment=4, m1=0x7f45d17aaba8, m2=0x7f45d17aabb0, request=0x7f45d17aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005571d85835cc in os_aio_handler (segment=4, m1=0x7f45d17aaba8, m2=0x7f45d17aabb0, request=0x7f45d17aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005571d87d64a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005571d867c909 in io_handler_thread (arg=0x5571d9803640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f45cefa6700 (LWP 75323)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858aa1a in os_event::timed_wait (this=0x5571dbdca8b0, abstime=0x7f45cefa5d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005571d858ad31 in os_event::wait_time_low (this=0x5571dbdca8b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005571d858aed1 in os_event_wait_time_low (event=0x5571dbdca8b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005571d8678ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f45ccfa2700 (LWP 75327)): #0 0x00007f4600844d47 in sched_yield () from /lib64/libc.so.6 #1 0x00005571d858b393 in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x00005571d86a4bd9 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x00005571d86a4de1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x00005571d867acca in srv_do_purge (n_total_purged=0x7f45ccfa1d98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x00005571d867b0f7 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f45d1fac700 (LWP 75316)): #0 0x00007f4602072644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005571d857e41c in LinuxAIOHandler::collect (this=0x7f45d1fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005571d857e945 in LinuxAIOHandler::poll (this=0x7f45d1fabb00, m1=0x7f45d1fabba8, m2=0x7f45d1fabbb0, request=0x7f45d1fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005571d857eaa8 in os_aio_linux_handler (global_segment=3, m1=0x7f45d1fabba8, m2=0x7f45d1fabbb0, request=0x7f45d1fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005571d85835cc in os_aio_handler (segment=3, m1=0x7f45d1fabba8, m2=0x7f45d1fabbb0, request=0x7f45d1fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005571d87d64a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005571d867c909 in io_handler_thread (arg=0x5571d9803638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f45f0421700 (LWP 75506)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d8a48997 in safe_cond_wait (cond=0x5571d97881e0 , mp=0x5571d978b560 , file=0x5571d8b13cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005571d7e464ee in inline_mysql_cond_wait (that=0x5571d97881e0 , mutex=0x5571d978b560 , src_file=0x5571d8b14830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005571d7e47d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005571d89db45f in pfs_spawn_thread (arg=0x5571dbd8e310) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f45d27ad700 (LWP 75315)): #0 0x00007f4602072644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005571d857e41c in LinuxAIOHandler::collect (this=0x7f45d27acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005571d857e945 in LinuxAIOHandler::poll (this=0x7f45d27acb00, m1=0x7f45d27acba8, m2=0x7f45d27acbb0, request=0x7f45d27acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005571d857eaa8 in os_aio_linux_handler (global_segment=2, m1=0x7f45d27acba8, m2=0x7f45d27acbb0, request=0x7f45d27acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005571d85835cc in os_aio_handler (segment=2, m1=0x7f45d27acba8, m2=0x7f45d27acbb0, request=0x7f45d27acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005571d87d64a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005571d867c909 in io_handler_thread (arg=0x5571d9803630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f45beffd700 (LWP 75330)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858b0ff in os_event::wait (this=0x5571dbdca7b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005571d858ab85 in os_event::wait_low (this=0x5571dbdca7b0, reset_sig_count=8367) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005571d858aef6 in os_event_wait_low (event=0x5571dbdca7b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005571d8676df3 in srv_resume_thread (slot=0x5571d95c57f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005571d867a9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f45bf7fe700 (LWP 75329)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858b0ff in os_event::wait (this=0x5571dbdca6b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005571d858ab85 in os_event::wait_low (this=0x5571dbdca6b0, reset_sig_count=10192) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005571d858aef6 in os_event_wait_low (event=0x5571dbdca6b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005571d8676df3 in srv_resume_thread (slot=0x5571d95c5780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005571d867a9f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f45d37af700 (LWP 75313)): #0 0x00007f4602072644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005571d857e41c in LinuxAIOHandler::collect (this=0x7f45d37aeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005571d857e945 in LinuxAIOHandler::poll (this=0x7f45d37aeb00, m1=0x7f45d37aeba8, m2=0x7f45d37aebb0, request=0x7f45d37aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005571d857eaa8 in os_aio_linux_handler (global_segment=0, m1=0x7f45d37aeba8, m2=0x7f45d37aebb0, request=0x7f45d37aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005571d85835cc in os_aio_handler (segment=0, m1=0x7f45d37aeba8, m2=0x7f45d37aebb0, request=0x7f45d37aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005571d87d64a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005571d867c909 in io_handler_thread (arg=0x5571d9803620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f45bffff700 (LWP 75328)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d858b0ff in os_event::wait (this=0x5571dbeb0800) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005571d858ab85 in os_event::wait_low (this=0x5571dbeb0800, reset_sig_count=891) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005571d858aef6 in os_event_wait_low (event=0x5571dbeb0800, reset_sig_count=891) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005571d8684bfc in sync_array_wait_event (arr=0x5571dbda1960, cell=@0x7f45bfffce70: 0x5571dbda1b50) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005571d84d60f7 in TTASEventMutex::enter (this=0x5571dbeb0130, max_spins=60, max_delay=4, filename=0x5571d8e1d2a8 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x00005571d84d4352 in PolicyMutex >::enter (this=0x5571dbeb0130, n_spins=30, n_delay=4, name=0x5571d8e1d2a8 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x00005571d87694cb in buf_LRU_get_free_block (buf_pool=0x5571dbeb0130) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1092 #8 0x00005571d8746d13 in buf_page_init_for_read (err=0x7f45bfffd884, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5332 #9 0x00005571d87700be in buf_read_page_low (err=0x7f45bfffd884, sync=false, type=192, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #10 0x00005571d8770de8 in buf_read_page_background (page_id=..., zip_size=0, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:444 #11 0x00005571d870a002 in btr_cur_prefetch_siblings (block=0x7f45d37b2700) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3304 #12 0x00005571d87127fd in btr_cur_optimistic_delete_func (cursor=0x5571dbfec168, flags=0, mtr=0x7f45bfffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5806 #13 0x00005571d8637ff4 in row_purge_remove_clust_if_poss_low (node=0x5571dbfec0d0, mode=2) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:168 #14 0x00005571d86381e3 in row_purge_remove_clust_if_poss (node=0x5571dbfec0d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:219 #15 0x00005571d8639a36 in row_purge_del_mark (node=0x5571dbfec0d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #16 0x00005571d863afbe in row_purge_record_func (node=0x5571dbfec0d0, undo_rec=0x5571dbfec5a8 "", thr=0x5571dbfec018, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #17 0x00005571d863b2fb in row_purge (node=0x5571dbfec0d0, undo_rec=0x5571dbfec5a8 "", thr=0x5571dbfec018) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #18 0x00005571d863b6a6 in row_purge_step (thr=0x5571dbfec018) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #19 0x00005571d85bda9a in que_thr_step (thr=0x5571dbfec018) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #20 0x00005571d85bdcca in que_run_threads_low (thr=0x5571dbfec018) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #21 0x00005571d85bdeba in que_run_threads (thr=0x5571dbfec018) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #22 0x00005571d867a855 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #23 0x00005571d867a9fc in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #24 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f45f824b700 (LWP 75291)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d8a48997 in safe_cond_wait (cond=0x7f45e8016858, mp=0x7f45e80167a8, file=0x5571d8bb2b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005571d81a0088 in inline_mysql_cond_wait (that=0x7f45e8016858, mutex=0x7f45e80167a8, src_file=0x5571d8bb2e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005571d81a2d20 in Wsrep_thd_queue::pop_front (this=0x7f45e8016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005571d81a1740 in wsrep_rollback_process (rollbacker=0x7f45e8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005571d8192282 in start_wsrep_THD (arg=0x5571dbcd2340) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f45f8200700 (LWP 75293)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d8a48997 in safe_cond_wait (cond=0x7f45dc015608, mp=0x7f45dc015558, file=0x5571d8bb2b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005571d81a0088 in inline_mysql_cond_wait (that=0x7f45dc015608, mutex=0x7f45dc015558, src_file=0x5571d8bb2e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005571d81a2d20 in Wsrep_thd_queue::pop_front (this=0x7f45dc015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005571d81a1b00 in wsrep_post_rollback_process (post_rollbacker=0x7f45dc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005571d8192282 in start_wsrep_THD (arg=0x5571dbcbc810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f45f81b5700 (LWP 75294)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45fa2a4132 in fifo_lock_get (q=q@entry=0x5571dbca8bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f45fa2a47a1 in gu_fifo_get_head (q=0x5571dbca8bd0, err=0x7f45f81b406c) at galerautils/src/gu_fifo.c:330 #3 0x00007f45fa3860cc in gcs_recv (conn=0x5571dbc82550, action=0x7f45f81b40b0) at gcs/src/gcs.cpp:2024 #4 0x00007f45fa3b316c in galera::GcsActionSource::process (this=0x5571dbc81f30, recv_ctx=0x7f45f81b44d0, exit_loop=@0x7f45f81b429e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f45fa3d2033 in galera::ReplicatorSMM::async_recv (this=0x5571dbc7a280, recv_ctx=0x7f45f81b44d0) at galera/src/replicator_smm.cpp:383 #6 0x00005571d8afd276 in wsrep::wsrep_provider_v26::run_applier (this=0x5571dbc01360, applier_ctx=0x7f45f81b44d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005571d81a0636 in wsrep_replication_process (thd=0x7f45e0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005571d8192282 in start_wsrep_THD (arg=0x5571dbcdc550) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f45faefa700 (LWP 75269)): #0 0x00007f46024a5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005571d8a48c99 in safe_cond_timedwait (cond=0x5571da027640 , mp=0x5571da027580 , abstime=0x7f45faef9d70, file=0x5571d8ed0840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005571d8a49c53 in inline_mysql_cond_timedwait (that=0x5571da027640 , mutex=0x5571da027580 , abstime=0x7f45faef9d70, src_file=0x5571d8ed0888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005571d8a4a7b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005571d89db45f in pfs_spawn_thread (arg=0x5571dbc43730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f45f247c700 (LWP 75290)): #0 0x00007f46024a5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45fa38a867 in gu::Lock::wait (cond=..., this=0x7f45f247b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f45fa38d3ab in RecvBuf::front (this=this@entry=0x5571dbcc28a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f45fa38af46 in gcomm_recv (backend=, msg=0x5571dbc82830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f45fa37eeec in core_msg_recv (backend=backend@entry=0x5571dbc82978, recv_msg=recv_msg@entry=0x5571dbc82830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f45fa382164 in gcs_core_recv (conn=0x5571dbc827d0, recv_act=recv_act@entry=0x7f45f247bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f45fa3893d8 in gcs_recv_thread (arg=0x5571dbc82550) at gcs/src/gcs.cpp:1398 #7 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f45f2c7d700 (LWP 75271)): #0 0x00007f4600860483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f45fa3516f8 in asio::detail::epoll_reactor::run (this=0x5571dbcc2fc0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f45fa355df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5571dbcc2ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f45fa3560a2 in asio::detail::task_io_service::run (this=0x5571dbcc2ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f45fa36dafd in asio::io_service::run (this=0x5571dbcc1fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f45fa36e251 in gcomm::AsioProtonet::event_loop (this=0x5571dbcc1f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f45fa38c2ed in GCommConn::run (this=0x5571dbcc26c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f45fa38f399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f46024a1dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f460085fead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f46028b2880 (LWP 75232)): #0 0x00007f46024a69d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005571d8a43368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005571d8281503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f460085520d in poll () from /lib64/libc.so.6 #5 0x00005571d7e21a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005571d7e20d25 in mysqld_main (argc=161, argv=0x5571dbafbd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005571d7e14ec5 in main (argc=23, argv=0x7ffd0588d368) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.75616' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.2/data/core.75616 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 75616] [New LWP 75640] [New LWP 75647] [New LWP 76118] [New LWP 75678] [New LWP 75717] [New LWP 76121] [New LWP 76120] [New LWP 76139] [New LWP 76128] [New LWP 76161] [New LWP 76133] [New LWP 76178] [New LWP 76134] [New LWP 76124] [New LWP 76136] [New LWP 76140] [New LWP 76165] [New LWP 76141] [New LWP 76138] [New LWP 76137] [New LWP 75718] [New LWP 76135] [New LWP 76125] [New LWP 76129] [New LWP 76132] [New LWP 76122] [New LWP 76123] [New LWP 76119] [New LWP 76131] [New LWP 76130] [New LWP 75716] [New LWP 75715] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f39f99089d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f39f99089d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f0081fb368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f007a39503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f39f7cb720d in poll () from /lib64/libc.so.6 #5 0x000055f0075d9a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f0075d8d25 in mysqld_main (argc=162, argv=0x55f00aa47d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f0075ccec5 in main (argc=23, argv=0x7ffcd3c80268) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f39e247b700 (LWP 75715)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f39f17ec867 in gu::Lock::wait (cond=..., this=0x7f39e247a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f39f17ef3ab in RecvBuf::front (this=this@entry=0x55f00ac0e818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f39f17ecf46 in gcomm_recv (backend=, msg=0x55f00abce720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f39f17e0eec in core_msg_recv (backend=backend@entry=0x55f00abce868, recv_msg=recv_msg@entry=0x55f00abce720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f39f17e4164 in gcs_core_recv (conn=0x55f00abce6c0, recv_act=recv_act@entry=0x7f39e247ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f39f17eb3d8 in gcs_recv_thread (arg=0x55f00abce440) at gcs/src/gcs.cpp:1398 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f39f00ae700 (LWP 75716)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f008200997 in safe_cond_wait (cond=0x7f39d8016858, mp=0x7f39d80167a8, file=0x55f00836ab60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f007958088 in inline_mysql_cond_wait (that=0x7f39d8016858, mutex=0x7f39d80167a8, src_file=0x55f00836ae80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f00795ad20 in Wsrep_thd_queue::pop_front (this=0x7f39d8016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f007959740 in wsrep_rollback_process (rollbacker=0x7f39d8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055f00794a282 in start_wsrep_THD (arg=0x55f00ac152d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f39b57fa700 (LWP 76130)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d42a1a in os_event::timed_wait (this=0x55f00ad1f570, abstime=0x7f39b57f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f007d42d31 in os_event::wait_time_low (this=0x55f00ad1f570, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f007d42ed1 in os_event_wait_time_low (event=0x55f00ad1f570, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f007e30ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f39b4ff9700 (LWP 76131)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d42a1a in os_event::timed_wait (this=0x55f00af29ae0, abstime=0x7f39b4ff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f007d42d31 in os_event::wait_time_low (this=0x55f00af29ae0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f007d42ed1 in os_event_wait_time_low (event=0x55f00af29ae0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f007f78142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f39c27ad700 (LWP 76119)): #0 0x00007f39f94d4644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f007d3641c in LinuxAIOHandler::collect (this=0x7f39c27acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f007d36945 in LinuxAIOHandler::poll (this=0x7f39c27acb00, m1=0x7f39c27acba8, m2=0x7f39c27acbb0, request=0x7f39c27acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f007d36aa8 in os_aio_linux_handler (global_segment=0, m1=0x7f39c27acba8, m2=0x7f39c27acbb0, request=0x7f39c27acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f007d3b5cc in os_aio_handler (segment=0, m1=0x7f39c27acba8, m2=0x7f39c27acbb0, request=0x7f39c27acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f007f8e4a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f007e34909 in io_handler_thread (arg=0x55f008fbb620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f39b7fff700 (LWP 76123)): #0 0x00007f39f94d4644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f007d3641c in LinuxAIOHandler::collect (this=0x7f39b7ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f007d36945 in LinuxAIOHandler::poll (this=0x7f39b7ffeb00, m1=0x7f39b7ffeba8, m2=0x7f39b7ffebb0, request=0x7f39b7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f007d36aa8 in os_aio_linux_handler (global_segment=4, m1=0x7f39b7ffeba8, m2=0x7f39b7ffebb0, request=0x7f39b7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f007d3b5cc in os_aio_handler (segment=4, m1=0x7f39b7ffeba8, m2=0x7f39b7ffebb0, request=0x7f39b7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f007f8e4a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f007e34909 in io_handler_thread (arg=0x55f008fbb640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f39c0faa700 (LWP 76122)): #0 0x00007f39f94d4644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f007d3641c in LinuxAIOHandler::collect (this=0x7f39c0fa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f007d36945 in LinuxAIOHandler::poll (this=0x7f39c0fa9b00, m1=0x7f39c0fa9ba8, m2=0x7f39c0fa9bb0, request=0x7f39c0fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f007d36aa8 in os_aio_linux_handler (global_segment=3, m1=0x7f39c0fa9ba8, m2=0x7f39c0fa9bb0, request=0x7f39c0fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f007d3b5cc in os_aio_handler (segment=3, m1=0x7f39c0fa9ba8, m2=0x7f39c0fa9bb0, request=0x7f39c0fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f007f8e4a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f007e34909 in io_handler_thread (arg=0x55f008fbb638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f39affff700 (LWP 76132)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d42a1a in os_event::timed_wait (this=0x55f00aebfab0, abstime=0x7f39afffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f007d42d31 in os_event::wait_time_low (this=0x55f00aebfab0, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f007d42ed1 in os_event_wait_time_low (event=0x55f00aebfab0, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f007e912fb in ib_wqueue_timedwait (wq=0x55f00aebf980, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055f007fd1ea4 in fts_optimize_thread (arg=0x55f00aebf980) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f39b67fc700 (LWP 76129)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d42a1a in os_event::timed_wait (this=0x55f00ad1f4f0, abstime=0x7f39b67fbba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f007d42d31 in os_event::wait_time_low (this=0x55f00ad1f4f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f007d42ed1 in os_event_wait_time_low (event=0x55f00ad1f4f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f007e30ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f39b6ffd700 (LWP 76125)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d42a1a in os_event::timed_wait (this=0x55f00ad1f670, abstime=0x7f39b6ffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f007d42d31 in os_event::wait_time_low (this=0x55f00ad1f670, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f007d42ed1 in os_event_wait_time_low (event=0x55f00ad1f670, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f007f18d1d in pc_sleep_if_needed (next_loop_time=1549883448914, sig_count=2, cur_time=1549883447914) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055f007f19c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f39ae7fc700 (LWP 76135)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d430ff in os_event::wait (this=0x55f00ad1f3f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f007d42b85 in os_event::wait_low (this=0x55f00ad1f3f0, reset_sig_count=8805) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f007d42ef6 in os_event_wait_low (event=0x55f00ad1f3f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f007e2edf3 in srv_resume_thread (slot=0x55f008d7d7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f007e329f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f39e1c7a700 (LWP 75718)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f39f1706132 in fifo_lock_get (q=q@entry=0x55f00abf4ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007f39f17067a1 in gu_fifo_get_head (q=0x55f00abf4ac0, err=0x7f39e1c7906c) at galerautils/src/gu_fifo.c:330 #3 0x00007f39f17e80cc in gcs_recv (conn=0x55f00abce440, action=0x7f39e1c790b0) at gcs/src/gcs.cpp:2024 #4 0x00007f39f181516c in galera::GcsActionSource::process (this=0x55f00ac06e60, recv_ctx=0x7f39e1c794d0, exit_loop=@0x7f39e1c7929e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f39f1834033 in galera::ReplicatorSMM::async_recv (this=0x55f00abc61f0, recv_ctx=0x7f39e1c794d0) at galera/src/replicator_smm.cpp:383 #6 0x000055f0082b5276 in wsrep::wsrep_provider_v26::run_applier (this=0x55f00ab3be50, applier_ctx=0x7f39e1c794d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055f007958636 in wsrep_replication_process (thd=0x7f39d0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055f00794a282 in start_wsrep_THD (arg=0x55f00ac30220) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f39ad7fa700 (LWP 76137)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d430ff in os_event::wait (this=0x55f00ad1f370) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f007d42b85 in os_event::wait_low (this=0x55f00ad1f370, reset_sig_count=9912) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f007d42ef6 in os_event_wait_low (event=0x55f00ad1f370, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f007e2edf3 in srv_resume_thread (slot=0x55f008d7d780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f007e329f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f39acff9700 (LWP 76138)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d430ff in os_event::wait (this=0x55f00ad1f5f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f007d42b85 in os_event::wait_low (this=0x55f00ad1f5f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f007d42ef6 in os_event_wait_low (event=0x55f00ad1f5f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f007f0eeb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f39a2ffd700 (LWP 76141)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f008200997 in safe_cond_wait (cond=0x7f39a2ffcc70, mp=0x7f39a2ffccb0, file=0x55f0084cec40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f007c5a487 in inline_mysql_cond_wait (that=0x7f39a2ffcc70, mutex=0x7f39a2ffccb0, src_file=0x55f0084d05d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f007c5f0c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055f00819345f in pfs_spawn_thread (arg=0x55f00af615f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f39e0079700 (LWP 76165)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f008200997 in safe_cond_wait (cond=0x55f008f401e0 , mp=0x55f008f43560 , file=0x55f0082cbcc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f0075fe4ee in inline_mysql_cond_wait (that=0x55f008f401e0 , mutex=0x55f008f43560 , src_file=0x55f0082cc830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055f0075ffd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055f00819345f in pfs_spawn_thread (arg=0x55f00b082360) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f39a37fe700 (LWP 76140)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d430ff in os_event::wait (this=0x55f00ad1f6f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f007d42b85 in os_event::wait_low (this=0x55f00ad1f6f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f007d42ef6 in os_event_wait_low (event=0x55f00ad1f6f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f007ef7b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f39adffb700 (LWP 76136)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d430ff in os_event::wait (this=0x55f00ad1f470) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f007d42b85 in os_event::wait_low (this=0x55f00ad1f470, reset_sig_count=8283) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f007d42ef6 in os_event_wait_low (event=0x55f00ad1f470, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f007e2edf3 in srv_resume_thread (slot=0x55f008d7d7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f007e329f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f39b77fe700 (LWP 76124)): #0 0x00007f39f94d4644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f007d3641c in LinuxAIOHandler::collect (this=0x7f39b77fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f007d36945 in LinuxAIOHandler::poll (this=0x7f39b77fdb00, m1=0x7f39b77fdba8, m2=0x7f39b77fdbb0, request=0x7f39b77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f007d36aa8 in os_aio_linux_handler (global_segment=5, m1=0x7f39b77fdba8, m2=0x7f39b77fdbb0, request=0x7f39b77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f007d3b5cc in os_aio_handler (segment=5, m1=0x7f39b77fdba8, m2=0x7f39b77fdbb0, request=0x7f39b77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f007f8e4a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f007e34909 in io_handler_thread (arg=0x55f008fbb648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f39aeffd700 (LWP 76134)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d430ff in os_event::wait (this=0x55f00ad1f2f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f007d42b85 in os_event::wait_low (this=0x55f00ad1f2f0, reset_sig_count=2343) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f007d42ef6 in os_event_wait_low (event=0x55f00ad1f2f0, reset_sig_count=2343) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f007e2edf3 in srv_resume_thread (slot=0x55f008d7d748 , sig_count=2343, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055f007e32e04 in srv_purge_coordinator_suspend (slot=0x55f008d7d748 , rseg_history_len=29) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055f007e3309a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f39c29ad700 (LWP 76178)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f008200c99 in safe_cond_timedwait (cond=0x55f008f401a0 , mp=0x55f008f42ba0 , abstime=0x7f39c29acc60, file=0x55f0082bc4c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f0075cd845 in inline_mysql_cond_timedwait (that=0x55f008f401a0 , mutex=0x55f008f42ba0 , abstime=0x7f39c29acc60, src_file=0x55f0082be538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f0075d261b in cache_thread (thd=0x7f39c4000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055f0075d28c9 in one_thread_per_connection_end (thd=0x7f39c4000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055f00786f0f3 in do_handle_one_connection (connect=0x55f00af9c0d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055f00786edad in handle_one_connection (arg=0x55f00af9c0d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055f00819345f in pfs_spawn_thread (arg=0x55f00ace5860) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f39af7fe700 (LWP 76133)): #0 0x00007f39f990ae3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055f007d43401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055f007e3235b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055f007e3248b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f39e00c4700 (LWP 76161)): #0 0x00007f39f7bfb40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055f0075ccf44 in my_sigwait (set=0x7f39e00c3cf0, sig=0x7f39e00c3cb4, code=0x7f39e00c3cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055f0075d3584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055f00819345f in pfs_spawn_thread (arg=0x55f00ace2d50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f39b5ffb700 (LWP 76128)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f007d42a1a in os_event::timed_wait (this=0x55f00aef7a20, abstime=0x7f39b5ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f007d42d31 in os_event::wait_time_low (this=0x55f00aef7a20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f007d42ed1 in os_event_wait_time_low (event=0x55f00aef7a20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f007d0f88e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f39a3fff700 (LWP 76139)): #0 0x00007f39f990ae3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055f007d43401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055f007ee9a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f39c1fac700 (LWP 76120)): #0 0x00007f39f94d4644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f007d3641c in LinuxAIOHandler::collect (this=0x7f39c1fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f007d36945 in LinuxAIOHandler::poll (this=0x7f39c1fabb00, m1=0x7f39c1fabba8, m2=0x7f39c1fabbb0, request=0x7f39c1fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f007d36aa8 in os_aio_linux_handler (global_segment=1, m1=0x7f39c1fabba8, m2=0x7f39c1fabbb0, request=0x7f39c1fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f007d3b5cc in os_aio_handler (segment=1, m1=0x7f39c1fabba8, m2=0x7f39c1fabbb0, request=0x7f39c1fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f007f8e4a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f007e34909 in io_handler_thread (arg=0x55f008fbb628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f39c17ab700 (LWP 76121)): #0 0x00007f39f94d4644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f007d3641c in LinuxAIOHandler::collect (this=0x7f39c17aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f007d36945 in LinuxAIOHandler::poll (this=0x7f39c17aab00, m1=0x7f39c17aaba8, m2=0x7f39c17aabb0, request=0x7f39c17aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f007d36aa8 in os_aio_linux_handler (global_segment=2, m1=0x7f39c17aaba8, m2=0x7f39c17aabb0, request=0x7f39c17aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f007d3b5cc in os_aio_handler (segment=2, m1=0x7f39c17aaba8, m2=0x7f39c17aabb0, request=0x7f39c17aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055f007f8e4a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055f007e34909 in io_handler_thread (arg=0x55f008fbb630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f39f0063700 (LWP 75717)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f008200997 in safe_cond_wait (cond=0x7f39cc016918, mp=0x7f39cc016868, file=0x55f00836ab60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f007958088 in inline_mysql_cond_wait (that=0x7f39cc016918, mutex=0x7f39cc016868, src_file=0x55f00836ae80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f00795ad20 in Wsrep_thd_queue::pop_front (this=0x7f39cc016810) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f007959b00 in wsrep_post_rollback_process (post_rollbacker=0x7f39cc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055f00794a282 in start_wsrep_THD (arg=0x55f00ac26ae0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f39e2c7c700 (LWP 75678)): #0 0x00007f39f7cc2483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f39f17b36f8 in asio::detail::epoll_reactor::run (this=0x55f00ac0f000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f39f17b7df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55f00ac0eea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f39f17b80a2 in asio::detail::task_io_service::run (this=0x55f00ac0eea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f39f17cfafd in asio::io_service::run (this=0x55f00ac0dbc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f39f17d0251 in gcomm::AsioProtonet::event_loop (this=0x55f00ac0db20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f39f17ee2ed in GCommConn::run (this=0x55f00ac0e630) at gcs/src/gcs_gcommworker[2] Trying to dump core for [mysqltest - pid: 105948, winpid: 105948] worker[2] Trying to dump core for [mysqld.1 - pid: 105042, winpid: 105042] worker[2] Trying to dump core for [mysqld.2 - pid: 105407, winpid: 105407] .cpp:498 #7 0x00007f39f17f1399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f39c3fff700 (LWP 76118)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f008200c99 in safe_cond_timedwait (cond=0x55f0097ccea0 , mp=0x55f0097ccde0 , abstime=0x7f39c3ffecb0, file=0x55f008644610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f00806aacb in inline_mysql_cond_timedwait (that=0x55f0097ccea0 , mutex=0x55f0097ccde0 , abstime=0x7f39c3ffecb0, src_file=0x55f008644658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f00806afef in my_service_thread_sleep (control=0x55f008d81b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055f00805f488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055f00819345f in pfs_spawn_thread (arg=0x55f00acd75c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f39f11b0700 (LWP 75647)): #0 0x00007f39f9907965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f39f1811fd7 in gu::Lock::wait (cond=..., this=0x7f39f11afbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f39f1812069 in galera::ServiceThd::thd_func (arg=0x55f00abc6908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f39f235c700 (LWP 75640)): #0 0x00007f39f9907d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f008200c99 in safe_cond_timedwait (cond=0x55f0097df640 , mp=0x55f0097df580 , abstime=0x7f39f235bd70, file=0x55f008688840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f008201c53 in inline_mysql_cond_timedwait (that=0x55f0097df640 , mutex=0x55f0097df580 , abstime=0x7f39f235bd70, src_file=0x55f008688888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f0082027b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055f00819345f in pfs_spawn_thread (arg=0x55f00ab8f7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f39f9903dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f39f7cc1ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f39f9d14880 (LWP 75616)): #0 0x00007f39f99089d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f0081fb368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f007a39503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f39f7cb720d in poll () from /lib64/libc.so.6 #5 0x000055f0075d9a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f0075d8d25 in mysqld_main (argc=162, argv=0x55f00aa47d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f0075ccec5 in main (argc=23, argv=0x7ffcd3c80268) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.mysql-wsrep-features#8-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#8 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1008 'innodb' w4 [ pass ] 13515 galera_sr.GCF-1018 'innodb' w4 [ pass ] 7826 galera_sr.GCF-571 'innodb' w4 [ pass ] 10796 galera_sr.GCF-1060 'innodb' w3 [ pass ] 432309 galera_sr.GCF-561 'innodb' w3 [ pass ] 9163 galera_sr.GCF-1043B 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 13:24:26 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/GCF-1043B.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 INSERT INTO t1 VALUES (1,'y'),(2,'x') UPDATE t1 SET f2 = 'y' WHERE f1 = 1 OR f1 = 2; connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.105043' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.GCF-1043B-innodb/mysqld.1/data/core.105043 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 105043] [New LWP 105092] [New LWP 105047] [New LWP 105094] [New LWP 105093] [New LWP 105048] [New LWP 105095] [New LWP 105105] [New LWP 105097] [New LWP 105107] [New LWP 105110] [New LWP 105098] [New LWP 105113] [New LWP 105099] [New LWP 105114] [New LWP 105100] [New LWP 105120] [New LWP 105102] [New LWP 105121] [New LWP 105109] [New LWP 105245] [New LWP 105112] [New LWP 105122] [New LWP 105244] [New LWP 106026] [New LWP 105067] [New LWP 105155] [New LWP 105119] [New LWP 105106] [New LWP 105111] [New LWP 105096] [New LWP 105108] [New LWP 105101] [New LWP 106027] [New LWP 105103] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f9fabe0b9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f9fabe0b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b4062d9368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b405b17503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9faa1ba20d in poll () from /lib64/libc.so.6 #5 0x000055b4056b7a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055b4056b6d25 in mysqld_main (argc=161, argv=0x55b4091f0d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055b4056aaec5 in main (argc=23, argv=0x7fffb3cd6af8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f9f79ffb700 (LWP 105103)): #0 0x00007f9fabe0dd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055b405e15acb in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055b405e15e21 in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055b40605e3a2 in pfs_os_file_flush_func (file=..., src_file=0x55b4066d8a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055b40606111d in fil_flush_low (space=0x55b4096c1420, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055b40606c9f7 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055b405fe9cbd in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x000055b405ff43a5 in buf_flush_end (buf_pool=0x55b40959d110, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x000055b405ff4563 in buf_flush_do_batch (buf_pool=0x55b40959d110, type=BUF_FLUSH_LIST, min_n=12, lsn_limit=18446744073709551615, n=0x7f9f79ffab70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x000055b405ff724d in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x000055b405ff819b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f9f99457700 (LWP 106027)): #0 0x00007f9fabe0dd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055b405e15acb in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055b405e15e21 in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055b40605e3a2 in pfs_os_file_flush_func (file=..., src_file=0x55b4066d8a70 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055b40606111d in fil_flush_low (space=0x55b4096c17c0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055b40606c9f7 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055b405df11cf in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055b405df1c31 in log_write_up_to (lsn=4513866, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055b405f5cd91 in trx_flush_log_if_needed_low (lsn=4513866) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000055b405f5cdda in trx_flush_log_if_needed (lsn=4513866, trx=0x7f9f99fd6268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000055b405f5ec03 in trx_commit_complete_for_mysql (trx=0x7f9f99fd6268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000055b405d43514 in innobase_commit (hton=0x55b40948bbb0, thd=0x7f9f2008a490, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x000055b405b1d039 in commit_one_phase_2 (thd=0x7f9f2008a490, all=true, trans=0x7f9f2008d9e0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000055b405b1cf3b in ha_commit_one_phase (thd=0x7f9f2008a490, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000055b405b1c2b7 in ha_commit_trans (thd=0x7f9f2008a490, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000055b405961e8e in trans_commit (thd=0x7f9f2008a490) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000055b405d2118c in Wsrep_storage_service::commit (this=0x7f9f20089320, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x000055b4063894d3 in wsrep::transaction::certify_fragment (this=0x7f9f200069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x000055b406388c37 in wsrep::transaction::streaming_step (this=0x7f9f200069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x000055b406385ff2 in wsrep::transaction::after_row (this=0x7f9f200069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x000055b405b2f3a8 in wsrep::client_state::after_row (this=0x7f9f20006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x000055b405b18625 in wsrep_after_row (thd=0x7f9f20000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x000055b405b2a52a in wsrep_after_row (thd=0x7f9f20000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x000055b405b2a91b in handler::ha_write_row (this=0x7f9f88039c08, buf=0x7f9f88029aa0 "\375\002") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x000055b4057a0443 in write_record (thd=0x7f9f20000b00, table=0x7f9f88028e60, info=0x7f9f99454f50) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x000055b40579d805 in mysql_insert (thd=0x7f9f20000b00, table_list=0x7f9f2001fa00, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:1067 #26 0x000055b4057e6b4a in mysql_execute_command (thd=0x7f9f20000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4754 #27 0x000055b4057f2935 in mysql_parse (thd=0x7f9f20000b00, rawbuf=0x7f9f2001f908 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7f9f994560d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #28 0x000055b4057f1f68 in wsrep_mysql_parse (thd=0x7f9f20000b00, rawbuf=0x7f9f2001f908 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7f9f994560d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #29 0x000055b4057dde25 in dispatch_command (command=COM_QUERY, thd=0x7f9f20000b00, packet=0x7f9f200177d1 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", packet_length=37, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #30 0x000055b4057dc80d in do_command (thd=0x7f9f20000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #31 0x000055b40594d069 in do_handle_one_connection (connect=0x55b40974ec90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #32 0x000055b40594cdad in handle_one_connection (arg=0x55b40974ec90) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #33 0x000055b40627145f in pfs_spawn_thread (arg=0x55b4097ae030) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #34 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #35 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f9f7affd700 (LWP 105101)): #0 0x00007f9fab9d7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b405e1441c in LinuxAIOHandler::collect (this=0x7f9f7affcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b405e14945 in LinuxAIOHandler::poll (this=0x7f9f7affcb00, m1=0x7f9f7affcba8, m2=0x7f9f7affcbb0, request=0x7f9f7affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b405e14aa8 in os_aio_linux_handler (global_segment=4, m1=0x7f9f7affcba8, m2=0x7f9f7affcbb0, request=0x7f9f7affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b405e195cc in os_aio_handler (segment=4, m1=0x7f9f7affcba8, m2=0x7f9f7affcbb0, request=0x7f9f7affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b40606c4a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055b405f12909 in io_handler_thread (arg=0x55b407099640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f9f737fe700 (LWP 105108)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e20a1a in os_event::timed_wait (this=0x55b4096c1e00, abstime=0x7f9f737fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b405e20d31 in os_event::wait_time_low (this=0x55b4096c1e00, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b405e20ed1 in os_event_wait_time_low (event=0x55b4096c1e00, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b406056142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f9f9abd5700 (LWP 105096)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062dec99 in safe_cond_timedwait (cond=0x55b4078aaea0 , mp=0x55b4078aade0 , abstime=0x7f9f9abd4cb0, file=0x55b406722610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b406148acb in inline_mysql_cond_timedwait (that=0x55b4078aaea0 , mutex=0x55b4078aade0 , abstime=0x7f9f9abd4cb0, src_file=0x55b406722658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b406148fef in my_service_thread_sleep (control=0x55b406e5fb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055b40613d488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055b40627145f in pfs_spawn_thread (arg=0x55b40946f880) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f9f71ffb700 (LWP 105111)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e210ff in os_event::wait (this=0x55b4094b7610) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b405e20b85 in os_event::wait_low (this=0x55b4094b7610, reset_sig_count=3797) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b405e20ef6 in os_event_wait_low (event=0x55b4094b7610, reset_sig_count=3797) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b405f0cdf3 in srv_resume_thread (slot=0x55b406e5b748 , sig_count=3797, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b405f10e04 in srv_purge_coordinator_suspend (slot=0x55b406e5b748 , rseg_history_len=13) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055b405f1109a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f9f73fff700 (LWP 105106)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e20a1a in os_event::timed_wait (this=0x55b4094b7810, abstime=0x7f9f73ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b405e20d31 in os_event::wait_time_low (this=0x55b4094b7810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b405e20ed1 in os_event_wait_time_low (event=0x55b4094b7810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b405f0effc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f9f5f7fe700 (LWP 105119)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e210ff in os_event::wait (this=0x55b4094b7910) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b405e20b85 in os_event::wait_low (this=0x55b4094b7910, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b405e20ef6 in os_event_wait_low (event=0x55b4094b7910, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b405feceb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f9f99583700 (LWP 105155)): #0 0x00007f9faa0fe40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055b4056aaf44 in my_sigwait (set=0x7f9f99582cf0, sig=0x7f9f99582cb4, code=0x7f9f99582cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055b4056b1584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055b40627145f in pfs_spawn_thread (arg=0x55b40947afb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f9f9c7f0700 (LWP 105067)): #0 0x00007f9faa1c5483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9f9fc576f8 in asio::detail::epoll_reactor::run (this=0x55b4093b0040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f9f9fc5bdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55b4093afee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f9f9fc5c0a2 in asio::detail::task_io_service::run (this=0x55b4093afee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f9f9fc73afd in asio::io_service::run (this=0x55b4093aefe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f9f9fc74251 in gcomm::AsioProtonet::event_loop (this=0x55b4093aef40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f9f9fc922ed in GCommConn::run (this=0x55b4093af6c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f9f9fc95399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f9f994a2700 (LWP 106026)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062dec99 in safe_cond_timedwait (cond=0x55b40701e1a0 , mp=0x55b407020ba0 , abstime=0x7f9f994a1c60, file=0x55b40639a4c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b4056ab845 in inline_mysql_cond_timedwait (that=0x55b40701e1a0 , mutex=0x55b407020ba0 , abstime=0x7f9f994a1c60, src_file=0x55b40639c538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b4056b061b in cache_thread (thd=0x7f9f2c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055b4056b08c9 in one_thread_per_connection_end (thd=0x7f9f2c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055b40594d0f3 in do_handle_one_connection (connect=0x55b40947bc10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055b40594cdad in handle_one_connection (arg=0x55b40947bc10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055b40627145f in pfs_spawn_thread (arg=0x55b40932ba70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f9f99538700 (LWP 105244)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062de997 in safe_cond_wait (cond=0x55b40701e1e0 , mp=0x55b407021560 , file=0x55b4063a9cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b4056dc4ee in inline_mysql_cond_wait (that=0x55b40701e1e0 , mutex=0x55b407021560 , src_file=0x55b4063aa830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055b4056ddd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055b40627145f in pfs_spawn_thread (arg=0x55b40947b4a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f9f5dffb700 (LWP 105122)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062de997 in safe_cond_wait (cond=0x7f9f5dffac70, mp=0x7f9f5dffacb0, file=0x55b4065acc40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b405d38487 in inline_mysql_cond_wait (that=0x7f9f5dffac70, mutex=0x7f9f5dffacb0, src_file=0x55b4065ae5d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b405d3d0c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055b40627145f in pfs_spawn_thread (arg=0x55b4096f8910) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f9f717fa700 (LWP 105112)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e210ff in os_event::wait (this=0x55b4094b7710) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b405e20b85 in os_event::wait_low (this=0x55b4094b7710, reset_sig_count=15205) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b405e20ef6 in os_event_wait_low (event=0x55b4094b7710, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b405f0cdf3 in srv_resume_thread (slot=0x55b406e5b7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b405f109f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f9f994ed700 (LWP 105245)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062dec99 in safe_cond_timedwait (cond=0x55b40701e1a0 , mp=0x55b407020ba0 , abstime=0x7f9f994ecc60, file=0x55b40639a4c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b4056ab845 in inline_mysql_cond_timedwait (that=0x55b40701e1a0 , mutex=0x55b407020ba0 , abstime=0x7f9f994ecc60, src_file=0x55b40639c538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b4056b061b in cache_thread (thd=0x7f9f34000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055b4056b08c9 in one_thread_per_connection_end (thd=0x7f9f34000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055b40594d0f3 in do_handle_one_connection (connect=0x55b409826860) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055b40594cdad in handle_one_connection (arg=0x55b409826860) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055b40627145f in pfs_spawn_thread (arg=0x55b409848670) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f9f72ffd700 (LWP 105109)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e20a1a in os_event::timed_wait (this=0x55b409657a90, abstime=0x7f9f72ffcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b405e20d31 in os_event::wait_time_low (this=0x55b409657a90, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b405e20ed1 in os_event_wait_time_low (event=0x55b409657a90, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b405f6f2fb in ib_wqueue_timedwait (wq=0x55b409657960, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055b4060afea4 in fts_optimize_thread (arg=0x55b409657960) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f9f5e7fc700 (LWP 105121)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e210ff in os_event::wait (this=0x55b4094b7a10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b405e20b85 in os_event::wait_low (this=0x55b4094b7a10, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b405e20ef6 in os_event_wait_low (event=0x55b4094b7a10, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b405fd5b26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f9f7a7fc700 (LWP 105102)): #0 0x00007f9fab9d7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b405e1441c in LinuxAIOHandler::collect (this=0x7f9f7a7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b405e14945 in LinuxAIOHandler::poll (this=0x7f9f7a7fbb00, m1=0x7f9f7a7fbba8, m2=0x7f9f7a7fbbb0, request=0x7f9f7a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b405e14aa8 in os_aio_linux_handler (global_segment=5, m1=0x7f9f7a7fbba8, m2=0x7f9f7a7fbbb0, request=0x7f9f7a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b405e195cc in os_aio_handler (segment=5, m1=0x7f9f7a7fbba8, m2=0x7f9f7a7fbbb0, request=0x7f9f7a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b40606c4a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055b405f12909 in io_handler_thread (arg=0x55b407099648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f9f5effd700 (LWP 105120)): #0 0x00007f9fabe0de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055b405e21401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055b405fc7a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f9f7b7fe700 (LWP 105100)): #0 0x00007f9fab9d7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b405e1441c in LinuxAIOHandler::collect (this=0x7f9f7b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b405e14945 in LinuxAIOHandler::poll (this=0x7f9f7b7fdb00, m1=0x7f9f7b7fdba8, m2=0x7f9f7b7fdbb0, request=0x7f9f7b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b405e14aa8 in os_aio_linux_handler (global_segment=3, m1=0x7f9f7b7fdba8, m2=0x7f9f7b7fdbb0, request=0x7f9f7b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b405e195cc in os_aio_handler (segment=3, m1=0x7f9f7b7fdba8, m2=0x7f9f7b7fdbb0, request=0x7f9f7b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b40606c4a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055b405f12909 in io_handler_thread (arg=0x55b407099638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f9f5ffff700 (LWP 105114)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e210ff in os_event::wait (this=0x55b4094b7790) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b405e20b85 in os_event::wait_low (this=0x55b4094b7790, reset_sig_count=13835) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b405e20ef6 in os_event_wait_low (event=0x55b4094b7790, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b405f0cdf3 in srv_resume_thread (slot=0x55b406e5b7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b405f109f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f9f7bfff700 (LWP 105099)): #0 0x00007f9fab9d7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b405e1441c in LinuxAIOHandler::collect (this=0x7f9f7bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b405e14945 in LinuxAIOHandler::poll (this=0x7f9f7bffeb00, m1=0x7f9f7bffeba8, m2=0x7f9f7bffebb0, request=0x7f9f7bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b405e14aa8 in os_aio_linux_handler (global_segment=2, m1=0x7f9f7bffeba8, m2=0x7f9f7bffebb0, request=0x7f9f7bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b405e195cc in os_aio_handler (segment=2, m1=0x7f9f7bffeba8, m2=0x7f9f7bffebb0, request=0x7f9f7bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b40606c4a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055b405f12909 in io_handler_thread (arg=0x55b407099630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f9f70ff9700 (LWP 105113)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e210ff in os_event::wait (this=0x55b4094b7690) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b405e20b85 in os_event::wait_low (this=0x55b4094b7690, reset_sig_count=17679) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b405e20ef6 in os_event_wait_low (event=0x55b4094b7690, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b405f0cdf3 in srv_resume_thread (slot=0x55b406e5b780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b405f109f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f9f98b82700 (LWP 105098)): #0 0x00007f9fab9d7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b405e1441c in LinuxAIOHandler::collect (this=0x7f9f98b81b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b405e14945 in LinuxAIOHandler::poll (this=0x7f9f98b81b00, m1=0x7f9f98b81ba8, m2=0x7f9f98b81bb0, request=0x7f9f98b81be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b405e14aa8 in os_aio_linux_handler (global_segment=1, m1=0x7f9f98b81ba8, m2=0x7f9f98b81bb0, request=0x7f9f98b81be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b405e195cc in os_aio_handler (segment=1, m1=0x7f9f98b81ba8, m2=0x7f9f98b81bb0, request=0x7f9f98b81be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b40606c4a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055b405f12909 in io_handler_thread (arg=0x55b407099628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f9f727fc700 (LWP 105110)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e210ff in os_event::wait (this=0x55b40964fd80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b405e20b85 in os_event::wait_low (this=0x55b40964fd80, reset_sig_count=4687) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b405e20ef6 in os_event_wait_low (event=0x55b40964fd80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b405df1761 in log_write_up_to (lsn=4513909, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000055b405df1d61 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #6 0x000055b405f0f430 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #7 0x000055b405f0f9f3 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #8 0x000055b405f10551 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #9 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f9f797fa700 (LWP 105107)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e20a1a in os_event::timed_wait (this=0x55b4094b7890, abstime=0x7f9f797f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b405e20d31 in os_event::wait_time_low (this=0x55b4094b7890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b405e20ed1 in os_event_wait_time_low (event=0x55b4094b7890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b405f0eada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f9f99383700 (LWP 105097)): #0 0x00007f9fab9d7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b405e1441c in LinuxAIOHandler::collect (this=0x7f9f99382b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b405e14945 in LinuxAIOHandler::poll (this=0x7f9f99382b00, m1=0x7f9f99382ba8, m2=0x7f9f99382bb0, request=0x7f9f99382be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b405e14aa8 in os_aio_linux_handler (global_segment=0, m1=0x7f9f99382ba8, m2=0x7f9f99382bb0, request=0x7f9f99382be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b405e195cc in os_aio_handler (segment=0, m1=0x7f9f99382ba8, m2=0x7f9f99382bb0, request=0x7f9f99382be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b40606c4a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055b405f12909 in io_handler_thread (arg=0x55b407099620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f9f78ff9700 (LWP 105105)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b405e20a1a in os_event::timed_wait (this=0x55b40968fd40, abstime=0x7f9f78ff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b405e20d31 in os_event::wait_time_low (this=0x55b40968fd40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b405e20ed1 in os_event_wait_time_low (event=0x55b40968fd40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b405ded88e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f9f9b758700 (LWP 105095)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9f9fbaa132 in fifo_lock_get (q=q@entry=0x55b40939dbd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f9f9fbaa7a1 in gu_fifo_get_head (q=0x55b40939dbd0, err=0x7f9f9b75706c) at galerautils/src/gu_fifo.c:330 #3 0x00007f9f9fc8c0cc in gcs_recv (conn=0x55b409377550, action=0x7f9f9b7570b0) at gcs/src/gcs.cpp:2024 #4 0x00007f9f9fcb916c in galera::GcsActionSource::process (this=0x55b409376f30, recv_ctx=0x7f9f9b7574d0, exit_loop=@0x7f9f9b75729e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f9f9fcd8033 in galera::ReplicatorSMM::async_recv (this=0x55b40936f280, recv_ctx=0x7f9f9b7574d0) at galera/src/replicator_smm.cpp:383 #6 0x000055b406393276 in wsrep::wsrep_provider_v26::run_applier (this=0x55b4092f6360, applier_ctx=0x7f9f9b7574d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055b405a36636 in wsrep_replication_process (thd=0x7f9f88000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055b405a28282 in start_wsrep_THD (arg=0x55b4093c82a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f9f9ec74700 (LWP 105048)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9f9fcb5fd7 in gu::Lock::wait (cond=..., this=0x7f9f9ec73bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f9f9fcb6069 in galera::ServiceThd::thd_func (arg=0x55b40936f998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f9f9b7ee700 (LWP 105093)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062de997 in safe_cond_wait (cond=0x7f9f90016858, mp=0x7f9f900167a8, file=0x55b406448b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b405a36088 in inline_mysql_cond_wait (that=0x7f9f90016858, mutex=0x7f9f900167a8, src_file=0x55b406448e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b405a38d20 in Wsrep_thd_queue::pop_front (this=0x7f9f90016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b405a37740 in wsrep_rollback_process (rollbacker=0x7f9f90001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055b405a28282 in start_wsrep_THD (arg=0x55b4093c8e90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f9f9b7a3700 (LWP 105094)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062de997 in safe_cond_wait (cond=0x7f9f84016918, mp=0x7f9f84016868, file=0x55b406448b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b405a36088 in inline_mysql_cond_wait (that=0x7f9f84016918, mutex=0x7f9f84016868, src_file=0x55b406448e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b405a38d20 in Wsrep_thd_queue::pop_front (this=0x7f9f84016810) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b405a37b00 in wsrep_post_rollback_process (post_rollbacker=0x7f9f84000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055b405a28282 in start_wsrep_THD (arg=0x55b4093bf320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f9fa485f700 (LWP 105047)): #0 0x00007f9fabe0ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b4062dec99 in safe_cond_timedwait (cond=0x55b4078bd640 , mp=0x55b4078bd580 , abstime=0x7f9fa485ed70, file=0x55b406766840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b4062dfc53 in inline_mysql_cond_timedwait (that=0x55b4078bd640 , mutex=0x55b4078bd580 , abstime=0x7f9fa485ed70, src_file=0x55b406766888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b4062e07b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055b40627145f in pfs_spawn_thread (arg=0x55b409338730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f9f9bfef700 (LWP 105092)): #0 0x00007f9fabe0a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9f9fc90867 in gu::Lock::wait (cond=..., this=0x7f9f9bfee850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f9f9fc933ab in RecvBuf::front (this=this@entry=0x55b4093af8a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f9f9fc90f46 in gcomm_recv (backend=, msg=0x55b409377830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f9f9fc84eec in core_msg_recv (backend=backend@entry=0x55b409377978, recv_msg=recv_msg@entry=0x55b409377830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f9f9fc88164 in gcs_core_recv (conn=0x55b4093777d0, recv_act=recv_act@entry=0x7f9f9bfeed70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f9f9fc8f3d8 in gcs_recv_thread (arg=0x55b409377550) at gcs/src/gcs.cpp:1398 #7 0x00007f9fabe06dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9faa1c4ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f9fac217880 (LWP 105043)): #0 0x00007f9fabe0b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b4062d9368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b405b17503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9faa1ba20d in poll () from /lib64/libc.so.6 #5 0x000055b4056b7a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055b4056b6d25 in mysqld_main (argc=161, argv=0x55b4091f0d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055b4056aaec5 in main (argc=23, argv=0x7fffb3cd6af8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.105409' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.GCF-1043B-innodb/mysqld.2/data/core.105409 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 105409] [New LWP 105494] [New LWP 105430] [New LWP 105876] [New LWP 105492] [New LWP 105932] [New LWP 105493] [New LWP 105934] [New LWP 105869] [New LWP 105917] [New LWP 105918] [New LWP 105921] [New LWP 105927] [New LWP 105928] [New LWP 105870] [New LWP 105933] [New LWP 105935] [New LWP 105875] [New LWP 105936] [New LWP 105871] [New LWP 105939] [New LWP 105874] [New LWP 105931] [New LWP 105926] [New LWP 105920] [New LWP 105873] [New LWP 105930] [New LWP 105929] [New LWP 105919] [New LWP 105872] [New LWP 105495] [New LWP 105411] [New LWP 105449] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fd0af7c39d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fd0af7c39d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556f74da2368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556f745e0503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd0adb7220d in poll () from /lib64/libc.so.6 #5 0x0000556f74180a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000556f7417fd25 in mysqld_main (argc=162, argv=0x556f771a9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000556f74173ec5 in main (argc=23, argv=0x7fff526d0978) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fd0a4be7700 (LWP 105449)): #0 0x00007fd0adb7d483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd0a766e6f8 in asio::detail::epoll_reactor::run (this=0x556f77369010, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fd0a7672df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x556f77368eb0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fd0a76730a2 in asio::detail::task_io_service::run (this=0x556f77368eb0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fd0a768aafd in asio::io_service::run (this=0x556f77367bd0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fd0a768b251 in gcomm::AsioProtonet::event_loop (this=0x556f77367b30, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fd0a76a92ed in GCommConn::run (this=0x556f77368640) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fd0a76ac399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fd0a8217700 (LWP 105411)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f74da7c99 in safe_cond_timedwait (cond=0x556f76386640 , mp=0x556f76386580 , abstime=0x7fd0a8216d70, file=0x556f7522f840 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556f74da8c53 in inline_mysql_cond_timedwait (that=0x556f76386640 , mutex=0x556f76386580 , abstime=0x7fd0a8216d70, src_file=0x556f7522f888 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556f74da97b6 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000556f74d3a45f in pfs_spawn_thread (arg=0x556f772f17d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fd0a4350700 (LWP 105495)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd0a75c1132 in fifo_lock_get (q=q@entry=0x556f77356ad0) at galerautils/src/gu_fifo.c:262 #2 0x00007fd0a75c17a1 in gu_fifo_get_head (q=0x556f77356ad0, err=0x7fd0a434f06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd0a76a30cc in gcs_recv (conn=0x556f77330450, action=0x7fd0a434f0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd0a76d016c in galera::GcsActionSource::process (this=0x556f77360e70, recv_ctx=0x7fd0a434f4d0, exit_loop=@0x7fd0a434f29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd0a76ef033 in galera::ReplicatorSMM::async_recv (this=0x556f77328200, recv_ctx=0x7fd0a434f4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000556f74e5c276 in wsrep::wsrep_provider_v26::run_applier (this=0x556f7729de50, applier_ctx=0x7fd0a434f4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000556f744ff636 in wsrep_replication_process (thd=0x7fd08c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000556f744f1282 in start_wsrep_THD (arg=0x556f77389710) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fd06effd700 (LWP 105872)): #0 0x00007fd0af38f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f748dd41c in LinuxAIOHandler::collect (this=0x7fd06effcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f748dd945 in LinuxAIOHandler::poll (this=0x7fd06effcb00, m1=0x7fd06effcba8, m2=0x7fd06effcbb0, request=0x7fd06effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f748ddaa8 in os_aio_linux_handler (global_segment=2, m1=0x7fd06effcba8, m2=0x7fd06effcbb0, request=0x7fd06effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f748e25cc in os_aio_handler (segment=2, m1=0x7fd06effcba8, m2=0x7fd06effcbb0, request=0x7fd06effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f74b354a5 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000556f749db909 in io_handler_thread (arg=0x556f75b62630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fd066ffd700 (LWP 105919)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748e9a1a in os_event::timed_wait (this=0x556f774794c0, abstime=0x7fd066ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f748e9d31 in os_event::wait_time_low (this=0x556f774794c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f748e9ed1 in os_event_wait_time_low (event=0x556f774794c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f749d7ada in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fd0577fe700 (LWP 105929)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748ea0ff in os_event::wait (this=0x556f774792c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f748e9b85 in os_event::wait_low (this=0x556f774792c0, reset_sig_count=17476) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f748e9ef6 in os_event_wait_low (event=0x556f774792c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f749d5df3 in srv_resume_thread (slot=0x556f75924780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f749d99f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fd056ffd700 (LWP 105930)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748ea0ff in os_event::wait (this=0x556f774793c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f748e9b85 in os_event::wait_low (this=0x556f774793c0, reset_sig_count=12981) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f748e9ef6 in os_event_wait_low (event=0x556f774793c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f749d5df3 in srv_resume_thread (slot=0x556f759247f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f749d99f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fd06e7fc700 (LWP 105873)): #0 0x00007fd0af38f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f748dd41c in LinuxAIOHandler::collect (this=0x7fd06e7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f748dd945 in LinuxAIOHandler::poll (this=0x7fd06e7fbb00, m1=0x7fd06e7fbba8, m2=0x7fd06e7fbbb0, request=0x7fd06e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f748ddaa8 in os_aio_linux_handler (global_segment=3, m1=0x7fd06e7fbba8, m2=0x7fd06e7fbbb0, request=0x7fd06e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f748e25cc in os_aio_handler (segment=3, m1=0x7fd06e7fbba8, m2=0x7fd06e7fbbb0, request=0x7fd06e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f74b354a5 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000556f749db909 in io_handler_thread (arg=0x556f75b62638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fd0667fc700 (LWP 105920)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748e9a1a in os_event::timed_wait (this=0x556f77683a30, abstime=0x7fd0667fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f748e9d31 in os_event::wait_time_low (this=0x556f77683a30, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f748e9ed1 in os_event_wait_time_low (event=0x556f77683a30, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f74b1f142 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fd0657fa700 (LWP 105926)): #0 0x00007fd0af7c5e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000556f748ea401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000556f749d935b in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000556f749d948b in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fd0567fc700 (LWP 105931)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748ea0ff in os_event::wait (this=0x556f77479540) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f748e9b85 in os_event::wait_low (this=0x556f77479540, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f748e9ef6 in os_event_wait_low (event=0x556f77479540, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f74ab5eb1 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fd06dffb700 (LWP 105874)): #0 0x00007fd0af38f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f748dd41c in LinuxAIOHandler::collect (this=0x7fd06dffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f748dd945 in LinuxAIOHandler::poll (this=0x7fd06dffab00, m1=0x7fd06dffaba8, m2=0x7fd06dffabb0, request=0x7fd06dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f748ddaa8 in os_aio_linux_handler (global_segment=4, m1=0x7fd06dffaba8, m2=0x7fd06dffabb0, request=0x7fd06dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f748e25cc in os_aio_handler (segment=4, m1=0x7fd06dffaba8, m2=0x7fd06dffabb0, request=0x7fd06dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f74b354a5 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000556f749db909 in io_handler_thread (arg=0x556f75b62640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fd0a4092700 (LWP 105939)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f74da7c99 in safe_cond_timedwait (cond=0x556f75ae71a0 , mp=0x556f75ae9ba0 , abstime=0x7fd0a4091c60, file=0x556f74e634c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556f74174845 in inline_mysql_cond_timedwait (that=0x556f75ae71a0 , mutex=0x556f75ae9ba0 , abstime=0x7fd0a4091c60, src_file=0x556f74e65538 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556f7417961b in cache_thread (thd=0x7fd078000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000556f741798c9 in one_thread_per_connection_end (thd=0x7fd078000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000556f744160f3 in do_handle_one_connection (connect=0x556f77779400) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000556f74415dad in handle_one_connection (arg=0x556f77779400) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000556f74d3a45f in pfs_spawn_thread (arg=0x556f777a7590) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fd06f7fe700 (LWP 105871)): #0 0x00007fd0af38f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f748dd41c in LinuxAIOHandler::collect (this=0x7fd06f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f748dd945 in LinuxAIOHandler::poll (this=0x7fd06f7fdb00, m1=0x7fd06f7fdba8, m2=0x7fd06f7fdbb0, request=0x7fd06f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f748ddaa8 in os_aio_linux_handler (global_segment=1, m1=0x7fd06f7fdba8, m2=0x7fd06f7fdbb0, request=0x7fd06f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f748e25cc in os_aio_handler (segment=1, m1=0x7fd06f7fdba8, m2=0x7fd06f7fdbb0, request=0x7fd06f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f74b354a5 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000556f749db909 in io_handler_thread (arg=0x556f75b62628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fd0a40dd700 (LWP 105936)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f74da7997 in safe_cond_wait (cond=0x556f75ae71e0 , mp=0x556f75aea560 , file=0x556f74e72cc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f741a54ee in inline_mysql_cond_wait (that=0x556f75ae71e0 , mutex=0x556f75aea560 , src_file=0x556f74e73830 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000556f741a6d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000556f74d3a45f in pfs_spawn_thread (arg=0x556f777dbd60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fd06d7fa700 (LWP 105875)): #0 0x00007fd0af38f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f748dd41c in LinuxAIOHandler::collect (this=0x7fd06d7f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f748dd945 in LinuxAIOHandler::poll (this=0x7fd06d7f9b00, m1=0x7fd06d7f9ba8, m2=0x7fd06d7f9bb0, request=0x7fd06d7f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f748ddaa8 in os_aio_linux_handler (global_segment=5, m1=0x7fd06d7f9ba8, m2=0x7fd06d7f9bb0, request=0x7fd06d7f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f748e25cc in os_aio_handler (segment=5, m1=0x7fd06d7f9ba8, m2=0x7fd06d7f9bb0, request=0x7fd06d7f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f74b354a5 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000556f749db909 in io_handler_thread (arg=0x556f75b62648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fd0a4128700 (LWP 105935)): #0 0x00007fd0adab640a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000556f74173f44 in my_sigwait (set=0x7fd0a4127cf0, sig=0x7fd0a4127cb4, code=0x7fd0a4127cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000556f7417a584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000556f74d3a45f in pfs_spawn_thread (arg=0x556f7743cc10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fd0557fa700 (LWP 105933)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748ea0ff in os_event::wait (this=0x556f77479640) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f748e9b85 in os_event::wait_low (this=0x556f77479640, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f748e9ef6 in os_event_wait_low (event=0x556f77479640, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f74a9eb26 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fd06ffff700 (LWP 105870)): #0 0x00007fd0af38f644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f748dd41c in LinuxAIOHandler::collect (this=0x7fd06fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f748dd945 in LinuxAIOHandler::poll (this=0x7fd06fffeb00, m1=0x7fd06fffeba8, m2=0x7fd06fffebb0, request=0x7fd06fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f748ddaa8 in os_aio_linux_handler (global_segment=0, m1=0x7fd06fffeba8, m2=0x7fd06fffebb0, request=0x7fd06fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f748e25cc in os_aio_handler (segment=0, m1=0x7fd06fffeba8, m2=0x7fd06fffebb0, request=0x7fd06fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f74b354a5 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000556f749db909 in io_handler_thread (arg=0x556f75b62620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fd057fff700 (LWP 105928)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748ea0ff in os_event::wait (this=0x556f77479340) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f748e9b85 in os_event::wait_low (this=0x556f77479340, reset_sig_count=14724) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f748e9ef6 in os_event_wait_low (event=0x556f77479340, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f749d5df3 in srv_resume_thread (slot=0x556f759247b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f749d99f7 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fd064ff9700 (LWP 105927)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748ea0ff in os_event::wait (this=0x556f77479240) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f748e9b85 in os_event::wait_low (this=0x556f77479240, reset_sig_count=4009) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f748e9ef6 in os_event_wait_low (event=0x556f77479240, reset_sig_count=4009) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f749d5df3 in srv_resume_thread (slot=0x556f75924748 , sig_count=4009, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f749d9e04 in srv_purge_coordinator_suspend (slot=0x556f75924748 , rseg_history_len=36) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000556f749da09a in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fd065ffb700 (LWP 105921)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748e9a1a in os_event::timed_wait (this=0x556f77621cf0, abstime=0x7fd065ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f748e9d31 in os_event::wait_time_low (this=0x556f77621cf0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f748e9ed1 in os_event_wait_time_low (event=0x556f77621cf0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f74a382fb in ib_wqueue_timedwait (wq=0x556f77621bc0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000556f74b78ea4 in fts_optimize_thread (arg=0x556f77621bc0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fd067fff700 (LWP 105918)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748e9a1a in os_event::timed_wait (this=0x556f77479440, abstime=0x7fd067ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f748e9d31 in os_event::wait_time_low (this=0x556f77479440, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f748e9ed1 in os_event_wait_time_low (event=0x556f77479440, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f749d7ffc in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fd0677fe700 (LWP 105917)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748e9a1a in os_event::timed_wait (this=0x556f77651970, abstime=0x7fd0677fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f748e9d31 in os_event::wait_time_low (this=0x556f77651970, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f748e9ed1 in os_event_wait_time_low (event=0x556f77651970, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f748b688e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fd095e6f700 (LWP 105869)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f74da7c99 in safe_cond_timedwait (cond=0x556f76373ea0 , mp=0x556f76373de0 , abstime=0x7fd095e6ecb0, file=0x556f751eb610 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556f74c11acb in inline_mysql_cond_timedwait (that=0x556f76373ea0 , mutex=0x556f76373de0 , abstime=0x7fd095e6ecb0, src_file=0x556f751eb658 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556f74c11fef in my_service_thread_sleep (control=0x556f75928b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000556f74c06488 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000556f74d3a45f in pfs_spawn_thread (arg=0x556f77431560) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fd054ff9700 (LWP 105934)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f74da7997 in safe_cond_wait (cond=0x7fd054ff8c70, mp=0x7fd054ff8cb0, file=0x556f75075c40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f74801487 in inline_mysql_cond_wait (that=0x7fd054ff8c70, mutex=0x7fd054ff8cb0, src_file=0x556f750775d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556f748060c3 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000556f74d3a45f in pfs_spawn_thread (arg=0x556f77634600) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fd0a43e6700 (LWP 105493)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f74da7997 in safe_cond_wait (cond=0x7fd080015608, mp=0x7fd080015558, file=0x556f74f11b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f744ff088 in inline_mysql_cond_wait (that=0x7fd080015608, mutex=0x7fd080015558, src_file=0x556f74f11e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556f74501d20 in Wsrep_thd_queue::pop_front (this=0x7fd080015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000556f74500740 in wsrep_rollback_process (rollbacker=0x7fd080000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000556f744f1282 in start_wsrep_THD (arg=0x556f77389a90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fd055ffb700 (LWP 105932)): #0 0x00007fd0af7c5e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000556f748ea401 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000556f74a90a3c in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fd097ffe700 (LWP 105492)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd0a76a7867 in gu::Lock::wait (cond=..., this=0x7fd097ffd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd0a76aa3ab in RecvBuf::front (this=this@entry=0x556f77368828, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fd0a76a7f46 in gcomm_recv (backend=, msg=0x556f77330730, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fd0a769beec in core_msg_recv (backend=backend@entry=0x556f77330878, recv_msg=recv_msg@entry=0x556f77330730, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fd0a769f164 in gcs_core_recv (conn=0x556f773306d0, recv_act=recv_act@entry=0x7fd097ffdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fd0a76a63d8 in gcs_recv_thread (arg=0x556f77330450) at gcs/src/gcs.cpp:1398 #7 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fd06cff9700 (LWP 105876)): #0 0x00007fd0af7c2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f748e9a1a in os_event::timed_wait (this=0x556f774795c0, abstime=0x7fd06cff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f748e9d31 in os_event::wait_time_low (this=0x556f774795c0, time_in_usec=726000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f748e9ed1 in os_event_wait_time_low (event=0x556f774795c0, time_in_usec=726000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f74abfd1d in pc_sleep_if_needed (next_loop_time=1549884212851, sig_count=2, cur_time=1549884212125) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000556f74ac0c56 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fd0a706b700 (LWP 105430)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd0a76ccfd7 in gu::Lock::wait (cond=..., this=0x7fd0a706abf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd0a76cd069 in galera::ServiceThd::thd_func (arg=0x556f77328918) at galera/src/galera_service_thd.cpp:30 #3 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fd0a439b700 (LWP 105494)): #0 0x00007fd0af7c2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f74da7997 in safe_cond_wait (cond=0x7fd084015608, mp=0x7fd084015558, file=0x556f74f11b60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f744ff088 in inline_mysql_cond_wait (that=0x7fd084015608, mutex=0x7fd084015558, src_file=0x556f74f11e80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556f74501d20 in Wsrep_thd_queue::pop_front (this=0x7fd084015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000556f74500b00 in wsrep_post_rollback_process (post_rollbacker=0x7fd084000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000556f744f1282 in start_wsrep_THD (arg=0x556f77380c00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007fd0af7bedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd0adb7cead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fd0afbcf880 (LWP 105409)): #0 0x00007fd0af7c39d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556f74da2368 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556f745e0503 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd0adb7220d in poll () from /lib64/libc.so.6 #5 0x0000556f74180a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000556f7417fd25 in mysqld_main (argc=162, argv=0x556f771a9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000556f74173ec5 in main (argc=23, argv=0x7fff526d0978) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.GCF-1043B-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043B-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1043B Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1051 'innodb' w2 [ pass ] 4158 -------------------------------------------------------------------------- The servers were restarted 28 times Spent 1401.736 of 4191 seconds executing testcases Check of testcase failed for: galera_sr.mysql-wsrep-features#148 galera_sr.galera_sr_cc_slave galera_sr.galera_sr_kill_all_norecovery Completed: Failed 17/83 tests, 79.52% were successful. Failing test(s): galera_sr.galera_sr_large_fragment galera_sr.GCF-1018B galera_sr.galera_sr_insert_select galera_sr.galera-features#56 galera_sr.galera_sr_cc_master galera_sr.GCF-437 galera_sr.galera_sr_rollback_statement galera_sr.mysql-wsrep-features#165 galera_sr.galera_sr_table_contents galera_sr.galera_sr_load_data galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_mysqldump_sst galera_sr.galera_sr_small_gcache galera_sr.GCF-1043A galera_sr.galera_sr_many_fragments galera_sr.mysql-wsrep-features#8 galera_sr.GCF-1043B The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html Errors/warnings were found in logfiles during server shutdown after running the following sequence(s) of tests: galera_sr.galera_sr_large_fragment galera_sr.GCF-1018B galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_conflict_with_rollback_master galera_sr.galera_sr_ddl_master galera_sr.galera_sr_insert_select galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.GCF-623 galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera-features#56 galera_sr.GCF-845 galera_sr.galera_sr_cc_master galera_sr.galera_sr_bf_abort galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 galera_sr.galera_sr_load_data galera_sr.galera_sr_blob galera_sr.galera_sr_kill_query galera_sr.galera_sr_kill_connection galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery galera_sr.galera_sr_small_gcache galera_sr.galera_sr_myisam galera_sr.GCF-1043A galera_sr.galera_sr_many_fragments galera_sr.mysql-wsrep-features#8 galera_sr.GCF-1043B mysql-test-run: *** ERROR: there were failing test cases