Checking supported features... MariaDB Version 10.4.4-MariaDB-debug - SSL connections supported - binaries are debug compiled Using suites: galera_sr Collecting tests... Installing system database... Using parallel: 4 worker[3] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 worker[1] Using MTR_BUILD_THREAD 301, with reserved ports 16020..16039 worker[2] Using MTR_BUILD_THREAD 302, with reserved ports 16040..16059 worker[4] Using MTR_BUILD_THREAD 303, with reserved ports 16060..16079 worker[3] Trying to dump core for [mysqltest - pid: 217313, winpid: 217313] worker[3] Trying to dump core for [mysqld.1 - pid: 215950, winpid: 215950] worker[3] Trying to dump core for [mysqld.2 - pid: 216307, winpid: 216307] ============================================================================== TEST WORKER RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- galera_sr.GCF-437 [ disabled ] test relies on InnoDB redo log size limtation galera_sr.galera_sr_table_contents [ disabled ] missing file galera_sr.galera_sr_gtid 'innodb' w2 [ pass ] 33187 galera_sr.mysql-wsrep-features#32 'innodb' w1 [ pass ] 20993 galera_sr.galera_sr_log_bin 'innodb' w4 [ pass ] 33725 galera_sr.mysql-wsrep-features#136 'innodb' w2 [ pass ] 6262 galera_sr.galera_sr_v1_row_events 'innodb' w1 [ pass ] 5269 galera_sr.GCF-627 'innodb' w2 [ pass ] 10361 galera_sr.galera_sr_create_drop 'innodb' w1 [ pass ] 7704 galera_sr.GCF-845 'innodb' w2 [ pass ] 8034 galera_sr.GCF-851 'innodb' w2 [ pass ] 6105 galera_sr.galera_sr_ddl_master 'innodb' w1 [ pass ] 12293 galera_sr.galera_sr_ddl_schema 'innodb' w1 [ pass ] 7196 galera_sr.galera_sr_ddl_slave 'innodb' w1 [ pass ] 9826 galera_sr.galera_sr_ddl_unrelated 'innodb' w1 [ pass ] 12746 galera_sr.GCF-867 'innodb' w2 [ pass ] 46558 galera_sr.galera_sr_dupkey_error 'innodb' w1 [ pass ] 11228 galera_sr.GCF-889 'innodb' w2 [ pass ] 6099 galera_sr.GCF-900 'innodb' w2 [ pass ] 7796 galera_sr.galera_sr_fk_conflict 'innodb' w1 [ pass ] 20631 galera_sr.GCF-1018B 'innodb' w4 [ pass ] 695479 galera_sr.galera_sr_kill_all_nobootstrap 'innodb' w4 [ pass ] 31615 galera_sr.galera_sr_kill_all_pcrecovery 'innodb' w4 [ pass ] 33066 galera_sr.galera_sr_load_data_splitting 'innodb' w4 [ pass ] 41477 galera_sr.galera_sr_large_fragment 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 16:54:16 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr_large_fragment.log == connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1024 * 1024 * 10; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; connection node_2; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT COUNT(*) > 50000 FROM t1; COUNT(*) > 50000 1 connection node_1; ROLLBACK; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.215953' (0/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.1/data/core.215953 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 215953] [New LWP 216074] [New LWP 216076] [New LWP 215958] [New LWP 216085] [New LWP 215963] [New LWP 216086] [New LWP 216073] [New LWP 216000] [New LWP 215966] [New LWP 215976] [New LWP 216002] [New LWP 216009] [New LWP 216078] [New LWP 216080] [New LWP 216084] [New LWP 216087] [New LWP 216138] [New LWP 216003] [New LWP 216010] [New LWP 216012] [New LWP 217315] [New LWP 216083] [New LWP 216082] [New LWP 216081] [New LWP 216075] [New LWP 216015] [New LWP 216137] [New LWP 216077] [New LWP 216014] [New LWP 215978] [New LWP 215982] [New LWP 215980] [New LWP 216079] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f6d33a3f9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f6d33a3f9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055710e304550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055710db39ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6d31dee20d in poll () from /lib64/libc.so.6 #5 0x000055710d6c3f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055710d6c31b4 in mysqld_main (argc=164, argv=0x5571108d1140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055710d6b7695 in main (argc=25, argv=0x7ffded77a768) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f6cf9ffb700 (LWP 216079)): #0 0x00007f6d33a41d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055710de3f749 in os_file_fsync_posix (file=51) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055710de3fa9f in os_file_flush_func (file=51) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055710e0881fa in pfs_os_file_flush_func (file=..., src_file=0x55710e70d588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055710e08b055 in fil_flush_low (space=0x557110f2b330, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x000055710e096b12 in fil_flush (space=0x557110f2b330) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4494 #6 0x000055710e01bdfa in buf_flush_write_block_low (bpage=0x7f6d2144b670, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1071 #7 0x000055710e01c497 in buf_flush_page (buf_pool=0x557110c75290, bpage=0x7f6d2144b670, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1209 #8 0x000055710e01ef7c in buf_flush_single_page_from_LRU (buf_pool=0x557110c75290) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2182 #9 0x000055710e029ee8 in buf_LRU_get_free_block (buf_pool=0x557110c75290) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1194 #10 0x000055710e006eaf in buf_page_init_for_read (err=0x7f6cf9ff90f4, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5387 #11 0x000055710e0305a0 in buf_read_page_low (err=0x7f6cf9ff90f4, sync=true, type=0, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #12 0x000055710e031182 in buf_read_page (page_id=..., zip_size=0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:412 #13 0x000055710e003d76 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x55710e6c4608 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=8113, mtr=0x7f6cf9ff9580, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4395 #14 0x000055710dfd822e in btr_free_externally_stored_field (index=0x557110f2cee8, field_ref=0x7f6d21838244 "", rec=0x7f6d218381ff "2f750a2d-3a9d-11e9-b42b-92e7bb51617e\200", offsets=0x7f6cec00f7c8, page_zip=0x0, i=6, rollback=false, local_mtr=0x7f6cf9ffa100) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8113 #15 0x000055710dfd8771 in btr_rec_free_externally_stored_fields (index=0x557110f2cee8, rec=0x7f6d218381ff "2f750a2d-3a9d-11e9-b42b-92e7bb51617e\200", offsets=0x7f6cec00f7c8, page_zip=0x0, rollback=false, mtr=0x7f6cf9ffa100) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8215 #16 0x000055710dfd2dc4 in btr_cur_pessimistic_delete (err=0x7f6cf9ff9dbc, has_reserved_extents=0, cursor=0x557110db1748, flags=0, rollback=false, mtr=0x7f6cf9ffa100) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5905 #17 0x000055710def7f8c in row_purge_remove_clust_if_poss_low (node=0x557110db16b0, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:175 #18 0x000055710def8111 in row_purge_remove_clust_if_poss (node=0x557110db16b0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #19 0x000055710def993e in row_purge_del_mark (node=0x557110db16b0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #20 0x000055710defaec6 in row_purge_record_func (node=0x557110db16b0, undo_rec=0x557110db21a8 "", thr=0x557110db15f8, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #21 0x000055710defb203 in row_purge (node=0x557110db16b0, undo_rec=0x557110db21a8 "", thr=0x557110db15f8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #22 0x000055710defb5ae in row_purge_step (thr=0x557110db15f8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #23 0x000055710de7d70e in que_thr_step (thr=0x557110db15f8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #24 0x000055710de7d93e in que_run_threads_low (thr=0x557110db15f8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #25 0x000055710de7db2e in que_run_threads (thr=0x557110db15f8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #26 0x000055710df64faa in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1325 #27 0x000055710df3adf8 in srv_do_purge (n_total_purged=0x7f6cf9ffad98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #28 0x000055710df3b225 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #29 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f6d28465700 (LWP 215980)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710e309b7f in safe_cond_wait (cond=0x7f6d0c015938, mp=0x7f6d0c015888, file=0x55710e4784c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055710da574ba in inline_mysql_cond_wait (that=0x7f6d0c015938, mutex=0x7f6d0c015888, src_file=0x55710e4787e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055710da5a34e in Wsrep_thd_queue::pop_front (this=0x7f6d0c015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055710da590d0 in wsrep_post_rollback_process (post_rollbacker=0x7f6d0c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x000055710da49587 in start_wsrep_THD (arg=0x557110a9fdd0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f6d2841a700 (LWP 215982)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d2b84f272 in fifo_lock_get (q=q@entry=0x7f6d2aa36010) at galerautils/src/gu_fifo.c:262 #2 0x00007f6d2b84f8e1 in gu_fifo_get_head (q=0x7f6d2aa36010, err=0x7f6d2841906c) at galerautils/src/gu_fifo.c:330 #3 0x00007f6d2b93143c in gcs_recv (conn=0x557110a57790, action=0x7f6d284190b0) at gcs/src/gcs.cpp:2024 #4 0x00007f6d2b95e50c in galera::GcsActionSource::process (this=0x557110a57380, recv_ctx=0x7f6d284194d0, exit_loop=@0x7f6d2841929e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f6d2b97d433 in galera::ReplicatorSMM::async_recv (this=0x557110a4f6b0, recv_ctx=0x7f6d284194d0) at galera/src/replicator_smm.cpp:383 #6 0x000055710e3be8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5571109d67f0, applier_ctx=0x7f6d284194d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055710da57a69 in wsrep_replication_process (thd=0x7f6d10000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055710da49587 in start_wsrep_THD (arg=0x557110aa02f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f6d284b0700 (LWP 215978)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710e309b7f in safe_cond_wait (cond=0x7f6d18017e98, mp=0x7f6d18017de8, file=0x55710e4784c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055710da574ba in inline_mysql_cond_wait (that=0x7f6d18017e98, mutex=0x7f6d18017de8, src_file=0x55710e4787e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055710da5a34e in Wsrep_thd_queue::pop_front (this=0x7f6d18017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055710da58c54 in wsrep_rollback_process (rollbacker=0x7f6d18001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x000055710da49587 in start_wsrep_THD (arg=0x557110a975a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f6d027fc700 (LWP 216014)): #0 0x00007f6d3360b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055710de3e09a in LinuxAIOHandler::collect (this=0x7f6d027fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055710de3e5c3 in LinuxAIOHandler::poll (this=0x7f6d027fbb00, m1=0x7f6d027fbba8, m2=0x7f6d027fbbb0, request=0x7f6d027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055710de3e726 in os_aio_linux_handler (global_segment=5, m1=0x7f6d027fbba8, m2=0x7f6d027fbbb0, request=0x7f6d027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055710de4324a in os_aio_handler (segment=5, m1=0x7f6d027fbba8, m2=0x7f6d027fbbb0, request=0x7f6d027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055710e0964a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055710df3ca47 in io_handler_thread (arg=0x55710f0d7188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f6cfaffd700 (LWP 216077)): #0 0x00007f6d33a3ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4a698 in os_event::timed_wait (this=0x557110d2fad0, abstime=0x7f6cfaffcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055710de4a9af in os_event::wait_time_low (this=0x557110d2fad0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055710de4ab4f in os_event_wait_time_low (event=0x557110d2fad0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055710df994c9 in ib_wqueue_timedwait (wq=0x557110d2f9a0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055710e0da73a in fts_optimize_thread (arg=0x557110d2f9a0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f6d211bb700 (LWP 216137)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710e309b7f in safe_cond_wait (cond=0x55710f05b4e0 , mp=0x55710f05e800 , file=0x55710e3d5488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055710d6e8fe2 in inline_mysql_cond_wait (that=0x55710f05b4e0 , mutex=0x55710f05e800 , src_file=0x55710e3d5ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055710d6ea874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055710e29bfb3 in pfs_spawn_thread (arg=0x557110e91a00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f6d01ffb700 (LWP 216015)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110c8e4e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110c8e4e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110c8e4e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710df44d96 in sync_array_wait_event (arr=0x557110b66b70, cell=@0x7f6d01ffa680: 0x557110b66ca0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055710df48615 in rw_lock_sx_lock_func (lock=0x7f6d21419c98, pass=2, file_name=0x55710e6e3c28 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1188) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x000055710e017b21 in pfs_rw_lock_sx_lock_func (lock=0x7f6d21419c98, pass=2, file_name=0x55710e6e3c28 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1188) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x000055710e01c395 in buf_flush_page (buf_pool=0x557110c75290, bpage=0x7f6d21419bd0, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1188 #8 0x000055710e01cddf in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=76, n_to_flush=200) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1433 #9 0x000055710e01d162 in buf_flush_page_and_try_neighbors (bpage=0x7f6d21419bd0, flush_type=BUF_FLUSH_LIST, n_to_flush=200, count=0x7f6d01ffa8b8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1505 #10 0x000055710e01dd7b in buf_do_flush_list_batch (buf_pool=0x557110c75290, min_n=200, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1763 #11 0x000055710e01e373 in buf_flush_batch (buf_pool=0x557110c75290, flush_type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7f6d01ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1833 #12 0x000055710e01ea33 in buf_flush_do_batch (buf_pool=0x557110c75290, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7f6d01ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1991 #13 0x000055710e01ed07 in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7f6d01ffabd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2097 #14 0x000055710e022a52 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3249 #15 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f6d015f5700 (LWP 216075)): #0 0x00007f6d33a3ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4a698 in os_event::timed_wait (this=0x557110b8fa10, abstime=0x7f6d015f4d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055710de4a9af in os_event::wait_time_low (this=0x557110b8fa10, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055710de4ab4f in os_event_wait_time_low (event=0x557110b8fa10, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055710df38c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f6cf8ff9700 (LWP 216081)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110b8f810) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110b8f810, reset_sig_count=52) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110b8f810, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710df36f21 in srv_resume_thread (slot=0x55710ee98980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055710df3ab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f6ce7fff700 (LWP 216082)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110b8f910) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110b8f910, reset_sig_count=46) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110b8f910, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710df36f21 in srv_resume_thread (slot=0x55710ee989f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055710df3ab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f6ce77fe700 (LWP 216083)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110b8fa90) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110b8fa90, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110b8fa90, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710e01719b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f6d21125700 (LWP 217315)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110da59c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110da59c0, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110da59c0, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710df44d96 in sync_array_wait_event (arr=0x557110b66b70, cell=@0x7f6d21122058: 0x557110b66d60) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055710df47aab in rw_lock_s_lock_spin (lock=0x557110d9a070, pass=0, file_name=0x55710e682c38 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000055710decfeda in rw_lock_s_lock_func (lock=0x557110d9a070, pass=0, file_name=0x55710e682c38 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000055710ded034b in pfs_rw_lock_s_lock_func (lock=0x557110d9a070, pass=0, file_name=0x55710e682c38 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000055710ded91e4 in row_mysql_freeze_data_dictionary_func (trx=0x7f6d21c59268, file=0x55710e682c38 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:2123 #9 0x000055710df1c777 in row_undo (node=0x7f6cb1507118, thr=0x7f6cb02c1358) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc:429 #10 0x000055710df1c9e4 in row_undo_step (thr=0x7f6cb02c1358) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc:497 #11 0x000055710de7d6f2 in que_thr_step (thr=0x7f6cb02c1358) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1040 #12 0x000055710de7d93e in que_run_threads_low (thr=0x7f6cb02c1358) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000055710de7db2e in que_run_threads (thr=0x7f6cb02c1358) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x000055710df7848b in trx_rollback_to_savepoint_low (trx=0x7f6d21c59268, savept=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:137 #15 0x000055710df78944 in trx_rollback_for_mysql_low (trx=0x7f6d21c59268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:211 #16 0x000055710df78c83 in trx_rollback_for_mysql (trx=0x7f6d21c59268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:237 #17 0x000055710dd6bf22 in innobase_rollback (hton=0x557110b63c40, thd=0x7f6cb0000b00, rollback_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4743 #18 0x000055710db3fe52 in ha_rollback_trans (thd=0x7f6cb0000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1851 #19 0x000055710d97d054 in trans_rollback (thd=0x7f6cb0000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:418 #20 0x000055710d7fc272 in mysql_execute_command (thd=0x7f6cb0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5903 #21 0x000055710d8030f1 in mysql_parse (thd=0x7f6cb0000b00, rawbuf=0x7f6cb0014198 "ROLLBACK", length=8, parser_state=0x7f6d21124040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #22 0x000055710d802720 in wsrep_mysql_parse (thd=0x7f6cb0000b00, rawbuf=0x7f6cb0014198 "ROLLBACK", length=8, parser_state=0x7f6d21124040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #23 0x000055710d7ee46d in dispatch_command (command=COM_QUERY, thd=0x7f6cb0000b00, packet=0x7f6cb000a761 "ROLLBACK", packet_length=8, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #24 0x000055710d7eccbb in do_command (thd=0x7f6cb0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #25 0x000055710d967c9e in do_handle_one_connection (connect=0x557110e24420) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #26 0x000055710d9679e2 in handle_one_connection (arg=0x557110e24420) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #27 0x000055710e29bfb3 in pfs_spawn_thread (arg=0x5571109d6c30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #28 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f6d02ffd700 (LWP 216012)): #0 0x00007f6d3360b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055710de3e09a in LinuxAIOHandler::collect (this=0x7f6d02ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055710de3e5c3 in LinuxAIOHandler::poll (this=0x7f6d02ffcb00, m1=0x7f6d02ffcba8, m2=0x7f6d02ffcbb0, request=0x7f6d02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055710de3e726 in os_aio_linux_handler (global_segment=4, m1=0x7f6d02ffcba8, m2=0x7f6d02ffcbb0, request=0x7f6d02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055710de4324a in os_aio_handler (segment=4, m1=0x7f6d02ffcba8, m2=0x7f6d02ffcbb0, request=0x7f6d02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055710e0964a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055710df3ca47 in io_handler_thread (arg=0x55710f0d7180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f6d037fe700 (LWP 216010)): #0 0x00007f6d3360b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055710de3e09a in LinuxAIOHandler::collect (this=0x7f6d037fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055710de3e5c3 in LinuxAIOHandler::poll (this=0x7f6d037fdb00, m1=0x7f6d037fdba8, m2=0x7f6d037fdbb0, request=0x7f6d037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055710de3e726 in os_aio_linux_handler (global_segment=3, m1=0x7f6d037fdba8, m2=0x7f6d037fdbb0, request=0x7f6d037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055710de4324a in os_aio_handler (segment=3, m1=0x7f6d037fdba8, m2=0x7f6d037fdbb0, request=0x7f6d037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055710e0964a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055710df3ca47 in io_handler_thread (arg=0x55710f0d7178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f6d20805700 (LWP 216003)): #0 0x00007f6d3360b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055710de3e09a in LinuxAIOHandler::collect (this=0x7f6d20804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055710de3e5c3 in LinuxAIOHandler::poll (this=0x7f6d20804b00, m1=0x7f6d20804ba8, m2=0x7f6d20804bb0, request=0x7f6d20804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055710de3e726 in os_aio_linux_handler (global_segment=1, m1=0x7f6d20804ba8, m2=0x7f6d20804bb0, request=0x7f6d20804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055710de4324a in os_aio_handler (segment=1, m1=0x7f6d20804ba8, m2=0x7f6d20804bb0, request=0x7f6d20804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055710e0964a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055710df3ca47 in io_handler_thread (arg=0x55710f0d7168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f6d21170700 (LWP 216138)): #0 0x00007f6d33a3ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710e309e81 in safe_cond_timedwait (cond=0x55710f05b4a0 , mp=0x55710f05de40 , abstime=0x7f6d2116fc60, file=0x55710e3c5b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055710d6b8015 in inline_mysql_cond_timedwait (that=0x55710f05b4a0 , mutex=0x55710f05de40 , abstime=0x7f6d2116fc60, src_file=0x55710e3c7b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055710d6bcb08 in cache_thread (thd=0x7f6cb8000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x000055710d6bcdb6 in one_thread_per_connection_end (thd=0x7f6cb8000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x000055710d967d28 in do_handle_one_connection (connect=0x557110f000d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055710d9679e2 in handle_one_connection (arg=0x557110f000d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055710e29bfb3 in pfs_spawn_thread (arg=0x557110f1de80) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f6d21206700 (LWP 216087)): #0 0x00007f6d31d3240a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055710d6b7714 in my_sigwait (set=0x7f6d21205cf0, sig=0x7f6d21205cb4, code=0x7f6d21205cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055710d6bda71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x000055710e29bfb3 in pfs_spawn_thread (arg=0x557110b53090) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f6ce6ffd700 (LWP 216084)): #0 0x00007f6d33a41e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055710de4b07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055710dff1b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f6cf97fa700 (LWP 216080)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110b8f890) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110b8f890, reset_sig_count=43) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110b8f890, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710df36f21 in srv_resume_thread (slot=0x55710ee989b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055710df3ab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f6cfa7fc700 (LWP 216078)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110da5a40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110da5a40, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110da5a40, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710df44d96 in sync_array_wait_event (arr=0x557110b66b70, cell=@0x7f6cfa7fbb98: 0x557110b66da0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055710df47c7c in rw_lock_x_lock_wait_func (lock=0x557110d9a070, pass=0, threshold=0, file_name=0x55710e693068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:459 #6 0x000055710df47dae in rw_lock_x_lock_low (lock=0x557110d9a070, pass=0, file_name=0x55710e693068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:514 #7 0x000055710df481cc in rw_lock_x_lock_func (lock=0x557110d9a070, pass=0, file_name=0x55710e693068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:673 #8 0x000055710df36571 in pfs_rw_lock_x_lock_func (lock=0x557110d9a070, pass=0, file_name=0x55710e693068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:552 #9 0x000055710df395b4 in srv_master_evict_from_table_cache (pct_check=50) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1999 #10 0x000055710df39c14 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2162 #11 0x000055710df3a67f in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f6d03fff700 (LWP 216009)): #0 0x00007f6d3360b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055710de3e09a in LinuxAIOHandler::collect (this=0x7f6d03ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055710de3e5c3 in LinuxAIOHandler::poll (this=0x7f6d03ffeb00, m1=0x7f6d03ffeba8, m2=0x7f6d03ffebb0, request=0x7f6d03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055710de3e726 in os_aio_linux_handler (global_segment=2, m1=0x7f6d03ffeba8, m2=0x7f6d03ffebb0, request=0x7f6d03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055710de4324a in os_aio_handler (segment=2, m1=0x7f6d03ffeba8, m2=0x7f6d03ffebb0, request=0x7f6d03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055710e0964a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055710df3ca47 in io_handler_thread (arg=0x55710f0d7170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f6d21006700 (LWP 216002)): #0 0x00007f6d3360b644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055710de3e09a in LinuxAIOHandler::collect (this=0x7f6d21005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055710de3e5c3 in LinuxAIOHandler::poll (this=0x7f6d21005b00, m1=0x7f6d21005ba8, m2=0x7f6d21005bb0, request=0x7f6d21005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055710de3e726 in os_aio_linux_handler (global_segment=0, m1=0x7f6d21005ba8, m2=0x7f6d21005bb0, request=0x7f6d21005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055710de4324a in os_aio_handler (segment=0, m1=0x7f6d21005ba8, m2=0x7f6d21005bb0, request=0x7f6d21005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055710e0964a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055710df3ca47 in io_handler_thread (arg=0x55710f0d7160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f6d237fe700 (LWP 215976)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d2b935bd7 in gu::Lock::wait (cond=..., this=0x7f6d237fd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6d2b93871b in RecvBuf::front (this=this@entry=0x557110a87928, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f6d2b9362b6 in gcomm_recv (backend=, msg=0x557110a57a70, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f6d2b92a25c in core_msg_recv (backend=backend@entry=0x557110a57bb8, recv_msg=recv_msg@entry=0x557110a57a70, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f6d2b92d4d4 in gcs_core_recv (conn=0x557110a57a10, recv_act=recv_act@entry=0x7f6d237fdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f6d2b934748 in gcs_recv_thread (arg=0x557110a57790) at gcs/src/gcs.cpp:1398 #7 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f6d23fff700 (LWP 215966)): #0 0x00007f6d31df9483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6d2b8fca68 in asio::detail::epoll_reactor::run (this=0x557110a880c0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f6d2b901160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x557110a87f60, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f6d2b901412 in asio::detail::task_io_service::run (this=0x557110a87f60, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f6d2b918e6d in asio::io_service::run (this=0x557110a86a90) at asio/asio/impl/io_service.ipp:58 #5 0x00007f6d2b9195c1 in gcomm::AsioProtonet::event_loop (this=0x557110a869f0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f6d2b93765d in GCommConn::run (this=0x557110a87740) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f6d2b93a709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f6d22858700 (LWP 216000)): #0 0x00007f6d33a3ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710e309e81 in safe_cond_timedwait (cond=0x55710f8e8a00 , mp=0x55710f8e8940 , abstime=0x7f6d22857cb0, file=0x55710e757630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055710e173309 in inline_mysql_cond_timedwait (that=0x55710f8e8a00 , mutex=0x55710f8e8940 , abstime=0x7f6d22857cb0, src_file=0x55710e757678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055710e17382d in my_service_thread_sleep (control=0x55710ee9cd00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055710e167cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055710e29bfb3 in pfs_spawn_thread (arg=0x557110b47910) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f6d00df4700 (LWP 216073)): #0 0x00007f6d33a3ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4a698 in os_event::timed_wait (this=0x557110d67a80, abstime=0x7f6d00df3d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055710de4a9af in os_event::wait_time_low (this=0x557110d67a80, time_in_usec=1000000, reset_sig_count=14) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055710de4ab4f in os_event_wait_time_low (event=0x557110d67a80, time_in_usec=1000000, reset_sig_count=14) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055710de174f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f6ce5ffb700 (LWP 216086)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710e309b7f in safe_cond_wait (cond=0x7f6ce5ffac70, mp=0x7f6ce5ffacb0, file=0x55710e5e0920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055710dd60cad in inline_mysql_cond_wait (that=0x7f6ce5ffac70, mutex=0x7f6ce5ffacb0, src_file=0x55710e5e2348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055710dd6579b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055710e29bfb3 in pfs_spawn_thread (arg=0x557110dd0830) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f6d2a934700 (LWP 215963)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d2b95b377 in gu::Lock::wait (cond=..., this=0x7f6d2a933bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6d2b95b409 in galera::ServiceThd::thd_func (arg=0x557110a4fdc8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f6ce67fc700 (LWP 216085)): #0 0x00007f6d33a3e965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4ad7d in os_event::wait (this=0x557110b8fb90) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055710de4a803 in os_event::wait_low (this=0x557110b8fb90, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055710de4ab74 in os_event_wait_low (event=0x557110b8fb90, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055710dfffcbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f6d2c493700 (LWP 215958)): #0 0x00007f6d33a3ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710e309e81 in safe_cond_timedwait (cond=0x55710f8fb1c0 , mp=0x55710f8fb100 , abstime=0x7f6d2c492d70, file=0x55710e79be90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055710e30ae3b in inline_mysql_cond_timedwait (that=0x55710f8fb1c0 , mutex=0x55710f8fb100 , abstime=0x7f6d2c492d70, src_file=0x55710e79bed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055710e30b99e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055710e29bfb3 in pfs_spawn_thread (arg=0x557110a18dd0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f6cfb7fe700 (LWP 216076)): #0 0x00007f6d33a3ed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055710de4a698 in os_event::timed_wait (this=0x557110d99b40, abstime=0x7f6cfb7fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055710de4a9af in os_event::wait_time_low (this=0x557110d99b40, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055710de4ab4f in os_event_wait_time_low (event=0x557110d99b40, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055710e0800e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f6cfbfff700 (LWP 216074)): #0 0x00007f6d33a41e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055710de4b07f in os_thread_sleep (tm=30000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055710df46202 in sync_array_print_long_waits (waiter=0x7f6cfbffec28, sema=0x7f6cfbffec30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:1099 #3 0x000055710df38fb6 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1801 #4 0x00007f6d33a3add5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6d31df8ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f6d33e4b880 (LWP 215953)): #0 0x00007f6d33a3f9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055710e304550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055710db39ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6d31dee20d in poll () from /lib64/libc.so.6 #5 0x000055710d6c3f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055710d6c31b4 in mysqld_main (argc=164, argv=0x5571108d1140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055710d6b7695 in main (argc=25, argv=0x7ffded77a768) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.1/data/core.215953 - found 'core.216309' (1/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.2/data/core.216309 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 216309] [New LWP 216546] [New LWP 216397] [New LWP 216547] [New LWP 216402] [New LWP 217264] [New LWP 216549] [New LWP 217266] [New LWP 217287] [New LWP 216550] [New LWP 217267] [New LWP 217293] [New LWP 217268] [New LWP 217270] [New LWP 217269] [New LWP 217289] [New LWP 217286] [New LWP 217291] [New LWP 217288] [New LWP 217292] [New LWP 217294] [New LWP 217299] [New LWP 217296] [New LWP 217301] [New LWP 217297] [New LWP 216504] [New LWP 217298] [New LWP 217265] [New LWP 217300] [New LWP 217304] [New LWP 217271] [New LWP 217290] [New LWP 217295] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007efc6ca069d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007efc6ca069d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000561aa20ad550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000561aa18e2ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007efc6adb520d in poll () from /lib64/libc.so.6 #5 0x0000561aa146cf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000561aa146c1b4 in mysqld_main (argc=165, argv=0x561aa4b77140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000561aa1460695 in main (argc=25, argv=0x7ffc724f0278) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7efc18ff9700 (LWP 217295)): #0 0x00007efc6ca08d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000561aa1be8749 in os_file_fsync_posix (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000561aa1be8a9f in os_file_flush_func (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000561aa1e311fa in pfs_os_file_flush_func (file=..., src_file=0x561aa24b6588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000561aa1e34055 in fil_flush_low (space=0x561aa50481d0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x0000561aa1e3f9f8 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x0000561aa1dbd959 in buf_dblwr_write_single_page (bpage=0x7efc35c0b330, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1332 #7 0x0000561aa1dc4d6b in buf_flush_write_block_low (bpage=0x7efc35c0b330, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1058 #8 0x0000561aa1dc5497 in buf_flush_page (buf_pool=0x561aa4f23ec0, bpage=0x7efc35c0b330, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1209 #9 0x0000561aa1dc7f7c in buf_flush_single_page_from_LRU (buf_pool=0x561aa4f23ec0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2182 #10 0x0000561aa1dd2ee8 in buf_LRU_get_free_block (buf_pool=0x561aa4f23ec0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1194 #11 0x0000561aa1dafeaf in buf_page_init_for_read (err=0x7efc18ff7194, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5387 #12 0x0000561aa1dd95a0 in buf_read_page_low (err=0x7efc18ff7194, sync=true, type=0, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #13 0x0000561aa1dda182 in buf_read_page (page_id=..., zip_size=0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:412 #14 0x0000561aa1dacd76 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x561aa246d608 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=8113, mtr=0x7efc18ff7620, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4395 #15 0x0000561aa1d8122e in btr_free_externally_stored_field (index=0x7efc38030768, field_ref=0x7efc3626c184 "", rec=0x7efc3626c13f "2f750a2d-3a9d-11e9-b42b-92e7bb51617e\200", offsets=0x7efc0000caa8, page_zip=0x0, i=6, rollback=false, local_mtr=0x7efc18ff81a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8113 #16 0x0000561aa1d81771 in btr_rec_free_externally_stored_fields (index=0x7efc38030768, rec=0x7efc3626c13f "2f750a2d-3a9d-11e9-b42b-92e7bb51617e\200", offsets=0x7efc0000caa8, page_zip=0x0, rollback=false, mtr=0x7efc18ff81a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8215 #17 0x0000561aa1d7bdc4 in btr_cur_pessimistic_delete (err=0x7efc18ff7e5c, has_reserved_extents=0, cursor=0x561aa5063370, flags=0, rollback=false, mtr=0x7efc18ff81a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5905 #18 0x0000561aa1ca0f8c in row_purge_remove_clust_if_poss_low (node=0x561aa50632d8, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:175 #19 0x0000561aa1ca1111 in row_purge_remove_clust_if_poss (node=0x561aa50632d8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #20 0x0000561aa1ca293e in row_purge_del_mark (node=0x561aa50632d8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #21 0x0000561aa1ca3ec6 in row_purge_record_func (node=0x561aa50632d8, undo_rec=0x561aa5063ea8 "", thr=0x561aa5062e30, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #22 0x0000561aa1ca4203 in row_purge (node=0x561aa50632d8, undo_rec=0x561aa5063ea8 "", thr=0x561aa5062e30) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #23 0x0000561aa1ca45ae in row_purge_step (thr=0x561aa5062e30) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #24 0x0000561aa1c2670e in que_thr_step (thr=0x561aa5062e30) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #25 0x0000561aa1c2693e in que_run_threads_low (thr=0x561aa5062e30) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #26 0x0000561aa1c26b2e in que_run_threads (thr=0x561aa5062e30) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #27 0x0000561aa1ce3983 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #28 0x0000561aa1ce3b2a in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #29 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7efc1b7fe700 (LWP 217290)): #0 0x00007efc6ca05d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3698 in os_event::timed_wait (this=0x561aa4fe6560, abstime=0x7efc1b7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561aa1bf39af in os_event::wait_time_low (this=0x561aa4fe6560, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561aa1bf3b4f in os_event_wait_time_low (event=0x561aa4fe6560, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561aa1d424c9 in ib_wqueue_timedwait (wq=0x561aa4fe6430, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000561aa1e8373a in fts_optimize_thread (arg=0x561aa4fe6430) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7efc2e7fc700 (LWP 217271)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3d7d in os_event::wait (this=0x561aa4f720f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561aa1bf3803 in os_event::wait_low (this=0x561aa4f720f0, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561aa1bf3b74 in os_event_wait_low (event=0x561aa4f720f0, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561aa1cedd96 in sync_array_wait_event (arr=0x561aa4e15780, cell=@0x7efc2e7fb690: 0x561aa4e158b0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000561aa1cf1615 in rw_lock_sx_lock_func (lock=0x7efc35c434f8, pass=2, file_name=0x561aa248cc28 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1188) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x0000561aa1dc0b21 in pfs_rw_lock_sx_lock_func (lock=0x7efc35c434f8, pass=2, file_name=0x561aa248cc28 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1188) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x0000561aa1dc5395 in buf_flush_page (buf_pool=0x561aa4f23ec0, bpage=0x7efc35c43430, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1188 #8 0x0000561aa1dc5ddf in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=84, n_to_flush=130) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1433 #9 0x0000561aa1dc6162 in buf_flush_page_and_try_neighbors (bpage=0x7efc35c43430, flush_type=BUF_FLUSH_LIST, n_to_flush=130, count=0x7efc2e7fb8c8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1505 #10 0x0000561aa1dc6d7b in buf_do_flush_list_batch (buf_pool=0x561aa4f23ec0, min_n=130, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1763 #11 0x0000561aa1dc7373 in buf_flush_batch (buf_pool=0x561aa4f23ec0, flush_type=BUF_FLUSH_LIST, min_n=130, lsn_limit=18446744073709551615, n=0x7efc2e7fbb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1833 #12 0x0000561aa1dc7a33 in buf_flush_do_batch (buf_pool=0x561aa4f23ec0, type=BUF_FLUSH_LIST, min_n=130, lsn_limit=18446744073709551615, n=0x7efc2e7fbb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1991 #13 0x0000561aa1dca72e in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2794 #14 0x0000561aa1dcb67c in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3203 #15 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7efc540fa700 (LWP 217304)): #0 0x00007efc6ca05d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa20b2e81 in safe_cond_timedwait (cond=0x561aa2e044a0 , mp=0x561aa2e06e40 , abstime=0x7efc540f9c60, file=0x561aa216eb08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000561aa1461015 in inline_mysql_cond_timedwait (that=0x561aa2e044a0 , mutex=0x561aa2e06e40 , abstime=0x7efc540f9c60, src_file=0x561aa2170b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000561aa1465b08 in cache_thread (thd=0x7efc38000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x0000561aa1465db6 in one_thread_per_connection_end (thd=0x7efc38000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x0000561aa1710d28 in do_handle_one_connection (connect=0x561aa51affd0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000561aa17109e2 in handle_one_connection (arg=0x561aa51affd0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000561aa2044fb3 in pfs_spawn_thread (arg=0x561aa51c8170) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7efc54190700 (LWP 217300)): #0 0x00007efc6acf940a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000561aa1460714 in my_sigwait (set=0x7efc5418fcf0, sig=0x7efc5418fcb4, code=0x7efc5418fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000561aa1466a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x0000561aa2044fb3 in pfs_spawn_thread (arg=0x561aa4e01b60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7efc35807700 (LWP 217265)): #0 0x00007efc6c5d2644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561aa1be709a in LinuxAIOHandler::collect (this=0x7efc35806b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561aa1be75c3 in LinuxAIOHandler::poll (this=0x7efc35806b00, m1=0x7efc35806ba8, m2=0x7efc35806bb0, request=0x7efc35806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561aa1be7726 in os_aio_linux_handler (global_segment=0, m1=0x7efc35806ba8, m2=0x7efc35806bb0, request=0x7efc35806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561aa1bec24a in os_aio_handler (segment=0, m1=0x7efc35806ba8, m2=0x7efc35806bb0, request=0x7efc35806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561aa1e3f4a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000561aa1ce5a47 in io_handler_thread (arg=0x561aa2e80160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7efc12ffd700 (LWP 217298)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3d7d in os_event::wait (this=0x561aa4e3e7c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561aa1bf3803 in os_event::wait_low (this=0x561aa4e3e7c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561aa1bf3b74 in os_event_wait_low (event=0x561aa4e3e7c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561aa1da8cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7efc55b7a700 (LWP 216504)): #0 0x00007efc6adc0483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007efc648c3a68 in asio::detail::epoll_reactor::run (this=0x561aa4d2e060, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007efc648c8160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x561aa4d2df00, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007efc648c8412 in asio::detail::task_io_service::run (this=0x561aa4d2df00, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007efc648dfe6d in asio::io_service::run (this=0x561aa4d2cf40) at asio/asio/impl/io_service.ipp:58 #5 0x00007efc648e05c1 in gcomm::AsioProtonet::event_loop (this=0x561aa4d2cea0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007efc648fe65d in GCommConn::run (this=0x561aa4d2d690) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007efc64901709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7efc137fe700 (LWP 217297)): #0 0x00007efc6ca08e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000561aa1bf407f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000561aa1d9ab55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7efc54145700 (LWP 217301)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa20b2b7f in safe_cond_wait (cond=0x561aa2e044e0 , mp=0x561aa2e07800 , file=0x561aa217e488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561aa1491fe2 in inline_mysql_cond_wait (that=0x561aa2e044e0 , mutex=0x561aa2e07800 , src_file=0x561aa217eff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000561aa1493874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000561aa2044fb3 in pfs_spawn_thread (arg=0x561aa5141840) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7efc13fff700 (LWP 217296)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3d7d in os_event::wait (this=0x561aa4e3e6c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561aa1bf3803 in os_event::wait_low (this=0x561aa4e3e6c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561aa1bf3b74 in os_event_wait_low (event=0x561aa4e3e6c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561aa1dc019b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7efc127fc700 (LWP 217299)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa20b2b7f in safe_cond_wait (cond=0x7efc127fbc70, mp=0x7efc127fbcb0, file=0x561aa2389920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561aa1b09cad in inline_mysql_cond_wait (that=0x7efc127fbc70, mutex=0x7efc127fbcb0, src_file=0x561aa238b348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000561aa1b0e79b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000561aa2044fb3 in pfs_spawn_thread (arg=0x561aa4ff93a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7efc197fa700 (LWP 217294)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3d7d in os_event::wait (this=0x561aa4e3e440) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561aa1bf3803 in os_event::wait_low (this=0x561aa4e3e440, reset_sig_count=51) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561aa1bf3b74 in os_event_wait_low (event=0x561aa4e3e440, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561aa1cdff21 in srv_resume_thread (slot=0x561aa2c41980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000561aa1ce3b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7efc1a7fc700 (LWP 217292)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3d7d in os_event::wait (this=0x561aa4f720f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561aa1bf3803 in os_event::wait_low (this=0x561aa4f720f0, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561aa1bf3b74 in os_event_wait_low (event=0x561aa4f720f0, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561aa1cedd96 in sync_array_wait_event (arr=0x561aa4e15780, cell=@0x7efc1a7f9888: 0x561aa4e15970) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000561aa1cf0aab in rw_lock_s_lock_spin (lock=0x7efc35c434f8, pass=0, file_name=0x561aa2421870 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x0000561aa1d9be18 in rw_lock_s_lock_func (lock=0x7efc35c434f8, pass=0, file_name=0x561aa2421870 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x0000561aa1d9c85c in pfs_rw_lock_s_lock_func (lock=0x7efc35c434f8, pass=0, file_name=0x561aa2421870 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:651 #8 0x0000561aa1dae3f8 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=1, guess=0x0, mode=10, file=0x561aa2421870 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7efc1a7fb100, err=0x7efc1a7f9c20) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4866 #9 0x0000561aa1d6dd43 in btr_cur_search_to_nth_level_func (index=0x7efc38030768, level=0, tuple=0x7efc0c011588, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x561aa50635d8, ahi_latch=0x0, file=0x561aa2421870 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7efc1a7fb100, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1548 #10 0x0000561aa1ca94da in btr_pcur_open_low (index=0x7efc38030768, level=0, tuple=0x7efc0c011588, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x561aa50635d8, file=0x561aa2421870 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, autoinc=0, mtr=0x7efc1a7fb100) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:464 #11 0x0000561aa1cac684 in row_search_on_row_ref (pcur=0x561aa50635d8, mode=2, table=0x7efc3802f118, ref=0x7efc0c011588, mtr=0x7efc1a7fb100) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc:1219 #12 0x0000561aa1ca0b29 in row_purge_reposition_pcur (mode=2, node=0x561aa5063540, mtr=0x7efc1a7fb100) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:104 #13 0x0000561aa1ca0cbb in row_purge_remove_clust_if_poss_low (node=0x561aa5063540, mode=2) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:140 #14 0x0000561aa1ca10eb in row_purge_remove_clust_if_poss (node=0x561aa5063540) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:219 #15 0x0000561aa1ca293e in row_purge_del_mark (node=0x561aa5063540) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #16 0x0000561aa1ca3ec6 in row_purge_record_func (node=0x561aa5063540, undo_rec=0x561aa5064038 "", thr=0x561aa5063488, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #17 0x0000561aa1ca4203 in row_purge (node=0x561aa5063540, undo_rec=0x561aa5064038 "", thr=0x561aa5063488) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #18 0x0000561aa1ca45ae in row_purge_step (thr=0x561aa5063488) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #19 0x0000561aa1c2670e in que_thr_step (thr=0x561aa5063488) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #20 0x0000561aa1c2693e in que_run_threads_low (thr=0x561aa5063488) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #21 0x0000561aa1c26b2e in que_run_threads (thr=0x561aa5063488) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #22 0x0000561aa1d0dfaa in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1325 #23 0x0000561aa1ce3df8 in srv_do_purge (n_total_purged=0x7efc1a7fbd98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #24 0x0000561aa1ce4225 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #25 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #26 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7efc2cdf4700 (LWP 217288)): #0 0x00007efc6ca05d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3698 in os_event::timed_wait (this=0x561aa4e3e640, abstime=0x7efc2cdf3d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561aa1bf39af in os_event::wait_time_low (this=0x561aa4e3e640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561aa1bf3b4f in os_event_wait_time_low (event=0x561aa4e3e640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561aa1ce1c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7efc1affd700 (LWP 217291)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3d7d in os_event::wait (this=0x561aa5050a40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561aa1bf3803 in os_event::wait_low (this=0x561aa5050a40, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561aa1bf3b74 in os_event_wait_low (event=0x561aa5050a40, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561aa1cedd96 in sync_array_wait_event (arr=0x561aa4e15780, cell=@0x7efc1affcb98: 0x561aa4e158f0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000561aa1cf0c7c in rw_lock_x_lock_wait_func (lock=0x561aa504bf80, pass=0, threshold=0, file_name=0x561aa243c068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:459 #6 0x0000561aa1cf0dae in rw_lock_x_lock_low (lock=0x561aa504bf80, pass=0, file_name=0x561aa243c068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:514 #7 0x0000561aa1cf11cc in rw_lock_x_lock_func (lock=0x561aa504bf80, pass=0, file_name=0x561aa243c068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:673 #8 0x0000561aa1cdf571 in pfs_rw_lock_x_lock_func (lock=0x561aa504bf80, pass=0, file_name=0x561aa243c068 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:552 #9 0x0000561aa1ce25b4 in srv_master_evict_from_table_cache (pct_check=100) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1999 #10 0x0000561aa1ce30b0 in srv_master_do_idle_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2237 #11 0x0000561aa1ce3686 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2358 #12 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7efc2d5f5700 (LWP 217286)): #0 0x00007efc6ca05d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3698 in os_event::timed_wait (this=0x561aa5016af0, abstime=0x7efc2d5f4d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561aa1bf39af in os_event::wait_time_low (this=0x561aa5016af0, time_in_usec=1000000, reset_sig_count=12) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561aa1bf3b4f in os_event_wait_time_low (event=0x561aa5016af0, time_in_usec=1000000, reset_sig_count=12) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561aa1bc04f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7efc1bfff700 (LWP 217289)): #0 0x00007efc6ca05d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3698 in os_event::timed_wait (this=0x561aa5048bb0, abstime=0x7efc1bffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561aa1bf39af in os_event::wait_time_low (this=0x561aa5048bb0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561aa1bf3b4f in os_event_wait_time_low (event=0x561aa5048bb0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561aa1e290e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7efc2f7fe700 (LWP 217269)): #0 0x00007efc6c5d2644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561aa1be709a in LinuxAIOHandler::collect (this=0x7efc2f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561aa1be75c3 in LinuxAIOHandler::poll (this=0x7efc2f7fdb00, m1=0x7efc2f7fdba8, m2=0x7efc2f7fdbb0, request=0x7efc2f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561aa1be7726 in os_aio_linux_handler (global_segment=4, m1=0x7efc2f7fdba8, m2=0x7efc2f7fdbb0, request=0x7efc2f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561aa1bec24a in os_aio_handler (segment=4, m1=0x7efc2f7fdba8, m2=0x7efc2f7fdbb0, request=0x7efc2f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561aa1e3f4a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000561aa1ce5a47 in io_handler_thread (arg=0x561aa2e80180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7efc2effd700 (LWP 217270)): #0 0x00007efc6c5d2644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561aa1be709a in LinuxAIOHandler::collect (this=0x7efc2effcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561aa1be75c3 in LinuxAIOHandler::poll (this=0x7efc2effcb00, m1=0x7efc2effcba8, m2=0x7efc2effcbb0, request=0x7efc2effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561aa1be7726 in os_aio_linux_handler (global_segment=5, m1=0x7efc2effcba8, m2=0x7efc2effcbb0, request=0x7efc2effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561aa1bec24a in os_aio_handler (segment=5, m1=0x7efc2effcba8, m2=0x7efc2effcbb0, request=0x7efc2effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561aa1e3f4a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000561aa1ce5a47 in io_handler_thread (arg=0x561aa2e80188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7efc2ffff700 (LWP 217268)): #0 0x00007efc6c5d2644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561aa1be709a in LinuxAIOHandler::collect (this=0x7efc2fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561aa1be75c3 in LinuxAIOHandler::poll (this=0x7efc2fffeb00, m1=0x7efc2fffeba8, m2=0x7efc2fffebb0, request=0x7efc2fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561aa1be7726 in os_aio_linux_handler (global_segment=3, m1=0x7efc2fffeba8, m2=0x7efc2fffebb0, request=0x7efc2fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561aa1bec24a in os_aio_handler (segment=3, m1=0x7efc2fffeba8, m2=0x7efc2fffebb0, request=0x7efc2fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561aa1e3f4a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000561aa1ce5a47 in io_handler_thread (arg=0x561aa2e80178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7efc19ffb700 (LWP 217293)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa1bf3d7d in os_event::wait (this=0x561aa4e3e4c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561aa1bf3803 in os_event::wait_low (this=0x561aa4e3e4c0, reset_sig_count=43) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561aa1bf3b74 in os_event_wait_low (event=0x561aa4e3e4c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561aa1cdff21 in srv_resume_thread (slot=0x561aa2c419b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000561aa1ce3b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7efc34805700 (LWP 217267)): #0 0x00007efc6c5d2644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561aa1be709a in LinuxAIOHandler::collect (this=0x7efc34804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561aa1be75c3 in LinuxAIOHandler::poll (this=0x7efc34804b00, m1=0x7efc34804ba8, m2=0x7efc34804bb0, request=0x7efc34804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561aa1be7726 in os_aio_linux_handler (global_segment=2, m1=0x7efc34804ba8, m2=0x7efc34804bb0, request=0x7efc34804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561aa1bec24a in os_aio_handler (segment=2, m1=0x7efc34804ba8, m2=0x7efc34804bb0, request=0x7efc34804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561aa1e3f4a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000561aa1ce5a47 in io_handler_thread (arg=0x561aa2e80170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7efc64266700 (LWP 216550)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007efc64816272 in fifo_lock_get (q=q@entry=0x7efc643fe010) at galerautils/src/gu_fifo.c:262 #2 0x00007efc648168e1 in gu_fifo_get_head (q=0x7efc643fe010, err=0x7efc6426506c) at galerautils/src/gu_fifo.c:330 #3 0x00007efc648f843c in gcs_recv (conn=0x561aa4cfd610, action=0x7efc642650b0) at gcs/src/gcs.cpp:2024 #4 0x00007efc6492550c in galera::GcsActionSource::process (this=0x561aa4d25f20, recv_ctx=0x7efc642654d0, exit_loop=@0x7efc6426529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007efc64944433 in galera::ReplicatorSMM::async_recv (this=0x561aa4cf53c0, recv_ctx=0x7efc642654d0) at galera/src/replicator_smm.cpp:383 #6 0x0000561aa21678a8 in wsrep::wsrep_provider_v26::run_applier (this=0x561aa4c6b2e0, applier_ctx=0x7efc642654d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000561aa1800a69 in wsrep_replication_process (thd=0x7efc44000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000561aa17f2587 in start_wsrep_THD (arg=0x561aa4d33d50) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7efc2ddf6700 (LWP 217287)): #0 0x00007efc6ca08e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000561aa1bf407f in os_thread_sleep (tm=30000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000561aa1cef202 in sync_array_print_long_waits (waiter=0x7efc2ddf5c28, sema=0x7efc2ddf5c30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:1099 #3 0x0000561aa1ce1fb6 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1801 #4 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7efc35006700 (LWP 217266)): #0 0x00007efc6c5d2644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561aa1be709a in LinuxAIOHandler::collect (this=0x7efc35005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561aa1be75c3 in LinuxAIOHandler::poll (this=0x7efc35005b00, m1=0x7efc35005ba8, m2=0x7efc35005bb0, request=0x7efc35005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561aa1be7726 in os_aio_linux_handler (global_segment=1, m1=0x7efc35005ba8, m2=0x7efc35005bb0, request=0x7efc35005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561aa1bec24a in os_aio_handler (segment=1, m1=0x7efc35005ba8, m2=0x7efc35005bb0, request=0x7efc35005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561aa1e3f4a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000561aa1ce5a47 in io_handler_thread (arg=0x561aa2e80168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7efc642b1700 (LWP 216549)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa20b2b7f in safe_cond_wait (cond=0x7efc40015938, mp=0x7efc40015888, file=0x561aa22214c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561aa18004ba in inline_mysql_cond_wait (that=0x7efc40015938, mutex=0x7efc40015888, src_file=0x561aa22217e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000561aa180334e in Wsrep_thd_queue::pop_front (this=0x7efc40015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000561aa18020d0 in wsrep_post_rollback_process (post_rollbacker=0x7efc40000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x0000561aa17f2587 in start_wsrep_THD (arg=0x561aa4d4e360) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7efc37059700 (LWP 217264)): #0 0x00007efc6ca05d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa20b2e81 in safe_cond_timedwait (cond=0x561aa3691a00 , mp=0x561aa3691940 , abstime=0x7efc37058cb0, file=0x561aa2500630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000561aa1f1c309 in inline_mysql_cond_timedwait (that=0x561aa3691a00 , mutex=0x561aa3691940 , abstime=0x7efc37058cb0, src_file=0x561aa2500678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000561aa1f1c82d in my_service_thread_sleep (control=0x561aa2c45d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000561aa1f10cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000561aa2044fb3 in pfs_spawn_thread (arg=0x561aa4df6680) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7efc57ffe700 (LWP 216402)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007efc64922377 in gu::Lock::wait (cond=..., this=0x7efc57ffdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007efc64922409 in galera::ServiceThd::thd_func (arg=0x561aa4cf5ad8) at galera/src/galera_service_thd.cpp:30 #3 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7efc642fc700 (LWP 216547)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa20b2b7f in safe_cond_wait (cond=0x7efc4c016b88, mp=0x7efc4c016ad8, file=0x561aa22214c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561aa18004ba in inline_mysql_cond_wait (that=0x7efc4c016b88, mutex=0x7efc4c016ad8, src_file=0x561aa22217e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000561aa180334e in Wsrep_thd_queue::pop_front (this=0x7efc4c016a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000561aa1801c54 in wsrep_rollback_process (rollbacker=0x7efc4c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x0000561aa17f2587 in start_wsrep_THD (arg=0x561aa4d4e060) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7efc6545a700 (LWP 216397)): #0 0x00007efc6ca05d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561aa20b2e81 in safe_cond_timedwait (cond=0x561aa36a41c0 , mp=0x561aa36a4100 , abstime=0x7efc65459d70, file=0x561aa2544e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000561aa20b3e3b in inline_mysql_cond_timedwait (that=0x561aa36a41c0 , mutex=0x561aa36a4100 , abstime=0x7efc65459d70, src_file=0x561aa2544ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000561aa20b499e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000561aa2044fb3 in pfs_spawn_thread (arg=0x561aa4cbee70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7efc55379700 (LWP 216546)): #0 0x00007efc6ca05965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007efc648fcbd7 in gu::Lock::wait (cond=..., this=0x7efc55378850) at galerautils/src/gu_lock.hpp:45 #2 0x00007efc648ff71b in RecvBuf::front (this=this@entry=0x561aa4d2d878, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007efc648fd2b6 in gcomm_recv (backend=, msg=0x561aa4cfd8f0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007efc648f125c in core_msg_recv (backend=backend@entry=0x561aa4cfda38, recv_msg=recv_msg@entry=0x561aa4cfd8f0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007efc648f44d4 in gcs_core_recv (conn=0x561aa4cfd890, recv_act=recv_act@entry=0x7efc55378d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007efc648fb748 in gcs_recv_thread (arg=0x561aa4cfd610) at gcs/src/gcs.cpp:1398 #7 0x00007efc6ca01dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007efc6adbfead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7efc6ce12880 (LWP 216309)): #0 0x00007efc6ca069d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000561aa20ad550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000561aa18e2ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007efc6adb520d in poll () from /lib64/libc.so.6 #5 0x0000561aa146cf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000561aa146c1b4 in mysqld_main (argc=165, argv=0x561aa4b77140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000561aa1460695 in main (argc=25, argv=0x7ffc724f0278) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 worker[2] Trying to dump core for [mysqltest - pid: 219017, winpid: 219017] worker[2] Trying to dump core for [mysqld.1 - pid: 218180, winpid: 218180] worker[2] Trying to dump core for [mysqld.2 - pid: 218217, winpid: 218217] worker[1] Trying to dump core for [mysqltest - pid: 219040, winpid: 219040] worker[1] Trying to dump core for [mysqld.1 - pid: 218341, winpid: 218341] worker[1] Trying to dump core for [mysqld.2 - pid: 218479, winpid: 218479] Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.2/data/core.216309 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_large_fragment-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_large_fragment-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_large_fragment 2019-02-27 16:46:51 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:46:51 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:47:22 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:47:22 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:47:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:47:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:47:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:47:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:47:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:48:24 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:48:24 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:48:55 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:48:55 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:26 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:26 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:57 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:57 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:50:28 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:50:28 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:50:59 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:50:59 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:51:30 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:51:30 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:01 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:01 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:32 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:32 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:32 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:32 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:53:03 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:53:03 0 [Warning] InnoDB: A long semaphore wait: Attempting backtrace. You can use the following information to find out 2019-02-27 16:47:49 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:48:20 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:48:51 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:22 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:49:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:50:24 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:50:55 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:51:26 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:51:57 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:28 0 [Warning] InnoDB: A long semaphore wait: 2019-02-27 16:52:59 0 [Warning] InnoDB: A long semaphore wait: Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_parallel_apply 'innodb' w3 [ pass ] 10275 galera_sr.galera_sr_rollback 'innodb' w3 [ pass ] 12898 galera_sr.galera_sr_rollback_retry 'innodb' w3 [ pass ] 15414 galera_sr.galera_sr_rollback_savepoint 'innodb' w3 [ pass ] 11814 galera_sr.galera_sr_rollback_statement 'innodb' w3 [ pass ] 8403 galera_sr.galera_sr_shutdown_master 'innodb' w3 [ pass ] 24708 galera_sr.galera-features#56 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 16:56:42 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera-features#56.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); CREATE TABLE t1 (f1 INTEGER AUTO_INCREMENT PRIMARY KEY, f2 INTEGER) Engine=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; connection node_2; SET GLOBAL wsrep_slave_threads = 4; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1a; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_2; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.218181' (2/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera-features#56-innodb/mysqld.1/data/core.218181 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 218181] [New LWP 218187] [New LWP 218182] [New LWP 218190] [New LWP 218186] [New LWP 218183] [New LWP 218191] [New LWP 218193] [New LWP 218189] [New LWP 218192] [New LWP 218198] [New LWP 218195] [New LWP 218199] [New LWP 218201] [New LWP 218202] [New LWP 218203] [New LWP 218210] [New LWP 218208] [New LWP 218810] [New LWP 218211] [New LWP 218811] [New LWP 218206] [New LWP 218212] [New LWP 218205] [New LWP 218185] [New LWP 218215] [New LWP 218207] [New LWP 218204] [New LWP 218184] [New LWP 218821] [New LWP 218209] [New LWP 218200] [New LWP 218194] [New LWP 218196] [New LWP 218188] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f94821459d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f94821459d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005585f579e550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005585f4fd3ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f94804f420d in poll () from /lib64/libc.so.6 #5 0x00005585f4b5df19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005585f4b5d1b4 in mysqld_main (argc=162, argv=0x5585f7f9d140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005585f4b51695 in main (argc=23, argv=0x7fff55996978) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f9471b30700 (LWP 218188)): #0 0x00007f9482147d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005585f52d9749 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005585f52d9a9f in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005585f55221fa in pfs_os_file_flush_func (file=..., src_file=0x5585f5ba7588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005585f5525055 in fil_flush_low (space=0x5585f846d9e0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005585f55309f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x00005585f52b4e31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005585f52b5893 in log_write_up_to (lsn=4578809, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005585f5420f5f in trx_flush_log_if_needed_low (lsn=4578809) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005585f5420fa8 in trx_flush_log_if_needed (lsn=4578809, trx=0x7f947037c140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005585f5422dd1 in trx_commit_complete_for_mysql (trx=0x7f947037c140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005585f5205ca6 in innobase_commit (hton=0x5585f8237d00, thd=0x7f9460000b00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4658 #12 0x00005585f4fd9a1d in commit_one_phase_2 (thd=0x7f9460000b00, all=true, trans=0x7f9460004078, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1741 #13 0x00005585f4fd991f in ha_commit_one_phase (thd=0x7f9460000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1721 #14 0x00005585f4fd8c98 in ha_commit_trans (thd=0x7f9460000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1534 #15 0x00005585f4e16ac6 in trans_commit (thd=0x7f9460000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005585f4ed11bc in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7f9471b2f4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:264 #17 0x00005585f5840e75 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:68 #18 0x00005585f58418e3 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:243 #19 0x00005585f5844d07 in wsrep::server_state::on_apply (this=0x5585f80e5cb0, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:966 #20 0x00005585f585a503 in wsrep::high_priority_service::apply (this=0x7f9471b2f4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x00005585f5857bea in (anonymous namespace)::apply_cb (ctx=0x7f9471b2f4d0, wsh=0x7f9471b2e3e0, flags=0, buf=0x7f9471b2e420, meta=0x7f9471b2e6b0, exit_loop=0x7f9471b2e5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007f947a04a721 in galera::TrxHandleSlave::apply (this=this@entry=0x7f94240451a0, recv_ctx=recv_ctx@entry=0x7f9471b2f4d0, apply_cb=0x5585f58579c7 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f9471b2e5ef: false) at galera/src/trx_handle.cpp:418 #23 0x00007f947a07d192 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5585f811b420, recv_ctx=recv_ctx@entry=0x7f9471b2f4d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007f947a082e40 in galera::ReplicatorSMM::process_trx (this=0x5585f811b420, recv_ctx=0x7f9471b2f4d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2114 #25 0x00007f947a06400b in galera::GcsActionSource::process_writeset (this=this@entry=0x5585f81230f0, recv_ctx=0x7f9471b2f4d0, act=..., exit_loop=@0x7f9471b2f29e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007f947a0641e5 in galera::GcsActionSource::dispatch (this=this@entry=0x5585f81230f0, recv_ctx=recv_ctx@entry=0x7f9471b2f4d0, act=..., exit_loop=@0x7f9471b2f29e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007f947a0645bd in galera::GcsActionSource::process (this=0x5585f81230f0, recv_ctx=0x7f9471b2f4d0, exit_loop=@0x7f9471b2f29e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007f947a083433 in galera::ReplicatorSMM::async_recv (this=0x5585f811b420, recv_ctx=0x7f9471b2f4d0) at galera/src/replicator_smm.cpp:383 #29 0x00005585f58588a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5585f80a2760, applier_ctx=0x7f9471b2f4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x00005585f4ef1a69 in wsrep_replication_process (thd=0x7f9460000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x00005585f4ee3587 in start_wsrep_THD (arg=0x5585f8174460) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #32 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f94505a9700 (LWP 218196)): #0 0x00007f9482147d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005585f52d9749 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005585f52d9a9f in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005585f55221fa in pfs_os_file_flush_func (file=..., src_file=0x5585f5ba7588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005585f5525055 in fil_flush_low (space=0x5585f846d640, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005585f55309f8 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x00005585f54adf60 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1095 #7 0x00005585f54b8886 in buf_flush_end (buf_pool=0x5585f8349330, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1924 #8 0x00005585f54b8a44 in buf_flush_do_batch (buf_pool=0x5585f8349330, type=BUF_FLUSH_LIST, min_n=4, lsn_limit=18446744073709551615, n=0x7f94505a8b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1993 #9 0x00005585f54bb72e in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2794 #10 0x00005585f54bc67c in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3203 #11 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f94515ab700 (LWP 218194)): #0 0x00007f9481d11644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005585f52d809a in LinuxAIOHandler::collect (this=0x7f94515aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005585f52d85c3 in LinuxAIOHandler::poll (this=0x7f94515aab00, m1=0x7f94515aaba8, m2=0x7f94515aabb0, request=0x7f94515aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005585f52d8726 in os_aio_linux_handler (global_segment=4, m1=0x7f94515aaba8, m2=0x7f94515aabb0, request=0x7f94515aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005585f52dd24a in os_aio_handler (segment=4, m1=0x7f94515aaba8, m2=0x7f94515aabb0, request=0x7f94515aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005585f55304a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005585f53d6a47 in io_handler_thread (arg=0x5585f6571180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f944e3a0700 (LWP 218200)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4698 in os_event::timed_wait (this=0x5585f8263ab0, abstime=0x7f944e39fd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005585f52e49af in os_event::wait_time_low (this=0x5585f8263ab0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005585f52e4b4f in os_event_wait_time_low (event=0x5585f8263ab0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005585f53d2c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f943dffb700 (LWP 218209)): #0 0x00007f9482147e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005585f52e507f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005585f548bb55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f943c2b2700 (LWP 218821)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f947a079417 in gu::Lock::wait (cond=..., this=0x7f943c2ae450) at galerautils/src/gu_lock.hpp:45 #2 0x00007f947a08a06c in galera::Monitor::enter (this=this@entry=0x5585f811c480, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f947a07eb17 in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x5585f811b420, trx=...) at galera/src/replicator_smm.cpp:1280 #4 0x00007f947a09d4c3 in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:658 #5 0x00005585f5858e0b in wsrep::wsrep_provider_v26::commit_order_enter (this=0x5585f80a2760, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:769 #6 0x00005585f584c09e in wsrep::transaction::before_commit (this=0x7f947404e2f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:435 #7 0x00005585f4ed385e in wsrep::client_state::before_commit (this=0x7f947404e280) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:401 #8 0x00005585f51e2604 in wsrep_before_commit (thd=0x7f9474048200, all=true) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:203 #9 0x00005585f51e3881 in Wsrep_storage_service::commit (this=0x7f9474067fa0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:171 #10 0x00005585f584eba1 in wsrep::transaction::certify_fragment (this=0x7f9474009260, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #11 0x00005585f584e37b in wsrep::transaction::streaming_step (this=0x7f9474009260, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #12 0x00005585f584b59a in wsrep::transaction::after_row (this=0x7f9474009260) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #13 0x00005585f4fed0b6 in wsrep::client_state::after_row (this=0x7f94740091f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #14 0x00005585f4fd5000 in wsrep_after_row (thd=0x7f9474003170) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #15 0x00005585f4fe70df in wsrep_after_row (thd=0x7f9474003170) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #16 0x00005585f4fe7e15 in handler::ha_write_row (this=0x7f94740347b8, buf=0x7f9474009910 "\375\365\023") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6673 #17 0x00005585f4c48acf in write_record (thd=0x7f9474003170, table=0x7f947400b380, info=0x7f9474019be0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #18 0x00005585f4c4ddce in select_insert::send_data (this=0x7f9474019b98, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #19 0x00005585f4d0bdcf in end_send (join=0x7f9474019c48, join_tab=0x7f9474023a58, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:21196 #20 0x00005585f4e50eff in JOIN_CACHE::generate_full_extensions (this=0x7f94740242f0, rec_ptr=0x7f9474026db8 "\254") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #21 0x00005585f4e50c1a in JOIN_CACHE::join_matching_records (this=0x7f94740242f0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #22 0x00005585f4e505e6 in JOIN_CACHE::join_records (this=0x7f94740242f0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #23 0x00005585f4e50780 in JOIN_CACHE::join_records (this=0x7f94740241c0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #24 0x00005585f4e50780 in JOIN_CACHE::join_records (this=0x7f947401c6b8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #25 0x00005585f4d08a6d in sub_select_cache (join=0x7f9474019c48, join_tab=0x7f9474022f00, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19786 #26 0x00005585f4d08c7a in sub_select (join=0x7f9474019c48, join_tab=0x7f9474022b38, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19957 #27 0x00005585f4d08445 in do_select (join=0x7f9474019c48, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19548 #28 0x00005585f4cdf079 in JOIN::exec_inner (this=0x7f9474019c48) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4364 #29 0x00005585f4cde226 in JOIN::exec (this=0x7f9474019c48) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4146 #30 0x00005585f4cdf90c in mysql_select (thd=0x7f9474003170, tables=0x7f9474017678, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f9474019b98, unit=0x7f9474007088, select_lex=0x7f9474017128) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4578 #31 0x00005585f4ccfea6 in handle_select (thd=0x7f9474003170, lex=0x7f9474006fc0, result=0x7f9474019b98, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:424 #32 0x00005585f4c91b44 in mysql_execute_command (thd=0x7f9474003170) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4896 #33 0x00005585f4c9d0f1 in mysql_parse (thd=0x7f9474003170, rawbuf=0x7f9474016808 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f943c2b1040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #34 0x00005585f4c9c720 in wsrep_mysql_parse (thd=0x7f9474003170, rawbuf=0x7f9474016808 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f943c2b1040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #35 0x00005585f4c8846d in dispatch_command (command=COM_QUERY, thd=0x7f9474003170, packet=0x7f947400cdd1 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #36 0x00005585f4c86cbb in do_command (thd=0x7f9474003170) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #37 0x00005585f4e01c9e in do_handle_one_connection (connect=0x5585f84f79a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #38 0x00005585f4e019e2 in handle_one_connection (arg=0x5585f84f79a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #39 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f859e0d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #40 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #41 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f947885a700 (LWP 218184)): #0 0x00007f94804ff483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f947a002a68 in asio::detail::epoll_reactor::run (this=0x5585f815c200, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f947a007160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5585f815c0a0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f947a007412 in asio::detail::task_io_service::run (this=0x5585f815c0a0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f947a01ee6d in asio::io_service::run (this=0x5585f815b1a0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f947a01f5c1 in gcomm::AsioProtonet::event_loop (this=0x5585f815b100, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f947a03d65d in GCommConn::run (this=0x5585f815b880) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f947a040709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f944cb9d700 (LWP 218204)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4d7d in os_event::wait (this=0x5585f8263830) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005585f52e4803 in os_event::wait_low (this=0x5585f8263830, reset_sig_count=3143) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005585f52e4b74 in os_event_wait_low (event=0x5585f8263830, reset_sig_count=3143) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005585f53d0f21 in srv_resume_thread (slot=0x5585f6332948 , sig_count=3143, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005585f53d4f32 in srv_purge_coordinator_suspend (slot=0x5585f6332948 , rseg_history_len=34) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005585f53d51c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f943effd700 (LWP 218207)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4d7d in os_event::wait (this=0x5585f82639b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005585f52e4803 in os_event::wait_low (this=0x5585f82639b0, reset_sig_count=11845) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005585f52e4b74 in os_event_wait_low (event=0x5585f82639b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005585f53d0f21 in srv_resume_thread (slot=0x5585f63329f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005585f53d4b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f947012f700 (LWP 218215)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f57a3b7f in safe_cond_wait (cond=0x5585f64f54e0 , mp=0x5585f64f8800 , file=0x5585f586f488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005585f4b82fe2 in inline_mysql_cond_wait (that=0x5585f64f54e0 , mutex=0x5585f64f8800 , src_file=0x5585f586fff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005585f4b84874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f8565d60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f947237c700 (LWP 218185)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f947a03bbd7 in gu::Lock::wait (cond=..., this=0x7f947237b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f947a03e71b in RecvBuf::front (this=this@entry=0x5585f815ba68, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f947a03c2b6 in gcomm_recv (backend=, msg=0x5585f81239f0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f947a03025c in core_msg_recv (backend=backend@entry=0x5585f8123b38, recv_msg=recv_msg@entry=0x5585f81239f0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f947a0334d4 in gcs_core_recv (conn=0x5585f8123990, recv_act=recv_act@entry=0x7f947237bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f947a03a748 in gcs_recv_thread (arg=0x5585f8123710) at gcs/src/gcs.cpp:1398 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f943ffff700 (LWP 218205)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4d7d in os_event::wait (this=0x5585f8263930) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005585f52e4803 in os_event::wait_low (this=0x5585f8263930, reset_sig_count=12149) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005585f52e4b74 in os_event_wait_low (event=0x5585f8263930, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005585f53d0f21 in srv_resume_thread (slot=0x5585f63329b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005585f53d4b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f947017a700 (LWP 218212)): #0 0x00007f948043840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005585f4b51714 in my_sigwait (set=0x7f9470179cf0, sig=0x7f9470179cb4, code=0x7f9470179cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005585f4b57a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f8227040) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f943f7fe700 (LWP 218206)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4d7d in os_event::wait (this=0x5585f82638b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005585f52e4803 in os_event::wait_low (this=0x5585f82638b0, reset_sig_count=13367) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005585f52e4b74 in os_event_wait_low (event=0x5585f82638b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005585f53d0f21 in srv_resume_thread (slot=0x5585f6332980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005585f53d4b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f943c5a0700 (LWP 218811)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f57a3e81 in safe_cond_timedwait (cond=0x5585f64f54a0 , mp=0x5585f64f7e40 , abstime=0x7f943c59fc60, file=0x5585f585fb08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005585f4b52015 in inline_mysql_cond_timedwait (that=0x5585f64f54a0 , mutex=0x5585f64f7e40 , abstime=0x7f943c59fc60, src_file=0x5585f5861b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005585f4b56b08 in cache_thread (thd=0x7f9444000c70) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x00005585f4b56db6 in one_thread_per_connection_end (thd=0x7f9444000c70, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x00005585f4e01d28 in do_handle_one_connection (connect=0x5585f84f79a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005585f4e019e2 in handle_one_connection (arg=0x5585f84f79a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f85a60a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f943cff9700 (LWP 218211)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f57a3b7f in safe_cond_wait (cond=0x7f943cff8c70, mp=0x7f943cff8cb0, file=0x5585f5a7a920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005585f51facad in inline_mysql_cond_wait (that=0x7f943cff8c70, mutex=0x7f943cff8cb0, src_file=0x5585f5a7c348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005585f51ff79b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f84a4b30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f943c5eb700 (LWP 218810)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f947a079417 in gu::Lock::wait (cond=..., this=0x7f943c5e7450) at galerautils/src/gu_lock.hpp:45 #2 0x00007f947a08a06c in galera::Monitor::enter (this=this@entry=0x5585f811c480, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f947a07eb17 in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x5585f811b420, trx=...) at galera/src/replicator_smm.cpp:1280 #4 0x00007f947a09d4c3 in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:658 #5 0x00005585f5858e0b in wsrep::wsrep_provider_v26::commit_order_enter (this=0x5585f80a2760, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:769 #6 0x00005585f584c09e in wsrep::transaction::before_commit (this=0x7f943006f390) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:435 #7 0x00005585f4ed385e in wsrep::client_state::before_commit (this=0x7f943006f320) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:401 #8 0x00005585f51e2604 in wsrep_before_commit (thd=0x7f94300692a0, all=true) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:203 #9 0x00005585f51e3881 in Wsrep_storage_service::commit (this=0x7f943002dcb0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:171 #10 0x00005585f584eba1 in wsrep::transaction::certify_fragment (this=0x7f9430006e50, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #11 0x00005585f584e37b in wsrep::transaction::streaming_step (this=0x7f9430006e50, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #12 0x00005585f584b59a in wsrep::transaction::after_row (this=0x7f9430006e50) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #13 0x00005585f4fed0b6 in wsrep::client_state::after_row (this=0x7f9430006de0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #14 0x00005585f4fd5000 in wsrep_after_row (thd=0x7f9430000d60) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #15 0x00005585f4fe70df in wsrep_after_row (thd=0x7f9430000d60) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #16 0x00005585f4fe7e15 in handler::ha_write_row (this=0x7f9430058bf8, buf=0x7f94300587a0 "\375\343\033") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6673 #17 0x00005585f4c48acf in write_record (thd=0x7f9430000d60, table=0x7f9430057930, info=0x7f9430017700) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #18 0x00005585f4c4ddce in select_insert::send_data (this=0x7f94300176b8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #19 0x00005585f4d0bdcf in end_send (join=0x7f9430017768, join_tab=0x7f9430044b08, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:21196 #20 0x00005585f4e50eff in JOIN_CACHE::generate_full_extensions (this=0x7f9430045300, rec_ptr=0x7f9430047e44 "\210") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #21 0x00005585f4e50c1a in JOIN_CACHE::join_matching_records (this=0x7f9430045300, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #22 0x00005585f4e505e6 in JOIN_CACHE::join_records (this=0x7f9430045300, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #23 0x00005585f4e50780 in JOIN_CACHE::join_records (this=0x7f94300451d0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #24 0x00005585f4e50780 in JOIN_CACHE::join_records (this=0x7f94300450a0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #25 0x00005585f4d08a6d in sub_select_cache (join=0x7f9430017768, join_tab=0x7f9430043fb0, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19786 #26 0x00005585f4d08c7a in sub_select (join=0x7f9430017768, join_tab=0x7f9430043be8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19957 #27 0x00005585f4d08445 in do_select (join=0x7f9430017768, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19548 #28 0x00005585f4cdf079 in JOIN::exec_inner (this=0x7f9430017768) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4364 #29 0x00005585f4cde226 in JOIN::exec (this=0x7f9430017768) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4146 #30 0x00005585f4cdf90c in mysql_select (thd=0x7f9430000d60, tables=0x7f9430015218, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f94300176b8, unit=0x7f9430004c78, select_lex=0x7f9430014cc8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4578 #31 0x00005585f4ccfea6 in handle_select (thd=0x7f9430000d60, lex=0x7f9430004bb0, result=0x7f94300176b8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:424 #32 0x00005585f4c91b44 in mysql_execute_command (thd=0x7f9430000d60) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4896 #33 0x00005585f4c9d0f1 in mysql_parse (thd=0x7f9430000d60, rawbuf=0x7f94300143a8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f943c5ea040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #34 0x00005585f4c9c720 in wsrep_mysql_parse (thd=0x7f9430000d60, rawbuf=0x7f94300143a8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f943c5ea040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #35 0x00005585f4c8846d in dispatch_command (command=COM_QUERY, thd=0x7f9430000d60, packet=0x7f943000a971 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #36 0x00005585f4c86cbb in do_command (thd=0x7f9430000d60) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #37 0x00005585f4e01c9e in do_handle_one_connection (connect=0x5585f84f79a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #38 0x00005585f4e019e2 in handle_one_connection (arg=0x5585f84f79a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #39 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f85ad660) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #40 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #41 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f943e7fc700 (LWP 218208)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4d7d in os_event::wait (this=0x5585f8263b30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005585f52e4803 in os_event::wait_low (this=0x5585f8263b30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005585f52e4b74 in os_event_wait_low (event=0x5585f8263b30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005585f54b119b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f943d7fa700 (LWP 218210)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4d7d in os_event::wait (this=0x5585f8263c30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005585f52e4803 in os_event::wait_low (this=0x5585f8263c30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005585f52e4b74 in os_event_wait_low (event=0x5585f8263c30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005585f5499cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f944d39e700 (LWP 218203)): #0 0x00007f9482147e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005585f52e507f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005585f53d4489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005585f53d45b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f944fba3700 (LWP 218202)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4698 in os_event::timed_wait (this=0x5585f8403f30, abstime=0x7f944fba2b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005585f52e49af in os_event::wait_time_low (this=0x5585f8403f30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005585f52e4b4f in os_event_wait_time_low (event=0x5585f8403f30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005585f54334c9 in ib_wqueue_timedwait (wq=0x5585f8403e00, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005585f557473a in fts_optimize_thread (arg=0x5585f8403e00) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f944db9f700 (LWP 218201)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4698 in os_event::timed_wait (this=0x5585f846e020, abstime=0x7f944db9ed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005585f52e49af in os_event::wait_time_low (this=0x5585f846e020, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005585f52e4b4f in os_event_wait_time_low (event=0x5585f846e020, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005585f551a0e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f944eba1700 (LWP 218199)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4698 in os_event::timed_wait (this=0x5585f8263a30, abstime=0x7f944eba0ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005585f52e49af in os_event::wait_time_low (this=0x5585f8263a30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005585f52e4b4f in os_event_wait_time_low (event=0x5585f8263a30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005585f53d312a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f9450daa700 (LWP 218195)): #0 0x00007f9481d11644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005585f52d809a in LinuxAIOHandler::collect (this=0x7f9450da9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005585f52d85c3 in LinuxAIOHandler::poll (this=0x7f9450da9b00, m1=0x7f9450da9ba8, m2=0x7f9450da9bb0, request=0x7f9450da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005585f52d8726 in os_aio_linux_handler (global_segment=5, m1=0x7f9450da9ba8, m2=0x7f9450da9bb0, request=0x7f9450da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005585f52dd24a in os_aio_handler (segment=5, m1=0x7f9450da9ba8, m2=0x7f9450da9bb0, request=0x7f9450da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005585f55304a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005585f53d6a47 in io_handler_thread (arg=0x5585f6571188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f944f3a2700 (LWP 218198)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f52e4698 in os_event::timed_wait (this=0x5585f843bf60, abstime=0x7f944f3a1d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005585f52e49af in os_event::wait_time_low (this=0x5585f843bf60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005585f52e4b4f in os_event_wait_time_low (event=0x5585f843bf60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005585f52b14f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f94525ad700 (LWP 218192)): #0 0x00007f9481d11644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005585f52d809a in LinuxAIOHandler::collect (this=0x7f94525acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005585f52d85c3 in LinuxAIOHandler::poll (this=0x7f94525acb00, m1=0x7f94525acba8, m2=0x7f94525acbb0, request=0x7f94525acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005585f52d8726 in os_aio_linux_handler (global_segment=2, m1=0x7f94525acba8, m2=0x7f94525acbb0, request=0x7f94525acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005585f52dd24a in os_aio_handler (segment=2, m1=0x7f94525acba8, m2=0x7f94525acbb0, request=0x7f94525acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005585f55304a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005585f53d6a47 in io_handler_thread (arg=0x5585f6571170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f9470f7c700 (LWP 218189)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f57a3e81 in safe_cond_timedwait (cond=0x5585f6d82a00 , mp=0x5585f6d82940 , abstime=0x7f9470f7bcb0, file=0x5585f5bf1630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005585f560d309 in inline_mysql_cond_timedwait (that=0x5585f6d82a00 , mutex=0x5585f6d82940 , abstime=0x7f9470f7bcb0, src_file=0x5585f5bf1678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005585f560d82d in my_service_thread_sleep (control=0x5585f6336d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005585f5601cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f821b9d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f9451dac700 (LWP 218193)): #0 0x00007f9481d11644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005585f52d809a in LinuxAIOHandler::collect (this=0x7f9451dabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005585f52d85c3 in LinuxAIOHandler::poll (this=0x7f9451dabb00, m1=0x7f9451dabba8, m2=0x7f9451dabbb0, request=0x7f9451dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005585f52d8726 in os_aio_linux_handler (global_segment=3, m1=0x7f9451dabba8, m2=0x7f9451dabbb0, request=0x7f9451dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005585f52dd24a in os_aio_handler (segment=3, m1=0x7f9451dabba8, m2=0x7f9451dabbb0, request=0x7f9451dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005585f55304a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005585f53d6a47 in io_handler_thread (arg=0x5585f6571178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f9452dae700 (LWP 218191)): #0 0x00007f9481d11644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005585f52d809a in LinuxAIOHandler::collect (this=0x7f9452dadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005585f52d85c3 in LinuxAIOHandler::poll (this=0x7f9452dadb00, m1=0x7f9452dadba8, m2=0x7f9452dadbb0, request=0x7f9452dadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005585f52d8726 in os_aio_linux_handler (global_segment=1, m1=0x7f9452dadba8, m2=0x7f9452dadbb0, request=0x7f9452dadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005585f52dd24a in os_aio_handler (segment=1, m1=0x7f9452dadba8, m2=0x7f9452dadbb0, request=0x7f9452dadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005585f55304a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005585f53d6a47 in io_handler_thread (arg=0x5585f6571168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f947905b700 (LWP 218183)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f947a061377 in gu::Lock::wait (cond=..., this=0x7f947905abf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f947a061409 in galera::ServiceThd::thd_func (arg=0x5585f811bb38) at galera/src/galera_service_thd.cpp:30 #3 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f9478059700 (LWP 218186)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f57a3b7f in safe_cond_wait (cond=0x7f9468017e98, mp=0x7f9468017de8, file=0x5585f59124c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005585f4ef14ba in inline_mysql_cond_wait (that=0x7f9468017e98, mutex=0x7f9468017de8, src_file=0x5585f59127e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005585f4ef434e in Wsrep_thd_queue::pop_front (this=0x7f9468017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005585f4ef2c54 in wsrep_rollback_process (rollbacker=0x7f9468001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x00005585f4ee3587 in start_wsrep_THD (arg=0x5585f8175050) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f94535af700 (LWP 218190)): #0 0x00007f9481d11644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005585f52d809a in LinuxAIOHandler::collect (this=0x7f94535aeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005585f52d85c3 in LinuxAIOHandler::poll (this=0x7f94535aeb00, m1=0x7f94535aeba8, m2=0x7f94535aebb0, request=0x7f94535aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005585f52d8726 in os_aio_linux_handler (global_segment=0, m1=0x7f94535aeba8, m2=0x7f94535aebb0, request=0x7f94535aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005585f52dd24a in os_aio_handler (segment=0, m1=0x7f94535aeba8, m2=0x7f94535aebb0, request=0x7f94535aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005585f55304a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005585f53d6a47 in io_handler_thread (arg=0x5585f6571160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f947ab99700 (LWP 218182)): #0 0x00007f9482144d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f57a3e81 in safe_cond_timedwait (cond=0x5585f6d951c0 , mp=0x5585f6d95100 , abstime=0x7f947ab98d70, file=0x5585f5c35e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005585f57a4e3b in inline_mysql_cond_timedwait (that=0x5585f6d951c0 , mutex=0x5585f6d95100 , abstime=0x7f947ab98d70, src_file=0x5585f5c35ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005585f57a599e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005585f5735fb3 in pfs_spawn_thread (arg=0x5585f80e4d40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f9471b7b700 (LWP 218187)): #0 0x00007f9482144965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005585f57a3b7f in safe_cond_wait (cond=0x7f945c015938, mp=0x7f945c015888, file=0x5585f59124c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005585f4ef14ba in inline_mysql_cond_wait (that=0x7f945c015938, mutex=0x7f945c015888, src_file=0x5585f59127e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005585f4ef434e in Wsrep_thd_queue::pop_front (this=0x7f945c015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005585f4ef30d0 in wsrep_post_rollback_process (post_rollbacker=0x7f945c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x00005585f4ee3587 in start_wsrep_THD (arg=0x5585f816b4e0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f9482140dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f94804feead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f9482551880 (LWP 218181)): #0 0x00007f94821459d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005585f579e550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005585f4fd3ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f94804f420d in poll () from /lib64/libc.so.6 #5 0x00005585f4b5df19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005585f4b5d1b4 in mysqld_main (argc=162, argv=0x5585f7f9d140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005585f4b51695 in main (argc=23, argv=0x7fff55996978) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera-features#56-innodb/mysqld.1/data/core.218181 - found 'core.218219' (3/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera-features#56-innodb/mysqld.2/data/core.218219 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 218219] [New LWP 218221] [New LWP 218222] [New LWP 218430] [New LWP 218437] [New LWP 218442] [New LWP 218444] [New LWP 218452] [New LWP 218453] [New LWP 218457] [New LWP 218448] [New LWP 219043] [New LWP 218460] [New LWP 218456] [New LWP 218445] [New LWP 218443] [New LWP 218455] [New LWP 218454] [New LWP 218451] [New LWP 218450] [New LWP 218436] [New LWP 218435] [New LWP 218449] [New LWP 218447] [New LWP 218434] [New LWP 218226] [New LWP 219042] [New LWP 218225] [New LWP 218433] [New LWP 218446] [New LWP 218432] [New LWP 218223] [New LWP 218224] [New LWP 218227] [New LWP 219044] [New LWP 218438] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f76903809d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f76903809d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005579adcc4550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005579ad4f9ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f768e72f20d in poll () from /lib64/libc.so.6 #5 0x00005579ad083f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005579ad0831b4 in mysqld_main (argc=163, argv=0x5579b08a4140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005579ad077695 in main (argc=23, argv=0x7ffc589bffe8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 36 (Thread 0x7f76517fa700 (LWP 218438)): #0 0x00007f7690382d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005579ad7ff749 in os_file_fsync_posix (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005579ad7ffa9f in os_file_flush_func (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005579ada481fa in pfs_os_file_flush_func (file=..., src_file=0x5579ae0cd588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005579ada4b055 in fil_flush_low (space=0x5579b0d79310, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005579ada569f8 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x00005579ad9d3f60 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1095 #7 0x00005579ad9de886 in buf_flush_end (buf_pool=0x5579b0c55000, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1924 #8 0x00005579ad9dea44 in buf_flush_do_batch (buf_pool=0x5579b0c55000, type=BUF_FLUSH_LIST, min_n=4, lsn_limit=18446744073709551615, n=0x7f76517f9b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1993 #9 0x00005579ad9e172e in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2794 #10 0x00005579ad9e267c in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3203 #11 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f7671bbc700 (LWP 219044)): #0 0x00007f7690382d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005579ad7ff749 in os_file_fsync_posix (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005579ad7ffa9f in os_file_flush_func (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005579ada481fa in pfs_os_file_flush_func (file=..., src_file=0x5579ae0cd588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005579ada4b055 in fil_flush_low (space=0x5579b0d796b0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005579ada569f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x00005579ad7dae31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005579ad7db893 in log_write_up_to (lsn=3045528, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005579ad946f5f in trx_flush_log_if_needed_low (lsn=3045528) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005579ad946fa8 in trx_flush_log_if_needed (lsn=3045528, trx=0x7f767245f830) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005579ad948dd1 in trx_commit_complete_for_mysql (trx=0x7f767245f830) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005579ad72bca6 in innobase_commit (hton=0x5579b0b43c00, thd=0x7f7604000b00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4658 #12 0x00005579ad4ffa1d in commit_one_phase_2 (thd=0x7f7604000b00, all=true, trans=0x7f7604004078, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1741 #13 0x00005579ad4ff91f in ha_commit_one_phase (thd=0x7f7604000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1721 #14 0x00005579ad4fec98 in ha_commit_trans (thd=0x7f7604000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1534 #15 0x00005579ad33cac6 in trans_commit (thd=0x7f7604000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005579ad3f71bc in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7f7671bbb4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:264 #17 0x00005579add66e75 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:68 #18 0x00005579add678e3 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:243 #19 0x00005579add6ad07 in wsrep::server_state::on_apply (this=0x5579b09ecd50, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:966 #20 0x00005579add80503 in wsrep::high_priority_service::apply (this=0x7f7671bbb4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x00005579add7dbea in (anonymous namespace)::apply_cb (ctx=0x7f7671bbb4d0, wsh=0x7f7671bba3e0, flags=0, buf=0x7f7671bba420, meta=0x7f7671bba6b0, exit_loop=0x7f7671bba5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007f7683cb1721 in galera::TrxHandleSlave::apply (this=this@entry=0x7f765c44fd70, recv_ctx=recv_ctx@entry=0x7f7671bbb4d0, apply_cb=0x5579add7d9c7 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f7671bba5ef: false) at galera/src/trx_handle.cpp:418 #23 0x00007f7683ce4192 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5579b0a223b0, recv_ctx=recv_ctx@entry=0x7f7671bbb4d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007f7683ce9e40 in galera::ReplicatorSMM::process_trx (this=0x5579b0a223b0, recv_ctx=0x7f7671bbb4d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2114 #25 0x00007f7683ccb00b in galera::GcsActionSource::process_writeset (this=this@entry=0x5579b0a57020, recv_ctx=0x7f7671bbb4d0, act=..., exit_loop=@0x7f7671bbb29e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007f7683ccb1e5 in galera::GcsActionSource::dispatch (this=this@entry=0x5579b0a57020, recv_ctx=recv_ctx@entry=0x7f7671bbb4d0, act=..., exit_loop=@0x7f7671bbb29e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007f7683ccb5bd in galera::GcsActionSource::process (this=0x5579b0a57020, recv_ctx=0x7f7671bbb4d0, exit_loop=@0x7f7671bbb29e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007f7683cea433 in galera::ReplicatorSMM::async_recv (this=0x5579b0a223b0, recv_ctx=0x7f7671bbb4d0) at galera/src/replicator_smm.cpp:383 #29 0x00005579add7e8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5579b0998250, applier_ctx=0x7f7671bbb4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x00005579ad417a69 in wsrep_replication_process (thd=0x7f7604000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x00005579ad409587 in start_wsrep_THD (arg=0x7f7660165540) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #32 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f7688213700 (LWP 218227)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7683ce0417 in gu::Lock::wait (cond=..., this=0x7f76882113f0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f7683cf05cd in galera::Monitor::enter (this=this@entry=0x5579b0a23350, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f7683ce40ca in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5579b0a223b0, recv_ctx=recv_ctx@entry=0x7f76882124d0, ts=...) at galera/src/replicator_smm.cpp:469 #4 0x00007f7683ce9e40 in galera::ReplicatorSMM::process_trx (this=0x5579b0a223b0, recv_ctx=0x7f76882124d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2114 #5 0x00007f7683ccb00b in galera::GcsActionSource::process_writeset (this=this@entry=0x5579b0a57020, recv_ctx=0x7f76882124d0, act=..., exit_loop=@0x7f768821229e: false) at galera/src/gcs_action_source.cpp:62 #6 0x00007f7683ccb1e5 in galera::GcsActionSource::dispatch (this=this@entry=0x5579b0a57020, recv_ctx=recv_ctx@entry=0x7f76882124d0, act=..., exit_loop=@0x7f768821229e: false) at galera/src/gcs_action_source.cpp:109 #7 0x00007f7683ccb5bd in galera::GcsActionSource::process (this=0x5579b0a57020, recv_ctx=0x7f76882124d0, exit_loop=@0x7f768821229e: false) at galera/src/gcs_action_source.cpp:182 #8 0x00007f7683cea433 in galera::ReplicatorSMM::async_recv (this=0x5579b0a223b0, recv_ctx=0x7f76882124d0) at galera/src/replicator_smm.cpp:383 #9 0x00005579add7e8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5579b0998250, applier_ctx=0x7f76882124d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #10 0x00005579ad417a69 in wsrep_replication_process (thd=0x7f765c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #11 0x00005579ad409587 in start_wsrep_THD (arg=0x5579b0a80b40) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #12 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f7670d67700 (LWP 218224)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7683ca2bd7 in gu::Lock::wait (cond=..., this=0x7f7670d66850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f7683ca571b in RecvBuf::front (this=this@entry=0x5579b0a5e9d8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f7683ca32b6 in gcomm_recv (backend=, msg=0x5579b0a2a8e0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f7683c9725c in core_msg_recv (backend=backend@entry=0x5579b0a2aa28, recv_msg=recv_msg@entry=0x5579b0a2a8e0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f7683c9a4d4 in gcs_core_recv (conn=0x5579b0a2a880, recv_act=recv_act@entry=0x7f7670d66d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f7683ca1748 in gcs_recv_thread (arg=0x5579b0a2a600) at gcs/src/gcs.cpp:1398 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f7679568700 (LWP 218223)): #0 0x00007f768e73a483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f7683c69a68 in asio::detail::epoll_reactor::run (this=0x5579b0a5f1c0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f7683c6e160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5579b0a5f060, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f7683c6e412 in asio::detail::task_io_service::run (this=0x5579b0a5f060, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f7683c85e6d in asio::io_service::run (this=0x5579b0a5dd80) at asio/asio/impl/io_service.ipp:58 #5 0x00007f7683c865c1 in gcomm::AsioProtonet::event_loop (this=0x5579b0a5dce0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f7683ca465d in GCommConn::run (this=0x5579b0a5e7f0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f7683ca7709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f7671a07700 (LWP 218432)): #0 0x00007f768ff4c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005579ad7fe09a in LinuxAIOHandler::collect (this=0x7f7671a06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005579ad7fe5c3 in LinuxAIOHandler::poll (this=0x7f7671a06b00, m1=0x7f7671a06ba8, m2=0x7f7671a06bb0, request=0x7f7671a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005579ad7fe726 in os_aio_linux_handler (global_segment=0, m1=0x7f7671a06ba8, m2=0x7f7671a06bb0, request=0x7f7671a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579ad80324a in os_aio_handler (segment=0, m1=0x7f7671a06ba8, m2=0x7f7671a06bb0, request=0x7f7671a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579ada564a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005579ad8fca47 in io_handler_thread (arg=0x5579aea97160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f763e7fc700 (LWP 218446)): #0 0x00007f769037fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80a698 in os_event::timed_wait (this=0x5579b0d17620, abstime=0x7f763e7fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005579ad80a9af in os_event::wait_time_low (this=0x5579b0d17620, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005579ad80ab4f in os_event_wait_time_low (event=0x5579b0d17620, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005579ad9594c9 in ib_wqueue_timedwait (wq=0x5579b0d17570, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005579ada9a73a in fts_optimize_thread (arg=0x5579b0d17570) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f7653fff700 (LWP 218433)): #0 0x00007f768ff4c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005579ad7fe09a in LinuxAIOHandler::collect (this=0x7f7653ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005579ad7fe5c3 in LinuxAIOHandler::poll (this=0x7f7653ffeb00, m1=0x7f7653ffeba8, m2=0x7f7653ffebb0, request=0x7f7653ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005579ad7fe726 in os_aio_linux_handler (global_segment=1, m1=0x7f7653ffeba8, m2=0x7f7653ffebb0, request=0x7f7653ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579ad80324a in os_aio_handler (segment=1, m1=0x7f7653ffeba8, m2=0x7f7653ffebb0, request=0x7f7653ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579ada564a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005579ad8fca47 in io_handler_thread (arg=0x5579aea97168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f76882a9700 (LWP 218225)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579adcc9b7f in safe_cond_wait (cond=0x7f7668015938, mp=0x7f7668015888, file=0x5579ade384c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005579ad4174ba in inline_mysql_cond_wait (that=0x7f7668015938, mutex=0x7f7668015888, src_file=0x5579ade387e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005579ad41a34e in Wsrep_thd_queue::pop_front (this=0x7f7668015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005579ad418c54 in wsrep_rollback_process (rollbacker=0x7f7668000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x00005579ad409587 in start_wsrep_THD (arg=0x5579b0a80410) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f7678068700 (LWP 219042)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0c9ff30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0c9ff30, reset_sig_count=27) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0c9ff30, reset_sig_count=27) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad904d96 in sync_array_wait_event (arr=0x5579b0b469d0, cell=@0x7f7678062c48: 0x5579b0b46b00) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005579ad908362 in rw_lock_x_lock_func (lock=0x7f7671c40df8, pass=0, file_name=0x5579ae071e88 "/home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0undo.cc", line=1461) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:735 #6 0x00005579ad9b36a3 in pfs_rw_lock_x_lock_func (lock=0x7f7671c40df8, pass=0, file_name=0x5579ae071e88 "/home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0undo.cc", line=1461) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:558 #7 0x00005579ad9c5495 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x5579ae071e88 "/home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0undo.cc", line=1461, mtr=0x7f7678063080, err=0x7f7678063008) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4879 #8 0x00005579ad95465d in trx_undo_assign_low (trx=0x7f767245c4b8, rseg=0x5579b0d9e3c0, undo=0x7f767245d4d8, err=0x7f7678063008, mtr=0x7f7678063080) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0undo.cc:1461 #9 0x00005579ad935bb4 in trx_undo_report_row_operation (thr=0x7f760003dad8, index=0x7f765c82a6d8, clust_entry=0x7f760003f8f8, update=0x0, cmpl_info=0, rec=0x0, offsets=0x0, roll_ptr=0x7f7678063610) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0rec.cc:2115 #10 0x00005579ad989fe0 in btr_cur_ins_lock_and_undo (flags=0, cursor=0x7f7678063ac0, entry=0x7f760003f8f8, thr=0x7f760003dad8, mtr=0x7f7678063ee0, inherit=0x7f7678063689) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3272 #11 0x00005579ad98a9db in btr_cur_optimistic_insert (flags=0, cursor=0x7f7678063ac0, offsets=0x7f7678063a58, heap=0x7f7678063a50, entry=0x7f760003f8f8, rec=0x7f7678063a60, big_rec=0x7f7678063a48, n_ext=0, thr=0x7f760003dad8, mtr=0x7f7678063ee0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3483 #12 0x00005579ad874852 in row_ins_clust_index_entry_low (flags=0, mode=2, index=0x7f765c82a6d8, n_uniq=1, entry=0x7f760003f8f8, n_ext=0, thr=0x7f760003dad8, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:2756 #13 0x00005579ad876224 in row_ins_clust_index_entry (index=0x7f765c82a6d8, entry=0x7f760003f8f8, thr=0x7f760003dad8, n_ext=0, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3280 #14 0x00005579ad876752 in row_ins_index_entry (index=0x7f765c82a6d8, entry=0x7f760003f8f8, thr=0x7f760003dad8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3405 #15 0x00005579ad876e91 in row_ins_index_entry_step (node=0x7f760003d860, thr=0x7f760003dad8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3574 #16 0x00005579ad8772c2 in row_ins (node=0x7f760003d860, thr=0x7f760003dad8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3717 #17 0x00005579ad877ac1 in row_ins_step (thr=0x7f760003dad8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3962 #18 0x00005579ad89762c in row_insert_for_mysql (mysql_rec=0x7f760003c6f0 "\375\361\023", prebuilt=0x7f760003d308, ins_mode=ROW_INS_NORMAL) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:1460 #19 0x00005579ad733ba8 in ha_innobase::write_row (this=0x7f760003cb48, record=0x7f760003c6f0 "\375\361\023") at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:8086 #20 0x00005579ad50dcc5 in handler::ha_write_row (this=0x7f760003cb48, buf=0x7f760003c6f0 "\375\361\023") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6663 #21 0x00005579ad652ad8 in Rows_log_event::write_row (this=0x7f760008a040, rgi=0x7f7600038d50, overwrite=false) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:13474 #22 0x00005579ad652dfa in Write_rows_log_event::do_exec_row (this=0x7f760008a040, rgi=0x7f7600038d50) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:13684 #23 0x00005579ad64dee8 in Rows_log_event::do_apply_event (this=0x7f760008a040, rgi=0x7f7600038d50) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:11573 #24 0x00005579ad0c227f in Log_event::apply_event (this=0x7f760008a040, rgi=0x7f7600038d50) at /home/stepan/galera/git/10.4/server/sql/log_event.h:1482 #25 0x00005579ad416d9f in wsrep_apply_events (thd=0x7f7600022b30, rli=0x7f760002d920, events_buf=0x7f767bab0d80, buf_len=0) at /home/stepan/galera/git/10.4/server/sql/wsrep_applier.cc:206 #26 0x00005579ad3f8479 in Wsrep_applier_service::apply_write_set (this=0x7f7600029900, ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:492 #27 0x00005579add66e09 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:63 #28 0x00005579add678e3 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:243 #29 0x00005579add6ad07 in wsrep::server_state::on_apply (this=0x5579b09ecd50, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:966 #30 0x00005579add80503 in wsrep::high_priority_service::apply (this=0x7f76780674d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #31 0x00005579add7dbea in (anonymous namespace)::apply_cb (ctx=0x7f76780674d0, wsh=0x7f76780663e0, flags=0, buf=0x7f7678066420, meta=0x7f76780666b0, exit_loop=0x7f76780665ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #32 0x00007f7683cb1721 in galera::TrxHandleSlave::apply (this=this@entry=0x7f765c6a5ff0, recv_ctx=recv_ctx@entry=0x7f76780674d0, apply_cb=0x5579add7d9c7 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f76780665ef: false) at galera/src/trx_handle.cpp:418 #33 0x00007f7683ce4192 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5579b0a223b0, recv_ctx=recv_ctx@entry=0x7f76780674d0, ts=...) at galera/src/replicator_smm.cpp:489 #34 0x00007f7683ce9e40 in galera::ReplicatorSMM::process_trx (this=0x5579b0a223b0, recv_ctx=0x7f76780674d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2114 #35 0x00007f7683ccb00b in galera::GcsActionSource::process_writeset (this=this@entry=0x5579b0a57020, recv_ctx=0x7f76780674d0, act=..., exit_loop=@0x7f767806729e: false) at galera/src/gcs_action_source.cpp:62 #36 0x00007f7683ccb1e5 in galera::GcsActionSource::dispatch (this=this@entry=0x5579b0a57020, recv_ctx=recv_ctx@entry=0x7f76780674d0, act=..., exit_loop=@0x7f767806729e: false) at galera/src/gcs_action_source.cpp:109 #37 0x00007f7683ccb5bd in galera::GcsActionSource::process (this=0x5579b0a57020, recv_ctx=0x7f76780674d0, exit_loop=@0x7f767806729e: false) at galera/src/gcs_action_source.cpp:182 #38 0x00007f7683cea433 in galera::ReplicatorSMM::async_recv (this=0x5579b0a223b0, recv_ctx=0x7f76780674d0) at galera/src/replicator_smm.cpp:383 #39 0x00005579add7e8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5579b0998250, applier_ctx=0x7f76780674d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #40 0x00005579ad417a69 in wsrep_replication_process (thd=0x7f7600000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #41 0x00005579ad409587 in start_wsrep_THD (arg=0x7f7660095500) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #42 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #43 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f768825e700 (LWP 218226)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579adcc9b7f in safe_cond_wait (cond=0x7f7664017e98, mp=0x7f7664017de8, file=0x5579ade384c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005579ad4174ba in inline_mysql_cond_wait (that=0x7f7664017e98, mutex=0x7f7664017de8, src_file=0x5579ade387e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005579ad41a34e in Wsrep_thd_queue::pop_front (this=0x7f7664017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005579ad4190d0 in wsrep_post_rollback_process (post_rollbacker=0x7f7664001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x00005579ad409587 in start_wsrep_THD (arg=0x5579b0a80d50) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f76537fe700 (LWP 218434)): #0 0x00007f768ff4c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005579ad7fe09a in LinuxAIOHandler::collect (this=0x7f76537fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005579ad7fe5c3 in LinuxAIOHandler::poll (this=0x7f76537fdb00, m1=0x7f76537fdba8, m2=0x7f76537fdbb0, request=0x7f76537fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005579ad7fe726 in os_aio_linux_handler (global_segment=2, m1=0x7f76537fdba8, m2=0x7f76537fdbb0, request=0x7f76537fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579ad80324a in os_aio_handler (segment=2, m1=0x7f76537fdba8, m2=0x7f76537fdbb0, request=0x7f76537fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579ada564a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005579ad8fca47 in io_handler_thread (arg=0x5579aea97170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f763dffb700 (LWP 218447)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0d07c70) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0d07c70, reset_sig_count=4047) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0d07c70, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad7db3c3 in log_write_up_to (lsn=3045928, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x00005579ad7db9c3 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #6 0x00005579ad8f955e in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #7 0x00005579ad8f9b21 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #8 0x00005579ad8fa67f in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #9 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f763cff9700 (LWP 218449)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0b6f680) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0b6f680, reset_sig_count=11512) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0b6f680, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad8f6f21 in srv_resume_thread (slot=0x5579ae8589f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005579ad8fab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f7652ffd700 (LWP 218435)): #0 0x00007f768ff4c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005579ad7fe09a in LinuxAIOHandler::collect (this=0x7f7652ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005579ad7fe5c3 in LinuxAIOHandler::poll (this=0x7f7652ffcb00, m1=0x7f7652ffcba8, m2=0x7f7652ffcbb0, request=0x7f7652ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005579ad7fe726 in os_aio_linux_handler (global_segment=3, m1=0x7f7652ffcba8, m2=0x7f7652ffcbb0, request=0x7f7652ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579ad80324a in os_aio_handler (segment=3, m1=0x7f7652ffcba8, m2=0x7f7652ffcbb0, request=0x7f7652ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579ada564a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005579ad8fca47 in io_handler_thread (arg=0x5579aea97178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f76527fc700 (LWP 218436)): #0 0x00007f768ff4c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005579ad7fe09a in LinuxAIOHandler::collect (this=0x7f76527fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005579ad7fe5c3 in LinuxAIOHandler::poll (this=0x7f76527fbb00, m1=0x7f76527fbba8, m2=0x7f76527fbbb0, request=0x7f76527fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005579ad7fe726 in os_aio_linux_handler (global_segment=4, m1=0x7f76527fbba8, m2=0x7f76527fbbb0, request=0x7f76527fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579ad80324a in os_aio_handler (segment=4, m1=0x7f76527fbba8, m2=0x7f76527fbbb0, request=0x7f76527fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579ada564a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005579ad8fca47 in io_handler_thread (arg=0x5579aea97180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f7637fff700 (LWP 218450)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0b6f580) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0b6f580, reset_sig_count=13386) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0b6f580, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad8f6f21 in srv_resume_thread (slot=0x5579ae858980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005579ad8fab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f76377fe700 (LWP 218451)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0b6f600) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0b6f600, reset_sig_count=12303) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0b6f600, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad8f6f21 in srv_resume_thread (slot=0x5579ae8589b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005579ad8fab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f7635ffb700 (LWP 218454)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0b6f900) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0b6f900, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0b6f900, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad9bfcbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f76357fa700 (LWP 218455)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579adcc9b7f in safe_cond_wait (cond=0x7f76357f9c70, mp=0x7f76357f9cb0, file=0x5579adfa0920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005579ad720cad in inline_mysql_cond_wait (that=0x7f76357f9c70, mutex=0x7f76357f9cb0, src_file=0x5579adfa2348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005579ad72579b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005579adc5bfb3 in pfs_spawn_thread (arg=0x5579b0d2a420) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f7650ff9700 (LWP 218443)): #0 0x00007f769037fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80a698 in os_event::timed_wait (this=0x5579b0b6f700, abstime=0x7f7650ff8ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005579ad80a9af in os_event::wait_time_low (this=0x5579b0b6f700, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005579ad80ab4f in os_event_wait_time_low (event=0x5579b0b6f700, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005579ad8f912a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f763effd700 (LWP 218445)): #0 0x00007f769037fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80a698 in os_event::timed_wait (this=0x5579b0d79cf0, abstime=0x7f763effcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005579ad80a9af in os_event::wait_time_low (this=0x5579b0d79cf0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005579ad80ab4f in os_event_wait_time_low (event=0x5579b0d79cf0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005579ada400e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f768806c700 (LWP 218456)): #0 0x00007f768e67340a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005579ad077714 in my_sigwait (set=0x7f768806bcf0, sig=0x7f768806bcb4, code=0x7f768806bcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005579ad07da71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x00005579adc5bfb3 in pfs_spawn_thread (arg=0x5579b0b32db0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f76780b3700 (LWP 218460)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7683ce0417 in gu::Lock::wait (cond=..., this=0x7f76780af450) at galerautils/src/gu_lock.hpp:45 #2 0x00007f7683cf106c in galera::Monitor::enter (this=this@entry=0x5579b0a23410, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f7683ce5b17 in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x5579b0a223b0, trx=...) at galera/src/replicator_smm.cpp:1280 #4 0x00007f7683d044c3 in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:658 #5 0x00005579add7ee0b in wsrep::wsrep_provider_v26::commit_order_enter (this=0x5579b0998250, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:769 #6 0x00005579add7209e in wsrep::transaction::before_commit (this=0x7f76600a50d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:435 #7 0x00005579ad3f985e in wsrep::client_state::before_commit (this=0x7f76600a5060) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:401 #8 0x00005579ad708604 in wsrep_before_commit (thd=0x7f766009efe0, all=true) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:203 #9 0x00005579ad709881 in Wsrep_storage_service::commit (this=0x7f766009c4d0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:171 #10 0x00005579add74ba1 in wsrep::transaction::certify_fragment (this=0x7f7660006d30, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #11 0x00005579add7437b in wsrep::transaction::streaming_step (this=0x7f7660006d30, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #12 0x00005579add7159a in wsrep::transaction::after_row (this=0x7f7660006d30) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #13 0x00005579ad5130b6 in wsrep::client_state::after_row (this=0x7f7660006cc0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #14 0x00005579ad4fb000 in wsrep_after_row (thd=0x7f7660000c40) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #15 0x00005579ad50d0df in wsrep_after_row (thd=0x7f7660000c40) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #16 0x00005579ad50de15 in handler::ha_write_row (this=0x7f76601dab38, buf=0x7f76601de3f0 "\375\304\t") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6673 #17 0x00005579ad16eacf in write_record (thd=0x7f7660000c40, table=0x7f766021d420, info=0x7f76600345a0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #18 0x00005579ad173dce in select_insert::send_data (this=0x7f7660034558, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #19 0x00005579ad231dcf in end_send (join=0x7f7660034608, join_tab=0x7f7660204da8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:21196 #20 0x00005579ad376eff in JOIN_CACHE::generate_full_extensions (this=0x7f7660205640, rec_ptr=0x7f766021ffa4 "(") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #21 0x00005579ad376c1a in JOIN_CACHE::join_matching_records (this=0x7f7660205640, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #22 0x00005579ad3765e6 in JOIN_CACHE::join_records (this=0x7f7660205640, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #23 0x00005579ad376780 in JOIN_CACHE::join_records (this=0x7f7660205510, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #24 0x00005579ad376780 in JOIN_CACHE::join_records (this=0x7f7660037078, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #25 0x00005579ad22ea6d in sub_select_cache (join=0x7f7660034608, join_tab=0x7f7660204250, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19786 #26 0x00005579ad22ec7a in sub_select (join=0x7f7660034608, join_tab=0x7f7660203e88, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19957 #27 0x00005579ad22e445 in do_select (join=0x7f7660034608, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19548 #28 0x00005579ad205079 in JOIN::exec_inner (this=0x7f7660034608) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4364 #29 0x00005579ad204226 in JOIN::exec (this=0x7f7660034608) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4146 #30 0x00005579ad20590c in mysql_select (thd=0x7f7660000c40, tables=0x7f7660032038, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f7660034558, unit=0x7f7660004b58, select_lex=0x7f7660031ae8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4578 #31 0x00005579ad1f5ea6 in handle_select (thd=0x7f7660000c40, lex=0x7f7660004a90, result=0x7f7660034558, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:424 #32 0x00005579ad1b7b44 in mysql_execute_command (thd=0x7f7660000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4896 #33 0x00005579ad1c30f1 in mysql_parse (thd=0x7f7660000c40, rawbuf=0x7f76600311c8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f76780b2040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #34 0x00005579ad1c2720 in wsrep_mysql_parse (thd=0x7f7660000c40, rawbuf=0x7f76600311c8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f76780b2040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #35 0x00005579ad1ae46d in dispatch_command (command=COM_QUERY, thd=0x7f7660000c40, packet=0x7f7660041de1 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #36 0x00005579ad1accbb in do_command (thd=0x7f7660000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #37 0x00005579ad327c9e in do_handle_one_connection (connect=0x5579b0b361d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #38 0x00005579ad3279e2 in handle_one_connection (arg=0x5579b0b361d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #39 0x00005579adc5bfb3 in pfs_spawn_thread (arg=0x5579b0ef8750) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #40 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #41 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f7671c07700 (LWP 219043)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7683ce0417 in gu::Lock::wait (cond=..., this=0x7f7671c053f0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f7683cf05cd in galera::Monitor::enter (this=this@entry=0x5579b0a23350, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f7683ce40ca in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5579b0a223b0, recv_ctx=recv_ctx@entry=0x7f7671c064d0, ts=...) at galera/src/replicator_smm.cpp:469 #4 0x00007f7683ce9e40 in galera::ReplicatorSMM::process_trx (this=0x5579b0a223b0, recv_ctx=0x7f7671c064d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2114 #5 0x00007f7683ccb00b in galera::GcsActionSource::process_writeset (this=this@entry=0x5579b0a57020, recv_ctx=0x7f7671c064d0, act=..., exit_loop=@0x7f7671c0629e: false) at galera/src/gcs_action_source.cpp:62 #6 0x00007f7683ccb1e5 in galera::GcsActionSource::dispatch (this=this@entry=0x5579b0a57020, recv_ctx=recv_ctx@entry=0x7f7671c064d0, act=..., exit_loop=@0x7f7671c0629e: false) at galera/src/gcs_action_source.cpp:109 #7 0x00007f7683ccb5bd in galera::GcsActionSource::process (this=0x5579b0a57020, recv_ctx=0x7f7671c064d0, exit_loop=@0x7f7671c0629e: false) at galera/src/gcs_action_source.cpp:182 #8 0x00007f7683cea433 in galera::ReplicatorSMM::async_recv (this=0x5579b0a223b0, recv_ctx=0x7f7671c064d0) at galera/src/replicator_smm.cpp:383 #9 0x00005579add7e8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5579b0998250, applier_ctx=0x7f7671c064d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #10 0x00005579ad417a69 in wsrep_replication_process (thd=0x7f75f8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #11 0x00005579ad409587 in start_wsrep_THD (arg=0x7f766002edd0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #12 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f763d7fa700 (LWP 218448)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0b6f500) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0b6f500, reset_sig_count=3222) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0b6f500, reset_sig_count=3222) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad8f6f21 in srv_resume_thread (slot=0x5579ae858948 , sig_count=3222, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005579ad8faf32 in srv_purge_coordinator_suspend (slot=0x5579ae858948 , rseg_history_len=42) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005579ad8fb1c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f76780fe700 (LWP 218457)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579adcc9b7f in safe_cond_wait (cond=0x5579aea1b4e0 , mp=0x5579aea1e800 , file=0x5579add95488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005579ad0a8fe2 in inline_mysql_cond_wait (that=0x5579aea1b4e0 , mutex=0x5579aea1e800 , src_file=0x5579add95ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005579ad0aa874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005579adc5bfb3 in pfs_spawn_thread (arg=0x5579b0e72980) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f76367fc700 (LWP 218453)): #0 0x00007f7690382e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005579ad80b07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005579ad9b1b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f7636ffd700 (LWP 218452)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80ad7d in os_event::wait (this=0x5579b0b6f800) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005579ad80a803 in os_event::wait_low (this=0x5579b0b6f800, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005579ad80ab74 in os_event_wait_low (event=0x5579b0b6f800, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005579ad9d719b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f763f7fe700 (LWP 218444)): #0 0x00007f769037fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80a698 in os_event::timed_wait (this=0x5579b0b6f780, abstime=0x7f763f7fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005579ad80a9af in os_event::wait_time_low (this=0x5579b0b6f780, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005579ad80ab4f in os_event_wait_time_low (event=0x5579b0b6f780, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005579ad8f8c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f763ffff700 (LWP 218442)): #0 0x00007f769037fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579ad80a698 in os_event::timed_wait (this=0x5579b0d47c30, abstime=0x7f763fffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005579ad80a9af in os_event::wait_time_low (this=0x5579b0d47c30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005579ad80ab4f in os_event_wait_time_low (event=0x5579b0d47c30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005579ad7d74f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f7651ffb700 (LWP 218437)): #0 0x00007f768ff4c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005579ad7fe09a in LinuxAIOHandler::collect (this=0x7f7651ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005579ad7fe5c3 in LinuxAIOHandler::poll (this=0x7f7651ffab00, m1=0x7f7651ffaba8, m2=0x7f7651ffabb0, request=0x7f7651ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005579ad7fe726 in os_aio_linux_handler (global_segment=5, m1=0x7f7651ffaba8, m2=0x7f7651ffabb0, request=0x7f7651ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005579ad80324a in os_aio_handler (segment=5, m1=0x7f7651ffaba8, m2=0x7f7651ffabb0, request=0x7f7651ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005579ada564a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005579ad8fca47 in io_handler_thread (arg=0x5579aea97188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f7673059700 (LWP 218430)): #0 0x00007f769037fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579adcc9e81 in safe_cond_timedwait (cond=0x5579af2a8a00 , mp=0x5579af2a8940 , abstime=0x7f7673058cb0, file=0x5579ae117630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005579adb33309 in inline_mysql_cond_timedwait (that=0x5579af2a8a00 , mutex=0x5579af2a8940 , abstime=0x7f7673058cb0, src_file=0x5579ae117678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005579adb3382d in my_service_thread_sleep (control=0x5579ae85cd00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005579adb27cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005579adc5bfb3 in pfs_spawn_thread (arg=0x5579b0b278d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f767b9ec700 (LWP 218222)): #0 0x00007f769037f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7683cc8377 in gu::Lock::wait (cond=..., this=0x7f767b9ebbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f7683cc8409 in galera::ServiceThd::thd_func (arg=0x5579b0a22ac8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f7688dd4700 (LWP 218221)): #0 0x00007f769037fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005579adcc9e81 in safe_cond_timedwait (cond=0x5579af2bb1c0 , mp=0x5579af2bb100 , abstime=0x7f7688dd3d70, file=0x5579ae15be90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005579adccae3b in inline_mysql_cond_timedwait (that=0x5579af2bb1c0 , mutex=0x5579af2bb100 , abstime=0x7f7688dd3d70, src_file=0x5579ae15bed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005579adccb99e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005579adc5bfb3 in pfs_spawn_thread (arg=0x5579b09ebde0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f769037bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f768e739ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f769078c880 (LWP 218219)): #0 0x00007f76903809d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005579adcc4550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005579ad4f9ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f768e72f20d in poll () from /lib64/libc.so.6 #5 0x00005579ad083f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005579ad0831b4 in mysqld_main (argc=163, argv=0x5579b08a4140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005579ad077695 in main (argc=23, argv=0x7ffc589bffe8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera-features#56-innodb/mysqld.2/data/core.218219 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera-features#56-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera-features#56-innodb/' galera_sr.galera_sr_insert_select 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 16:57:05 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr_insert_select.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.218342' (4/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_insert_select-innodb/mysqld.1/data/core.218342 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 218342] [New LWP 218350] [New LWP 218358] [New LWP 218362] [New LWP 218363] [New LWP 218413] [New LWP 218389] [New LWP 218364] [New LWP 218414] [New LWP 218393] [New LWP 218416] [New LWP 218365] [New LWP 218406] [New LWP 218471] [New LWP 218392] [New LWP 218472] [New LWP 218409] [New LWP 218395] [New LWP 218410] [New LWP 218408] [New LWP 218412] [New LWP 218415] [New LWP 218419] [New LWP 218720] [New LWP 218420] [New LWP 218418] [New LWP 218417] [New LWP 218407] [New LWP 218394] [New LWP 218391] [New LWP 218390] [New LWP 218388] [New LWP 218345] [New LWP 218411] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f41a6aaa9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f41a6aaa9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559c1cc2e550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559c1c463ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f41a4e5920d in poll () from /lib64/libc.so.6 #5 0x0000559c1bfedf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000559c1bfed1b4 in mysqld_main (argc=162, argv=0x559c1e24a140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000559c1bfe1695 in main (argc=23, argv=0x7fff50eff398) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f416d7fa700 (LWP 218411)): #0 0x00007f41a6aacd1d in fsync () from /lib64/libpthread.so.0 #1 0x0000559c1c769749 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000559c1c769a9f in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000559c1c9b21fa in pfs_os_file_flush_func (file=..., src_file=0x559c1d037588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559c1c9b5055 in fil_flush_low (space=0x559c1e716590, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x0000559c1c9c09f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x0000559c1c744e31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000559c1c745893 in log_write_up_to (lsn=5036192, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000559c1c7459c3 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x0000559c1c86355e in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #10 0x0000559c1c863b21 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #11 0x0000559c1c86467f in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f419f4fe700 (LWP 218345)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1cc33e81 in safe_cond_timedwait (cond=0x559c1e2251c0 , mp=0x559c1e225100 , abstime=0x7f419f4fdd70, file=0x559c1d0c5e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559c1cc34e3b in inline_mysql_cond_timedwait (that=0x559c1e2251c0 , mutex=0x559c1e225100 , abstime=0x7f419f4fdd70, src_file=0x559c1d0c5ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559c1cc3599e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559c1cbc5fb3 in pfs_spawn_thread (arg=0x559c1e391d40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f41958c8700 (LWP 218388)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1cc33e81 in safe_cond_timedwait (cond=0x559c1e212a00 , mp=0x559c1e212940 , abstime=0x7f41958c7cb0, file=0x559c1d081630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559c1ca9d309 in inline_mysql_cond_timedwait (that=0x559c1e212a00 , mutex=0x559c1e212940 , abstime=0x7f41958c7cb0, src_file=0x559c1d081678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559c1ca9d82d in my_service_thread_sleep (control=0x559c1d7c6d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559c1ca91cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559c1cbc5fb3 in pfs_spawn_thread (arg=0x559c1e4c49f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f418f7fe700 (LWP 218390)): #0 0x00007f41a6676644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559c1c76809a in LinuxAIOHandler::collect (this=0x7f418f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559c1c7685c3 in LinuxAIOHandler::poll (this=0x7f418f7fdb00, m1=0x7f418f7fdba8, m2=0x7f418f7fdbb0, request=0x7f418f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559c1c768726 in os_aio_linux_handler (global_segment=1, m1=0x7f418f7fdba8, m2=0x7f418f7fdbb0, request=0x7f418f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559c1c76d24a in os_aio_handler (segment=1, m1=0x7f418f7fdba8, m2=0x7f418f7fdbb0, request=0x7f418f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559c1c9c04a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000559c1c866a47 in io_handler_thread (arg=0x559c1da01168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f418effd700 (LWP 218391)): #0 0x00007f41a6676644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559c1c76809a in LinuxAIOHandler::collect (this=0x7f418effcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559c1c7685c3 in LinuxAIOHandler::poll (this=0x7f418effcb00, m1=0x7f418effcba8, m2=0x7f418effcbb0, request=0x7f418effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559c1c768726 in os_aio_linux_handler (global_segment=2, m1=0x7f418effcba8, m2=0x7f418effcbb0, request=0x7f418effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559c1c76d24a in os_aio_handler (segment=2, m1=0x7f418effcba8, m2=0x7f418effcbb0, request=0x7f418effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559c1c9c04a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000559c1c866a47 in io_handler_thread (arg=0x559c1da01170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f418d7fa700 (LWP 218394)): #0 0x00007f41a6676644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559c1c76809a in LinuxAIOHandler::collect (this=0x7f418d7f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559c1c7685c3 in LinuxAIOHandler::poll (this=0x7f418d7f9b00, m1=0x7f418d7f9ba8, m2=0x7f418d7f9bb0, request=0x7f418d7f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559c1c768726 in os_aio_linux_handler (global_segment=5, m1=0x7f418d7f9ba8, m2=0x7f418d7f9bb0, request=0x7f418d7f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559c1c76d24a in os_aio_handler (segment=5, m1=0x7f418d7f9ba8, m2=0x7f418d7f9bb0, request=0x7f418d7f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559c1c9c04a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000559c1c866a47 in io_handler_thread (arg=0x559c1da01188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f416ffff700 (LWP 218407)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774698 in os_event::timed_wait (this=0x559c1e50ca20, abstime=0x7f416fffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559c1c7749af in os_event::wait_time_low (this=0x559c1e50ca20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559c1c774b4f in os_event_wait_time_low (event=0x559c1e50ca20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559c1c86312a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f415dffb700 (LWP 218417)): #0 0x00007f41a6aace3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559c1c77507f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559c1c91bb55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f415d7fa700 (LWP 218418)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774d7d in os_event::wait (this=0x559c1e50cc20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559c1c774803 in os_event::wait_low (this=0x559c1e50cc20, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559c1c774b74 in os_event_wait_low (event=0x559c1e50cc20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559c1c929cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4194276700 (LWP 218420)): #0 0x00007f41a4d9d40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000559c1bfe1714 in my_sigwait (set=0x7f4194275cf0, sig=0x7f4194275cb4, code=0x7f4194275cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000559c1bfe7a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x0000559c1cbc5fb3 in pfs_spawn_thread (arg=0x559c1e4d0060) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4194195700 (LWP 218720)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1cc33e81 in safe_cond_timedwait (cond=0x559c1d9854a0 , mp=0x559c1d987e40 , abstime=0x7f4194194c60, file=0x559c1ccefb08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559c1bfe2015 in inline_mysql_cond_timedwait (that=0x559c1d9854a0 , mutex=0x559c1d987e40 , abstime=0x7f4194194c60, src_file=0x559c1ccf1b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559c1bfe6b08 in cache_thread (thd=0x7f4128000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x0000559c1bfe6db6 in one_thread_per_connection_end (thd=0x7f4128000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x0000559c1c291d28 in do_handle_one_connection (connect=0x559c1e7a0690) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000559c1c2919e2 in handle_one_connection (arg=0x559c1e7a0690) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000559c1cbc5fb3 in pfs_spawn_thread (arg=0x559c1e34fba0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f415cff9700 (LWP 218419)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1cc33b7f in safe_cond_wait (cond=0x7f415cff8c70, mp=0x7f415cff8cb0, file=0x559c1cf0a920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559c1c68acad in inline_mysql_cond_wait (that=0x7f415cff8c70, mutex=0x7f415cff8cb0, src_file=0x559c1cf0c348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559c1c68f79b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000559c1cbc5fb3 in pfs_spawn_thread (arg=0x559c1e74d820) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f415effd700 (LWP 218415)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774d7d in os_event::wait (this=0x559c1e50c8a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559c1c774803 in os_event::wait_low (this=0x559c1e50c8a0, reset_sig_count=10899) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559c1c774b74 in os_event_wait_low (event=0x559c1e50c8a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559c1c860f21 in srv_resume_thread (slot=0x559c1d7c2980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559c1c864b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f416cff9700 (LWP 218412)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774d7d in os_event::wait (this=0x559c1e50c820) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559c1c774803 in os_event::wait_low (this=0x559c1e50c820, reset_sig_count=2426) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559c1c774b74 in os_event_wait_low (event=0x559c1e50c820, reset_sig_count=2426) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559c1c860f21 in srv_resume_thread (slot=0x559c1d7c2948 , sig_count=2426, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559c1c864f32 in srv_purge_coordinator_suspend (slot=0x559c1d7c2948 , rseg_history_len=12) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000559c1c8651c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f416effd700 (LWP 218408)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774698 in os_event::timed_wait (this=0x559c1e50caa0, abstime=0x7f416effcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559c1c7749af in os_event::wait_time_low (this=0x559c1e50caa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559c1c774b4f in os_event_wait_time_low (event=0x559c1e50caa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559c1c862c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f416dffb700 (LWP 218410)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774698 in os_event::timed_wait (this=0x559c1e6ac970, abstime=0x7f416dffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559c1c7749af in os_event::wait_time_low (this=0x559c1e6ac970, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559c1c774b4f in os_event_wait_time_low (event=0x559c1e6ac970, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559c1c8c34c9 in ib_wqueue_timedwait (wq=0x559c1e6ac840, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559c1ca0473a in fts_optimize_thread (arg=0x559c1e6ac840) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f418cff9700 (LWP 218395)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774698 in os_event::timed_wait (this=0x559c1e50cba0, abstime=0x7f418cff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559c1c7749af in os_event::wait_time_low (this=0x559c1e50cba0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559c1c774b4f in os_event_wait_time_low (event=0x559c1e50cba0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559c1c94b1fe in pc_sleep_if_needed (next_loop_time=1551279339816, sig_count=2, cur_time=1551279338816) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x0000559c1c94c137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f416e7fc700 (LWP 218409)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774698 in os_event::timed_wait (this=0x559c1e716bd0, abstime=0x7f416e7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559c1c7749af in os_event::wait_time_low (this=0x559c1e716bd0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559c1c774b4f in os_event_wait_time_low (event=0x559c1e716bd0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559c1c9aa0e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f41941e0700 (LWP 218472)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774d7d in os_event::wait (this=0x559c1e6a4c60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559c1c774803 in os_event::wait_low (this=0x559c1e6a4c60, reset_sig_count=3421) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559c1c774b74 in os_event_wait_low (event=0x559c1e6a4c60, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559c1c7453c3 in log_write_up_to (lsn=5037954, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x0000559c1c8b0f5f in trx_flush_log_if_needed_low (lsn=5037954) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #6 0x0000559c1c8b0fa8 in trx_flush_log_if_needed (lsn=5037954, trx=0x7f4194cca390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #7 0x0000559c1c8b2dd1 in trx_commit_complete_for_mysql (trx=0x7f4194cca390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #8 0x0000559c1c695ca6 in innobase_commit (hton=0x559c1e4e0d20, thd=0x7f4134076cf0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4658 #9 0x0000559c1c469a1d in commit_one_phase_2 (thd=0x7f4134076cf0, all=true, trans=0x7f413407a268, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1741 #10 0x0000559c1c46991f in ha_commit_one_phase (thd=0x7f4134076cf0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1721 #11 0x0000559c1c468c98 in ha_commit_trans (thd=0x7f4134076cf0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1534 #12 0x0000559c1c2a6ac6 in trans_commit (thd=0x7f4134076cf0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #13 0x0000559c1c6738a3 in Wsrep_storage_service::commit (this=0x7f41342a0f60, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #14 0x0000559c1ccdeba1 in wsrep::transaction::certify_fragment (this=0x7f4134006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #15 0x0000559c1ccde37b in wsrep::transaction::streaming_step (this=0x7f4134006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #16 0x0000559c1ccdb59a in wsrep::transaction::after_row (this=0x7f4134006bf0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #17 0x0000559c1c47d0b6 in wsrep::client_state::after_row (this=0x7f4134006b80) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #18 0x0000559c1c465000 in wsrep_after_row (thd=0x7f4134000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #19 0x0000559c1c4770df in wsrep_after_row (thd=0x7f4134000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #20 0x0000559c1c477e15 in handler::ha_write_row (this=0x7f4134281638, buf=0x7f413415fa50 "\375\331\022") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6673 #21 0x0000559c1c0d8acf in write_record (thd=0x7f4134000b00, table=0x7f41342807d0, info=0x7f4134018a50) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #22 0x0000559c1c0dddce in select_insert::send_data (this=0x7f4134018a08, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #23 0x0000559c1c19bdcf in end_send (join=0x7f4134018ab8, join_tab=0x7f41340531e8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:21196 #24 0x0000559c1c2e0eff in JOIN_CACHE::generate_full_extensions (this=0x7f4134053d80, rec_ptr=0x7f4134056980 "4") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #25 0x0000559c1c2e0c1a in JOIN_CACHE::join_matching_records (this=0x7f4134053d80, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #26 0x0000559c1c2e05e6 in JOIN_CACHE::join_records (this=0x7f4134053d80, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #27 0x0000559c1c2e0780 in JOIN_CACHE::join_records (this=0x7f4134053c50, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #28 0x0000559c1c2e0780 in JOIN_CACHE::join_records (this=0x7f4134053b20, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #29 0x0000559c1c198a6d in sub_select_cache (join=0x7f4134018ab8, join_tab=0x7f4134052690, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19786 #30 0x0000559c1c198c7a in sub_select (join=0x7f4134018ab8, join_tab=0x7f41340522c8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19957 #31 0x0000559c1c198445 in do_select (join=0x7f4134018ab8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19548 #32 0x0000559c1c16f079 in JOIN::exec_inner (this=0x7f4134018ab8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4364 #33 0x0000559c1c16e226 in JOIN::exec (this=0x7f4134018ab8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4146 #34 0x0000559c1c16f90c in mysql_select (thd=0x7f4134000b00, tables=0x7f41340164e8, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7f4134018a08, unit=0x7f4134004a18, select_lex=0x7f4134015de8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4578 #35 0x0000559c1c15fea6 in handle_select (thd=0x7f4134000b00, lex=0x7f4134004950, result=0x7f4134018a08, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:424 #36 0x0000559c1c121b44 in mysql_execute_command (thd=0x7f4134000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4896 #37 0x0000559c1c12d0f1 in mysql_parse (thd=0x7f4134000b00, rawbuf=0x7f41340154a8 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f41941df040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #38 0x0000559c1c12c720 in wsrep_mysql_parse (thd=0x7f4134000b00, rawbuf=0x7f41340154a8 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f41941df040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #39 0x0000559c1c11846d in dispatch_command (command=COM_QUERY, thd=0x7f4134000b00, packet=0x7f4134028541 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #40 0x0000559c1c116cbb in do_command (thd=0x7f4134000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #41 0x0000559c1c291c9e in do_handle_one_connection (connect=0x559c1e87d0b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #42 0x0000559c1c2919e2 in handle_one_connection (arg=0x559c1e87d0b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #43 0x0000559c1cbc5fb3 in pfs_spawn_thread (arg=0x559c1e89ae30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #44 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #45 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f418e7fc700 (LWP 218392)): #0 0x00007f41a6676644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559c1c76809a in LinuxAIOHandler::collect (this=0x7f418e7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559c1c7685c3 in LinuxAIOHandler::poll (this=0x7f418e7fbb00, m1=0x7f418e7fbba8, m2=0x7f418e7fbbb0, request=0x7f418e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559c1c768726 in os_aio_linux_handler (global_segment=3, m1=0x7f418e7fbba8, m2=0x7f418e7fbbb0, request=0x7f418e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559c1c76d24a in os_aio_handler (segment=3, m1=0x7f418e7fbba8, m2=0x7f418e7fbbb0, request=0x7f418e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559c1c9c04a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000559c1c866a47 in io_handler_thread (arg=0x559c1da01178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f419422b700 (LWP 218471)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1cc33b7f in safe_cond_wait (cond=0x559c1d9854e0 , mp=0x559c1d988800 , file=0x559c1ccff488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559c1c012fe2 in inline_mysql_cond_wait (that=0x559c1d9854e0 , mutex=0x559c1d988800 , src_file=0x559c1ccffff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000559c1c014874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559c1cbc5fb3 in pfs_spawn_thread (arg=0x559c1e834260) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f416f7fe700 (LWP 218406)): #0 0x00007f41a6aa9d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774698 in os_event::timed_wait (this=0x559c1e6e4b10, abstime=0x7f416f7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559c1c7749af in os_event::wait_time_low (this=0x559c1e6e4b10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559c1c774b4f in os_event_wait_time_low (event=0x559c1e6e4b10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559c1c7414f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f419647c700 (LWP 218365)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f419e8ba272 in fifo_lock_get (q=q@entry=0x559c1e3f6d90) at galerautils/src/gu_fifo.c:262 #2 0x00007f419e8ba8e1 in gu_fifo_get_head (q=0x559c1e3f6d90, err=0x7f419647b06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f419e99c43c in gcs_recv (conn=0x559c1e3d0710, action=0x7f419647b0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f419e9c950c in galera::GcsActionSource::process (this=0x559c1e3d00f0, recv_ctx=0x7f419647b4d0, exit_loop=@0x7f419647b29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f419e9e8433 in galera::ReplicatorSMM::async_recv (this=0x559c1e3c8420, recv_ctx=0x7f419647b4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000559c1cce88a8 in wsrep::wsrep_provider_v26::run_applier (this=0x559c1e34f760, applier_ctx=0x7f419647b4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000559c1c381a69 in wsrep_replication_process (thd=0x7f4184000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000559c1c373587 in start_wsrep_THD (arg=0x559c1e41d460) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f415e7fc700 (LWP 218416)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774d7d in os_event::wait (this=0x559c1e50cb20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559c1c774803 in os_event::wait_low (this=0x559c1e50cb20, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559c1c774b74 in os_event_wait_low (event=0x559c1e50cb20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559c1c94119b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f418dffb700 (LWP 218393)): #0 0x00007f41a6676644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559c1c76809a in LinuxAIOHandler::collect (this=0x7f418dffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559c1c7685c3 in LinuxAIOHandler::poll (this=0x7f418dffab00, m1=0x7f418dffaba8, m2=0x7f418dffabb0, request=0x7f418dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559c1c768726 in os_aio_linux_handler (global_segment=4, m1=0x7f418dffaba8, m2=0x7f418dffabb0, request=0x7f418dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559c1c76d24a in os_aio_handler (segment=4, m1=0x7f418dffaba8, m2=0x7f418dffabb0, request=0x7f418dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559c1c9c04a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000559c1c866a47 in io_handler_thread (arg=0x559c1da01180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f415f7fe700 (LWP 218414)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774d7d in os_event::wait (this=0x559c1e50c920) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559c1c774803 in os_event::wait_low (this=0x559c1e50c920, reset_sig_count=9504) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559c1c774b74 in os_event_wait_low (event=0x559c1e50c920, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559c1c860f21 in srv_resume_thread (slot=0x559c1d7c29b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559c1c864b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f419c072700 (LWP 218364)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1cc33b7f in safe_cond_wait (cond=0x7f4178015938, mp=0x7f4178015888, file=0x559c1cda24c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559c1c3814ba in inline_mysql_cond_wait (that=0x7f4178015938, mutex=0x7f4178015888, src_file=0x559c1cda27e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559c1c38434e in Wsrep_thd_queue::pop_front (this=0x7f4178015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559c1c3830d0 in wsrep_post_rollback_process (post_rollbacker=0x7f4178000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x0000559c1c373587 in start_wsrep_THD (arg=0x559c1e3fe9d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f418ffff700 (LWP 218389)): #0 0x00007f41a6676644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559c1c76809a in LinuxAIOHandler::collect (this=0x7f418fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559c1c7685c3 in LinuxAIOHandler::poll (this=0x7f418fffeb00, m1=0x7f418fffeba8, m2=0x7f418fffebb0, request=0x7f418fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559c1c768726 in os_aio_linux_handler (global_segment=0, m1=0x7f418fffeba8, m2=0x7f418fffebb0, request=0x7f418fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559c1c76d24a in os_aio_handler (segment=0, m1=0x7f418fffeba8, m2=0x7f418fffebb0, request=0x7f418fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559c1c9c04a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000559c1c866a47 in io_handler_thread (arg=0x559c1da01160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f415ffff700 (LWP 218413)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1c774d7d in os_event::wait (this=0x559c1e50c9a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559c1c774803 in os_event::wait_low (this=0x559c1e50c9a0, reset_sig_count=8697) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559c1c774b74 in os_event_wait_low (event=0x559c1e50c9a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559c1c860f21 in srv_resume_thread (slot=0x559c1d7c29f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559c1c864b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f419c0bd700 (LWP 218363)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559c1cc33b7f in safe_cond_wait (cond=0x7f4180016b88, mp=0x7f4180016ad8, file=0x559c1cda24c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559c1c3814ba in inline_mysql_cond_wait (that=0x7f4180016b88, mutex=0x7f4180016ad8, src_file=0x559c1cda27e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559c1c38434e in Wsrep_thd_queue::pop_front (this=0x7f4180016a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559c1c382c54 in wsrep_rollback_process (rollbacker=0x7f4180001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x0000559c1c373587 in start_wsrep_THD (arg=0x559c1e4144e0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4196c7d700 (LWP 218362)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f419e9a0bd7 in gu::Lock::wait (cond=..., this=0x7f4196c7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f419e9a371b in RecvBuf::front (this=this@entry=0x559c1e404a68, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f419e9a12b6 in gcomm_recv (backend=, msg=0x559c1e3d09f0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f419e99525c in core_msg_recv (backend=backend@entry=0x559c1e3d0b38, recv_msg=recv_msg@entry=0x559c1e3d09f0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f419e9984d4 in gcs_core_recv (conn=0x559c1e3d0990, recv_act=recv_act@entry=0x7f4196c7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f419e99f748 in gcs_recv_thread (arg=0x559c1e3d0710) at gcs/src/gcs.cpp:1398 #7 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f419c8be700 (LWP 218358)): #0 0x00007f41a4e64483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f419e967a68 in asio::detail::epoll_reactor::run (this=0x559c1e405200, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f419e96c160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559c1e4050a0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f419e96c412 in asio::detail::task_io_service::run (this=0x559c1e4050a0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f419e983e6d in asio::io_service::run (this=0x559c1e4041a0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f419e9845c1 in gcomm::AsioProtonet::event_loop (this=0x559c1e404100, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f419e9a265d in GCommConn::run (this=0x559c1e404880) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f419e9a5709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f419d9c0700 (LWP 218350)): #0 0x00007f41a6aa9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f419e9c6377 in gu::Lock::wait (cond=..., this=0x7f419d9bfbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f419e9c6409 in galera::ServiceThd::thd_func (arg=0x559c1e3c8b38) at galera/src/galera_service_thd.cpp:30 #3 0x00007f41a6aa5dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f41a4e63ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f41a6eb6880 (LWP 218342)): #0 0x00007f41a6aaa9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559c1cc2e550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559c1c463ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f41a4e5920d in poll () from /lib64/libc.so.6 #5 0x0000559c1bfedf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000559c1bfed1b4 in mysqld_main (argc=162, argv=0x559c1e24a140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000559c1bfe1695 in main (argc=23, argv=0x7fff50eff398) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_insert_select-innodb/mysqld.1/data/core.218342 - found 'core.218481' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_insert_select-innodb/mysqld.2/data/core.218481 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 218481] [New LWP 218483] [New LWP 218487] [New LWP 218486] [New LWP 218489] [New LWP 218484] [New LWP 218488] [New LWP 218490] [New LWP 218661] [New LWP 218660] [New LWP 218662] [New LWP 218675] [New LWP 218667] [New LWP 218663] [New LWP 218679] [New LWP 218664] [New LWP 218677] [New LWP 218671] [New LWP 218683] [New LWP 218684] [New LWP 218676] [New LWP 218672] [New LWP 218685] [New LWP 218673] [New LWP 218666] [New LWP 218686] [New LWP 218674] [New LWP 218678] [New LWP 218688] [New LWP 218680] [New LWP 219023] [New LWP 218687] [New LWP 218665] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f79fd5599d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f79fd5599d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055fb8f5bb550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055fb8edf0ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f79fb90820d in poll () from /lib64/libc.so.6 #5 0x000055fb8e97af19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055fb8e97a1b4 in mysqld_main (argc=163, argv=0x55fb91c4f140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055fb8e96e695 in main (argc=23, argv=0x7ffd54ba27d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f79bbfff700 (LWP 218665)): #0 0x00007f79fd55bd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055fb8f0f6749 in os_file_fsync_posix (file=29) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055fb8f0f6a9f in os_file_flush_func (file=29) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055fb8f33f1fa in pfs_os_file_flush_func (file=..., src_file=0x55fb8f9c4588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055fb8f342055 in fil_flush_low (space=0x55fb921238e0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x000055fb8f34d9f8 in fil_flush (space_id=7) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x000055fb8f34dc7f in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4548 #7 0x000055fb8f2ca256 in buf_dblwr_update (bpage=0x7f79c641b6f0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:767 #8 0x000055fb8f2d1d28 in buf_flush_write_complete (bpage=0x7f79c641b6f0, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000055fb8f2c0910 in buf_page_io_complete (bpage=0x7f79c641b6f0, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6216 #10 0x000055fb8f34d752 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4435 #11 0x000055fb8f1f3a47 in io_handler_thread (arg=0x55fb9038e180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f79f41ca700 (LWP 218687)): #0 0x00007f79fb84c40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055fb8e96e714 in my_sigwait (set=0x7f79f41c9cf0, sig=0x7f79f41c9cb4, code=0x7f79f41c9cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055fb8e974a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x000055fb8f552fb3 in pfs_spawn_thread (arg=0x55fb921980e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f79f40e9700 (LWP 219023)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f5c0e81 in safe_cond_timedwait (cond=0x55fb903124a0 , mp=0x55fb90314e40 , abstime=0x7f79f40e8c60, file=0x55fb8f67cb08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fb8e96f015 in inline_mysql_cond_timedwait (that=0x55fb903124a0 , mutex=0x55fb90314e40 , abstime=0x7f79f40e8c60, src_file=0x55fb8f67eb50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fb8e973b08 in cache_thread (thd=0x7f796c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x000055fb8e973db6 in one_thread_per_connection_end (thd=0x7f796c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x000055fb8ec1ed28 in do_handle_one_connection (connect=0x55fb92124db0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055fb8ec1e9e2 in handle_one_connection (arg=0x55fb92124db0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055fb8f552fb3 in pfs_spawn_thread (arg=0x55fb922a1580) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f79a97fa700 (LWP 218680)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101d7d in os_event::wait (this=0x55fb91f18630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb8f101803 in os_event::wait_low (this=0x55fb91f18630, reset_sig_count=8613) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb8f101b74 in os_event_wait_low (event=0x55fb91f18630, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb8f1edf21 in srv_resume_thread (slot=0x55fb9014f9f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb8f1f1b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f79f417f700 (LWP 218688)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f5c0b7f in safe_cond_wait (cond=0x55fb903124e0 , mp=0x55fb90315800 , file=0x55fb8f68c488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb8e99ffe2 in inline_mysql_cond_wait (that=0x55fb903124e0 , mutex=0x55fb90315800 , src_file=0x55fb8f68cff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055fb8e9a1874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055fb8f552fb3 in pfs_spawn_thread (arg=0x55fb922403c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f79aa7fc700 (LWP 218678)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101d7d in os_event::wait (this=0x55fb91f18530) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb8f101803 in os_event::wait_low (this=0x55fb91f18530, reset_sig_count=10389) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb8f101b74 in os_event_wait_low (event=0x55fb91f18530, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb8f1edf21 in srv_resume_thread (slot=0x55fb9014f980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb8f1f1b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f79abfff700 (LWP 218674)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101698 in os_event::timed_wait (this=0x55fb92122ca0, abstime=0x7f79abffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb8f1019af in os_event::wait_time_low (this=0x55fb92122ca0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb8f101b4f in os_event_wait_time_low (event=0x55fb92122ca0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb8f3370e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f798affd700 (LWP 218686)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f5c0b7f in safe_cond_wait (cond=0x7f798affcc70, mp=0x7f798affccb0, file=0x55fb8f897920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb8f017cad in inline_mysql_cond_wait (that=0x7f798affcc70, mutex=0x7f798affccb0, src_file=0x55fb8f899348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fb8f01c79b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055fb8f552fb3 in pfs_spawn_thread (arg=0x55fb920d3460) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f79bb7fe700 (LWP 218666)): #0 0x00007f79fd125644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb8f0f509a in LinuxAIOHandler::collect (this=0x7f79bb7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb8f0f55c3 in LinuxAIOHandler::poll (this=0x7f79bb7fdb00, m1=0x7f79bb7fdba8, m2=0x7f79bb7fdbb0, request=0x7f79bb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb8f0f5726 in os_aio_linux_handler (global_segment=5, m1=0x7f79bb7fdba8, m2=0x7f79bb7fdbb0, request=0x7f79bb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb8f0fa24a in os_aio_handler (segment=5, m1=0x7f79bb7fdba8, m2=0x7f79bb7fdbb0, request=0x7f79bb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb8f34d4a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055fb8f1f3a47 in io_handler_thread (arg=0x55fb9038e188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f79b8ff9700 (LWP 218673)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101698 in os_event::timed_wait (this=0x55fb91f18730, abstime=0x7f79b8ff8d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb8f1019af in os_event::wait_time_low (this=0x55fb91f18730, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb8f101b4f in os_event_wait_time_low (event=0x55fb91f18730, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb8f1efc08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f798b7fe700 (LWP 218685)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101d7d in os_event::wait (this=0x55fb91f188b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb8f101803 in os_event::wait_low (this=0x55fb91f188b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb8f101b74 in os_event_wait_low (event=0x55fb91f188b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb8f2b6cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f79b97fa700 (LWP 218672)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101698 in os_event::timed_wait (this=0x55fb91f186b0, abstime=0x7f79b97f9ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb8f1019af in os_event::wait_time_low (this=0x55fb91f186b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb8f101b4f in os_event_wait_time_low (event=0x55fb91f186b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb8f1f012a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f79ba7fc700 (LWP 218676)): #0 0x00007f79fd55be3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055fb8f10207f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055fb8f1f1489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055fb8f1f15b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f798bfff700 (LWP 218684)): #0 0x00007f79fd55be3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055fb8f10207f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055fb8f2a8b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f79a8ff9700 (LWP 218683)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101d7d in os_event::wait (this=0x55fb91f187b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb8f101803 in os_event::wait_low (this=0x55fb91f187b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb8f101b74 in os_event_wait_low (event=0x55fb91f187b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb8f2ce19b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f79b9ffb700 (LWP 218671)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101698 in os_event::timed_wait (this=0x55fb920f0be0, abstime=0x7f79b9ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb8f1019af in os_event::wait_time_low (this=0x55fb920f0be0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb8f101b4f in os_event_wait_time_low (event=0x55fb920f0be0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb8f0ce4f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f79aaffd700 (LWP 218677)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101d7d in os_event::wait (this=0x55fb91f184b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb8f101803 in os_event::wait_low (this=0x55fb91f184b0, reset_sig_count=2500) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb8f101b74 in os_event_wait_low (event=0x55fb91f184b0, reset_sig_count=2500) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb8f1edf21 in srv_resume_thread (slot=0x55fb9014f948 , sig_count=2500, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb8f1f1f32 in srv_purge_coordinator_suspend (slot=0x55fb9014f948 , rseg_history_len=4) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055fb8f1f21c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f79c4a05700 (LWP 218664)): #0 0x00007f79fd125644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb8f0f509a in LinuxAIOHandler::collect (this=0x7f79c4a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb8f0f55c3 in LinuxAIOHandler::poll (this=0x7f79c4a04b00, m1=0x7f79c4a04ba8, m2=0x7f79c4a04bb0, request=0x7f79c4a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb8f0f5726 in os_aio_linux_handler (global_segment=3, m1=0x7f79c4a04ba8, m2=0x7f79c4a04bb0, request=0x7f79c4a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb8f0fa24a in os_aio_handler (segment=3, m1=0x7f79c4a04ba8, m2=0x7f79c4a04bb0, request=0x7f79c4a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb8f34d4a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055fb8f1f3a47 in io_handler_thread (arg=0x55fb9038e178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f79a9ffb700 (LWP 218679)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101d7d in os_event::wait (this=0x55fb91f185b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb8f101803 in os_event::wait_low (this=0x55fb91f185b0, reset_sig_count=9437) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb8f101b74 in os_event_wait_low (event=0x55fb91f185b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb8f1edf21 in srv_resume_thread (slot=0x55fb9014f9b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb8f1f1b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f79c5206700 (LWP 218663)): #0 0x00007f79fd125644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb8f0f509a in LinuxAIOHandler::collect (this=0x7f79c5205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb8f0f55c3 in LinuxAIOHandler::poll (this=0x7f79c5205b00, m1=0x7f79c5205ba8, m2=0x7f79c5205bb0, request=0x7f79c5205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb8f0f5726 in os_aio_linux_handler (global_segment=2, m1=0x7f79c5205ba8, m2=0x7f79c5205bb0, request=0x7f79c5205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb8f0fa24a in os_aio_handler (segment=2, m1=0x7f79c5205ba8, m2=0x7f79c5205bb0, request=0x7f79c5205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb8f34d4a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055fb8f1f3a47 in io_handler_thread (arg=0x55fb9038e170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f79baffd700 (LWP 218667)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101d7d in os_event::wait (this=0x55fb91ffe680) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb8f101803 in os_event::wait_low (this=0x55fb91ffe680, reset_sig_count=441) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb8f101b74 in os_event_wait_low (event=0x55fb91ffe680, reset_sig_count=441) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb8f1fbd96 in sync_array_wait_event (arr=0x55fb91eefa50, cell=@0x7f79baffca20: 0x55fb91eefbc0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055fb8f04bf53 in TTASEventMutex::enter (this=0x55fb91ffdfb0, max_spins=60, max_delay=4, filename=0x55fb8f99ac28 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1877) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x000055fb8f04a1ae in PolicyMutex >::enter (this=0x55fb91ffdfb0, n_spins=30, n_delay=4, name=0x55fb8f99ac28 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1877) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x000055fb8f2d5664 in buf_flush_start (buf_pool=0x55fb91ffdfb0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1877 #8 0x000055fb8f2d5a04 in buf_flush_do_batch (buf_pool=0x55fb91ffdfb0, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7f79baffcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1987 #9 0x000055fb8f2d5d07 in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7f79baffcbd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2097 #10 0x000055fb8f2d9a52 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3249 #11 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f79ab7fe700 (LWP 218675)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f101698 in os_event::timed_wait (this=0x55fb920c08d0, abstime=0x7f79ab7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb8f1019af in os_event::wait_time_low (this=0x55fb920c08d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb8f101b4f in os_event_wait_time_low (event=0x55fb920c08d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb8f2504c9 in ib_wqueue_timedwait (wq=0x55fb920c07a0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055fb8f39173a in fts_optimize_thread (arg=0x55fb920c07a0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f79c5a07700 (LWP 218662)): #0 0x00007f79fd125644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb8f0f509a in LinuxAIOHandler::collect (this=0x7f79c5a06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb8f0f55c3 in LinuxAIOHandler::poll (this=0x7f79c5a06b00, m1=0x7f79c5a06ba8, m2=0x7f79c5a06bb0, request=0x7f79c5a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb8f0f5726 in os_aio_linux_handler (global_segment=1, m1=0x7f79c5a06ba8, m2=0x7f79c5a06bb0, request=0x7f79c5a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb8f0fa24a in os_aio_handler (segment=1, m1=0x7f79c5a06ba8, m2=0x7f79c5a06bb0, request=0x7f79c5a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb8f34d4a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055fb8f1f3a47 in io_handler_thread (arg=0x55fb9038e168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f79c785a700 (LWP 218660)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f5c0e81 in safe_cond_timedwait (cond=0x55fb90b9fa00 , mp=0x55fb90b9f940 , abstime=0x7f79c7859cb0, file=0x55fb8fa0e630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fb8f42a309 in inline_mysql_cond_timedwait (that=0x55fb90b9fa00 , mutex=0x55fb90b9f940 , abstime=0x7f79c7859cb0, src_file=0x55fb8fa0e678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fb8f42a82d in my_service_thread_sleep (control=0x55fb90153d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055fb8f41ecc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055fb8f552fb3 in pfs_spawn_thread (arg=0x55fb91ed0700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f79c6208700 (LWP 218661)): #0 0x00007f79fd125644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb8f0f509a in LinuxAIOHandler::collect (this=0x7f79c6207b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb8f0f55c3 in LinuxAIOHandler::poll (this=0x7f79c6207b00, m1=0x7f79c6207ba8, m2=0x7f79c6207bb0, request=0x7f79c6207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb8f0f5726 in os_aio_linux_handler (global_segment=0, m1=0x7f79c6207ba8, m2=0x7f79c6207bb0, request=0x7f79c6207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb8f0fa24a in os_aio_handler (segment=0, m1=0x7f79c6207ba8, m2=0x7f79c6207bb0, request=0x7f79c6207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb8f34d4a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055fb8f1f3a47 in io_handler_thread (arg=0x55fb9038e160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f79f45d9700 (LWP 218490)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f79f5369272 in fifo_lock_get (q=q@entry=0x55fb91dfbc80) at galerautils/src/gu_fifo.c:262 #2 0x00007f79f53698e1 in gu_fifo_get_head (q=0x55fb91dfbc80, err=0x7f79f45d806c) at galerautils/src/gu_fifo.c:330 #3 0x00007f79f544b43c in gcs_recv (conn=0x55fb91dd5600, action=0x7f79f45d80b0) at gcs/src/gcs.cpp:2024 #4 0x00007f79f547850c in galera::GcsActionSource::process (this=0x55fb91e00020, recv_ctx=0x7f79f45d84d0, exit_loop=@0x7f79f45d829e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f79f5497433 in galera::ReplicatorSMM::async_recv (this=0x55fb91dcd3b0, recv_ctx=0x7f79f45d84d0) at galera/src/replicator_smm.cpp:383 #6 0x000055fb8f6758a8 in wsrep::wsrep_provider_v26::run_applier (this=0x55fb91d43250, applier_ctx=0x7f79f45d84d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055fb8ed0ea69 in wsrep_replication_process (thd=0x7f79d4000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055fb8ed00587 in start_wsrep_THD (arg=0x55fb91e29d00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f79f466f700 (LWP 218488)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f5c0b7f in safe_cond_wait (cond=0x7f79dc017e98, mp=0x7f79dc017de8, file=0x55fb8f72f4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb8ed0e4ba in inline_mysql_cond_wait (that=0x7f79dc017e98, mutex=0x7f79dc017de8, src_file=0x55fb8f72f7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fb8ed1134e in Wsrep_thd_queue::pop_front (this=0x7f79dc017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055fb8ed0fc54 in wsrep_rollback_process (rollbacker=0x7f79dc001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x000055fb8ed00587 in start_wsrep_THD (arg=0x55fb91e28c40) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f79fd554dd5 in start_thread () from /lib6worker[1] mysql-test-run: WARNING: Test reserved for w3 picked up by w1 4/libpthread.so.0 #7 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f79f4e70700 (LWP 218484)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f79f5475377 in gu::Lock::wait (cond=..., this=0x7f79f4e6fbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f79f5475409 in galera::ServiceThd::thd_func (arg=0x55fb91dcdac8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f79f4624700 (LWP 218489)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f5c0b7f in safe_cond_wait (cond=0x7f79d0015938, mp=0x7f79d0015888, file=0x55fb8f72f4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb8ed0e4ba in inline_mysql_cond_wait (that=0x7f79d0015938, mutex=0x7f79d0015888, src_file=0x55fb8f72f7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fb8ed1134e in Wsrep_thd_queue::pop_front (this=0x7f79d0015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055fb8ed100d0 in wsrep_post_rollback_process (post_rollbacker=0x7f79d0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x000055fb8ed00587 in start_wsrep_THD (arg=0x55fb91e29710) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f79e637b700 (LWP 218486)): #0 0x00007f79fb913483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f79f5416a68 in asio::detail::epoll_reactor::run (this=0x55fb91e081c0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f79f541b160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55fb91e08060, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f79f541b412 in asio::detail::task_io_service::run (this=0x55fb91e08060, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f79f5432e6d in asio::io_service::run (this=0x55fb91e06d80) at asio/asio/impl/io_service.ipp:58 #5 0x00007f79f54335c1 in gcomm::AsioProtonet::event_loop (this=0x55fb91e06ce0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f79f545165d in GCommConn::run (this=0x55fb91e077f0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f79f5454709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f79e5b7a700 (LWP 218487)): #0 0x00007f79fd558965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f79f544fbd7 in gu::Lock::wait (cond=..., this=0x7f79e5b79850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f79f545271b in RecvBuf::front (this=this@entry=0x55fb91e079d8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f79f54502b6 in gcomm_recv (backend=, msg=0x55fb91dd58e0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f79f544425c in core_msg_recv (backend=backend@entry=0x55fb91dd5a28, recv_msg=recv_msg@entry=0x55fb91dd58e0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f79f54474d4 in gcs_core_recv (conn=0x55fb91dd5880, recv_act=recv_act@entry=0x7f79e5b79d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f79f544e748 in gcs_recv_thread (arg=0x55fb91dd5600) at gcs/src/gcs.cpp:1398 #7 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f79f5fad700 (LWP 218483)): #0 0x00007f79fd558d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb8f5c0e81 in safe_cond_timedwait (cond=0x55fb90bb21c0 , mp=0x55fb90bb2100 , abstime=0x7f79f5facd70, file=0x55fb8fa52e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fb8f5c1e3b in inline_mysql_cond_timedwait (that=0x55fb90bb21c0 , mutex=0x55fb90bb2100 , abstime=0x7f79f5facd70, src_file=0x55fb8fa52ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fb8f5c299e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055fb8f552fb3 in pfs_spawn_thread (arg=0x55fb91d96de0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f79fd554dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f79fb912ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f79fd965880 (LWP 218481)): #0 0x00007f79fd5599d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055fb8f5bb550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055fb8edf0ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f79fb90820d in poll () from /lib64/libc.so.6 #5 0x000055fb8e97af19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055fb8e97a1b4 in mysqld_main (argc=163, argv=0x55fb91c4f140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055fb8e96e695 in main (argc=23, argv=0x7ffd54ba27d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_insert_select-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_insert_select-innodb/' galera_sr.galera_sr_shutdown_slave 'innodb' w3 [ pass ] 27411 ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera-features#56 galera_sr.GCF-851 galera_sr.GCF-845 galera_sr.GCF-867 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_create_drop galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_ddl_master galera_sr.galera_sr_insert_select Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_transaction_replay 'innodb' w3 [ pass ] 10120 galera_sr.galera_sr_unit_statements 'innodb' w3 [ pass ] 6004 galera_sr.galera_var_ignore_apply_errors_sr 'innodb' w3 [ pass ] 7951 galera_sr.galera_sr_kill_connection 'innodb' w1 [ pass ] 8102 galera_sr.galera_sr_ws_size2 'innodb' w1 [ pass ] 12824 galera_sr.galera_sr_bf_abort 'innodb' w2 [ pass ] 50374 galera_sr.galera_sr_kill_query 'innodb' w1 [ pass ] 14615 galera_sr.mysql-wsrep#215 'innodb' w3 [ fail ] Test ended at 2019-02-27 16:58:27 CURRENT_TEST: galera_sr.mysql-wsrep#215 --- /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/mysql-wsrep#215.result 2019-01-23 13:59:37.387821400 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/mysql-wsrep#215.reject 2019-02-27 16:58:27.057647200 +0200 @@ -115,6 +115,19 @@ INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES (2); connection node_1a; +Timeout in wait_condition.inc for SELECT VARIABLE_VALUE = 3 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_local_cert_failures' +Id User Host db Command Time State Info Progress +3 system user NULL Sleep 38 debug sworker[4] Trying to dump core for [mysqltest - pid: 219721, winpid: 219721] worker[4] Trying to dump core for [mysqld.1 - pid: 219439, winpid: 219439] worker[4] Trying to dump core for [mysqld.2 - pid: 219476, winpid: 219476] ync point: now NULL 0.000 +1 system user NULL Sleep 220 wsrep aborter idle NULL 0.000 +2 system user NULL Sleep 220 Waiting on cond NULL 0.000 +4 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000 +7 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000 +161 root localhost test Sleep 47 NULL 0.000 +162 root localhost:37174 test Query 31 Update INSERT INTO t1 VALUES (2) 0.000 +163 root localhost:37176 test Query 0 Init show full processlist 0.000 SET GLOBAL DEBUG_DBUG = ''; SET DEBUG_SYNC='now SIGNAL signal.wsrep_apply_cb'; connection node_1; mysqltest: Result length mismatch - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.mysql-wsrep#215-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep#215-innodb/' galera_sr.galera_sr_load_data 'innodb' w1 [ pass ] 19873 galera_sr.galera_sr_ws_size 'innodb' w2 [ pass ] 25337 galera_sr.galera_sr_myisam 'innodb' w1 [ pass ] 4562 galera_sr.galera_sr_blob 'innodb' w2 [ pass ] 9510 galera_sr.mysql-wsrep-features#138 'innodb' w3 [ pass ] 6973 galera_sr.mysql-wsrep-features#148 'innodb' w1 [ pass ] 19336 galera_sr.mysql-wsrep-features#14 'innodb' w3 [ pass ] 5782 galera_sr.mysql-wsrep-features#15 'innodb' w1 [ pass ] 5228 galera_sr.mysql-wsrep-features#35 'innodb' w3 [ pass ] 8744 galera_sr.galera_sr_cc_master 'innodb' w2 [ pass ] 25603 galera_sr.mysql-wsrep-features#93 'innodb' w3 [ pass ] 7615 galera_sr.mysql-wsrep-features#96 'innodb' w3 [ pass ] 9036 galera_sr.galera_sr_cc_slave 'innodb' w2 [ pass ] 29781 galera_sr.galera_sr_concurrent 'innodb' w2 [ pass ] 16281 galera_sr.MDEV-18585 'innodb' w3 [ pass ] 7295 galera_sr.galera_sr_conflict 'innodb' w2 [ pass ] 8506 galera_sr.galera_sr_conflict_on_commit 'innodb' w2 [ pass ] 8048 galera_sr.galera_sr_conflict_on_commit2 'innodb' w2 [ pass ] 10309 galera_sr.galera_sr_conflict_with_rollback_master 'innodb' w2 [ pass ] 7494 galera_sr.galera_sr_kill_all_norecovery 'innodb' w3 [ fail ] Test ended at 2019-02-27 17:00:45 CURRENT_TEST: galera_sr.galera_sr_kill_all_norecovery mysqltest: In included file "./include/kill_galera.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/galera_sr_kill_all_norecovery.test at line 32: At line 4: query 'let $_server_id= `SELECT @@server_id`' failed: 1205: Lock wait timeout exceeded; try restarting transaction The result from queries just before the failure was: < snip > connection node_1; connection node_1; connection node_2; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES (2); INSERT INTO t1 VALUES (3); INSERT INTO t1 VALUES (4); INSERT INTO t1 VALUES (5); connection node_2; Killing server ... connection node_1; Killing server ... More results from queries before failure can be found in /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr_kill_all_norecovery.log - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_kill_all_norecovery-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_kill_all_norecovery-innodb/' galera_sr.mysql-wsrep-features#165 'innodb' w1 [ pass ] 119078 galera_sr.mysql-wsrep-features#22 'innodb' w1 [ pass ] 12150 galera_sr.mysql-wsrep-features#27 'innodb' w1 [ pass ] 9563 galera_sr.galera_sr_kill_slave 'innodb' w3 [ pass ] 36789 galera_sr.mysql-wsrep-features#9 'innodb' w1 [ pass ] 25643 galera_sr.GCF-1008 'innodb' w1 [ pass ] 15370 galera_sr.galera_sr_many_fragments 'innodb' w4 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 17:08:08 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr_many_fragments.log == connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) - found 'core.219440' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.1/data/core.219440 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 219440] [New LWP 219442] [New LWP 219444] [New LWP 219448] [New LWP 219449] [New LWP 219452] [New LWP 219453] [New LWP 219454] [New LWP 219458] [New LWP 219460] [New LWP 219461] [New LWP 219465] [New LWP 219467] [New LWP 219470] [New LWP 219472] [New LWP 219469] [New LWP 219468] [New LWP 219466] [New LWP 219464] [New LWP 219463] [New LWP 219474] [New LWP 219473] [New LWP 219471] [New LWP 219462] [New LWP 219450] [New LWP 219447] [New LWP 219446] [New LWP 219459] [New LWP 219441] [New LWP 219443] [New LWP 219445] [New LWP 219451] [New LWP 219695] [New LWP 219455] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f641de619d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f641de619d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055a7865f1550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055a785e26ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f641c21020d in poll () from /lib64/libc.so.6 #5 0x000055a7859b0f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055a7859b01b4 in mysqld_main (argc=162, argv=0x55a78954c140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055a7859a4695 in main (argc=23, argv=0x7ffc0474ba18) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f63ec1a8700 (LWP 219455)): #0 0x00007f641de63d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055a78612c749 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055a78612ca9f in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055a7863751fa in pfs_os_file_flush_func (file=..., src_file=0x55a7869fa588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055a786378055 in fil_flush_low (space=0x55a789a18350, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x000055a7863839f8 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x000055a786300f60 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1095 #7 0x000055a78630b886 in buf_flush_end (buf_pool=0x55a7898f4480, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1924 #8 0x000055a78630ba44 in buf_flush_do_batch (buf_pool=0x55a7898f4480, type=BUF_FLUSH_LIST, min_n=10, lsn_limit=18446744073709551615, n=0x7f63ec1a7b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1993 #9 0x000055a78630e72e in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2794 #10 0x000055a78630f67c in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3203 #11 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f640c189700 (LWP 219695)): #0 0x00007f641de63d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055a78612c749 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055a78612ca9f in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055a7863751fa in pfs_os_file_flush_func (file=..., src_file=0x55a7869fa588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055a786378055 in fil_flush_low (space=0x55a789a186f0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x000055a7863839f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x000055a786107e31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055a786108893 in log_write_up_to (lsn=13747679, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055a786273f5f in trx_flush_log_if_needed_low (lsn=13747679) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000055a786273fa8 in trx_flush_log_if_needed (lsn=13747679, trx=0x7f63efc02390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000055a786275dd1 in trx_commit_complete_for_mysql (trx=0x7f63efc02390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000055a786058ca6 in innobase_commit (hton=0x55a7897e2d40, thd=0x7f639c0c0880, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4658 #12 0x000055a785e2ca1d in commit_one_phase_2 (thd=0x7f639c0c0880, all=true, trans=0x7f639c0c3df8, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1741 #13 0x000055a785e2c91f in ha_commit_one_phase (thd=0x7f639c0c0880, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1721 #14 0x000055a785e2bc98 in ha_commit_trans (thd=0x7f639c0c0880, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1534 #15 0x000055a785c69ac6 in trans_commit (thd=0x7f639c0c0880) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000055a7860368a3 in Wsrep_storage_service::commit (this=0x7f639c3aa660, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x000055a7866a1ba1 in wsrep::transaction::certify_fragment (this=0x7f639c006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #18 0x000055a7866a137b in wsrep::transaction::streaming_step (this=0x7f639c006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #19 0x000055a78669e59a in wsrep::transaction::after_row (this=0x7f639c006bf0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #20 0x000055a785e400b6 in wsrep::client_state::after_row (this=0x7f639c006b80) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x000055a785e28000 in wsrep_after_row (thd=0x7f639c000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #22 0x000055a785e3a0df in wsrep_after_row (thd=0x7f639c000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #23 0x000055a785e3ae15 in handler::ha_write_row (this=0x7f639c2be7a8, buf=0x7f639c2bef68 "\376\001\022") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6673 #24 0x000055a785a9bacf in write_record (thd=0x7f639c000b00, table=0x7f639c2bd940, info=0x7f639c017740) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x000055a785aa0dce in select_insert::send_data (this=0x7f639c0176f8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x000055a785b5edcf in end_send (join=0x7f639c0177a8, join_tab=0x7f639c0a12e8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:21196 #27 0x000055a785ca3eff in JOIN_CACHE::generate_full_extensions (this=0x7f639c0a1e80, rec_ptr=0x7f639c25f710 "\024") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x000055a785ca3c1a in JOIN_CACHE::join_matching_records (this=0x7f639c0a1e80, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x000055a785ca35e6 in JOIN_CACHE::join_records (this=0x7f639c0a1e80, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x000055a785ca3780 in JOIN_CACHE::join_records (this=0x7f639c0a1d50, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x000055a785ca3780 in JOIN_CACHE::join_records (this=0x7f639c0a1c20, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x000055a785b5ba6d in sub_select_cache (join=0x7f639c0177a8, join_tab=0x7f639c0a0790, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19786 #33 0x000055a785b5bc7a in sub_select (join=0x7f639c0177a8, join_tab=0x7f639c0a03c8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19957 #34 0x000055a785b5b445 in do_select (join=0x7f639c0177a8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19548 #35 0x000055a785b32079 in JOIN::exec_inner (this=0x7f639c0177a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4364 #36 0x000055a785b31226 in JOIN::exec (this=0x7f639c0177a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4146 #37 0x000055a785b3290c in mysql_select (thd=0x7f639c000b00, tables=0x7f639c0151d8, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7f639c0176f8, unit=0x7f639c004a18, select_lex=0x7f639c014ad8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4578 #38 0x000055a785b22ea6 in handle_select (thd=0x7f639c000b00, lex=0x7f639c004950, result=0x7f639c0176f8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:424 #39 0x000055a785ae4b44 in mysql_execute_command (thd=0x7f639c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4896 #40 0x000055a785af00f1 in mysql_parse (thd=0x7f639c000b00, rawbuf=0x7f639c014198 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f640c188040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #41 0x000055a785aef720 in wsrep_mysql_parse (thd=0x7f639c000b00, rawbuf=0x7f639c014198 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f640c188040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #42 0x000055a785adb46d in dispatch_command (command=COM_QUERY, thd=0x7f639c000b00, packet=0x7f639c00a761 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #43 0x000055a785ad9cbb in do_command (thd=0x7f639c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #44 0x000055a785c54c9e in do_handle_one_connection (connect=0x55a789af86f0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #45 0x000055a785c549e2 in handle_one_connection (arg=0x55a789af86f0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #46 0x000055a786588fb3 in pfs_spawn_thread (arg=0x55a789651ba0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f63ee1ac700 (LWP 219451)): #0 0x00007f641da2d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a78612b09a in LinuxAIOHandler::collect (this=0x7f63ee1abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a78612b5c3 in LinuxAIOHandler::poll (this=0x7f63ee1abb00, m1=0x7f63ee1abba8, m2=0x7f63ee1abbb0, request=0x7f63ee1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a78612b726 in os_aio_linux_handler (global_segment=2, m1=0x7f63ee1abba8, m2=0x7f63ee1abbb0, request=0x7f63ee1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a78613024a in os_aio_handler (segment=2, m1=0x7f63ee1abba8, m2=0x7f63ee1abbb0, request=0x7f63ee1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a7863834a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055a786229a47 in io_handler_thread (arg=0x55a7873c4170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f6414576700 (LWP 219445)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a7865f6b7f in safe_cond_wait (cond=0x7f63f8015938, mp=0x7f63f8015888, file=0x55a7867654c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a785d444ba in inline_mysql_cond_wait (that=0x7f63f8015938, mutex=0x7f63f8015888, src_file=0x55a7867657e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055a785d4734e in Wsrep_thd_queue::pop_front (this=0x7f63f8015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055a785d45c54 in wsrep_rollback_process (rollbacker=0x7f63f8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x000055a785d36587 in start_wsrep_THD (arg=0x55a789716680) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f640e37c700 (LWP 219443)): #0 0x00007f641c21b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6415d1ea68 in asio::detail::epoll_reactor::run (this=0x55a789707200, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f6415d23160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55a7897070a0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f6415d23412 in asio::detail::task_io_service::run (this=0x55a7897070a0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f6415d3ae6d in asio::io_service::run (this=0x55a7897061a0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f6415d3b5c1 in gcomm::AsioProtonet::event_loop (this=0x55a789706100, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f6415d5965d in GCommConn::run (this=0x55a789706880) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f6415d5c709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f64168b5700 (LWP 219441)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a7865f6e81 in safe_cond_timedwait (cond=0x55a787be81c0 , mp=0x55a787be8100 , abstime=0x7f64168b4d70, file=0x55a786a88e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055a7865f7e3b in inline_mysql_cond_timedwait (that=0x55a787be81c0 , mutex=0x55a787be8100 , abstime=0x7f64168b4d70, src_file=0x55a786a88ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055a7865f899e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055a786588fb3 in pfs_spawn_thread (arg=0x55a789693d40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f63ea7a0700 (LWP 219459)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137698 in os_event::timed_wait (this=0x55a78980eb80, abstime=0x7f63ea79fba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a7861379af in os_event::wait_time_low (this=0x55a78980eb80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a786137b4f in os_event_wait_time_low (event=0x55a78980eb80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a78622612a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f641452b700 (LWP 219446)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a7865f6b7f in safe_cond_wait (cond=0x7f63fc015938, mp=0x7f63fc015888, file=0x55a7867654c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a785d444ba in inline_mysql_cond_wait (that=0x7f63fc015938, mutex=0x7f63fc015888, src_file=0x55a7867657e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055a785d4734e in Wsrep_thd_queue::pop_front (this=0x7f63fc015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055a785d460d0 in wsrep_post_rollback_process (post_rollbacker=0x7f63fc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x000055a785d36587 in start_wsrep_THD (arg=0x55a7897009d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f64144e0700 (LWP 219447)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6415c71272 in fifo_lock_get (q=q@entry=0x55a7896f8d90) at galerautils/src/gu_fifo.c:262 #2 0x00007f6415c718e1 in gu_fifo_get_head (q=0x55a7896f8d90, err=0x7f64144df06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f6415d5343c in gcs_recv (conn=0x55a7896d2710, action=0x7f64144df0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f6415d8050c in galera::GcsActionSource::process (this=0x55a7896d20f0, recv_ctx=0x7f64144df4d0, exit_loop=@0x7f64144df29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f6415d9f433 in galera::ReplicatorSMM::async_recv (this=0x55a7896ca420, recv_ctx=0x7f64144df4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055a7866ab8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x55a789651760, applier_ctx=0x7f64144df4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055a785d44a69 in wsrep_replication_process (thd=0x7f6404001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055a785d36587 in start_wsrep_THD (arg=0x55a78971f600) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f63ee9ad700 (LWP 219450)): #0 0x00007f641da2d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a78612b09a in LinuxAIOHandler::collect (this=0x7f63ee9acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a78612b5c3 in LinuxAIOHandler::poll (this=0x7f63ee9acb00, m1=0x7f63ee9acba8, m2=0x7f63ee9acbb0, request=0x7f63ee9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a78612b726 in os_aio_linux_handler (global_segment=1, m1=0x7f63ee9acba8, m2=0x7f63ee9acbb0, request=0x7f63ee9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a78613024a in os_aio_handler (segment=1, m1=0x7f63ee9acba8, m2=0x7f63ee9acbb0, request=0x7f63ee9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a7863834a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055a786229a47 in io_handler_thread (arg=0x55a7873c4168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f63e8f9d700 (LWP 219462)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137698 in os_event::timed_wait (this=0x55a7899c9570, abstime=0x7f63e8f9cb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a7861379af in os_event::wait_time_low (this=0x55a7899c9570, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a786137b4f in os_event_wait_time_low (event=0x55a7899c9570, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a7862864c9 in ib_wqueue_timedwait (wq=0x55a7899c9440, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055a7863c773a in fts_optimize_thread (arg=0x55a7899c9440) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f63b77fe700 (LWP 219471)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a7865f6b7f in safe_cond_wait (cond=0x7f63b77fdc70, mp=0x7f63b77fdcb0, file=0x55a7868cd920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a78604dcad in inline_mysql_cond_wait (that=0x7f63b77fdc70, mutex=0x7f63b77fdcb0, src_file=0x55a7868cf348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055a78605279b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055a786588fb3 in pfs_spawn_thread (arg=0x55a7899b4100) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f6414086700 (LWP 219473)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a7865f6b7f in safe_cond_wait (cond=0x55a7873484e0 , mp=0x55a78734b800 , file=0x55a7866c2488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055a7859d5fe2 in inline_mysql_cond_wait (that=0x55a7873484e0 , mutex=0x55a78734b800 , src_file=0x55a7866c2ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055a7859d7874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055a786588fb3 in pfs_spawn_thread (arg=0x55a789b8f670) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f640c1d4700 (LWP 219474)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a7865f6e81 in safe_cond_timedwait (cond=0x55a7873484a0 , mp=0x55a78734ae40 , abstime=0x7f640c1d3c60, file=0x55a7866b2b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055a7859a5015 in inline_mysql_cond_timedwait (that=0x55a7873484a0 , mutex=0x55a78734ae40 , abstime=0x7f640c1d3c60, src_file=0x55a7866b4b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055a7859a9b08 in cache_thread (thd=0x7f63a0000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x000055a7859a9db6 in one_thread_per_connection_end (thd=0x7f63a0000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x000055a785c54d28 in do_handle_one_connection (connect=0x55a789b1a920) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055a785c549e2 in handle_one_connection (arg=0x55a789b1a920) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055a786588fb3 in pfs_spawn_thread (arg=0x55a789ba20c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f63dbfff700 (LWP 219463)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137d7d in os_event::wait (this=0x55a7899a6dc0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a786137803 in os_event::wait_low (this=0x55a7899a6dc0, reset_sig_count=3214) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a786137b74 in os_event_wait_low (event=0x55a7899a6dc0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a7861083c3 in log_write_up_to (lsn=13747739, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000055a7861089c3 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #6 0x000055a78622655e in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #7 0x000055a786226b21 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #8 0x000055a78622767f in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #9 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f63db7fe700 (LWP 219464)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137d7d in os_event::wait (this=0x55a78980e980) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a786137803 in os_event::wait_low (this=0x55a78980e980, reset_sig_count=2238) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a786137b74 in os_event_wait_low (event=0x55a78980e980, reset_sig_count=2238) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a786223f21 in srv_resume_thread (slot=0x55a787185948 , sig_count=2238, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a786227f32 in srv_purge_coordinator_suspend (slot=0x55a787185948 , rseg_history_len=8) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055a7862281c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f63da7fc700 (LWP 219466)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137d7d in os_event::wait (this=0x55a78980ea80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a786137803 in os_event::wait_low (this=0x55a78980ea80, reset_sig_count=8596) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a786137b74 in os_event_wait_low (event=0x55a78980ea80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a786223f21 in srv_resume_thread (slot=0x55a7871859b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a786227b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f63d97fa700 (LWP 219468)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137d7d in os_event::wait (this=0x55a78980ec80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a786137803 in os_event::wait_low (this=0x55a78980ec80, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a786137b74 in os_event_wait_low (event=0x55a78980ec80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a78630419b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f63d8ff9700 (LWP 219469)): #0 0x00007f641de63e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055a78613807f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055a7862deb55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f64140d1700 (LWP 219472)): #0 0x00007f641c15440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055a7859a4714 in my_sigwait (set=0x7f64140d0cf0, sig=0x7f64140d0cb4, code=0x7f64140d0cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055a7859aaa71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x000055a786588fb3 in pfs_spawn_thread (arg=0x55a7897d1f60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f63b7fff700 (LWP 219470)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137d7d in os_event::wait (this=0x55a78980ed80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a786137803 in os_event::wait_low (this=0x55a78980ed80, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a786137b74 in os_event_wait_low (event=0x55a78980ed80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a7862eccbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f63d9ffb700 (LWP 219467)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137d7d in os_event::wait (this=0x55a78980eb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a786137803 in os_event::wait_low (this=0x55a78980eb00, reset_sig_count=8123) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a786137b74 in os_event_wait_low (event=0x55a78980eb00, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a786223f21 in srv_resume_thread (slot=0x55a7871859f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a786227b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f63daffd700 (LWP 219465)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137d7d in os_event::wait (this=0x55a78980ea00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055a786137803 in os_event::wait_low (this=0x55a78980ea00, reset_sig_count=10055) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055a786137b74 in os_event_wait_low (event=0x55a78980ea00, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055a786223f21 in srv_resume_thread (slot=0x55a787185980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055a786227b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f63e979e700 (LWP 219461)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137698 in os_event::timed_wait (this=0x55a789a18d30, abstime=0x7f63e979dd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a7861379af in os_event::wait_time_low (this=0x55a789a18d30, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a786137b4f in os_event_wait_time_low (event=0x55a789a18d30, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a78636d0e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f63e9f9f700 (LWP 219460)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137698 in os_event::timed_wait (this=0x55a78980ec00, abstime=0x7f63e9f9ed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a7861379af in os_event::wait_time_low (this=0x55a78980ec00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a786137b4f in os_event_wait_time_low (event=0x55a78980ec00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a786225c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f63eafa1700 (LWP 219458)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a786137698 in os_event::timed_wait (this=0x55a7899e6c70, abstime=0x7f63eafa0d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055a7861379af in os_event::wait_time_low (this=0x55a7899e6c70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055a786137b4f in os_event_wait_time_low (event=0x55a7899e6c70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055a7861044f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f63ec9a9700 (LWP 219454)): #0 0x00007f641da2d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a78612b09a in LinuxAIOHandler::collect (this=0x7f63ec9a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a78612b5c3 in LinuxAIOHandler::poll (this=0x7f63ec9a8b00, m1=0x7f63ec9a8ba8, m2=0x7f63ec9a8bb0, request=0x7f63ec9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a78612b726 in os_aio_linux_handler (global_segment=5, m1=0x7f63ec9a8ba8, m2=0x7f63ec9a8bb0, request=0x7f63ec9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a78613024a in os_aio_handler (segment=5, m1=0x7f63ec9a8ba8, m2=0x7f63ec9a8bb0, request=0x7f63ec9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a7863834a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055a786229a47 in io_handler_thread (arg=0x55a7873c4188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f63ed1aa700 (LWP 219453)): #0 0x00007f641da2d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a78612b09a in LinuxAIOHandler::collect (this=0x7f63ed1a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a78612b5c3 in LinuxAIOHandler::poll (this=0x7f63ed1a9b00, m1=0x7f63ed1a9ba8, m2=0x7f63ed1a9bb0, request=0x7f63ed1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a78612b726 in os_aio_linux_handler (global_segment=4, m1=0x7f63ed1a9ba8, m2=0x7f63ed1a9bb0, request=0x7f63ed1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a78613024a in os_aio_handler (segment=4, m1=0x7f63ed1a9ba8, m2=0x7f63ed1a9bb0, request=0x7f63ed1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a7863834a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055a786229a47 in io_handler_thread (arg=0x55a7873c4180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f63ed9ab700 (LWP 219452)): #0 0x00007f641da2d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a78612b09a in LinuxAIOHandler::collect (this=0x7f63ed9aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a78612b5c3 in LinuxAIOHandler::poll (this=0x7f63ed9aab00, m1=0x7f63ed9aaba8, m2=0x7f63ed9aabb0, request=0x7f63ed9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a78612b726 in os_aio_linux_handler (global_segment=3, m1=0x7f63ed9aaba8, m2=0x7f63ed9aabb0, request=0x7f63ed9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a78613024a in os_aio_handler (segment=3, m1=0x7f63ed9aaba8, m2=0x7f63ed9aabb0, request=0x7f63ed9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a7863834a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055a786229a47 in io_handler_thread (arg=0x55a7873c4178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f63ef1ae700 (LWP 219449)): #0 0x00007f641da2d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055a78612b09a in LinuxAIOHandler::collect (this=0x7f63ef1adb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055a78612b5c3 in LinuxAIOHandler::poll (this=0x7f63ef1adb00, m1=0x7f63ef1adba8, m2=0x7f63ef1adbb0, request=0x7f63ef1adbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055a78612b726 in os_aio_linux_handler (global_segment=0, m1=0x7f63ef1adba8, m2=0x7f63ef1adbb0, request=0x7f63ef1adbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055a78613024a in os_aio_handler (segment=0, m1=0x7f63ef1adba8, m2=0x7f63ef1adbb0, request=0x7f63ef1adbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055a7863834a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055a786229a47 in io_handler_thread (arg=0x55a7873c4160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f640cbd5700 (LWP 219448)): #0 0x00007f641de60d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055a7865f6e81 in safe_cond_timedwait (cond=0x55a787bd5a00 , mp=0x55a787bd5940 , abstime=0x7f640cbd4cb0, file=0x55a786a44630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055a786460309 in inline_mysql_cond_timedwait (that=0x55a787bd5a00 , mutex=0x55a787bd5940 , abstime=0x7f640cbd4cb0, src_file=0x55a786a44678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055a78646082d in my_service_thread_sleep (control=0x55a787189d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055a786454cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055a786588fb3 in pfs_spawn_thread (arg=0x55a7897c6a10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f640db7b700 (LWP 219444)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6415d57bd7 in gu::Lock::wait (cond=..., this=0x7f640db7a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6415d5a71b in RecvBuf::front (this=this@entry=0x55a789706a68, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f6415d582b6 in gcomm_recv (backend=, msg=0x55a7896d29f0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f6415d4c25c in core_msg_recv (backend=backend@entry=0x55a7896d2b38, recv_msg=recv_msg@entry=0x55a7896d29f0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f6415d4f4d4 in gcs_core_recv (conn=0x55a7896d2990, recv_act=recv_act@entry=0x7f640db7ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f6415d56748 in gcs_recv_thread (arg=0x55a7896d2710) at gcs/src/gcs.cpp:1398 #7 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f6414d77700 (LWP 219442)): #0 0x00007f641de60965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6415d7d377 in gu::Lock::wait (cond=..., this=0x7f6414d76bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6415d7d409 in galera::ServiceThd::thd_func (arg=0x55a7896cab38) at galera/src/galera_service_thd.cpp:30 #3 0x00007f641de5cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f641c21aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f641e26d880 (LWP 219440)): #0 0x00007f641de619d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055a7865f1550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055a785e26ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f641c21020d in poll () from /lib64/libc.so.6 #5 0x000055a7859b0f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055a7859b01b4 in mysqld_main (argc=162, argv=0x55a78954c140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055a7859a4695 in main (argc=23, argv=0x7ffc0474ba18) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.219477' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.2/data/core.219477 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 219477] [New LWP 219654] [New LWP 219478] [New LWP 219479] [New LWP 219651] [New LWP 219653] [New LWP 219655] [New LWP 219662] [New LWP 219663] [New LWP 219665] [New LWP 219672] [New LWP 219674] [New LWP 219671] [New LWP 219668] [New LWP 219664] [New LWP 219666] [New LWP 219667] [New LWP 219669] [New LWP 219676] [New LWP 219673] [New LWP 219670] [New LWP 219675] [New LWP 219657] [New LWP 219661] [New LWP 219656] [New LWP 219660] [New LWP 219652] [New LWP 219650] [New LWP 219480] [New LWP 219482] [New LWP 219483] [New LWP 219484] [New LWP 219481] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fb813e509d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fb813e509d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557e69783550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557e68fb8ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb8121ff20d in poll () from /lib64/libc.so.6 #5 0x0000557e68b42f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000557e68b421b4 in mysqld_main (argc=163, argv=0x557e6ccac140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000557e68b36695 in main (argc=23, argv=0x7fffcbe9e498) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fb7f4a5e700 (LWP 219481)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb807ca2bd7 in gu::Lock::wait (cond=..., this=0x7fb7f4a5d850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb807ca571b in RecvBuf::front (this=this@entry=0x557e6ce36238, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fb807ca32b6 in gcomm_recv (backend=, msg=0x557e6ce53b40, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fb807c9725c in core_msg_recv (backend=backend@entry=0x557e6ce53c88, recv_msg=recv_msg@entry=0x557e6ce53b40, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fb807c9a4d4 in gcs_core_recv (conn=0x557e6ce53ae0, recv_act=recv_act@entry=0x7fb7f4a5dd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fb807ca1748 in gcs_recv_thread (arg=0x557e6ce53860) at gcs/src/gcs.cpp:1398 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fb7fca13700 (LWP 219484)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb807bbc272 in fifo_lock_get (q=q@entry=0x557e6ce79e60) at galerautils/src/gu_fifo.c:262 #2 0x00007fb807bbc8e1 in gu_fifo_get_head (q=0x557e6ce79e60, err=0x7fb7fca1206c) at galerautils/src/gu_fifo.c:330 #3 0x00007fb807c9e43c in gcs_recv (conn=0x557e6ce53860, action=0x7fb7fca120b0) at gcs/src/gcs.cpp:2024 #4 0x00007fb807ccb50c in galera::GcsActionSource::process (this=0x557e6ce31ca0, recv_ctx=0x7fb7fca124d0, exit_loop=@0x7fb7fca1229e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fb807cea433 in galera::ReplicatorSMM::async_recv (this=0x557e6ce2a3b0, recv_ctx=0x7fb7fca124d0) at galera/src/replicator_smm.cpp:383 #6 0x0000557e6983d8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x557e6cda0250, applier_ctx=0x7fb7fca124d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000557e68ed6a69 in wsrep_replication_process (thd=0x7fb7e0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000557e68ec8587 in start_wsrep_THD (arg=0x557e6cdf50f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fb7fca5e700 (LWP 219483)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e69788b7f in safe_cond_wait (cond=0x7fb7f0015938, mp=0x7fb7f0015888, file=0x557e698f74c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557e68ed64ba in inline_mysql_cond_wait (that=0x7fb7f0015938, mutex=0x7fb7f0015888, src_file=0x557e698f77e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557e68ed934e in Wsrep_thd_queue::pop_front (this=0x7fb7f0015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000557e68ed80d0 in wsrep_post_rollback_process (post_rollbacker=0x7fb7f0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x0000557e68ec8587 in start_wsrep_THD (arg=0x557e6ce38530) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fb80c082700 (LWP 219482)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e69788b7f in safe_cond_wait (cond=0x7fb7e8017e98, mp=0x7fb7e8017de8, file=0x557e698f74c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557e68ed64ba in inline_mysql_cond_wait (that=0x7fb7e8017e98, mutex=0x7fb7e8017de8, src_file=0x557e698f77e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557e68ed934e in Wsrep_thd_queue::pop_front (this=0x7fb7e8017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000557e68ed7c54 in wsrep_rollback_process (rollbacker=0x7fb7e8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x0000557e68ec8587 in start_wsrep_THD (arg=0x557e6ce4ea80) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fb7fd25f700 (LWP 219480)): #0 0x00007fb81220a483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb807c69a68 in asio::detail::epoll_reactor::run (this=0x557e6ce36da0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fb807c6e160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x557e6ce36bd0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fb807c6e412 in asio::detail::task_io_service::run (this=0x557e6ce36bd0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fb807c85e6d in asio::io_service::run (this=0x557e6ce35f30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fb807c865c1 in gcomm::AsioProtonet::event_loop (this=0x557e6ce35e90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fb807ca465d in GCommConn::run (this=0x557e6ce36050) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fb807ca7709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fb7f6e72700 (LWP 219650)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e69788e81 in safe_cond_timedwait (cond=0x557e6ad67a00 , mp=0x557e6ad67940 , abstime=0x7fb7f6e71cb0, file=0x557e69bd6630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557e695f2309 in inline_mysql_cond_timedwait (that=0x557e6ad67a00 , mutex=0x557e6ad67940 , abstime=0x7fb7f6e71cb0, src_file=0x557e69bd6678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557e695f282d in my_service_thread_sleep (control=0x557e6a31bd00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000557e695e6cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000557e6971afb3 in pfs_spawn_thread (arg=0x557e6cf2cf90) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fb7effff700 (LWP 219652)): #0 0x00007fb813a1c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557e692bd09a in LinuxAIOHandler::collect (this=0x7fb7efffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557e692bd5c3 in LinuxAIOHandler::poll (this=0x7fb7efffeb00, m1=0x7fb7efffeba8, m2=0x7fb7efffebb0, request=0x7fb7efffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557e692bd726 in os_aio_linux_handler (global_segment=1, m1=0x7fb7efffeba8, m2=0x7fb7efffebb0, request=0x7fb7efffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557e692c224a in os_aio_handler (segment=1, m1=0x7fb7efffeba8, m2=0x7fb7efffebb0, request=0x7fb7efffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557e695154a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000557e693bba47 in io_handler_thread (arg=0x557e6a556168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fb7c7fff700 (LWP 219660)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9698 in os_event::timed_wait (this=0x557e6d14d450, abstime=0x7fb7c7ffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557e692c99af in os_event::wait_time_low (this=0x557e6d14d450, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557e692c9b4f in os_event_wait_time_low (event=0x557e6d14d450, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557e692964f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fb7edffb700 (LWP 219656)): #0 0x00007fb813a1c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557e692bd09a in LinuxAIOHandler::collect (this=0x7fb7edffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557e692bd5c3 in LinuxAIOHandler::poll (this=0x7fb7edffab00, m1=0x7fb7edffaba8, m2=0x7fb7edffabb0, request=0x7fb7edffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557e692bd726 in os_aio_linux_handler (global_segment=5, m1=0x7fb7edffaba8, m2=0x7fb7edffabb0, request=0x7fb7edffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557e692c224a in os_aio_handler (segment=5, m1=0x7fb7edffaba8, m2=0x7fb7edffabb0, request=0x7fb7edffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557e695154a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000557e693bba47 in io_handler_thread (arg=0x557e6a556188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fb7ecff9700 (LWP 219661)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9698 in os_event::timed_wait (this=0x557e6cf74f20, abstime=0x7fb7ecff8ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557e692c99af in os_event::wait_time_low (this=0x557e6cf74f20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557e692c9b4f in os_event_wait_time_low (event=0x557e6cf74f20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557e693b812a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fb7ed7fa700 (LWP 219657)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9698 in os_event::timed_wait (this=0x557e6cf750a0, abstime=0x7fb7ed7f9af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557e692c99af in os_event::wait_time_low (this=0x557e6cf750a0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557e692c9b4f in os_event_wait_time_low (event=0x557e6cf750a0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557e694a01fe in pc_sleep_if_needed (next_loop_time=1551280033307, sig_count=2, cur_time=1551280032307) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x0000557e694a1137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fb7f57d5700 (LWP 219675)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e69788b7f in safe_cond_wait (cond=0x557e6a4da4e0 , mp=0x557e6a4dd800 , file=0x557e69854488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557e68b67fe2 in inline_mysql_cond_wait (that=0x557e6a4da4e0 , mutex=0x557e6a4dd800 , src_file=0x557e69854ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000557e68b69874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000557e6971afb3 in pfs_spawn_thread (arg=0x557e6d2e6eb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fb7b37fe700 (LWP 219670)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9d7d in os_event::wait (this=0x557e6cf75020) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557e692c9803 in os_event::wait_low (this=0x557e6cf75020, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557e692c9b74 in os_event_wait_low (event=0x557e6cf75020, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557e6949619b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fb7b1ffb700 (LWP 219673)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e69788b7f in safe_cond_wait (cond=0x7fb7b1ffac70, mp=0x7fb7b1ffacb0, file=0x557e69a5f920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000557e691dfcad in inline_mysql_cond_wait (that=0x7fb7b1ffac70, mutex=0x7fb7b1ffacb0, src_file=0x557e69a61348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000557e691e479b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000557e6971afb3 in pfs_spawn_thread (arg=0x557e6d12ff70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fb7f578a700 (LWP 219676)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e69788e81 in safe_cond_timedwait (cond=0x557e6a4da4a0 , mp=0x557e6a4dce40 , abstime=0x7fb7f5789c60, file=0x557e69844b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557e68b37015 in inline_mysql_cond_timedwait (that=0x557e6a4da4a0 , mutex=0x557e6a4dce40 , abstime=0x7fb7f5789c60, src_file=0x557e69846b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557e68b3bb08 in cache_thread (thd=0x7fb7d801b560) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x0000557e68b3bdb6 in one_thread_per_connection_end (thd=0x7fb7d801b560, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x0000557e68de6d28 in do_handle_one_connection (connect=0x557e6d2e7640) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000557e68de69e2 in handle_one_connection (arg=0x557e6d2e7640) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000557e6971afb3 in pfs_spawn_thread (arg=0x557e6d3069f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fb7ab7fe700 (LWP 219669)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9d7d in os_event::wait (this=0x557e6cf74ea0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557e692c9803 in os_event::wait_low (this=0x557e6cf74ea0, reset_sig_count=8101) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557e692c9b74 in os_event_wait_low (event=0x557e6cf74ea0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557e693b5f21 in srv_resume_thread (slot=0x557e6a3179f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557e693b9b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fb7c4ff9700 (LWP 219667)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9d7d in os_event::wait (this=0x557e6cf74da0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557e692c9803 in os_event::wait_low (this=0x557e6cf74da0, reset_sig_count=9815) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557e692c9b74 in os_event_wait_low (event=0x557e6cf74da0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557e693b5f21 in srv_resume_thread (slot=0x557e6a317980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557e693b9b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fb7c57fa700 (LWP 219666)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9d7d in os_event::wait (this=0x557e6cf74d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557e692c9803 in os_event::wait_low (this=0x557e6cf74d20, reset_sig_count=2301) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557e692c9b74 in os_event_wait_low (event=0x557e6cf74d20, reset_sig_count=2301) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557e693b5f21 in srv_resume_thread (slot=0x557e6a317948 , sig_count=2301, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557e693b9f32 in srv_purge_coordinator_suspend (slot=0x557e6a317948 , rseg_history_len=36) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000557e693ba1c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fb7c67fc700 (LWP 219664)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9698 in os_event::timed_wait (this=0x557e6d11e060, abstime=0x7fb7c67fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557e692c99af in os_event::wait_time_low (this=0x557e6d11e060, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557e692c9b4f in os_event_wait_time_low (event=0x557e6d11e060, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557e694184c9 in ib_wqueue_timedwait (wq=0x557e6d11df30, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000557e6955973a in fts_optimize_thread (arg=0x557e6d11df30) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fb7b3fff700 (LWP 219668)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9d7d in os_event::wait (this=0x557e6cf74e20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557e692c9803 in os_event::wait_low (this=0x557e6cf74e20, reset_sig_count=8630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557e692c9b74 in os_event_wait_low (event=0x557e6cf74e20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557e693b5f21 in srv_resume_thread (slot=0x557e6a3179b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000557e693b9b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fb7b2ffd700 (LWP 219671)): #0 0x00007fb813e52e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000557e692ca07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557e69470b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fb7f5820700 (LWP 219674)): #0 0x00007fb81214340a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000557e68b36714 in my_sigwait (set=0x7fb7f581fcf0, sig=0x7fb7f581fcb4, code=0x7fb7f581fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000557e68b3ca71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x0000557e6971afb3 in pfs_spawn_thread (arg=0x557e6cf371f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fb7b27fc700 (LWP 219672)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9d7d in os_event::wait (this=0x557e6cf75120) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000557e692c9803 in os_event::wait_low (this=0x557e6cf75120, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000557e692c9b74 in os_event_wait_low (event=0x557e6cf75120, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000557e6947ecbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fb7c5ffb700 (LWP 219665)): #0 0x00007fb813e52e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000557e692ca07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000557e693b9489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000557e693b95b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fb7c6ffd700 (LWP 219663)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9698 in os_event::timed_wait (this=0x557e6d17f510, abstime=0x7fb7c6ffcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557e692c99af in os_event::wait_time_low (this=0x557e6d17f510, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557e692c9b4f in os_event_wait_time_low (event=0x557e6d17f510, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557e694ff0e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fb7c77fe700 (LWP 219662)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e692c9698 in os_event::timed_wait (this=0x557e6cf74fa0, abstime=0x7fb7c77fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000557e692c99af in os_event::wait_time_low (this=0x557e6cf74fa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000557e692c9b4f in os_event_wait_time_low (event=0x557e6cf74fa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000557e693b7c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb7ee7fc700 (LWP 219655)): #0 0x00007fb813a1c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557e692bd09a in LinuxAIOHandler::collect (this=0x7fb7ee7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557e692bd5c3 in LinuxAIOHandler::poll (this=0x7fb7ee7fbb00, m1=0x7fb7ee7fbba8, m2=0x7fb7ee7fbbb0, request=0x7fb7ee7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557e692bd726 in os_aio_linux_handler (global_segment=4, m1=0x7fb7ee7fbba8, m2=0x7fb7ee7fbbb0, request=0x7fb7ee7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557e692c224a in os_aio_handler (segment=4, m1=0x7fb7ee7fbba8, m2=0x7fb7ee7fbbb0, request=0x7fb7ee7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557e695154a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000557e693bba47 in io_handler_thread (arg=0x557e6a556180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb7ef7fe700 (LWP 219653)): #0 0x00007fb813a1c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557e692bd09a in LinuxAIOHandler::collect (this=0x7fb7ef7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557e692bd5c3 in LinuxAIOHandler::poll (this=0x7fb7ef7fdb00, m1=0x7fb7ef7fdba8, m2=0x7fb7ef7fdbb0, request=0x7fb7ef7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557e692bd726 in os_aio_linux_handler (global_segment=2, m1=0x7fb7ef7fdba8, m2=0x7fb7ef7fdbb0, request=0x7fb7ef7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557e692c224a in os_aio_handler (segment=2, m1=0x7fb7ef7fdba8, m2=0x7fb7ef7fdbb0, request=0x7fb7ef7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557e695154a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000557e693bba47 in io_handler_thread (arg=0x557e6a556170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb7f5620700 (LWP 219651)): #0 0x00007fb813a1c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557e692bd09a in LinuxAIOHandler::collect (this=0x7fb7f561fb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557e692bd5c3 in LinuxAIOHandler::poll (this=0x7fb7f561fb00, m1=0x7fb7f561fba8, m2=0x7fb7f561fbb0, request=0x7fb7f561fbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557e692bd726 in os_aio_linux_handler (global_segment=0, m1=0x7fb7f561fba8, m2=0x7fb7f561fbb0, request=0x7fb7f561fbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557e692c224a in os_aio_handler (segment=0, m1=0x7fb7f561fba8, m2=0x7fb7f561fbb0, request=0x7fb7f561fbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0fiworker[1] Trying to dump core for [mysqltest - pid: 223987, winpid: 223987] worker[1] Trying to dump core for [mysqld.1 - pid: 220938, winpid: 220938] worker[1] Trying to dump core for [mysqld.2 - pid: 223478, winpid: 223478] worker[3] Trying to dump core for [mysqltest - pid: 223969, winpid: 223969] worker[3] Trying to dump core for [mysqld.1 - pid: 223680, winpid: 223680] worker[3] Trying to dump core for [mysqld.2 - pid: 223746, winpid: 223746] le.cc:5665 #5 0x0000557e695154a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000557e693bba47 in io_handler_thread (arg=0x557e6a556160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fb7ff6e3700 (LWP 219479)): #0 0x00007fb813e4f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb807cc8377 in gu::Lock::wait (cond=..., this=0x7fb7ff6e2bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb807cc8409 in galera::ServiceThd::thd_func (arg=0x557e6ce2aac8) at galera/src/galera_service_thd.cpp:30 #3 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fb80c8a4700 (LWP 219478)): #0 0x00007fb813e4fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000557e69788e81 in safe_cond_timedwait (cond=0x557e6ad7a1c0 , mp=0x557e6ad7a100 , abstime=0x7fb80c8a3d70, file=0x557e69c1ae90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000557e69789e3b in inline_mysql_cond_timedwait (that=0x557e6ad7a1c0 , mutex=0x557e6ad7a100 , abstime=0x7fb80c8a3d70, src_file=0x557e69c1aed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000557e6978a99e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000557e6971afb3 in pfs_spawn_thread (arg=0x557e6cdf3de0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb7eeffd700 (LWP 219654)): #0 0x00007fb813a1c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000557e692bd09a in LinuxAIOHandler::collect (this=0x7fb7eeffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000557e692bd5c3 in LinuxAIOHandler::poll (this=0x7fb7eeffcb00, m1=0x7fb7eeffcba8, m2=0x7fb7eeffcbb0, request=0x7fb7eeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000557e692bd726 in os_aio_linux_handler (global_segment=3, m1=0x7fb7eeffcba8, m2=0x7fb7eeffcbb0, request=0x7fb7eeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000557e692c224a in os_aio_handler (segment=3, m1=0x7fb7eeffcba8, m2=0x7fb7eeffcbb0, request=0x7fb7eeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000557e695154a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000557e693bba47 in io_handler_thread (arg=0x557e6a556178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb813e4bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb812209ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb81425c880 (LWP 219477)): #0 0x00007fb813e509d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000557e69783550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000557e68fb8ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb8121ff20d in poll () from /lib64/libc.so.6 #5 0x0000557e68b42f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000557e68b421b4 in mysqld_main (argc=163, argv=0x557e6ccac140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000557e68b36695 in main (argc=23, argv=0x7fffcbe9e498) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_many_fragments-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery galera_sr.galera_sr_many_fragments Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1051 'innodb' w4 [ pass ] 9014 galera_sr.galera_sr_mysqldump_sst 'innodb' w2 [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-27 17:10:42 line 2019-02-27 17:07:18 14 [Warning] WSREP: no THD for trx: 50 2019-02-27 17:07:18 14 [Warning] WSREP: no THD for trx: 50 ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/mysqld.2.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_mysqldump_sst-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' galera_sr.GCF-1043A 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 17:18:15 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/GCF-1043A.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 REPLACE INTO t1 VALUES (1,'y'),(2,'x') REPLACE INTO t1 VALUES (1,'y'),(2,'y'),(3,'y') connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.220939' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.GCF-1043A-innodb/mysqld.1/data/core.220939 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 220939] [New LWP 220945] [New LWP 220942] [New LWP 220944] [New LWP 220943] [New LWP 220950] [New LWP 220947] [New LWP 220962] [New LWP 220956] [New LWP 220952] [New LWP 220963] [New LWP 220958] [New LWP 220953] [New LWP 220972] [New LWP 220959] [New LWP 220961] [New LWP 220983] [New LWP 220960] [New LWP 220967] [New LWP 220969] [New LWP 220971] [New LWP 220968] [New LWP 221763] [New LWP 220965] [New LWP 221762] [New LWP 220951] [New LWP 220954] [New LWP 220970] [New LWP 220966] [New LWP 220964] [New LWP 220955] [New LWP 220984] [New LWP 220949] [New LWP 220946] [New LWP 220948] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fdd0ac209d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fdd0ac209d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005590e6488550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005590e5cbdebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdd08fcf20d in poll () from /lib64/libc.so.6 #5 0x00005590e5847f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005590e58471b4 in mysqld_main (argc=162, argv=0x5590e7b9c140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005590e583b695 in main (argc=23, argv=0x7ffedab7c3b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7fdcfa5e6700 (LWP 220948)): #0 0x00007fdd0ac22d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005590e5fc3749 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005590e5fc3a9f in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005590e620c1fa in pfs_os_file_flush_func (file=..., src_file=0x5590e6891588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005590e620f055 in fil_flush_low (space=0x5590e8074a30, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005590e621a9f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x00005590e5f9ee31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005590e5f9f893 in log_write_up_to (lsn=4522907, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005590e610af5f in trx_flush_log_if_needed_low (lsn=4522907) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005590e610afa8 in trx_flush_log_if_needed (lsn=4522907, trx=0x7fdcf8e664b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005590e610cdd1 in trx_commit_complete_for_mysql (trx=0x7fdcf8e664b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005590e5eefca6 in innobase_commit (hton=0x5590e7e3ec30, thd=0x7fdce4048a50, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4658 #12 0x00005590e5cc3a1d in commit_one_phase_2 (thd=0x7fdce4048a50, all=true, trans=0x7fdce404bfc8, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1741 #13 0x00005590e5cc391f in ha_commit_one_phase (thd=0x7fdce4048a50, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1721 #14 0x00005590e5cc2c98 in ha_commit_trans (thd=0x7fdce4048a50, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1534 #15 0x00005590e5b00ac6 in trans_commit (thd=0x7fdce4048a50) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005590e5bbb5cf in Wsrep_high_priority_service::commit (this=0x7fdce40741d0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:318 #17 0x00005590e652b001 in commit_fragment (server_state=..., high_priority_service=..., streaming_applier=0x7fdce40741d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:96 #18 0x00005590e652bb16 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:282 #19 0x00005590e652ed07 in wsrep::server_state::on_apply (this=0x5590e7ce4cb0, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:966 #20 0x00005590e6544503 in wsrep::high_priority_service::apply (this=0x7fdcfa5e54d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x00005590e6541bea in (anonymous namespace)::apply_cb (ctx=0x7fdcfa5e54d0, wsh=0x7fdcfa5e43e0, flags=9, buf=0x7fdcfa5e4420, meta=0x7fdcfa5e46b0, exit_loop=0x7fdcfa5e45ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007fdd02b2584b in galera::TrxHandleSlave::apply (this=this@entry=0x7fdc90279590, recv_ctx=recv_ctx@entry=0x7fdcfa5e54d0, apply_cb=0x5590e65419c7 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7fdcfa5e45ef: false) at galera/src/trx_handle.cpp:426 #23 0x00007fdd02b58192 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5590e7d1a420, recv_ctx=recv_ctx@entry=0x7fdcfa5e54d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007fdd02b5de40 in galera::ReplicatorSMM::process_trx (this=0x5590e7d1a420, recv_ctx=0x7fdcfa5e54d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2114 #25 0x00007fdd02b3f00b in galera::GcsActionSource::process_writeset (this=this@entry=0x5590e7d220f0, recv_ctx=0x7fdcfa5e54d0, act=..., exit_loop=@0x7fdcfa5e529e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007fdd02b3f1e5 in galera::GcsActionSource::dispatch (this=this@entry=0x5590e7d220f0, recv_ctx=recv_ctx@entry=0x7fdcfa5e54d0, act=..., exit_loop=@0x7fdcfa5e529e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007fdd02b3f5bd in galera::GcsActionSource::process (this=0x5590e7d220f0, recv_ctx=0x7fdcfa5e54d0, exit_loop=@0x7fdcfa5e529e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007fdd02b5e433 in galera::ReplicatorSMM::async_recv (this=0x5590e7d1a420, recv_ctx=0x7fdcfa5e54d0) at galera/src/replicator_smm.cpp:383 #29 0x00005590e65428a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5590e7ca1760, applier_ctx=0x7fdcfa5e54d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x00005590e5bdba69 in wsrep_replication_process (thd=0x7fdce4000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x00005590e5bcd587 in start_wsrep_THD (arg=0x5590e7d7c710) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #32 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fdcfa67c700 (LWP 220946)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648db7f in safe_cond_wait (cond=0x7fdcf0016b88, mp=0x7fdcf0016ad8, file=0x5590e65fc4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005590e5bdb4ba in inline_mysql_cond_wait (that=0x7fdcf0016b88, mutex=0x7fdcf0016ad8, src_file=0x5590e65fc7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005590e5bde34e in Wsrep_thd_queue::pop_front (this=0x7fdcf0016a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005590e5bdcc54 in wsrep_rollback_process (rollbacker=0x7fdcf0001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x00005590e5bcd587 in start_wsrep_THD (arg=0x5590e7d72500) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fdcf9a63700 (LWP 220949)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648de81 in safe_cond_timedwait (cond=0x5590e7a6ca00 , mp=0x5590e7a6c940 , abstime=0x7fdcf9a62cb0, file=0x5590e68db630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005590e62f7309 in inline_mysql_cond_timedwait (that=0x5590e7a6ca00 , mutex=0x5590e7a6c940 , abstime=0x7fdcf9a62cb0, src_file=0x5590e68db678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005590e62f782d in my_service_thread_sleep (control=0x5590e7020d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005590e62ebcc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e7e22900) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fdcf837b700 (LWP 220984)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd02b54417 in gu::Lock::wait (cond=..., this=0x7fdcf83778d0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fdd02b6506c in galera::Monitor::enter (this=this@entry=0x5590e7d1b480, obj=...) at galera/src/monitor.hpp:156 #3 0x00007fdd02b59b17 in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x5590e7d1a420, trx=...) at galera/src/replicator_smm.cpp:1280 #4 0x00007fdd02b784c3 in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:658 #5 0x00005590e6542e0b in wsrep::wsrep_provider_v26::commit_order_enter (this=0x5590e7ca1760, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:769 #6 0x00005590e653609e in wsrep::transaction::before_commit (this=0x7fdc90151c50) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:435 #7 0x00005590e5bbd85e in wsrep::client_state::before_commit (this=0x7fdc90151be0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:401 #8 0x00005590e5ecc604 in wsrep_before_commit (thd=0x7fdc9014bb60, all=true) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:203 #9 0x00005590e5ecd881 in Wsrep_storage_service::commit (this=0x7fdc901054d0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:171 #10 0x00005590e6538ba1 in wsrep::transaction::certify_fragment (this=0x7fdc90006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #11 0x00005590e653837b in wsrep::transaction::streaming_step (this=0x7fdc90006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #12 0x00005590e653559a in wsrep::transaction::after_row (this=0x7fdc90006bf0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #13 0x00005590e5cd70b6 in wsrep::client_state::after_row (this=0x7fdc90006b80) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #14 0x00005590e5cbf000 in wsrep_after_row (thd=0x7fdc90000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #15 0x00005590e5cd10df in wsrep_after_row (thd=0x7fdc90000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #16 0x00005590e5cd2284 in handler::ha_update_row (this=0x7fdc78069688, old_data=0x7fdc78008e38 "\375\002", new_data=0x7fdc78008d30 "\375\002") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6718 #17 0x00005590e593263c in write_record (thd=0x7fdc90000b00, table=0x7fdc780565c0, info=0x7fdcf8378e10) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:1944 #18 0x00005590e592fe9d in mysql_insert (thd=0x7fdc90000b00, table_list=0x7fdc90016eb0, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_REPLACE, ignore=false) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:1067 #19 0x00005590e597b292 in mysql_execute_command (thd=0x7fdc90000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4791 #20 0x00005590e59870f1 in mysql_parse (thd=0x7fdc90000b00, rawbuf=0x7fdc900154a8 "REPLACE INTO t1 VALUES (1,'y'),(2,'x')", length=38, parser_state=0x7fdcf837a040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #21 0x00005590e5986720 in wsrep_mysql_parse (thd=0x7fdc90000b00, rawbuf=0x7fdc900154a8 "REPLACE INTO t1 VALUES (1,'y'),(2,'x')", length=38, parser_state=0x7fdcf837a040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #22 0x00005590e597246d in dispatch_command (command=COM_QUERY, thd=0x7fdc90000b00, packet=0x7fdc900e07c1 "REPLACE INTO t1 VALUES (1,'y'),(2,'x')", packet_length=38, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #23 0x00005590e5970cbb in do_command (thd=0x7fdc90000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #24 0x00005590e5aebc9e in do_handle_one_connection (connect=0x5590e81db450) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #25 0x00005590e5aeb9e2 in handle_one_connection (arg=0x5590e81db450) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #26 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e81f91d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #27 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #28 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fdcd97fa700 (LWP 220955)): #0 0x00007fdd0a7ec644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005590e5fc209a in LinuxAIOHandler::collect (this=0x7fdcd97f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005590e5fc25c3 in LinuxAIOHandler::poll (this=0x7fdcd97f9b00, m1=0x7fdcd97f9ba8, m2=0x7fdcd97f9bb0, request=0x7fdcd97f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005590e5fc2726 in os_aio_linux_handler (global_segment=5, m1=0x7fdcd97f9ba8, m2=0x7fdcd97f9bb0, request=0x7fdcd97f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005590e5fc724a in os_aio_handler (segment=5, m1=0x7fdcd97f9ba8, m2=0x7fdcd97f9bb0, request=0x7fdcd97f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005590e621a4a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005590e60c0a47 in io_handler_thread (arg=0x5590e725b188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fdcd0ff9700 (LWP 220964)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fced7d in os_event::wait (this=0x5590e7e6a880) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005590e5fce803 in os_event::wait_low (this=0x5590e7e6a880, reset_sig_count=2584) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005590e5fceb74 in os_event_wait_low (event=0x5590e7e6a880, reset_sig_count=2584) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005590e60baf21 in srv_resume_thread (slot=0x5590e701c948 , sig_count=2584, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005590e60bef32 in srv_purge_coordinator_suspend (slot=0x5590e701c948 , rseg_history_len=36) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005590e60bf1c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fdcc37fe700 (LWP 220966)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fced7d in os_event::wait (this=0x5590e7e6a900) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005590e5fce803 in os_event::wait_low (this=0x5590e7e6a900, reset_sig_count=10563) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005590e5fceb74 in os_event_wait_low (event=0x5590e7e6a900, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005590e60baf21 in srv_resume_thread (slot=0x5590e701c980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005590e60beb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fdcc17fa700 (LWP 220970)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fced7d in os_event::wait (this=0x5590e7e6ac80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005590e5fce803 in os_event::wait_low (this=0x5590e7e6ac80, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005590e5fceb74 in os_event_wait_low (event=0x5590e7e6ac80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005590e6183cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fdcd9ffb700 (LWP 220954)): #0 0x00007fdd0a7ec644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005590e5fc209a in LinuxAIOHandler::collect (this=0x7fdcd9ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005590e5fc25c3 in LinuxAIOHandler::poll (this=0x7fdcd9ffab00, m1=0x7fdcd9ffaba8, m2=0x7fdcd9ffabb0, request=0x7fdcd9ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005590e5fc2726 in os_aio_linux_handler (global_segment=4, m1=0x7fdcd9ffaba8, m2=0x7fdcd9ffabb0, request=0x7fdcd9ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005590e5fc724a in os_aio_handler (segment=4, m1=0x7fdcd9ffaba8, m2=0x7fdcd9ffabb0, request=0x7fdcd9ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005590e621a4a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005590e60c0a47 in io_handler_thread (arg=0x5590e725b180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fdcdb7fe700 (LWP 220951)): #0 0x00007fdd0a7ec644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005590e5fc209a in LinuxAIOHandler::collect (this=0x7fdcdb7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005590e5fc25c3 in LinuxAIOHandler::poll (this=0x7fdcdb7fdb00, m1=0x7fdcdb7fdba8, m2=0x7fdcdb7fdbb0, request=0x7fdcdb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005590e5fc2726 in os_aio_linux_handler (global_segment=1, m1=0x7fdcdb7fdba8, m2=0x7fdcdb7fdbb0, request=0x7fdcdb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005590e5fc724a in os_aio_handler (segment=1, m1=0x7fdcdb7fdba8, m2=0x7fdcdb7fdbb0, request=0x7fdcdb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005590e621a4a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005590e60c0a47 in io_handler_thread (arg=0x5590e725b168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fdcf829a700 (LWP 221762)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648de81 in safe_cond_timedwait (cond=0x5590e71df4a0 , mp=0x5590e71e1e40 , abstime=0x7fdcf8299c60, file=0x5590e6549b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005590e583c015 in inline_mysql_cond_timedwait (that=0x5590e71df4a0 , mutex=0x5590e71e1e40 , abstime=0x7fdcf8299c60, src_file=0x5590e654bb50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005590e5840b08 in cache_thread (thd=0x7fdc84000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x00005590e5840db6 in one_thread_per_connection_end (thd=0x7fdc84000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x00005590e5aebd28 in do_handle_one_connection (connect=0x5590e8102550) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005590e5aeb9e2 in handle_one_connection (arg=0x5590e8102550) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e7e2ea30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fdcc3fff700 (LWP 220965)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fced7d in os_event::wait (this=0x5590e7e6a980) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005590e5fce803 in os_event::wait_low (this=0x5590e7e6a980, reset_sig_count=9775) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005590e5fceb74 in os_event_wait_low (event=0x5590e7e6a980, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005590e60baf21 in srv_resume_thread (slot=0x5590e701c9b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005590e60beb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fdcd87f8700 (LWP 221763)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648de81 in safe_cond_timedwait (cond=0x5590e71df4a0 , mp=0x5590e71e1e40 , abstime=0x7fdcd87f7c60, file=0x5590e6549b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005590e583c015 in inline_mysql_cond_timedwait (that=0x5590e71df4a0 , mutex=0x5590e71e1e40 , abstime=0x7fdcd87f7c60, src_file=0x5590e654bb50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005590e5840b08 in cache_thread (thd=0x7fdc78000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x00005590e5840db6 in one_thread_per_connection_end (thd=0x7fdc78000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x00005590e5aebd28 in do_handle_one_connection (connect=0x5590e80fea30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005590e5aeb9e2 in handle_one_connection (arg=0x5590e80fea30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e7e2ea30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fdcc27fc700 (LWP 220968)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fced7d in os_event::wait (this=0x5590e7e6ab80) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005590e5fce803 in os_event::wait_low (this=0x5590e7e6ab80, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005590e5fceb74 in os_event_wait_low (event=0x5590e7e6ab80, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005590e619b19b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fdcc0ff9700 (LWP 220971)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648db7f in safe_cond_wait (cond=0x7fdcc0ff8c70, mp=0x7fdcc0ff8cb0, file=0x5590e6764920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005590e5ee4cad in inline_mysql_cond_wait (that=0x7fdcc0ff8c70, mutex=0x7fdcc0ff8cb0, src_file=0x5590e6766348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005590e5ee979b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e80abc80) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fdcc1ffb700 (LWP 220969)): #0 0x00007fdd0ac22e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005590e5fcf07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005590e6175b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fdcc2ffd700 (LWP 220967)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fced7d in os_event::wait (this=0x5590e7e6aa00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005590e5fce803 in os_event::wait_low (this=0x5590e7e6aa00, reset_sig_count=9175) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005590e5fceb74 in os_event_wait_low (event=0x5590e7e6aa00, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005590e60baf21 in srv_resume_thread (slot=0x5590e701c9f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005590e60beb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fdcd2ffd700 (LWP 220960)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fce698 in os_event::timed_wait (this=0x5590e7e6ab00, abstime=0x7fdcd2ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005590e5fce9af in os_event::wait_time_low (this=0x5590e7e6ab00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005590e5fceb4f in os_event_wait_time_low (event=0x5590e7e6ab00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005590e60bcc08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fdcf83c6700 (LWP 220983)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648db7f in safe_cond_wait (cond=0x5590e71df4e0 , mp=0x5590e71e2800 , file=0x5590e6559488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005590e586cfe2 in inline_mysql_cond_wait (that=0x5590e71df4e0 , mutex=0x5590e71e2800 , src_file=0x5590e6559ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005590e586e874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e8192600) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fdcd27fc700 (LWP 220961)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fce698 in os_event::timed_wait (this=0x5590e8075070, abstime=0x7fdcd27fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005590e5fce9af in os_event::wait_time_low (this=0x5590e8075070, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005590e5fceb4f in os_event_wait_time_low (event=0x5590e8075070, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005590e62040e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fdcd3fff700 (LWP 220959)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fce698 in os_event::timed_wait (this=0x5590e7e6aa80, abstime=0x7fdcd3ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005590e5fce9af in os_event::wait_time_low (this=0x5590e7e6aa80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005590e5fceb4f in os_event_wait_time_low (event=0x5590e7e6aa80, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005590e60bd12a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fdcf8411700 (LWP 220972)): #0 0x00007fdd08f1340a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005590e583b714 in my_sigwait (set=0x7fdcf8410cf0, sig=0x7fdcf8410cb4, code=0x7fdcf8410cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005590e5841a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e7e2df70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fdcda7fc700 (LWP 220953)): #0 0x00007fdd0a7ec644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005590e5fc209a in LinuxAIOHandler::collect (this=0x7fdcda7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005590e5fc25c3 in LinuxAIOHandler::poll (this=0x7fdcda7fbb00, m1=0x7fdcda7fbba8, m2=0x7fdcda7fbbb0, request=0x7fdcda7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005590e5fc2726 in os_aio_linux_handler (global_segment=3, m1=0x7fdcda7fbba8, m2=0x7fdcda7fbbb0, request=0x7fdcda7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005590e5fc724a in os_aio_handler (segment=3, m1=0x7fdcda7fbba8, m2=0x7fdcda7fbbb0, request=0x7fdcda7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005590e621a4a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005590e60c0a47 in io_handler_thread (arg=0x5590e725b178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fdcd37fe700 (LWP 220958)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fce698 in os_event::timed_wait (this=0x5590e8042fb0, abstime=0x7fdcd37fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005590e5fce9af in os_event::wait_time_low (this=0x5590e8042fb0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005590e5fceb4f in os_event_wait_time_low (event=0x5590e8042fb0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005590e5f9b4f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fdcd17fa700 (LWP 220963)): #0 0x00007fdd0ac22e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005590e5fcf07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005590e60be489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005590e60be5b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fdcdaffd700 (LWP 220952)): #0 0x00007fdd0a7ec644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005590e5fc209a in LinuxAIOHandler::collect (this=0x7fdcdaffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005590e5fc25c3 in LinuxAIOHandler::poll (this=0x7fdcdaffcb00, m1=0x7fdcdaffcba8, m2=0x7fdcdaffcbb0, request=0x7fdcdaffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005590e5fc2726 in os_aio_linux_handler (global_segment=2, m1=0x7fdcdaffcba8, m2=0x7fdcdaffcbb0, request=0x7fdcdaffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005590e5fc724a in os_aio_handler (segment=2, m1=0x7fdcdaffcba8, m2=0x7fdcdaffcbb0, request=0x7fdcdaffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005590e621a4a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005590e60c0a47 in io_handler_thread (arg=0x5590e725b170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fdcd8ff9700 (LWP 220956)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fce698 in os_event::timed_wait (this=0x5590e7e6ac00, abstime=0x7fdcd8ff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005590e5fce9af in os_event::wait_time_low (this=0x5590e7e6ac00, time_in_usec=332000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005590e5fceb4f in os_event_wait_time_low (event=0x5590e7e6ac00, time_in_usec=332000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005590e61a51fe in pc_sleep_if_needed (next_loop_time=1551280628956, sig_count=2, cur_time=1551280628624) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x00005590e61a6137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fdcd1ffb700 (LWP 220962)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e5fce698 in os_event::timed_wait (this=0x5590e800ad00, abstime=0x7fdcd1ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005590e5fce9af in os_event::wait_time_low (this=0x5590e800ad00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005590e5fceb4f in os_event_wait_time_low (event=0x5590e800ad00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005590e611d4c9 in ib_wqueue_timedwait (wq=0x5590e800abd0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005590e625e73a in fts_optimize_thread (arg=0x5590e800abd0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fdcfa631700 (LWP 220947)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648db7f in safe_cond_wait (cond=0x7fdce8015938, mp=0x7fdce8015888, file=0x5590e65fc4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005590e5bdb4ba in inline_mysql_cond_wait (that=0x7fdce8015938, mutex=0x7fdce8015888, src_file=0x5590e65fc7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005590e5bde34e in Wsrep_thd_queue::pop_front (this=0x7fdce8015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005590e5bdd0d0 in wsrep_post_rollback_process (post_rollbacker=0x7fdce8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x00005590e5bcd587 in start_wsrep_THD (arg=0x5590e7d5c9d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fdcdbfff700 (LWP 220950)): #0 0x00007fdd0a7ec644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005590e5fc209a in LinuxAIOHandler::collect (this=0x7fdcdbffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005590e5fc25c3 in LinuxAIOHandler::poll (this=0x7fdcdbffeb00, m1=0x7fdcdbffeba8, m2=0x7fdcdbffebb0, request=0x7fdcdbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005590e5fc2726 in os_aio_linux_handler (global_segment=0, m1=0x7fdcdbffeba8, m2=0x7fdcdbffebb0, request=0x7fdcdbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005590e5fc724a in os_aio_handler (segment=0, m1=0x7fdcdbffeba8, m2=0x7fdcdbffebb0, request=0x7fdcdbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005590e621a4a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005590e60c0a47 in io_handler_thread (arg=0x5590e725b160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fdd01b36700 (LWP 220943)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd02b3c377 in gu::Lock::wait (cond=..., this=0x7fdd01b35bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fdd02b3c409 in galera::ServiceThd::thd_func (arg=0x5590e7d1ab38) at galera/src/galera_service_thd.cpp:30 #3 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fdcfb67e700 (LWP 220944)): #0 0x00007fdd08fda483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fdd02adda68 in asio::detail::epoll_reactor::run (this=0x5590e7d63180, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fdd02ae2160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5590e7d630a0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fdd02ae2412 in asio::detail::task_io_service::run (this=0x5590e7d630a0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fdd02af9e6d in asio::io_service::run (this=0x5590e7d621a0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fdd02afa5c1 in gcomm::AsioProtonet::event_loop (this=0x5590e7d62100, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fdd02b1865d in GCommConn::run (this=0x5590e7d62880) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fdd02b1b709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fdd03674700 (LWP 220942)): #0 0x00007fdd0ac1fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590e648de81 in safe_cond_timedwait (cond=0x5590e7a7f1c0 , mp=0x5590e7a7f100 , abstime=0x7fdd03673d70, file=0x5590e691fe90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005590e648ee3b in inline_mysql_cond_timedwait (that=0x5590e7a7f1c0 , mutex=0x5590e7a7f100 , abstime=0x7fdd03673d70, src_file=0x5590e691fed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005590e648f99e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005590e641ffb3 in pfs_spawn_thread (arg=0x5590e7ce3d40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fdcfae7d700 (LWP 220945)): #0 0x00007fdd0ac1f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd02b16bd7 in gu::Lock::wait (cond=..., this=0x7fdcfae7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fdd02b1971b in RecvBuf::front (this=this@entry=0x5590e7d62a68, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fdd02b172b6 in gcomm_recv (backend=, msg=0x5590e7d229f0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fdd02b0b25c in core_msg_recv (backend=backend@entry=0x5590e7d22b38, recv_msg=recv_msg@entry=0x5590e7d229f0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fdd02b0e4d4 in gcs_core_recv (conn=0x5590e7d22990, recv_act=recv_act@entry=0x7fdcfae7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fdd02b15748 in gcs_recv_thread (arg=0x5590e7d22710) at gcs/src/gcs.cpp:1398 #7 0x00007fdd0ac1bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd08fd9ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fdd0b02c880 (LWP 220939)): #0 0x00007fdd0ac209d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005590e6488550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005590e5cbdebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdd08fcf20d in poll () from /lib64/libc.so.6 #5 0x00005590e5847f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005590e58471b4 in mysqld_main (argc=162, argv=0x5590e7b9c140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005590e583b695 in main (argc=23, argv=0x7ffedab7c3b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.223479' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.GCF-1043A-innodb/mysqld.2/data/core.223479 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 223479] [New LWP 223480] [New LWP 223485] [New LWP 223481] [New LWP 223652] [New LWP 223658] [New LWP 223653] [New LWP 223671] [New LWP 223664] [New LWP 223676] [New LWP 223665] [New LWP 223667] [New LWP 223672] [New LWP 223677] [New LWP 223482] [New LWP 223669] [New LWP 223655] [New LWP 223670] [New LWP 223673] [New LWP 223656] [New LWP 223678] [New LWP 223659] [New LWP 223663] [New LWP 223662] [New LWP 223675] [New LWP 223486] [New LWP 223674] [New LWP 223668] [New LWP 223666] [New LWP 223483] [New LWP 223484] [New LWP 223657] [New LWP 223654] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f087f47d9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f087f47d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b1108d0550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b110105ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f087d82c20d in poll () from /lib64/libc.so.6 #5 0x000055b10fc8ff19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055b10fc8f1b4 in mysqld_main (argc=163, argv=0x55b113e6a140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055b10fc83695 in main (argc=23, argv=0x7ffc48dc9298) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f08437fe700 (LWP 223654)): #0 0x00007f087f47fd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055b11040b749 in os_file_fsync_posix (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055b11040ba9f in os_file_flush_func (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055b1106541fa in pfs_os_file_flush_func (file=..., src_file=0x55b110cd9588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055b110657055 in fil_flush_low (space=0x55b114342f40, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x000055b1106629f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x000055b1106626a8 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4416 #7 0x000055b110508a47 in io_handler_thread (arg=0x55b1116a3168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #8 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f0841ffb700 (LWP 223657)): #0 0x00007f087f47fd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055b11040b749 in os_file_fsync_posix (file=29) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055b11040ba9f in os_file_flush_func (file=29) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055b1106541fa in pfs_os_file_flush_func (file=..., src_file=0x55b110cd9588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055b110657055 in fil_flush_low (space=0x55b1142dc510, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x000055b1106629f8 in fil_flush (space_id=7) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x000055b110662c7f in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4548 #7 0x000055b1105df256 in buf_dblwr_update (bpage=0x7f08648689c0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:767 #8 0x000055b1105e6d28 in buf_flush_write_complete (bpage=0x7f08648689c0, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000055b1105d5910 in buf_page_io_complete (bpage=0x7f08648689c0, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6216 #10 0x000055b110662752 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4435 #11 0x000055b110508a47 in io_handler_thread (arg=0x55b1116a3180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f087410f700 (LWP 223484)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b1108d5b7f in safe_cond_wait (cond=0x7f0854015938, mp=0x7f0854015888, file=0x55b110a444c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b1100234ba in inline_mysql_cond_wait (that=0x7f0854015938, mutex=0x7f0854015888, src_file=0x55b110a447e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b11002634e in Wsrep_thd_queue::pop_front (this=0x7f0854015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b110024c54 in wsrep_rollback_process (rollbacker=0x7f0854000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x000055b110015587 in start_wsrep_THD (arg=0x55b114049a90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f0867ffe700 (LWP 223483)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0877373bd7 in gu::Lock::wait (cond=..., this=0x7f0867ffd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f087737671b in RecvBuf::front (this=this@entry=0x55b113ff4038, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f08773742b6 in gcomm_recv (backend=, msg=0x55b113ff7ab0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f087736825c in core_msg_recv (backend=backend@entry=0x55b113ff7bf8, recv_msg=recv_msg@entry=0x55b113ff7ab0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f087736b4d4 in gcs_core_recv (conn=0x55b113ff7a50, recv_act=recv_act@entry=0x7f0867ffdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f0877372748 in gcs_recv_thread (arg=0x55b113ff77d0) at gcs/src/gcs.cpp:1398 #7 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f0835ffb700 (LWP 223666)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416698 in os_event::timed_wait (this=0x55b1142e16e0, abstime=0x7f0835ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b1104169af in os_event::wait_time_low (this=0x55b1142e16e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b110416b4f in os_event_wait_time_low (event=0x55b1142e16e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b1105654c9 in ib_wqueue_timedwait (wq=0x55b1142d69c0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055b1106a673a in fts_optimize_thread (arg=0x55b1142d69c0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f0834ff9700 (LWP 223668)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416d7d in os_event::wait (this=0x55b114138d90) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b110416803 in os_event::wait_low (this=0x55b114138d90, reset_sig_count=2157) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b110416b74 in os_event_wait_low (event=0x55b114138d90, reset_sig_count=2157) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b110502f21 in srv_resume_thread (slot=0x55b111464948 , sig_count=2157, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b110506f32 in srv_purge_coordinator_suspend (slot=0x55b111464948 , rseg_history_len=37) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055b1105071c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f08257fa700 (LWP 223674)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416d7d in os_event::wait (this=0x55b114139190) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b110416803 in os_event::wait_low (this=0x55b114139190, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b110416b74 in os_event_wait_low (event=0x55b114139190, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b1105cbcbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f0874079700 (LWP 223486)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f087728d272 in fifo_lock_get (q=q@entry=0x55b11401dc90) at galerautils/src/gu_fifo.c:262 #2 0x00007f087728d8e1 in gu_fifo_get_head (q=0x55b11401dc90, err=0x7f087407806c) at galerautils/src/gu_fifo.c:330 #3 0x00007f087736f43c in gcs_recv (conn=0x55b113ff77d0, action=0x7f08740780b0) at gcs/src/gcs.cpp:2024 #4 0x00007f087739c50c in galera::GcsActionSource::process (this=0x55b113ff7570, recv_ctx=0x7f08740784d0, exit_loop=@0x7f087407829e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f08773bb433 in galera::ReplicatorSMM::async_recv (this=0x55b113fe83b0, recv_ctx=0x7f08740784d0) at galera/src/replicator_smm.cpp:383 #6 0x000055b11098a8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x55b113f5e250, applier_ctx=0x7f08740784d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055b110023a69 in wsrep_replication_process (thd=0x7f085c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055b110015587 in start_wsrep_THD (arg=0x55b1140104d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f0824ff9700 (LWP 223675)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b1108d5b7f in safe_cond_wait (cond=0x7f0824ff8c70, mp=0x7f0824ff8cb0, file=0x55b110bac920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b11032ccad in inline_mysql_cond_wait (that=0x7f0824ff8c70, mutex=0x7f0824ff8cb0, src_file=0x55b110bae348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b11033179b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055b110867fb3 in pfs_spawn_thread (arg=0x55b114383ad0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f08377fe700 (LWP 223662)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416698 in os_event::timed_wait (this=0x55b1143114c0, abstime=0x7f08377fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b1104169af in os_event::wait_time_low (this=0x55b1143114c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b110416b4f in os_event_wait_time_low (event=0x55b1143114c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b1103e34f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f0837fff700 (LWP 223663)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416698 in os_event::timed_wait (this=0x55b114138f90, abstime=0x7f0837ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b1104169af in os_event::wait_time_low (this=0x55b114138f90, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b110416b4f in os_event_wait_time_low (event=0x55b114138f90, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b11050512a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f0840ff9700 (LWP 223659)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416698 in os_event::timed_wait (this=0x55b114139110, abstime=0x7f0840ff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b1104169af in os_event::wait_time_low (this=0x55b114139110, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b110416b4f in os_event_wait_time_low (event=0x55b114139110, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b1105ed1fe in pc_sleep_if_needed (next_loop_time=1551280637613, sig_count=2, cur_time=1551280636613) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x000055b1105ee137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f08645aa700 (LWP 223678)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b1108d5e81 in safe_cond_timedwait (cond=0x55b1116274a0 , mp=0x55b111629e40 , abstime=0x7f08645a9c60, file=0x55b110991b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b10fc84015 in inline_mysql_cond_timedwait (that=0x55b1116274a0 , mutex=0x55b111629e40 , abstime=0x7f08645a9c60, src_file=0x55b110993b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b10fc88b08 in cache_thread (thd=0x7f0850000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x000055b10fc88db6 in one_thread_per_connection_end (thd=0x7f0850000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x000055b10ff33d28 in do_handle_one_connection (connect=0x55b1140ffa50) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055b10ff339e2 in handle_one_connection (arg=0x55b1140ffa50) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055b110867fb3 in pfs_spawn_thread (arg=0x55b1144ca1a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f08427fc700 (LWP 223656)): #0 0x00007f087f049644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b11040a09a in LinuxAIOHandler::collect (this=0x7f08427fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b11040a5c3 in LinuxAIOHandler::poll (this=0x7f08427fbb00, m1=0x7f08427fbba8, m2=0x7f08427fbbb0, request=0x7f08427fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b11040a726 in os_aio_linux_handler (global_segment=3, m1=0x7f08427fbba8, m2=0x7f08427fbbb0, request=0x7f08427fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b11040f24a in os_aio_handler (segment=3, m1=0x7f08427fbba8, m2=0x7f08427fbbb0, request=0x7f08427fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b1106624a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b110508a47 in io_handler_thread (arg=0x55b1116a3178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f0825ffb700 (LWP 223673)): #0 0x00007f087f47fe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055b11041707f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055b1105bdb55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f08277fe700 (LWP 223670)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416d7d in os_event::wait (this=0x55b114138e10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b110416803 in os_event::wait_low (this=0x55b114138e10, reset_sig_count=8778) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b110416b74 in os_event_wait_low (event=0x55b114138e10, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b110502f21 in srv_resume_thread (slot=0x55b111464980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b110506b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f0842ffd700 (LWP 223655)): #0 0x00007f087f049644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b11040a09a in LinuxAIOHandler::collect (this=0x7f0842ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b11040a5c3 in LinuxAIOHandler::poll (this=0x7f0842ffcb00, m1=0x7f0842ffcba8, m2=0x7f0842ffcbb0, request=0x7f0842ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b11040a726 in os_aio_linux_handler (global_segment=2, m1=0x7f0842ffcba8, m2=0x7f0842ffcbb0, request=0x7f0842ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b11040f24a in os_aio_handler (segment=2, m1=0x7f0842ffcba8, m2=0x7f0842ffcbb0, request=0x7f0842ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b1106624a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b110508a47 in io_handler_thread (arg=0x55b1116a3170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f0827fff700 (LWP 223669)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416d7d in os_event::wait (this=0x55b114138f10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b110416803 in os_event::wait_low (this=0x55b114138f10, reset_sig_count=7467) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b110416b74 in os_event_wait_low (event=0x55b114138f10, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b110502f21 in srv_resume_thread (slot=0x55b1114649f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b110506b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f0874910700 (LWP 223482)): #0 0x00007f087d837483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f087733aa68 in asio::detail::epoll_reactor::run (this=0x55b113ff4c30, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f087733f160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55b113ff47a0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f087733f412 in asio::detail::task_io_service::run (this=0x55b113ff47a0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f0877356e6d in asio::io_service::run (this=0x55b1140299f0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f08773575c1 in gcomm::AsioProtonet::event_loop (this=0x55b114029950, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f087737565d in GCommConn::run (this=0x55b113ff3e50) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f0877378709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f08645f5700 (LWP 223677)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b1108d5b7f in safe_cond_wait (cond=0x55b1116274e0 , mp=0x55b11162a800 , file=0x55b1109a1488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b10fcb4fe2 in inline_mysql_cond_wait (that=0x55b1116274e0 , mutex=0x55b11162a800 , src_file=0x55b1109a1ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055b10fcb6874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055b110867fb3 in pfs_spawn_thread (arg=0x55b11446a4e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f08267fc700 (LWP 223672)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416d7d in os_event::wait (this=0x55b114139090) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b110416803 in os_event::wait_low (this=0x55b114139090, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b110416b74 in os_event_wait_low (event=0x55b114139090, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b1105e319b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f08357fa700 (LWP 223667)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416d7d in os_event::wait (this=0x55b1142d1580) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b110416803 in os_event::wait_low (this=0x55b1142d1580, reset_sig_count=118) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b110416b74 in os_event_wait_low (event=0x55b1142d1580, reset_sig_count=118) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b110510d96 in sync_array_wait_event (arr=0x55b1141102e0, cell=@0x7f08357f9bc8: 0x55b114110590) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055b110513aab in rw_lock_s_lock_spin (lock=0x55b111461740 , pass=0, file_name=0x55b110be1a98 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x000055b1103e3ec1 in rw_lock_s_lock_func (lock=0x55b111461740 , pass=0, file_name=0x55b110be1a98 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x000055b1103e43f4 in pfs_rw_lock_s_lock_func (lock=0x55b111461740 , pass=0, file_name=0x55b110be1a98 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x000055b1103e8903 in log_write_checkpoint_info (sync=true, end_lsn=4245269) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1385 #9 0x000055b1103e8e0b in log_checkpoint (sync=true, write_always=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1534 #10 0x000055b1105062a8 in srv_master_do_idle_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2256 #11 0x000055b110506686 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2358 #12 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f08367fc700 (LWP 223665)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416698 in os_event::timed_wait (this=0x55b114343580, abstime=0x7f08367fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b1104169af in os_event::wait_time_low (this=0x55b114343580, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b110416b4f in os_event_wait_time_low (event=0x55b114343580, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b11064c0e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f0864640700 (LWP 223676)): #0 0x00007f087d77040a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055b10fc83714 in my_sigwait (set=0x7f086463fcf0, sig=0x7f086463fcb4, code=0x7f086463fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055b10fc89a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x000055b110867fb3 in pfs_spawn_thread (arg=0x55b1140fc630) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f0836ffd700 (LWP 223664)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416698 in os_event::timed_wait (this=0x55b114139010, abstime=0x7f0836ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b1104169af in os_event::wait_time_low (this=0x55b114139010, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b110416b4f in os_event_wait_time_low (event=0x55b114139010, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b110504c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f0826ffd700 (LWP 223671)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b110416d7d in os_event::wait (this=0x55b114138e90) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b110416803 in os_event::wait_low (this=0x55b114138e90, reset_sig_count=7859) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b110416b74 in os_event_wait_low (event=0x55b114138e90, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b110502f21 in srv_resume_thread (slot=0x55b1114649b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b110506b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f0843fff700 (LWP 223653)): #0 0x00007f087f049644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b11040a09a in LinuxAIOHandler::collect (this=0x7f0843ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b11040a5c3 in LinuxAIOHandler::poll (this=0x7f0843ffeb00, m1=0x7f0843ffeba8, m2=0x7f0843ffebb0, request=0x7f0843ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b11040a726 in os_aio_linux_handler (global_segment=0, m1=0x7f0843ffeba8, m2=0x7f0843ffebb0, request=0x7f0843ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b11040f24a in os_aio_handler (segment=0, m1=0x7f0843ffeba8, m2=0x7f0843ffebb0, request=0x7f0843ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b1106624a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b110508a47 in io_handler_thread (arg=0x55b1116a3160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f08417fa700 (LWP 223658)): #0 0x00007f087f049644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b11040a09a in LinuxAIOHandler::collect (this=0x7f08417f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b11040a5c3 in LinuxAIOHandler::poll (this=0x7f08417f9b00, m1=0x7f08417f9ba8, m2=0x7f08417f9bb0, request=0x7f08417f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b11040a726 in os_aio_linux_handler (global_segment=5, m1=0x7f08417f9ba8, m2=0x7f08417f9bb0, request=0x7f08417f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b11040f24a in os_aio_handler (segment=5, m1=0x7f08417f9ba8, m2=0x7f08417f9bb0, request=0x7f08417f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b1106624a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b110508a47 in io_handler_thread (arg=0x55b1116a3188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f0865c92700 (LWP 223652)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b1108d5e81 in safe_cond_timedwait (cond=0x55b111eb4a00 , mp=0x55b111eb4940 , abstime=0x7f0865c91cb0, file=0x55b110d23630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b11073f309 in inline_mysql_cond_timedwait (that=0x55b111eb4a00 , mutex=0x55b111eb4940 , abstime=0x7f0865c91cb0, src_file=0x55b110d23678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b11073f82d in my_service_thread_sleep (control=0x55b111468d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055b110733cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055b110867fb3 in pfs_spawn_thread (arg=0x55b1140f1000) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f0876d94700 (LWP 223481)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f0877399377 in gu::Lock::wait (cond=..., this=0x7f0876d93bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f0877399409 in galera::ServiceThd::thd_func (arg=0x55b113fe8ac8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f08740c4700 (LWP 223485)): #0 0x00007f087f47c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b1108d5b7f in safe_cond_wait (cond=0x7f0858015938, mp=0x7f0858015888, file=0x55b110a444c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b1100234ba in inline_mysql_cond_wait (that=0x7f0858015938, mutex=0x7f0858015888, src_file=0x55b110a447e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b11002634e in Wsrep_thd_queue::pop_front (this=0x7f0858015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b1100250d0 in wsrep_post_rollback_process (post_rollbacker=0x7f0858000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x000055b110015587 in start_wsrep_THD (arg=0x55b114037cb0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f0877ed1700 (LWP 223480)): #0 0x00007f087f47cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b1108d5e81 in safe_cond_timedwait (cond=0x55b111ec71c0 , mp=0x55b111ec7100 , abstime=0x7f0877ed0d70, file=0x55b110d67e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b1108d6e3b in inline_mysql_cond_timedwait (that=0x55b111ec71c0 , mutex=0x55b111ec7100 , abstime=0x7f0877ed0d70, src_file=0x55b110d67ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b1108d799e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055b110867fb3 in pfs_spawn_thread (arg=0x55b113fb1de0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f087f478dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f087d836ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f087f889880 (LWP 223479)): #0 0x00007f087f47d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b1108d0550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b110105ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f087d82c20d in poll () from /lib64/libc.so.6 #5 0x000055b10fc8ff19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055b10fc8f1b4 in mysqld_main (argc=163, argv=0x55b113e6a140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055b10fc83695 in main (argc=23, argv=0x7ffc48dc9298) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.GCF-1043A-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043A-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#22 galera_sr.galera_sr_myisam galera_sr.galera_sr_kill_connection galera_sr.galera_sr_ws_size2 galera_sr.GCF-1008 galera_sr.mysql-wsrep-features#27 galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#148 galera_sr.galera_sr_load_data galera_sr.mysql-wsrep-features#15 galera_sr.GCF-1043A galera_sr.galera_sr_kill_query galera_sr.mysql-wsrep-features#9 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.mysql-wsrep-features#8 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 17:18:29 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/mysql-wsrep-features#8.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); connection node_1; CREATE TABLE t1 (f1 INT PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(100), FULLTEXT (f2)) ENGINE=InnoDB; connection node_2; SELECT COUNT(*) = 13 FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE name LIKE 'test/%'; COUNT(*) = 13 1 connection node_1; INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.223681' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.1/data/core.223681 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 223681] [New LWP 223682] [New LWP 223695] [New LWP 223708] [New LWP 223710] [New LWP 223712] [New LWP 223728] [New LWP 223989] [New LWP 223722] [New LWP 223707] [New LWP 223711] [New LWP 223713] [New LWP 223721] [New LWP 223724] [New LWP 223729] [New LWP 223730] [New LWP 223735] [New LWP 223688] [New LWP 223734] [New LWP 223723] [New LWP 223727] [New LWP 223726] [New LWP 223725] [New LWP 223720] [New LWP 223718] [New LWP 223709] [New LWP 223719] [New LWP 223698] [New LWP 223717] [New LWP 223697] [New LWP 223716] [New LWP 223696] [New LWP 223683] [New LWP 223714] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007ff9a7c6a9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007ff9a7c6a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559680c02550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559680437ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff9a601920d in poll () from /lib64/libc.so.6 #5 0x000055967ffc1f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055967ffc11b4 in mysqld_main (argc=162, argv=0x5596829cb140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055967ffb5695 in main (argc=23, argv=0x7fff4d452b88) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7ff975ffb700 (LWP 223714)): #0 0x00007ff9a7c6cd1d in fsync () from /lib64/libpthread.so.0 #1 0x000055968073d749 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055968073da9f in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005596809861fa in pfs_os_file_flush_func (file=..., src_file=0x55968100b588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559680989055 in fil_flush_low (space=0x559682ea35e0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005596809949f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x0000559680718e31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000559680719893 in log_write_up_to (lsn=2794571, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000559680919a8d in buf_flush_write_block_low (bpage=0x7ff99542d160, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:998 #9 0x000055968091a497 in buf_flush_page (buf_pool=0x559682d7f370, bpage=0x7ff99542d160, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1209 #10 0x000055968091addf in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=7, n_to_flush=200) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1433 #11 0x000055968091b162 in buf_flush_page_and_try_neighbors (bpage=0x7ff99542d160, flush_type=BUF_FLUSH_LIST, n_to_flush=200, count=0x7ff975ffa8b8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1505 #12 0x000055968091bd7b in buf_do_flush_list_batch (buf_pool=0x559682d7f370, min_n=200, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1763 #13 0x000055968091c373 in buf_flush_batch (buf_pool=0x559682d7f370, flush_type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7ff975ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1833 #14 0x000055968091ca33 in buf_flush_do_batch (buf_pool=0x559682d7f370, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7ff975ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1991 #15 0x000055968091cd07 in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7ff975ffabd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2097 #16 0x0000559680920a52 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3249 #17 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7ff99eb80700 (LWP 223683)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff99fb86377 in gu::Lock::wait (cond=..., this=0x7ff99eb7fbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff99fb86409 in galera::ServiceThd::thd_func (arg=0x559682b49b38) at galera/src/galera_service_thd.cpp:30 #3 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff99c6fc700 (LWP 223696)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680c07b7f in safe_cond_wait (cond=0x7ff98c016b88, mp=0x7ff98c016ad8, file=0x559680d764c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005596803554ba in inline_mysql_cond_wait (that=0x7ff98c016b88, mutex=0x7ff98c016ad8, src_file=0x559680d767e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055968035834e in Wsrep_thd_queue::pop_front (this=0x7ff98c016a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559680356c54 in wsrep_rollback_process (rollbacker=0x7ff98c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x0000559680347587 in start_wsrep_THD (arg=0x559682baaf30) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff974ff9700 (LWP 223716)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748698 in os_event::timed_wait (this=0x559682e71b60, abstime=0x7ff974ff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005596807489af in os_event::wait_time_low (this=0x559682e71b60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559680748b4f in os_event_wait_time_low (event=0x559682e71b60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005596807154f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff99c6b1700 (LWP 223697)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680c07b7f in safe_cond_wait (cond=0x7ff980016c48, mp=0x7ff980016b98, file=0x559680d764c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005596803554ba in inline_mysql_cond_wait (that=0x7ff980016c48, mutex=0x7ff980016b98, src_file=0x559680d767e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055968035834e in Wsrep_thd_queue::pop_front (this=0x7ff980016b40) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005596803570d0 in wsrep_post_rollback_process (post_rollbacker=0x7ff980000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x0000559680347587 in start_wsrep_THD (arg=0x559682ba1500) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff96ffff700 (LWP 223717)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748698 in os_event::timed_wait (this=0x559682c99a70, abstime=0x7ff96fffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005596807489af in os_event::wait_time_low (this=0x559682c99a70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559680748b4f in os_event_wait_time_low (event=0x559682c99a70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055968083712a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff99c666700 (LWP 223698)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff99fa7a272 in fifo_lock_get (q=q@entry=0x559682b77d90) at galerautils/src/gu_fifo.c:262 #2 0x00007ff99fa7a8e1 in gu_fifo_get_head (q=0x559682b77d90, err=0x7ff99c66506c) at galerautils/src/gu_fifo.c:330 #3 0x00007ff99fb5c43c in gcs_recv (conn=0x559682b51710, action=0x7ff99c6650b0) at gcs/src/gcs.cpp:2024 #4 0x00007ff99fb8950c in galera::GcsActionSource::process (this=0x559682b510f0, recv_ctx=0x7ff99c6654d0, exit_loop=@0x7ff99c66529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007ff99fba8433 in galera::ReplicatorSMM::async_recv (this=0x559682b49420, recv_ctx=0x7ff99c6654d0) at galera/src/replicator_smm.cpp:383 #6 0x0000559680cbc8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x559682ad0760, applier_ctx=0x7ff99c6654d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000559680355a69 in wsrep_replication_process (thd=0x7ff984000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000559680347587 in start_wsrep_THD (arg=0x559682bab710) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff9757fa700 (LWP 223719)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748698 in os_event::timed_wait (this=0x559682ea3c20, abstime=0x7ff9757f9d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005596807489af in os_event::wait_time_low (this=0x559682ea3c20, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559680748b4f in os_event_wait_time_low (event=0x559682ea3c20, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055968097e0e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff994a05700 (LWP 223709)): #0 0x00007ff9a7836644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055968073c09a in LinuxAIOHandler::collect (this=0x7ff994a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055968073c5c3 in LinuxAIOHandler::poll (this=0x7ff994a04b00, m1=0x7ff994a04ba8, m2=0x7ff994a04bb0, request=0x7ff994a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055968073c726 in os_aio_linux_handler (global_segment=1, m1=0x7ff994a04ba8, m2=0x7ff994a04bb0, request=0x7ff994a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055968074124a in os_aio_handler (segment=1, m1=0x7ff994a04ba8, m2=0x7ff994a04bb0, request=0x7ff994a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005596809944a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055968083aa47 in io_handler_thread (arg=0x5596819d5168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff96f7fe700 (LWP 223718)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748698 in os_event::timed_wait (this=0x559682c99af0, abstime=0x7ff96f7fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005596807489af in os_event::wait_time_low (this=0x559682c99af0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559680748b4f in os_event_wait_time_low (event=0x559682c99af0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559680836c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff96effd700 (LWP 223720)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748698 in os_event::timed_wait (this=0x559682e399c0, abstime=0x7ff96effcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005596807489af in os_event::wait_time_low (this=0x559682e399c0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559680748b4f in os_event_wait_time_low (event=0x559682e399c0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005596808974c9 in ib_wqueue_timedwait (wq=0x559682e39890, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005596809d873a in fts_optimize_thread (arg=0x559682e39890) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff95bfff700 (LWP 223725)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748d7d in os_event::wait (this=0x559682c99970) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559680748803 in os_event::wait_low (this=0x559682c99970, reset_sig_count=7985) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559680748b74 in os_event_wait_low (event=0x559682c99970, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559680834f21 in srv_resume_thread (slot=0x5596817969b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559680838b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff95b7fe700 (LWP 223726)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748d7d in os_event::wait (this=0x559682c99b70) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559680748803 in os_event::wait_low (this=0x559682c99b70, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559680748b74 in os_event_wait_low (event=0x559682c99b70, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055968091519b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff95affd700 (LWP 223727)): #0 0x00007ff9a7c6ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055968074907f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005596808efb55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff96d7fa700 (LWP 223723)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748d7d in os_event::wait (this=0x559682c998f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559680748803 in os_event::wait_low (this=0x559682c998f0, reset_sig_count=9073) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559680748b74 in os_event_wait_low (event=0x559682c998f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559680834f21 in srv_resume_thread (slot=0x559681796980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559680838b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff995406700 (LWP 223734)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680c07b7f in safe_cond_wait (cond=0x5596819594e0 , mp=0x55968195c800 , file=0x559680cd3488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055967ffe6fe2 in inline_mysql_cond_wait (that=0x5596819594e0 , mutex=0x55968195c800 , src_file=0x559680cd3ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055967ffe8874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559680b99fb3 in pfs_spawn_thread (arg=0x559682fc1270) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff997fff700 (LWP 223688)): #0 0x00007ff9a6024483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff99fb27a68 in asio::detail::epoll_reactor::run (this=0x559682b92180, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007ff99fb2c160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559682b920a0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007ff99fb2c412 in asio::detail::task_io_service::run (this=0x559682b920a0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007ff99fb43e6d in asio::io_service::run (this=0x559682b911a0) at asio/asio/impl/io_service.ipp:58 #5 0x00007ff99fb445c1 in gcomm::AsioProtonet::event_loop (this=0x559682b91100, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007ff99fb6265d in GCommConn::run (this=0x559682b91880) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007ff99fb65709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff9953bb700 (LWP 223735)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680c07e81 in safe_cond_timedwait (cond=0x5596819594a0 , mp=0x55968195be40 , abstime=0x7ff9953bac60, file=0x559680cc3b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055967ffb6015 in inline_mysql_cond_timedwait (that=0x5596819594a0 , mutex=0x55968195be40 , abstime=0x7ff9953bac60, src_file=0x559680cc5b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055967ffbab08 in cache_thread (thd=0x7ff934000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x000055967ffbadb6 in one_thread_per_connection_end (thd=0x7ff934000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x0000559680265d28 in do_handle_one_connection (connect=0x55968300a0c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005596802659e2 in handle_one_connection (arg=0x55968300a0c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000559680b99fb3 in pfs_spawn_thread (arg=0x559683027e40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff99c057700 (LWP 223730)): #0 0x00007ff9a5f5d40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055967ffb5714 in my_sigwait (set=0x7ff99c056cf0, sig=0x7ff99c056cb4, code=0x7ff99c056cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055967ffbba71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x0000559680b99fb3 in pfs_spawn_thread (arg=0x559682c5cf70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff959ffb700 (LWP 223729)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680c07b7f in safe_cond_wait (cond=0x7ff959ffac70, mp=0x7ff959ffacb0, file=0x559680ede920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055968065ecad in inline_mysql_cond_wait (that=0x7ff959ffac70, mutex=0x7ff959ffacb0, src_file=0x559680ee0348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055968066379b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000559680b99fb3 in pfs_spawn_thread (arg=0x559682eda830) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff96cff9700 (LWP 223724)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748d7d in os_event::wait (this=0x559682dbfce0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559680748803 in os_event::wait_low (this=0x559682dbfce0, reset_sig_count=300) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559680748b74 in os_event_wait_low (event=0x559682dbfce0, reset_sig_count=300) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559680842d96 in sync_array_wait_event (arr=0x559682c70ae0, cell=@0x7ff96cff6b48: 0x559682c70c10) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000559680846362 in rw_lock_x_lock_func (lock=0x7ff995437f38, pass=0, file_name=0x559680fc0408 "/home/stepan/galera/git/10.4/server/storage/innobase/include/mtr0mtr.ic", line=147) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:735 #6 0x00005596808b7c44 in pfs_rw_lock_x_lock_func (lock=0x7ff995437f38, pass=0, file_name=0x559680fc0408 "/home/stepan/galera/git/10.4/server/storage/innobase/include/mtr0mtr.ic", line=147) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:558 #7 0x00005596808d7f75 in mtr_t::x_latch_at_savepoint (this=0x7ff96cff81a0, savepoint=16, block=0x7ff995437e70) at /home/stepan/galera/git/10.4/server/storage/innobase/include/mtr0mtr.ic:147 #8 0x00005596808c4c60 in btr_cur_search_to_nth_level_func (index=0x559683036f28, level=0, tuple=0x7ff95400c958, mode=PAGE_CUR_LE, latch_mode=33, cursor=0x559682ebaf68, ahi_latch=0x0, file=0x559680f732a0 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7ff96cff81a0, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:2190 #9 0x00005596808db14e in btr_pcur_open_with_no_init_func (index=0x559683036f28, tuple=0x7ff95400c958, mode=PAGE_CUR_LE, latch_mode=65569, cursor=0x559682ebaf68, ahi_latch=0x0, file=0x559680f732a0 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7ff96cff81a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:527 #10 0x00005596808dc457 in btr_pcur_restore_position_func (latch_mode=65569, cursor=0x559682ebaf68, file=0x559680f732a0 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7ff96cff81a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0pcur.cc:391 #11 0x00005596807f5aee in row_purge_reposition_pcur (mode=65569, node=0x559682ebaed0, mtr=0x7ff96cff81a0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:100 #12 0x00005596807f5cbb in row_purge_remove_clust_if_poss_low (node=0x559682ebaed0, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:140 #13 0x00005596807f6111 in row_purge_remove_clust_if_poss (node=0x559682ebaed0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #14 0x00005596807f793e in row_purge_del_mark (node=0x559682ebaed0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #15 0x00005596807f8ec6 in row_purge_record_func (node=0x559682ebaed0, undo_rec=0x559682ebb3a8 "", thr=0x559682ebae18, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #16 0x00005596807f9203 in row_purge (node=0x559682ebaed0, undo_rec=0x559682ebb3a8 "", thr=0x559682ebae18) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #17 0x00005596807f95ae in row_purge_step (thr=0x559682ebae18) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #18 0x000055968077b70e in que_thr_step (thr=0x559682ebae18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #19 0x000055968077b93e in que_run_threads_low (thr=0x559682ebae18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #20 0x000055968077bb2e in que_run_threads (thr=0x559682ebae18) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #21 0x0000559680838983 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #22 0x0000559680838b2a in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #23 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff96e7fc700 (LWP 223721)): #0 0x00007ff9a7c6ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055968074907f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559680838489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005596808385b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff9767fc700 (LWP 223713)): #0 0x00007ff9a7836644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055968073c09a in LinuxAIOHandler::collect (this=0x7ff9767fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055968073c5c3 in LinuxAIOHandler::poll (this=0x7ff9767fbb00, m1=0x7ff9767fbba8, m2=0x7ff9767fbbb0, request=0x7ff9767fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055968073c726 in os_aio_linux_handler (global_segment=5, m1=0x7ff9767fbba8, m2=0x7ff9767fbbb0, request=0x7ff9767fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055968074124a in os_aio_handler (segment=5, m1=0x7ff9767fbba8, m2=0x7ff9767fbbb0, request=0x7ff9767fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005596809944a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055968083aa47 in io_handler_thread (arg=0x5596819d5188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff9777fe700 (LWP 223711)): #0 0x00007ff9a7836644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055968073c09a in LinuxAIOHandler::collect (this=0x7ff9777fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055968073c5c3 in LinuxAIOHandler::poll (this=0x7ff9777fdb00, m1=0x7ff9777fdba8, m2=0x7ff9777fdbb0, request=0x7ff9777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055968073c726 in os_aio_linux_handler (global_segment=3, m1=0x7ff9777fdba8, m2=0x7ff9777fdbb0, request=0x7ff9777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055968074124a in os_aio_handler (segment=3, m1=0x7ff9777fdba8, m2=0x7ff9777fdbb0, request=0x7ff9777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005596809944a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055968083aa47 in io_handler_thread (arg=0x5596819d5178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff996858700 (LWP 223707)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680c07e81 in safe_cond_timedwait (cond=0x5596821e6a00 , mp=0x5596821e6940 , abstime=0x7ff996857cb0, file=0x559681055630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559680a71309 in inline_mysql_cond_timedwait (that=0x5596821e6a00 , mutex=0x5596821e6940 , abstime=0x7ff996857cb0, src_file=0x559681055678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559680a7182d in my_service_thread_sleep (control=0x55968179ad00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559680a65cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559680b99fb3 in pfs_spawn_thread (arg=0x559682c51900) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff96dffb700 (LWP 223722)): #0 0x00007ff9a6008d47 in sched_yield () from /lib64/libc.so.6 #1 0x0000559680749011 in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x0000559680862da7 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x0000559680862faf in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x0000559680838df8 in srv_do_purge (n_total_purged=0x7ff96dffad98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x0000559680839225 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff995370700 (LWP 223989)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748d7d in os_event::wait (this=0x559682e31cb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559680748803 in os_event::wait_low (this=0x559682e31cb0, reset_sig_count=2966) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559680748b74 in os_event_wait_low (event=0x559682e31cb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005596807193c3 in log_write_up_to (lsn=2795494, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x0000559680884f5f in trx_flush_log_if_needed_low (lsn=2795494) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #6 0x0000559680884fa8 in trx_flush_log_if_needed (lsn=2795494, trx=0x7ff995c5a390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #7 0x0000559680886dd1 in trx_commit_complete_for_mysql (trx=0x7ff995c5a390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #8 0x0000559680669ca6 in innobase_commit (hton=0x559682c6dc30, thd=0x7ff92c0b53f0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4658 #9 0x000055968043da1d in commit_one_phase_2 (thd=0x7ff92c0b53f0, all=true, trans=0x7ff92c0b8968, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1741 #10 0x000055968043d91f in ha_commit_one_phase (thd=0x7ff92c0b53f0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1721 #11 0x000055968043cc98 in ha_commit_trans (thd=0x7ff92c0b53f0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1534 #12 0x000055968027aac6 in trans_commit (thd=0x7ff92c0b53f0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #13 0x00005596806478a3 in Wsrep_storage_service::commit (this=0x7ff92c0cbbc0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #14 0x0000559680cb2ba1 in wsrep::transaction::certify_fragment (this=0x7ff92c006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #15 0x0000559680cb237b in wsrep::transaction::streaming_step (this=0x7ff92c006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #16 0x0000559680caf59a in wsrep::transaction::after_row (this=0x7ff92c006bf0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #17 0x00005596804510b6 in wsrep::client_state::after_row (this=0x7ff92c006b80) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #18 0x0000559680439000 in wsrep_after_row (thd=0x7ff92c000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #19 0x000055968044b0df in wsrep_after_row (thd=0x7ff92c000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #20 0x000055968044be15 in handler::ha_write_row (this=0x7ff92c0a0278, buf=0x7ff92c07f490 "\376\257\020") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6673 #21 0x00005596800acacf in write_record (thd=0x7ff92c000b00, table=0x7ff92c09f410, info=0x7ff92c0175a0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #22 0x00005596800b1dce in select_insert::send_data (this=0x7ff92c017558, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #23 0x000055968016fdcf in end_send (join=0x7ff92c017608, join_tab=0x7ff92c0ac328, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:21196 #24 0x00005596802b4eff in JOIN_CACHE::generate_full_extensions (this=0x7ff92c0accf0, rec_ptr=0x7ff92c0af66c "\220") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #25 0x00005596802b4c1a in JOIN_CACHE::join_matching_records (this=0x7ff92c0accf0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #26 0x00005596802b45e6 in JOIN_CACHE::join_records (this=0x7ff92c0accf0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #27 0x00005596802b4780 in JOIN_CACHE::join_records (this=0x7ff92c0acbc0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #28 0x00005596802b4780 in JOIN_CACHE::join_records (this=0x7ff92c0aca90, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #29 0x000055968016ca6d in sub_select_cache (join=0x7ff92c017608, join_tab=0x7ff92c0ab7d0, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19786 #30 0x000055968016cc7a in sub_select (join=0x7ff92c017608, join_tab=0x7ff92c0ab408, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19957 #31 0x000055968016c445 in do_select (join=0x7ff92c017608, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19548 #32 0x0000559680143079 in JOIN::exec_inner (this=0x7ff92c017608) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4364 #33 0x0000559680142226 in JOIN::exec (this=0x7ff92c017608) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4146 #34 0x000055968014390c in mysql_select (thd=0x7ff92c000b00, tables=0x7ff92c015038, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7ff92c017558, unit=0x7ff92c004a18, select_lex=0x7ff92c014ac8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4578 #35 0x0000559680133ea6 in handle_select (thd=0x7ff92c000b00, lex=0x7ff92c004950, result=0x7ff92c017558, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:424 #36 0x00005596800f5b44 in mysql_execute_command (thd=0x7ff92c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4896 #37 0x00005596801010f1 in mysql_parse (thd=0x7ff92c000b00, rawbuf=0x7ff92c014198 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7ff99536f040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #38 0x0000559680100720 in wsrep_mysql_parse (thd=0x7ff92c000b00, rawbuf=0x7ff92c014198 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7ff99536f040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #39 0x00005596800ec46d in dispatch_command (command=COM_QUERY, thd=0x7ff92c000b00, packet=0x7ff92c00a761 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=86, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #40 0x00005596800eacbb in do_command (thd=0x7ff92c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #41 0x0000559680265c9e in do_handle_one_connection (connect=0x559682f2d6a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #42 0x00005596802659e2 in handle_one_connection (arg=0x559682f2d6a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #43 0x0000559680b99fb3 in pfs_spawn_thread (arg=0x559682ad0ba0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #44 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #45 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff95a7fc700 (LWP 223728)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680748d7d in os_event::wait (this=0x559682c99c70) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559680748803 in os_event::wait_low (this=0x559682c99c70, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559680748b74 in os_event_wait_low (event=0x559682c99c70, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005596808fdcbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff976ffd700 (LWP 223712)): #0 0x00007ff9a7836644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055968073c09a in LinuxAIOHandler::collect (this=0x7ff976ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055968073c5c3 in LinuxAIOHandler::poll (this=0x7ff976ffcb00, m1=0x7ff976ffcba8, m2=0x7ff976ffcbb0, request=0x7ff976ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055968073c726 in os_aio_linux_handler (global_segment=4, m1=0x7ff976ffcba8, m2=0x7ff976ffcbb0, request=0x7ff976ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055968074124a in os_aio_handler (segment=4, m1=0x7ff976ffcba8, m2=0x7ff976ffcbb0, request=0x7ff976ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005596809944a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055968083aa47 in io_handler_thread (arg=0x5596819d5180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff977fff700 (LWP 223710)): #0 0x00007ff9a7836644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055968073c09a in LinuxAIOHandler::collect (this=0x7ff977ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055968073c5c3 in LinuxAIOHandler::poll (this=0x7ff977ffeb00, m1=0x7ff977ffeba8, m2=0x7ff977ffebb0, request=0x7ff977ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055968073c726 in os_aio_linux_handler (global_segment=2, m1=0x7ff977ffeba8, m2=0x7ff977ffebb0, request=0x7ff977ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055968074124a in os_aio_handler (segment=2, m1=0x7ff977ffeba8, m2=0x7ff977ffebb0, request=0x7ff977ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005596809944a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055968083aa47 in io_handler_thread (arg=0x5596819d5170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff995206700 (LWP 223708)): #0 0x00007ff9a7836644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055968073c09a in LinuxAIOHandler::collect (this=0x7ff995205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055968073c5c3 in LinuxAIOHandler::poll (this=0x7ff995205b00, m1=0x7ff995205ba8, m2=0x7ff995205bb0, request=0x7ff995205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055968073c726 in os_aio_linux_handler (global_segment=0, m1=0x7ff995205ba8, m2=0x7ff995205bb0, request=0x7ff995205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055968074124a in os_aio_handler (segment=0, m1=0x7ff995205ba8, m2=0x7ff995205bb0, request=0x7ff995205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005596809944a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055968083aa47 in io_handler_thread (arg=0x5596819d5160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff9977fe700 (LWP 223695)): #0 0x00007ff9a7c69965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff99fb60bd7 in gu::Lock::wait (cond=..., this=0x7ff9977fd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff99fb6371b in RecvBuf::front (this=this@entry=0x559682b91a68, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007ff99fb612b6 in gcomm_recv (backend=, msg=0x559682b519f0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007ff99fb5525c in core_msg_recv (backend=backend@entry=0x559682b51b38, recv_msg=recv_msg@entry=0x559682b519f0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007ff99fb584d4 in gcs_core_recv (conn=0x559682b51990, recv_act=recv_act@entry=0x7ff9977fdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007ff99fb5f748 in gcs_recv_thread (arg=0x559682b51710) at gcs/src/gcs.cpp:1398 #7 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff9a06be700 (LWP 223682)): #0 0x00007ff9a7c69d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559680c07e81 in safe_cond_timedwait (cond=0x5596821f91c0 , mp=0x5596821f9100 , abstime=0x7ff9a06bdd70, file=0x559681099e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559680c08e3b in inline_mysql_cond_timedwait (that=0x5596821f91c0 , mutex=0x5596821f9100 , abstime=0x7ff9a06bdd70, src_file=0x559681099ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559680c0999e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559680b99fb3 in pfs_spawn_thread (arg=0x559682b12d40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff9a7c65dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff9a6023ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff9a8076880 (LWP 223681)): #0 0x00007ff9a7c6a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559680c02550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559680437ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff9a601920d in poll () from /lib64/libc.so.6 #5 0x000055967ffc1f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055967ffc11b4 in mysqld_main (argc=162, argv=0x5596829cb140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055967ffb5695 in main (argc=23, argv=0x7fff4d452b88) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.223748' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.2/data/core.223748 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 223748] [New LWP 223750] [New LWP 223752] [New LWP 223751] [New LWP 223753] [New LWP 223754] [New LWP 223933] [New LWP 223932] [New LWP 223934] [New LWP 223935] [New LWP 223944] [New LWP 223938] [New LWP 223946] [New LWP 223950] [New LWP 223951] [New LWP 223954] [New LWP 223957] [New LWP 223956] [New LWP 223960] [New LWP 223756] [New LWP 223936] [New LWP 223943] [New LWP 223947] [New LWP 223948] [New LWP 223955] [New LWP 223755] [New LWP 223939] [New LWP 223942] [New LWP 223945] [New LWP 223949] [New LWP 223952] [New LWP 223953] [New LWP 223937] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fbb6772d9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fbb6772d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000564943996550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005649431cbebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fbb65adc20d in poll () from /lib64/libc.so.6 #5 0x0000564942d55f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000564942d551b4 in mysqld_main (argc=163, argv=0x5649453d6140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000564942d49695 in main (argc=23, argv=0x7ffd219d2d48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fbb45ffa700 (LWP 223937)): #0 0x00007fbb6772fd1d in fsync () from /lib64/libpthread.so.0 #1 0x00005649434d1749 in os_file_fsync_posix (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005649434d1a9f in os_file_flush_func (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056494371a1fa in pfs_os_file_flush_func (file=..., src_file=0x564943d9f588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056494371d055 in fil_flush_low (space=0x5649458aeea0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005649437289f8 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x0000564943728c7f in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4548 #7 0x00005649436a5256 in buf_dblwr_update (bpage=0x7fbb4d265d70, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:767 #8 0x00005649436acd28 in buf_flush_write_complete (bpage=0x7fbb4d265d70, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000056494369b910 in buf_page_io_complete (bpage=0x7fbb4d265d70, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6216 #10 0x0000564943728752 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4435 #11 0x00005649435cea47 in io_handler_thread (arg=0x564944769180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fbb0dffb700 (LWP 223953)): #0 0x00007fbb6772fe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005649434dd07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000564943683b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fbb0e7fc700 (LWP 223952)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dcd7d in os_event::wait (this=0x5649456a57d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649434dc803 in os_event::wait_low (this=0x5649456a57d0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649434dcb74 in os_event_wait_low (event=0x5649456a57d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649436a919b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fbb0ffff700 (LWP 223949)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dcd7d in os_event::wait (this=0x5649456a5650) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649434dc803 in os_event::wait_low (this=0x5649456a5650, reset_sig_count=7457) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649434dcb74 in os_event_wait_low (event=0x5649456a5650, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649435c8f21 in srv_resume_thread (slot=0x56494452a9f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005649435ccb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fbb1e7fc700 (LWP 223945)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dc698 in os_event::timed_wait (this=0x5649458af880, abstime=0x7fbb1e7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649434dc9af in os_event::wait_time_low (this=0x5649458af880, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649434dcb4f in os_event_wait_time_low (event=0x5649458af880, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649437120e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fbb1f7fe700 (LWP 223942)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dc698 in os_event::timed_wait (this=0x56494587d7c0, abstime=0x7fbb1f7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649434dc9af in os_event::wait_time_low (this=0x56494587d7c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649434dcb4f in os_event_wait_time_low (event=0x56494587d7c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649434a94f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fbb44ff8700 (LWP 223939)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dc698 in os_event::timed_wait (this=0x5649456a5850, abstime=0x7fbb44ff7af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649434dc9af in os_event::wait_time_low (this=0x5649456a5850, time_in_usec=594000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649434dcb4f in os_event_wait_time_low (event=0x5649456a5850, time_in_usec=594000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649436b31fe in pc_sleep_if_needed (next_loop_time=1551280639260, sig_count=2, cur_time=1551280638666) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x00005649436b4137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fbb5c374700 (LWP 223755)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056494399bb7f in safe_cond_wait (cond=0x7fbb3c015938, mp=0x7fbb3c015888, file=0x564943b0a4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005649430e94ba in inline_mysql_cond_wait (that=0x7fbb3c015938, mutex=0x7fbb3c015888, src_file=0x564943b0a7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005649430ec34e in Wsrep_thd_queue::pop_front (this=0x7fbb3c015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005649430eb0d0 in wsrep_post_rollback_process (post_rollbacker=0x7fbb3c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x00005649430db587 in start_wsrep_THD (arg=0x5649455b68b0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fbb0cff9700 (LWP 223955)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056494399bb7f in safe_cond_wait (cond=0x7fbb0cff8c70, mp=0x7fbb0cff8cb0, file=0x564943c72920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005649433f2cad in inline_mysql_cond_wait (that=0x7fbb0cff8c70, mutex=0x7fbb0cff8cb0, src_file=0x564943c74348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005649433f779b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000056494392dfb3 in pfs_spawn_thread (arg=0x564945860140) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fbb1cff9700 (LWP 223948)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dcd7d in os_event::wait (this=0x5649456a54d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649434dc803 in os_event::wait_low (this=0x5649456a54d0, reset_sig_count=2130) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649434dcb74 in os_event_wait_low (event=0x5649456a54d0, reset_sig_count=2130) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649435c8f21 in srv_resume_thread (slot=0x56494452a948 , sig_count=2130, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005649435ccf32 in srv_purge_coordinator_suspend (slot=0x56494452a948 , rseg_history_len=34) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005649435cd1c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fbb1d7fa700 (LWP 223947)): #0 0x00007fbb6772fe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005649434dd07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005649435cc489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005649435cc5b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fbb1ffff700 (LWP 223943)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dc698 in os_event::timed_wait (this=0x5649456a56d0, abstime=0x7fbb1fffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649434dc9af in os_event::wait_time_low (this=0x5649456a56d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649434dcb4f in os_event_wait_time_low (event=0x5649456a56d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649435cb12a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fbb467fb700 (LWP 223936)): #0 0x00007fbb672f9644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649434d009a in LinuxAIOHandler::collect (this=0x7fbb467fab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649434d05c3 in LinuxAIOHandler::poll (this=0x7fbb467fab00, m1=0x7fbb467faba8, m2=0x7fbb467fabb0, request=0x7fbb467fabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649434d0726 in os_aio_linux_handler (global_segment=3, m1=0x7fbb467faba8, m2=0x7fbb467fabb0, request=0x7fbb467fabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649434d524a in os_aio_handler (segment=3, m1=0x7fbb467faba8, m2=0x7fbb467fabb0, request=0x7fbb467fabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005649437284a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005649435cea47 in io_handler_thread (arg=0x564944769178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fbb5c329700 (LWP 223756)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbb5f53d272 in fifo_lock_get (q=q@entry=0x564945582c80) at galerautils/src/gu_fifo.c:262 #2 0x00007fbb5f53d8e1 in gu_fifo_get_head (q=0x564945582c80, err=0x7fbb5c32806c) at galerautils/src/gu_fifo.c:330 #3 0x00007fbb5f61f43c in gcs_recv (conn=0x56494555c600, action=0x7fbb5c3280b0) at gcs/src/gcs.cpp:2024 #4 0x00007fbb5f64c50c in galera::GcsActionSource::process (this=0x56494558d020, recv_ctx=0x7fbb5c3284d0, exit_loop=@0x7fbb5c32829e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fbb5f66b433 in galera::ReplicatorSMM::async_recv (this=0x5649455543b0, recv_ctx=0x7fbb5c3284d0) at galera/src/replicator_smm.cpp:383 #6 0x0000564943a508a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5649454ca250, applier_ctx=0x7fbb5c3284d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005649430e9a69 in wsrep_replication_process (thd=0x7fbb30000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005649430db587 in start_wsrep_THD (arg=0x5649455acdb0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fbb5c06b700 (LWP 223960)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056494399be81 in safe_cond_timedwait (cond=0x5649446ed4a0 , mp=0x5649446efe40 , abstime=0x7fbb5c06ac60, file=0x564943a57b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564942d4a015 in inline_mysql_cond_timedwait (that=0x5649446ed4a0 , mutex=0x5649446efe40 , abstime=0x7fbb5c06ac60, src_file=0x564943a59b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564942d4eb08 in cache_thread (thd=0x7fbb34000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x0000564942d4edb6 in one_thread_per_connection_end (thd=0x7fbb34000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x0000564942ff9d28 in do_handle_one_connection (connect=0x564945a16d40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000564942ff99e2 in handle_one_connection (arg=0x564945a16d40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000056494392dfb3 in pfs_spawn_thread (arg=0x564945a2eee0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fbb5c101700 (LWP 223956)): #0 0x00007fbb65a2040a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000564942d49714 in my_sigwait (set=0x7fbb5c100cf0, sig=0x7fbb5c100cb4, code=0x7fbb5c100cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000564942d4fa71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x000056494392dfb3 in pfs_spawn_thread (arg=0x564945668cb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fbb5c0b6700 (LWP 223957)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056494399bb7f in safe_cond_wait (cond=0x5649446ed4e0 , mp=0x5649446f0800 , file=0x564943a67488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564942d7afe2 in inline_mysql_cond_wait (that=0x5649446ed4e0 , mutex=0x5649446f0800 , src_file=0x564943a67ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000564942d7c874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056494392dfb3 in pfs_spawn_thread (arg=0x5649459a85b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fbb0d7fa700 (LWP 223954)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dcd7d in os_event::wait (this=0x5649456a58d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649434dc803 in os_event::wait_low (this=0x5649456a58d0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649434dcb74 in os_event_wait_low (event=0x5649456a58d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564943691cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fbb0effd700 (LWP 223951)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dcd7d in os_event::wait (this=0x5649456a55d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649434dc803 in os_event::wait_low (this=0x5649456a55d0, reset_sig_count=8056) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649434dcb74 in os_event_wait_low (event=0x5649456a55d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649435c8f21 in srv_resume_thread (slot=0x56494452a9b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005649435ccb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fbb0f7fe700 (LWP 223950)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dcd7d in os_event::wait (this=0x5649456a5550) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649434dc803 in os_event::wait_low (this=0x5649456a5550, reset_sig_count=8896) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649434dcb74 in os_event_wait_low (event=0x5649456a5550, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649435c8f21 in srv_resume_thread (slot=0x56494452a980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005649435ccb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fbb1dffb700 (LWP 223946)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dc698 in os_event::timed_wait (this=0x56494584d340, abstime=0x7fbb1dffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649434dc9af in os_event::wait_time_low (this=0x56494584d340, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649434dcb4f in os_event_wait_time_low (event=0x56494584d340, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056494362b4c9 in ib_wqueue_timedwait (wq=0x5649458b2970, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056494376c73a in fts_optimize_thread (arg=0x5649458b2970) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fbb457f9700 (LWP 223938)): #0 0x00007fbb672f9644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649434d009a in LinuxAIOHandler::collect (this=0x7fbb457f8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649434d05c3 in LinuxAIOHandler::poll (this=0x7fbb457f8b00, m1=0x7fbb457f8ba8, m2=0x7fbb457f8bb0, request=0x7fbb457f8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649434d0726 in os_aio_linux_handler (global_segment=5, m1=0x7fbb457f8ba8, m2=0x7fbb457f8bb0, request=0x7fbb457f8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649434d524a in os_aio_handler (segment=5, m1=0x7fbb457f8ba8, m2=0x7fbb457f8bb0, request=0x7fbb457f8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005649437284a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005649435cea47 in io_handler_thread (arg=0x564944769188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fbb1effd700 (LWP 223944)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649434dc698 in os_event::timed_wait (this=0x5649456a5750, abstime=0x7fbb1effcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649434dc9af in os_event::wait_time_low (this=0x5649456a5750, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649434dcb4f in os_event_wait_time_low (event=0x5649456a5750, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649435cac08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fbb46ffc700 (LWP 223935)): #0 0x00007fbb672f9644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649434d009a in LinuxAIOHandler::collect (this=0x7fbb46ffbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649434d05c3 in LinuxAIOHandler::poll (this=0x7fbb46ffbb00, m1=0x7fbb46ffbba8, m2=0x7fbb46ffbbb0, request=0x7fbb46ffbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649434d0726 in os_aio_linux_handler (global_segment=2, m1=0x7fbb46ffbba8, m2=0x7fbb46ffbbb0, request=0x7fbb46ffbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649434d524a in os_aio_handler (segment=2, m1=0x7fbb46ffbba8, m2=0x7fbb46ffbbb0, request=0x7fbb46ffbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005649437284a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005649435cea47 in io_handler_thread (arg=0x564944769170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fbb477fd700 (LWP 223934)): #0 0x00007fbb672f9644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649434d009a in LinuxAIOHandler::collect (this=0x7fbb477fcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649434d05c3 in LinuxAIOHandler::poll (this=0x7fbb477fcb00, m1=0x7fbb477fcba8, m2=0x7fbb477fcbb0, request=0x7fbb477fcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649434d0726 in os_aio_linux_handler (global_segment=1, m1=0x7fbb477fcba8, m2=0x7fbb477fcbb0, request=0x7fbb477fcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649434d524a in os_aio_handler (segment=1, m1=0x7fbb477fcba8, m2=0x7fbb477fcbb0, request=0x7fbb477fcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005649437284a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005649435cea47 in io_handler_thread (arg=0x564944769168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fbb4e670700 (LWP 223932)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056494399be81 in safe_cond_timedwait (cond=0x564944f7aa00 , mp=0x564944f7a940 , abstime=0x7fbb4e66fcb0, file=0x564943de9630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564943805309 in inline_mysql_cond_timedwait (that=0x564944f7aa00 , mutex=0x564944f7a940 , abstime=0x7fbb4e66fcb0, src_file=0x564943de9678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056494380582d in my_service_thread_sleep (control=0x56494452ed00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005649437f9cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056494392dfb3 in pfs_spawn_thread (arg=0x56494565d7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fbb4ce1e700 (LWP 223933)): #0 0x00007fbb672f9644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649434d009a in LinuxAIOHandler::collect (this=0x7fbb4ce1db00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649434d05c3 in LinuxAIOHandler::poll (this=0x7fbb4ce1db00, m1=0x7fbb4ce1dba8, m2=0x7fbb4ce1dbb0, request=0x7fbb4ce1dbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649434d0726 in os_aio_linux_handler (global_segment=0, m1=0x7fbb4ce1dba8, m2=0x7fbb4ce1dbb0, request=0x7fbb4ce1dbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649434d524a in os_aio_handler (segment=0, m1=0x7fbb4ce1dba8, m2=0x7fbb4ce1dbb0, request=0x7fbb4ce1dbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005649437284a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005649435cea47 in io_handler_thread (arg=0x564944769160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fbb5c3bf700 (LWP 223754)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056494399bb7f in safe_cond_wait (cond=0x7fbb38017e98, mp=0x7fbb38017de8, file=0x564943b0a4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005649430e94ba in inline_mysql_cond_wait (that=0x7fbb38017e98, mutex=0x7fbb38017de8, src_file=0x564943b0a7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005649430ec34e in Wsrep_thd_queue::pop_front (this=0x7fbb38017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005649430eac54 in wsrep_rollback_process (rollbacker=0x7fbb38001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x00005649430db587 in start_wsrep_THD (arg=0x5649455b6d50) at /home/stepan/galera/git/10.4worker[2] Trying to dump core for [mysqltest - pid: 224987, winpid: 224987] worker[2] Trying to dump core for [mysqld.1 - pid: 224697, winpid: 224697] worker[2] Trying to dump core for [mysqld.2 - pid: 224760, winpid: 224760] /server/sql/wsrep_mysqld.cc:2693 #6 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fbb47ffe700 (LWP 223753)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbb5f623bd7 in gu::Lock::wait (cond=..., this=0x7fbb47ffd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fbb5f62671b in RecvBuf::front (this=this@entry=0x5649455949d8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fbb5f6242b6 in gcomm_recv (backend=, msg=0x56494555c8e0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fbb5f61825c in core_msg_recv (backend=backend@entry=0x56494555ca28, recv_msg=recv_msg@entry=0x56494555c8e0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fbb5f61b4d4 in gcs_core_recv (conn=0x56494555c880, recv_act=recv_act@entry=0x7fbb47ffdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fbb5f622748 in gcs_recv_thread (arg=0x56494555c600) at gcs/src/gcs.cpp:1398 #7 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fbb5f044700 (LWP 223751)): #0 0x00007fbb6772c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbb5f649377 in gu::Lock::wait (cond=..., this=0x7fbb5f043bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fbb5f649409 in galera::ServiceThd::thd_func (arg=0x564945554ac8) at galera/src/galera_service_thd.cpp:30 #3 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fbb5cbc0700 (LWP 223752)): #0 0x00007fbb65ae7483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fbb5f5eaa68 in asio::detail::epoll_reactor::run (this=0x5649455951c0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fbb5f5ef160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x564945595060, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fbb5f5ef412 in asio::detail::task_io_service::run (this=0x564945595060, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fbb5f606e6d in asio::io_service::run (this=0x564945593d80) at asio/asio/impl/io_service.ipp:58 #5 0x00007fbb5f6075c1 in gcomm::AsioProtonet::event_loop (this=0x564945593ce0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fbb5f62565d in GCommConn::run (this=0x5649455947f0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fbb5f628709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fbb60181700 (LWP 223750)): #0 0x00007fbb6772cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056494399be81 in safe_cond_timedwait (cond=0x564944f8d1c0 , mp=0x564944f8d100 , abstime=0x7fbb60180d70, file=0x564943e2de90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056494399ce3b in inline_mysql_cond_timedwait (that=0x564944f8d1c0 , mutex=0x564944f8d100 , abstime=0x7fbb60180d70, src_file=0x564943e2ded8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056494399d99e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056494392dfb3 in pfs_spawn_thread (arg=0x56494551dde0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbb67728dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbb65ae6ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fbb67b39880 (LWP 223748)): #0 0x00007fbb6772d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000564943996550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005649431cbebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fbb65adc20d in poll () from /lib64/libc.so.6 #5 0x0000564942d55f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000564942d551b4 in mysqld_main (argc=163, argv=0x5649453d6140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000564942d49695 in main (argc=23, argv=0x7ffd219d2d48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.mysql-wsrep-features#8-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#8 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1018 'innodb' w3 [ pass ] 9546 galera_sr.GCF-1060 'innodb' w4 [ pass ] 615135 galera_sr.GCF-585 'innodb' w3 [ pass ] 30605 galera_sr.GCF-561 'innodb' w4 [ pass ] 11848 galera_sr.GCF-597 'innodb' w3 [ pass ] 9117 galera_sr.GCF-623 'innodb' w4 [ pass ] 9035 galera_sr.GCF-620 'innodb' w3 [ pass ] 6741 galera_sr.galera_sr_small_gcache 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 17:27:04 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr_small_gcache.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE t1 (f1 INTEGER) ENGINE=InnoDB; INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.224698' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.1/data/core.224698 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 224698] [New LWP 224703] [New LWP 224704] [New LWP 224706] [New LWP 224707] [New LWP 224719] [New LWP 224715] [New LWP 224720] [New LWP 224716] [New LWP 224728] [New LWP 224717] [New LWP 224729] [New LWP 224699] [New LWP 224726] [New LWP 224701] [New LWP 224712] [New LWP 224721] [New LWP 224702] [New LWP 224711] [New LWP 224724] [New LWP 224722] [New LWP 224725] [New LWP 224733] [New LWP 224718] [New LWP 224723] [New LWP 224708] [New LWP 224732] [New LWP 224709] [New LWP 224727] [New LWP 224710] [New LWP 224713] [New LWP 224700] [New LWP 224705] [New LWP 224989] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f45c1f0a9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f45c1f0a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005620daacc550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005620da301ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f45c02b920d in poll () from /lib64/libc.so.6 #5 0x00005620d9e8bf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005620d9e8b1b4 in mysqld_main (argc=162, argv=0x5620dcfc8140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005620d9e7f695 in main (argc=23, argv=0x7ffdd8f69578) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f4593fb4700 (LWP 224989)): #0 0x00007f45c1f0cd1d in fsync () from /lib64/libpthread.so.0 #1 0x00005620da607749 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005620da607a9f in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005620da8501fa in pfs_os_file_flush_func (file=..., src_file=0x5620daed5588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005620da853055 in fil_flush_low (space=0x5620dd498a20, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x00005620da85e9f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x00005620da5e2e31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005620da5e3893 in log_write_up_to (lsn=3482346, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005620da74ef5f in trx_flush_log_if_needed_low (lsn=3482346) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005620da74efa8 in trx_flush_log_if_needed (lsn=3482346, trx=0x7f45b08d8390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005620da750dd1 in trx_commit_complete_for_mysql (trx=0x7f45b08d8390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005620da533ca6 in innobase_commit (hton=0x5620dd262ce0, thd=0x7f454408d690, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4658 #12 0x00005620da307a1d in commit_one_phase_2 (thd=0x7f454408d690, all=true, trans=0x7f4544090c08, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1741 #13 0x00005620da30791f in ha_commit_one_phase (thd=0x7f454408d690, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1721 #14 0x00005620da306c98 in ha_commit_trans (thd=0x7f454408d690, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1534 #15 0x00005620da144ac6 in trans_commit (thd=0x7f454408d690) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005620da5118a3 in Wsrep_storage_service::commit (this=0x7f454409ebb0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x00005620dab7cba1 in wsrep::transaction::certify_fragment (this=0x7f4544006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1190 #18 0x00005620dab7c37b in wsrep::transaction::streaming_step (this=0x7f4544006bf0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1073 #19 0x00005620dab7959a in wsrep::transaction::after_row (this=0x7f4544006bf0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:240 #20 0x00005620da31b0b6 in wsrep::client_state::after_row (this=0x7f4544006b80) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x00005620da303000 in wsrep_after_row (thd=0x7f4544000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:131 #22 0x00005620da3150df in wsrep_after_row (thd=0x7f4544000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6483 #23 0x00005620da315e15 in handler::ha_write_row (this=0x7f454407d918, buf=0x7f454407b530 "\375\001") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6673 #24 0x00005620d9f76acf in write_record (thd=0x7f4544000b00, table=0x7f454407cab0, info=0x7f4544017438) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x00005620d9f7bdce in select_insert::send_data (this=0x7f45440173f0, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x00005620da039dcf in end_send (join=0x7f45440174a0, join_tab=0x7f4544088fa8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:21196 #27 0x00005620da17eeff in JOIN_CACHE::generate_full_extensions (this=0x7f45440897a0, rec_ptr=0x7f454408d038 "\\\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x00005620da17ec1a in JOIN_CACHE::join_matching_records (this=0x7f45440897a0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x00005620da17e5e6 in JOIN_CACHE::join_records (this=0x7f45440897a0, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x00005620da17e780 in JOIN_CACHE::join_records (this=0x7f4544089670, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x00005620da17e780 in JOIN_CACHE::join_records (this=0x7f4544089540, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x00005620da036a6d in sub_select_cache (join=0x7f45440174a0, join_tab=0x7f4544088450, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19786 #33 0x00005620da036c7a in sub_select (join=0x7f45440174a0, join_tab=0x7f4544088088, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19957 #34 0x00005620da036445 in do_select (join=0x7f45440174a0, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19548 #35 0x00005620da00d079 in JOIN::exec_inner (this=0x7f45440174a0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4364 #36 0x00005620da00c226 in JOIN::exec (this=0x7f45440174a0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4146 #37 0x00005620da00d90c in mysql_select (thd=0x7f4544000b00, tables=0x7f4544014ed0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f45440173f0, unit=0x7f4544004a18, select_lex=0x7f4544014980) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4578 #38 0x00005620d9ffdea6 in handle_select (thd=0x7f4544000b00, lex=0x7f4544004950, result=0x7f45440173f0, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:424 #39 0x00005620d9fbfb44 in mysql_execute_command (thd=0x7f4544000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4896 #40 0x00005620d9fcb0f1 in mysql_parse (thd=0x7f4544000b00, rawbuf=0x7f4544014198 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7f4593fb3040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #41 0x00005620d9fca720 in wsrep_mysql_parse (thd=0x7f4544000b00, rawbuf=0x7f4544014198 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7f4593fb3040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #42 0x00005620d9fb646d in dispatch_command (command=COM_QUERY, thd=0x7f4544000b00, packet=0x7f454400a761 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=71, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #43 0x00005620d9fb4cbb in do_command (thd=0x7f4544000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #44 0x00005620da12fc9e in do_handle_one_connection (connect=0x5620dd5229e0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #45 0x00005620da12f9e2 in handle_one_connection (arg=0x5620dd5229e0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #46 0x00005620daa63fb3 in pfs_spawn_thread (arg=0x5620dd524c40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f45b8689700 (LWP 224705)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45b9d1a272 in fifo_lock_get (q=q@entry=0x5620dd174c50) at galerautils/src/gu_fifo.c:262 #2 0x00007f45b9d1a8e1 in gu_fifo_get_head (q=0x5620dd174c50, err=0x7f45b868806c) at galerautils/src/gu_fifo.c:330 #3 0x00007f45b9dfc43c in gcs_recv (conn=0x5620dd14e6b0, action=0x7f45b86880b0) at gcs/src/gcs.cpp:2024 #4 0x00007f45b9e2950c in galera::GcsActionSource::process (this=0x5620dd14e300, recv_ctx=0x7f45b86884d0, exit_loop=@0x7f45b868829e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f45b9e48433 in galera::ReplicatorSMM::async_recv (this=0x5620dd146330, recv_ctx=0x7f45b86884d0) at galera/src/replicator_smm.cpp:383 #6 0x00005620dab868a8 in wsrep::wsrep_provider_v26::run_applier (this=0x5620dd0cd760, applier_ctx=0x7f45b86884d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005620da21fa69 in wsrep_replication_process (thd=0x7f45a0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005620da211587 in start_wsrep_THD (arg=0x5620dd1963f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f45b9821700 (LWP 224700)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45b9e26377 in gu::Lock::wait (cond=..., this=0x7f45b9820bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f45b9e26409 in galera::ServiceThd::thd_func (arg=0x5620dd146a48) at galera/src/galera_service_thd.cpp:30 #3 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f4590df9700 (LWP 224713)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612698 in os_event::timed_wait (this=0x5620dd28ec30, abstime=0x7f4590df8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005620da6129af in os_event::wait_time_low (this=0x5620dd28ec30, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005620da612b4f in os_event_wait_time_low (event=0x5620dd28ec30, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005620da7e91fe in pc_sleep_if_needed (next_loop_time=1551281163086, sig_count=2, cur_time=1551281162086) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x00005620da7ea137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f45925fc700 (LWP 224710)): #0 0x00007f45c1ad6644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005620da60609a in LinuxAIOHandler::collect (this=0x7f45925fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005620da6065c3 in LinuxAIOHandler::poll (this=0x7f45925fbb00, m1=0x7f45925fbba8, m2=0x7f45925fbbb0, request=0x7f45925fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005620da606726 in os_aio_linux_handler (global_segment=3, m1=0x7f45925fbba8, m2=0x7f45925fbbb0, request=0x7f45925fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005620da60b24a in os_aio_handler (segment=3, m1=0x7f45925fbba8, m2=0x7f45925fbbb0, request=0x7f45925fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005620da85e4a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005620da704a47 in io_handler_thread (arg=0x5620db89f178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f45797fa700 (LWP 224727)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612d7d in os_event::wait (this=0x5620dd28ecb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005620da612803 in os_event::wait_low (this=0x5620dd28ecb0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005620da612b74 in os_event_wait_low (event=0x5620dd28ecb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005620da7c7cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4592dfd700 (LWP 224709)): #0 0x00007f45c1ad6644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005620da60609a in LinuxAIOHandler::collect (this=0x7f4592dfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005620da6065c3 in LinuxAIOHandler::poll (this=0x7f4592dfcb00, m1=0x7f4592dfcba8, m2=0x7f4592dfcbb0, request=0x7f4592dfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005620da606726 in os_aio_linux_handler (global_segment=2, m1=0x7f4592dfcba8, m2=0x7f4592dfcbb0, request=0x7f4592dfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005620da60b24a in os_aio_handler (segment=2, m1=0x7f4592dfcba8, m2=0x7f4592dfcbb0, request=0x7f4592dfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005620da85e4a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005620da704a47 in io_handler_thread (arg=0x5620db89f170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f45b0084700 (LWP 224732)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620daad1b7f in safe_cond_wait (cond=0x5620db8234e0 , mp=0x5620db826800 , file=0x5620dab9d488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005620d9eb0fe2 in inline_mysql_cond_wait (that=0x5620db8234e0 , mutex=0x5620db826800 , src_file=0x5620dab9dff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005620d9eb2874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005620daa63fb3 in pfs_spawn_thread (arg=0x5620dd590da0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f45935fe700 (LWP 224708)): #0 0x00007f45c1ad6644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005620da60609a in LinuxAIOHandler::collect (this=0x7f45935fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005620da6065c3 in LinuxAIOHandler::poll (this=0x7f45935fdb00, m1=0x7f45935fdba8, m2=0x7f45935fdbb0, request=0x7f45935fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005620da606726 in os_aio_linux_handler (global_segment=1, m1=0x7f45935fdba8, m2=0x7f45935fdbb0, request=0x7f45935fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005620da60b24a in os_aio_handler (segment=1, m1=0x7f45935fdba8, m2=0x7f45935fdbb0, request=0x7f45935fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005620da85e4a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005620da704a47 in io_handler_thread (arg=0x5620db89f168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f457b7fe700 (LWP 224723)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612d7d in os_event::wait (this=0x5620dd28e9b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005620da612803 in os_event::wait_low (this=0x5620dd28e9b0, reset_sig_count=11630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005620da612b74 in os_event_wait_low (event=0x5620dd28e9b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005620da6fef21 in srv_resume_thread (slot=0x5620db6609b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005620da702b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f458bfff700 (LWP 224718)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612698 in os_event::timed_wait (this=0x5620dd499060, abstime=0x7f458bffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005620da6129af in os_event::wait_time_low (this=0x5620dd499060, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005620da612b4f in os_event_wait_time_low (event=0x5620dd499060, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005620da8480e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f4593fff700 (LWP 224733)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620daad1e81 in safe_cond_timedwait (cond=0x5620db8234a0 , mp=0x5620db825e40 , abstime=0x7f4593ffec60, file=0x5620dab8db08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005620d9e80015 in inline_mysql_cond_timedwait (that=0x5620db8234a0 , mutex=0x5620db825e40 , abstime=0x7f4593ffec60, src_file=0x5620dab8fb50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005620d9e84b08 in cache_thread (thd=0x7f454c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x00005620d9e84db6 in one_thread_per_connection_end (thd=0x7f454c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x00005620da12fd28 in do_handle_one_connection (connect=0x5620dd5f0f10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005620da12f9e2 in handle_one_connection (arg=0x5620dd5f0f10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005620daa63fb3 in pfs_spawn_thread (arg=0x5620dd50a9e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f457a7fc700 (LWP 224725)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612d7d in os_event::wait (this=0x5620dd28ebb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005620da612803 in os_event::wait_low (this=0x5620dd28ebb0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005620da612b74 in os_event_wait_low (event=0x5620dd28ebb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005620da7df19b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f457bfff700 (LWP 224722)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612d7d in os_event::wait (this=0x5620dd28e930) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005620da612803 in os_event::wait_low (this=0x5620dd28e930, reset_sig_count=13461) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005620da612b74 in os_event_wait_low (event=0x5620dd28e930, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005620da6fef21 in srv_resume_thread (slot=0x5620db660980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005620da702b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f457affd700 (LWP 224724)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612d7d in os_event::wait (this=0x5620dd28ea30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005620da612803 in os_event::wait_low (this=0x5620dd28ea30, reset_sig_count=10156) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005620da612b74 in os_event_wait_low (event=0x5620dd28ea30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005620da6fef21 in srv_resume_thread (slot=0x5620db6609f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005620da702b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f4591dfb700 (LWP 224711)): #0 0x00007f45c1ad6644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005620da60609a in LinuxAIOHandler::collect (this=0x7f4591dfab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005620da6065c3 in LinuxAIOHandler::poll (this=0x7f4591dfab00, m1=0x7f4591dfaba8, m2=0x7f4591dfabb0, request=0x7f4591dfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005620da606726 in os_aio_linux_handler (global_segment=4, m1=0x7f4591dfaba8, m2=0x7f4591dfabb0, request=0x7f4591dfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005620da60b24a in os_aio_handler (segment=4, m1=0x7f4591dfaba8, m2=0x7f4591dfabb0, request=0x7f4591dfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005620da85e4a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005620da704a47 in io_handler_thread (arg=0x5620db89f180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f45b247c700 (LWP 224702)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45b9e00bd7 in gu::Lock::wait (cond=..., this=0x7f45b247b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f45b9e0371b in RecvBuf::front (this=this@entry=0x5620dd1868d8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f45b9e012b6 in gcomm_recv (backend=, msg=0x5620dd14e990, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f45b9df525c in core_msg_recv (backend=backend@entry=0x5620dd14ead8, recv_msg=recv_msg@entry=0x5620dd14e990, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f45b9df84d4 in gcs_core_recv (conn=0x5620dd14e930, recv_act=recv_act@entry=0x7f45b247bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f45b9dff748 in gcs_recv_thread (arg=0x5620dd14e6b0) at gcs/src/gcs.cpp:1398 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4588ff9700 (LWP 224721)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612d7d in os_event::wait (this=0x5620dd28e8b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005620da612803 in os_event::wait_low (this=0x5620dd28e8b0, reset_sig_count=2906) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005620da612b74 in os_event_wait_low (event=0x5620dd28e8b0, reset_sig_count=2906) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005620da6fef21 in srv_resume_thread (slot=0x5620db660948 , sig_count=2906, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005620da702f32 in srv_purge_coordinator_suspend (slot=0x5620db660948 , rseg_history_len=30) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005620da7031c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f45915fa700 (LWP 224712)): #0 0x00007f45c1ad6644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005620da60609a in LinuxAIOHandler::collect (this=0x7f45915f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005620da6065c3 in LinuxAIOHandler::poll (this=0x7f45915f9b00, m1=0x7f45915f9ba8, m2=0x7f45915f9bb0, request=0x7f45915f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005620da606726 in os_aio_linux_handler (global_segment=5, m1=0x7f45915f9ba8, m2=0x7f45915f9bb0, request=0x7f45915f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005620da60b24a in os_aio_handler (segment=5, m1=0x7f45915f9ba8, m2=0x7f45915f9bb0, request=0x7f45915f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005620da85e4a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005620da704a47 in io_handler_thread (arg=0x5620db89f188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f45b2c7d700 (LWP 224701)): #0 0x00007f45c02c4483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f45b9dc7a68 in asio::detail::epoll_reactor::run (this=0x5620dd1870c0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f45b9dcc160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5620dd185dd0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f45b9dcc412 in asio::detail::task_io_service::run (this=0x5620dd185dd0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f45b9de3e6d in asio::io_service::run (this=0x5620dd186da0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f45b9de45c1 in gcomm::AsioProtonet::event_loop (this=0x5620dd186d00, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f45b9e0265d in GCommConn::run (this=0x5620dd1866f0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f45b9e05709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4579ffb700 (LWP 224726)): #0 0x00007f45c1f0ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005620da61307f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005620da7b9b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f45ba95e700 (LWP 224699)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620daad1e81 in safe_cond_timedwait (cond=0x5620dc0c31c0 , mp=0x5620dc0c3100 , abstime=0x7f45ba95dd70, file=0x5620daf63e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005620daad2e3b in inline_mysql_cond_timedwait (that=0x5620dc0c31c0 , mutex=0x5620dc0c3100 , abstime=0x7f45ba95dd70, src_file=0x5620daf63ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005620daad399e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005620daa63fb3 in pfs_spawn_thread (arg=0x5620dd10fd40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f45b807a700 (LWP 224729)): #0 0x00007f45c01fd40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005620d9e7f714 in my_sigwait (set=0x7f45b8079cf0, sig=0x7f45b8079cb4, code=0x7f45b8079cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005620d9e85a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x00005620daa63fb3 in pfs_spawn_thread (arg=0x5620dd252020) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f458a7fc700 (LWP 224717)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612698 in os_event::timed_wait (this=0x5620dd28eb30, abstime=0x7f458a7fbd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005620da6129af in os_event::wait_time_low (this=0x5620dd28eb30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005620da612b4f in os_event_wait_time_low (event=0x5620dd28eb30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005620da700c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4578ff9700 (LWP 224728)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620daad1b7f in safe_cond_wait (cond=0x7f4578ff8c70, mp=0x7f4578ff8cb0, file=0x5620dada8920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005620da528cad in inline_mysql_cond_wait (that=0x7f4578ff8c70, mutex=0x7f4578ff8cb0, src_file=0x5620dadaa348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005620da52d79b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005620daa63fb3 in pfs_spawn_thread (arg=0x5620dd4cfb70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f458affd700 (LWP 224716)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612698 in os_event::timed_wait (this=0x5620dd28eab0, abstime=0x7f458affcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005620da6129af in os_event::wait_time_low (this=0x5620dd28eab0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005620da612b4f in os_event_wait_time_low (event=0x5620dd28eab0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005620da70112a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f45897fa700 (LWP 224720)): #0 0x00007f45c1f0ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005620da61307f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005620da702489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005620da7025b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f458b7fe700 (LWP 224715)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612698 in os_event::timed_wait (this=0x5620dd466fa0, abstime=0x7f458b7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005620da6129af in os_event::wait_time_low (this=0x5620dd466fa0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005620da612b4f in os_event_wait_time_low (event=0x5620dd466fa0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005620da5df4f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4589ffb700 (LWP 224719)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620da612698 in os_event::timed_wait (this=0x5620dd42ecf0, abstime=0x7f4589ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005620da6129af in os_event::wait_time_low (this=0x5620dd42ecf0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005620da612b4f in os_event_wait_time_low (event=0x5620dd42ecf0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005620da7614c9 in ib_wqueue_timedwait (wq=0x5620dd42ebc0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005620da8a273a in fts_optimize_thread (arg=0x5620dd42ebc0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4593dff700 (LWP 224707)): #0 0x00007f45c1ad6644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005620da60609a in LinuxAIOHandler::collect (this=0x7f4593dfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005620da6065c3 in LinuxAIOHandler::poll (this=0x7f4593dfeb00, m1=0x7f4593dfeba8, m2=0x7f4593dfebb0, request=0x7f4593dfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005620da606726 in os_aio_linux_handler (global_segment=0, m1=0x7f4593dfeba8, m2=0x7f4593dfebb0, request=0x7f4593dfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005620da60b24a in os_aio_handler (segment=0, m1=0x7f4593dfeba8, m2=0x7f4593dfebb0, request=0x7f4593dfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005620da85e4a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x00005620da704a47 in io_handler_thread (arg=0x5620db89f160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f45b14d6700 (LWP 224706)): #0 0x00007f45c1f09d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620daad1e81 in safe_cond_timedwait (cond=0x5620dc0b0a00 , mp=0x5620dc0b0940 , abstime=0x7f45b14d5cb0, file=0x5620daf1f630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005620da93b309 in inline_mysql_cond_timedwait (that=0x5620dc0b0a00 , mutex=0x5620dc0b0940 , abstime=0x7f45b14d5cb0, src_file=0x5620daf1f678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005620da93b82d in my_service_thread_sleep (control=0x5620db664d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005620da92fcc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005620daa63fb3 in pfs_spawn_thread (arg=0x5620dd2469b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f45b86d4700 (LWP 224704)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620daad1b7f in safe_cond_wait (cond=0x7f459c015938, mp=0x7f459c015888, file=0x5620dac404c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005620da21f4ba in inline_mysql_cond_wait (that=0x7f459c015938, mutex=0x7f459c015888, src_file=0x5620dac407e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005620da22234e in Wsrep_thd_queue::pop_front (this=0x7f459c015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005620da2210d0 in wsrep_post_rollback_process (post_rollbacker=0x7f459c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x00005620da211587 in start_wsrep_THD (arg=0x5620dd180840) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f45b871f700 (LWP 224703)): #0 0x00007f45c1f09965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005620daad1b7f in safe_cond_wait (cond=0x7f45a8017e98, mp=0x7f45a8017de8, file=0x5620dac404c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005620da21f4ba in inline_mysql_cond_wait (that=0x7f45a8017e98, mutex=0x7f45a8017de8, src_file=0x5620dac407e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005620da22234e in Wsrep_thd_queue::pop_front (this=0x7f45a8017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005620da220c54 in wsrep_rollback_process (rollbacker=0x7f45a8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x00005620da211587 in start_wsrep_THD (arg=0x5620dd196490) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f45c1f05dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f45c02c3ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f45c2316880 (LWP 224698)): #0 0x00007f45c1f0a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005620daacc550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005620da301ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f45c02b920d in poll () from /lib64/libc.so.6 #5 0x00005620d9e8bf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x00005620d9e8b1b4 in mysqld_main (argc=162, argv=0x5620dcfc8140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x00005620d9e7f695 in main (argc=23, argv=0x7ffdd8f69578) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.224762' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.2/data/core.224762 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 224762] [New LWP 224764] [New LWP 224765] [New LWP 224766] [New LWP 224768] [New LWP 224770] [New LWP 224767] [New LWP 224946] [New LWP 224958] [New LWP 224947] [New LWP 224769] [New LWP 224959] [New LWP 224951] [New LWP 224949] [New LWP 224968] [New LWP 224953] [New LWP 224952] [New LWP 224950] [New LWP 224967] [New LWP 224957] [New LWP 224969] [New LWP 224965] [New LWP 224964] [New LWP 224974] [New LWP 224966] [New LWP 224963] [New LWP 224956] [New LWP 224970] [New LWP 224962] [New LWP 224972] [New LWP 224960] [New LWP 224948] [New LWP 224961] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fb8cb5319d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fb8cb5319d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b42736c550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b426ba1ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb8c98e020d in poll () from /lib64/libc.so.6 #5 0x000055b42672bf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055b42672b1b4 in mysqld_main (argc=163, argv=0x55b42a5e6140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055b42671f695 in main (argc=23, argv=0x7ffe9a0e9018) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fb8897fa700 (LWP 224961)): #0 0x00007fb8cb533e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055b426eb307f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055b426fa2489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055b426fa25b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fb8977fe700 (LWP 224948)): #0 0x00007fb8cb0fd644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b426ea609a in LinuxAIOHandler::collect (this=0x7fb8977fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b426ea65c3 in LinuxAIOHandler::poll (this=0x7fb8977fdb00, m1=0x7fb8977fdba8, m2=0x7fb8977fdbb0, request=0x7fb8977fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b426ea6726 in os_aio_linux_handler (global_segment=1, m1=0x7fb8977fdba8, m2=0x7fb8977fdbb0, request=0x7fb8977fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b426eab24a in os_aio_handler (segment=1, m1=0x7fb8977fdba8, m2=0x7fb8977fdbb0, request=0x7fb8977fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b4270fe4a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b426fa4a47 in io_handler_thread (arg=0x55b42813f168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fb889ffb700 (LWP 224960)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2698 in os_event::timed_wait (this=0x55b42aa55c80, abstime=0x7fb889ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b426eb29af in os_event::wait_time_low (this=0x55b42aa55c80, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b426eb2b4f in os_event_wait_time_low (event=0x55b42aa55c80, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b4270014c9 in ib_wqueue_timedwait (wq=0x55b42aa55b50, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055b42714273a in fts_optimize_thread (arg=0x55b42aa55b50) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fb8b86c6700 (LWP 224972)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b427371b7f in safe_cond_wait (cond=0x55b4280c34e0 , mp=0x55b4280c6800 , file=0x55b42743d488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b426750fe2 in inline_mysql_cond_wait (that=0x55b4280c34e0 , mutex=0x55b4280c6800 , src_file=0x55b42743dff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055b426752874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055b427303fb3 in pfs_spawn_thread (arg=0x55b42abb88c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fb888ff9700 (LWP 224962)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2d7d in os_event::wait (this=0x55b42a8b54c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b426eb2803 in os_event::wait_low (this=0x55b42a8b54c0, reset_sig_count=2974) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b426eb2b74 in os_event_wait_low (event=0x55b42a8b54c0, reset_sig_count=2974) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b426f9ef21 in srv_resume_thread (slot=0x55b427f00948 , sig_count=2974, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b426fa2f32 in srv_purge_coordinator_suspend (slot=0x55b427f00948 , rseg_history_len=19) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055b426fa31c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fb8b8711700 (LWP 224970)): #0 0x00007fb8c982440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055b42671f714 in my_sigwait (set=0x7fb8b8710cf0, sig=0x7fb8b8710cb4, code=0x7fb8b8710cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055b426725a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x000055b427303fb3 in pfs_spawn_thread (arg=0x55b42a879130) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fb88b7fe700 (LWP 224956)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2698 in os_event::timed_wait (this=0x55b42aa8dbf0, abstime=0x7fb88b7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b426eb29af in os_event::wait_time_low (this=0x55b42aa8dbf0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b426eb2b4f in os_event_wait_time_low (event=0x55b42aa8dbf0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b426e7f4f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fb87bfff700 (LWP 224963)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2d7d in os_event::wait (this=0x55b42a8b5540) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b426eb2803 in os_event::wait_low (this=0x55b42a8b5540, reset_sig_count=12580) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b426eb2b74 in os_event_wait_low (event=0x55b42a8b5540, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b426f9ef21 in srv_resume_thread (slot=0x55b427f00980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b426fa2b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fb87a7fc700 (LWP 224966)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2d7d in os_event::wait (this=0x55b42a8b57c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b426eb2803 in os_event::wait_low (this=0x55b42a8b57c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b426eb2b74 in os_event_wait_low (event=0x55b42a8b57c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b42707f19b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fb8b867b700 (LWP 224974)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b427371e81 in safe_cond_timedwait (cond=0x55b4280c34a0 , mp=0x55b4280c5e40 , abstime=0x7fb8b867ac60, file=0x55b42742db08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b426720015 in inline_mysql_cond_timedwait (that=0x55b4280c34a0 , mutex=0x55b4280c5e40 , abstime=0x7fb8b867ac60, src_file=0x55b42742fb50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b426724b08 in cache_thread (thd=0x7fb8a4000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x000055b426724db6 in one_thread_per_connection_end (thd=0x7fb8a4000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x000055b4269cfd28 in do_handle_one_connection (connect=0x55b42abb2b30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055b4269cf9e2 in handle_one_connection (arg=0x55b42abb2b30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055b427303fb3 in pfs_spawn_thread (arg=0x55b42ac3e260) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fb87b7fe700 (LWP 224964)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2d7d in os_event::wait (this=0x55b42a8b5640) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b426eb2803 in os_event::wait_low (this=0x55b42a8b5640, reset_sig_count=10513) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b426eb2b74 in os_event_wait_low (event=0x55b42a8b5640, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b426f9ef21 in srv_resume_thread (slot=0x55b427f009f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b426fa2b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fb87affd700 (LWP 224965)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2d7d in os_event::wait (this=0x55b42a8b55c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b426eb2803 in os_event::wait_low (this=0x55b42a8b55c0, reset_sig_count=11261) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b426eb2b74 in os_event_wait_low (event=0x55b42a8b55c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b426f9ef21 in srv_resume_thread (slot=0x55b427f009b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055b426fa2b25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fb878ff9700 (LWP 224969)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b427371b7f in safe_cond_wait (cond=0x7fb878ff8c70, mp=0x7fb878ff8cb0, file=0x55b427648920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b426dc8cad in inline_mysql_cond_wait (that=0x7fb878ff8c70, mutex=0x7fb878ff8cb0, src_file=0x55b42764a348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b426dcd79b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055b427303fb3 in pfs_spawn_thread (arg=0x55b42aaf77c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fb88bfff700 (LWP 224957)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2698 in os_event::timed_wait (this=0x55b42a8b56c0, abstime=0x7fb88bffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b426eb29af in os_event::wait_time_low (this=0x55b42a8b56c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b426eb2b4f in os_event_wait_time_low (event=0x55b42a8b56c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b426fa112a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fb879ffb700 (LWP 224967)): #0 0x00007fb8cb533e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055b426eb307f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055b427059b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fb8967fc700 (LWP 224950)): #0 0x00007fb8cb0fd644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b426ea609a in LinuxAIOHandler::collect (this=0x7fb8967fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b426ea65c3 in LinuxAIOHandler::poll (this=0x7fb8967fbb00, m1=0x7fb8967fbba8, m2=0x7fb8967fbbb0, request=0x7fb8967fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b426ea6726 in os_aio_linux_handler (global_segment=3, m1=0x7fb8967fbba8, m2=0x7fb8967fbbb0, request=0x7fb8967fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b426eab24a in os_aio_handler (segment=3, m1=0x7fb8967fbba8, m2=0x7fb8967fbbb0, request=0x7fb8967fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b4270fe4a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b426fa4a47 in io_handler_thread (arg=0x55b42813f178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fb8957fa700 (LWP 224952)): #0 0x00007fb8cb0fd644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b426ea609a in LinuxAIOHandler::collect (this=0x7fb8957f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b426ea65c3 in LinuxAIOHandler::poll (this=0x7fb8957f9b00, m1=0x7fb8957f9ba8, m2=0x7fb8957f9bb0, request=0x7fb8957f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b426ea6726 in os_aio_linux_handler (global_segment=5, m1=0x7fb8957f9ba8, m2=0x7fb8957f9bb0, request=0x7fb8957f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b426eab24a in os_aio_handler (segment=5, m1=0x7fb8957f9ba8, m2=0x7fb8957f9bb0, request=0x7fb8957f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b4270fe4a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b426fa4a47 in io_handler_thread (arg=0x55b42813f188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fb894ff9700 (LWP 224953)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2698 in os_event::timed_wait (this=0x55b42a8b5840, abstime=0x7fb894ff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b426eb29af in os_event::wait_time_low (this=0x55b42a8b5840, time_in_usec=523000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b426eb2b4f in os_event_wait_time_low (event=0x55b42a8b5840, time_in_usec=523000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b4270891fe in pc_sleep_if_needed (next_loop_time=1551281169890, sig_count=2, cur_time=1551281169367) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x000055b42708a137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fb8797fa700 (LWP 224968)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2d7d in os_event::wait (this=0x55b42a8b58c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b426eb2803 in os_event::wait_low (this=0x55b42a8b58c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b426eb2b74 in os_event_wait_low (event=0x55b42a8b58c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b427067cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fb896ffd700 (LWP 224949)): #0 0x00007fb8cb0fd644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b426ea609a in LinuxAIOHandler::collect (this=0x7fb896ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b426ea65c3 in LinuxAIOHandler::poll (this=0x7fb896ffcb00, m1=0x7fb896ffcba8, m2=0x7fb896ffcbb0, request=0x7fb896ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b426ea6726 in os_aio_linux_handler (global_segment=2, m1=0x7fb896ffcba8, m2=0x7fb896ffcbb0, request=0x7fb896ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b426eab24a in os_aio_handler (segment=2, m1=0x7fb896ffcba8, m2=0x7fb896ffcbb0, request=0x7fb896ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b4270fe4a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b426fa4a47 in io_handler_thread (arg=0x55b42813f170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fb895ffb700 (LWP 224951)): #0 0x00007fb8cb0fd644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b426ea609a in LinuxAIOHandler::collect (this=0x7fb895ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b426ea65c3 in LinuxAIOHandler::poll (this=0x7fb895ffab00, m1=0x7fb895ffaba8, m2=0x7fb895ffabb0, request=0x7fb895ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b426ea6726 in os_aio_linux_handler (global_segment=4, m1=0x7fb895ffaba8, m2=0x7fb895ffabb0, request=0x7fb895ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b426eab24a in os_aio_handler (segment=4, m1=0x7fb895ffaba8, m2=0x7fb895ffabb0, request=0x7fb895ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b4270fe4a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b426fa4a47 in io_handler_thread (arg=0x55b42813f180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fb88a7fc700 (LWP 224959)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2698 in os_event::timed_wait (this=0x55b42aabfcb0, abstime=0x7fb88a7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b426eb29af in os_event::wait_time_low (this=0x55b42aabfcb0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b426eb2b4f in os_event_wait_time_low (event=0x55b42aabfcb0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b4270e80e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fb8c0178700 (LWP 224769)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b427371b7f in safe_cond_wait (cond=0x7fb8ac015938, mp=0x7fb8ac015888, file=0x55b4274e04c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b426abf4ba in inline_mysql_cond_wait (that=0x7fb8ac015938, mutex=0x7fb8ac015888, src_file=0x55b4274e07e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b426ac234e in Wsrep_thd_queue::pop_front (this=0x7fb8ac015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b426ac10d0 in wsrep_post_rollback_process (post_rollbacker=0x7fb8ac000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x000055b426ab1587 in start_wsrep_THD (arg=0x55b42a7c5490) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fb897fff700 (LWP 224947)): #0 0x00007fb8cb0fd644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b426ea609a in LinuxAIOHandler::collect (this=0x7fb897ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b426ea65c3 in LinuxAIOHandler::poll (this=0x7fb897ffeb00, m1=0x7fb897ffeba8, m2=0x7fb897ffebb0, request=0x7fb897ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b426ea6726 in os_aio_linux_handler (global_segment=0, m1=0x7fb897ffeba8, m2=0x7fb897ffebb0, request=0x7fb897ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b426eab24a in os_aio_handler (segment=0, m1=0x7fb897ffeba8, m2=0x7fb897ffebb0, request=0x7fb897ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055b4270fe4a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x000055b426fa4a47 in io_handler_thread (arg=0x55b42813f160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fb88affd700 (LWP 224958)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b426eb2698 in os_event::timed_wait (this=0x55b42a8b5740, abstime=0x7fb88affcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b426eb29af in os_event::wait_time_low (this=0x55b42a8b5740, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b426eb2b4f in os_event_wait_time_low (event=0x55b42a8b5740, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b426fa0c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fb8b9d63700 (LWP 224946)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b427371e81 in safe_cond_timedwait (cond=0x55b428950a00 , mp=0x55b428950940 , abstime=0x7fb8b9d62cb0, file=0x55b4277bf630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b4271db309 in inline_mysql_cond_timedwait (that=0x55b428950a00 , mutex=0x55b428950940 , abstime=0x7fb8b9d62cb0, src_file=0x55b4277bf678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b4271db82d in my_service_thread_sleep (control=0x55b427f04d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055b4271cfcc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055b427303fb3 in pfs_spawn_thread (arg=0x55b42a86d9b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb8bbfff700 (LWP 224767)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb8c3427bd7 in gu::Lock::wait (cond=..., this=0x7fb8bbffe850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb8c342a71b in RecvBuf::front (this=this@entry=0x55b42a7a48a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fb8c34282b6 in gcomm_recv (backend=, msg=0x55b42a76c960, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fb8c341c25c in core_msg_recv (backend=backend@entry=0x55b42a76caa8, recv_msg=recv_msg@entry=0x55b42a76c960, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fb8c341f4d4 in gcs_core_recv (conn=0x55b42a76c900, recv_act=recv_act@entry=0x7fb8bbffed70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fb8c3426748 in gcs_recv_thread (arg=0x55b42a76c680) at gcs/src/gcs.cpp:1398 #7 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb8c012d700 (LWP 224770)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb8c3341272 in fifo_lock_get (q=q@entry=0x55b42a792c20) at galerautils/src/gu_fifo.c:262 #2 0x00007fb8c33418e1 in gu_fifo_get_head (q=0x55b42a792c20, err=0x7fb8c012c06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fb8c342343c in gcs_recv (conn=0x55b42a76c680, action=0x7fb8c012c0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fb8c345050c in galera::GcsActionSource::process (this=0x55b42a76c2d0, recv_ctx=0x7fb8c012c4d0, exit_loop=@0x7fb8c012c29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fb8c346f433 in galera::ReplicatorSMM::async_recv (this=0x55b42a764300, recv_ctx=0x7fb8c012c4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055b4274268a8 in wsrep::wsrep_provider_v26::run_applier (this=0x55b42a6eb760, applier_ctx=0x7fb8c012c4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055b426abfa69 in wsrep_replication_process (thd=0x7fb8a0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055b426ab1587 in start_wsrep_THD (arg=0x55b42a7c5840) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb8c01c3700 (LWP 224768)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b427371b7f in safe_cond_wait (cond=0x7fb8a8017e98, mp=0x7fb8a8017de8, file=0x55b4274e04c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b426abf4ba in inline_mysql_cond_wait (that=0x7fb8a8017e98, mutex=0x7fb8a8017de8, src_file=0x55b4274e07e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.worker[1] Trying to dump core for [mysqltest - pid: 225695, winpid: 225695] worker[1] Trying to dump core for [mysqld.1 - pid: 225116, winpid: 225116] worker[1] Trying to dump core for [mysqld.2 - pid: 225230, winpid: 225230] h:1178 #3 0x000055b426ac234e in Wsrep_thd_queue::pop_front (this=0x7fb8a8017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b426ac0c54 in wsrep_rollback_process (rollbacker=0x7fb8a8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x000055b426ab1587 in start_wsrep_THD (arg=0x55b42a7c6cd0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fb8c09c4700 (LWP 224766)): #0 0x00007fb8c98eb483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb8c33eea68 in asio::detail::epoll_reactor::run (this=0x55b42a7a5170, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fb8c33f3160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55b42a7a5010, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fb8c33f3412 in asio::detail::task_io_service::run (this=0x55b42a7a5010, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fb8c340ae6d in asio::io_service::run (this=0x55b42a7a4d70) at asio/asio/impl/io_service.ipp:58 #5 0x00007fb8c340b5c1 in gcomm::AsioProtonet::event_loop (this=0x55b42a7a4cd0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fb8c342965d in GCommConn::run (this=0x55b42a7a46c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fb8c342c709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fb8c2e48700 (LWP 224765)): #0 0x00007fb8cb530965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb8c344d377 in gu::Lock::wait (cond=..., this=0x7fb8c2e47bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb8c344d409 in galera::ServiceThd::thd_func (arg=0x55b42a764a18) at galera/src/galera_service_thd.cpp:30 #3 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb8c3f85700 (LWP 224764)): #0 0x00007fb8cb530d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b427371e81 in safe_cond_timedwait (cond=0x55b4289631c0 , mp=0x55b428963100 , abstime=0x7fb8c3f84d70, file=0x55b427803e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b427372e3b in inline_mysql_cond_timedwait (that=0x55b4289631c0 , mutex=0x55b428963100 , abstime=0x7fb8c3f84d70, src_file=0x55b427803ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b42737399e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055b427303fb3 in pfs_spawn_thread (arg=0x55b42a72dde0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb8cb52cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8c98eaead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb8cb93d880 (LWP 224762)): #0 0x00007fb8cb5319d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b42736c550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b426ba1ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb8c98e020d in poll () from /lib64/libc.so.6 #5 0x000055b42672bf19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x000055b42672b1b4 in mysqld_main (argc=163, argv=0x55b42a5e6140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x000055b42671f695 in main (argc=23, argv=0x7ffe9a0e9018) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_small_gcache-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_small_gcache-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_small_gcache Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-571 'innodb' w2 [ pass ] 6783 galera_sr.GCF-1043B 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-27 17:34:32 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/GCF-1043B.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 INSERT INTO t1 VALUES (1,'y'),(2,'x') UPDATE t1 SET f2 = 'y' WHERE f1 = 1 OR f1 = 2; connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.225118' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.GCF-1043B-innodb/mysqld.1/data/core.225118 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 225118] [New LWP 225131] [New LWP 225124] [New LWP 225138] [New LWP 225130] [New LWP 225141] [New LWP 225133] [New LWP 225143] [New LWP 225140] [New LWP 225145] [New LWP 225173] [New LWP 225163] [New LWP 225175] [New LWP 225165] [New LWP 225183] [New LWP 225174] [New LWP 225184] [New LWP 225180] [New LWP 225182] [New LWP 225198] [New LWP 225199] [New LWP 225700] [New LWP 225159] [New LWP 225128] [New LWP 225171] [New LWP 225181] [New LWP 225157] [New LWP 225172] [New LWP 225132] [New LWP 225161] [New LWP 225144] [New LWP 225142] [New LWP 225139] [New LWP 225126] [New LWP 225699] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f3d7c3629d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f3d7c3629d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556f6d0c9550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556f6c8feebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f3d7a71120d in poll () from /lib64/libc.so.6 #5 0x0000556f6c488f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000556f6c4881b4 in mysqld_main (argc=162, argv=0x556f703c8140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000556f6c47c695 in main (argc=23, argv=0x7ffdd7579ee8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f3d7006d700 (LWP 225699)): #0 0x00007f3d7c364d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000556f6cc04749 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000556f6cc04a9f in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000556f6ce4d1fa in pfs_os_file_flush_func (file=..., src_file=0x556f6d4d2588 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=832) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000556f6ce50055 in fil_flush_low (space=0x556f70898a50, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:832 #5 0x0000556f6ce5b9f8 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4475 #6 0x0000556f6cbdfe31 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000556f6cbe0893 in log_write_up_to (lsn=4867756, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000556f6cd4bf5f in trx_flush_log_if_needed_low (lsn=4867756) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000556f6cd4bfa8 in trx_flush_log_if_needed (lsn=4867756, trx=0x7f3d6a45a268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000556f6cd4ea0f in trx_prepare (trx=0x7f3d6a45a268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:2053 #11 0x0000556f6cd4ea6c in trx_prepare_for_mysql (trx=0x7f3d6a45a268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:2065 #12 0x0000556f6cb4ab5b in innobase_xa_prepare (hton=0x556f70662d70, thd=0x7f3d00000b00, prepare_trx=false) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:16931 #13 0x0000556f6c90301f in prepare_or_error (ht=0x556f70662d70, thd=0x7f3d00000b00, all=false) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1218 #14 0x0000556f6c903d58 in ha_commit_trans (thd=0x7f3d00000b00, all=false) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1560 #15 0x0000556f6c742383 in trans_commit_stmt (thd=0x7f3d00000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:513 #16 0x0000556f6c5c318c in mysql_execute_command (thd=0x7f3d00000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:6403 #17 0x0000556f6c5c80f1 in mysql_parse (thd=0x7f3d00000b00, rawbuf=0x7f3d00014198 "DELETE FROM t1", length=14, parser_state=0x7f3d7006c040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #18 0x0000556f6c5c7720 in wsrep_mysql_parse (thd=0x7f3d00000b00, rawbuf=0x7f3d00014198 "DELETE FROM t1", length=14, parser_state=0x7f3d7006c040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #19 0x0000556f6c5b346d in dispatch_command (command=COM_QUERY, thd=0x7f3d00000b00, packet=0x7f3d0000a761 "DELETE FROM t1", packet_length=14, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #20 0x0000556f6c5b1cbb in do_command (thd=0x7f3d00000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #21 0x0000556f6c72cc9e in do_handle_one_connection (connect=0x556f70922a10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #22 0x0000556f6c72c9e2 in handle_one_connection (arg=0x556f70922a10) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #23 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f709d12f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #24 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f3d73278700 (LWP 225126)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3d7427e377 in gu::Lock::wait (cond=..., this=0x7f3d73277bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f3d7427e409 in galera::ServiceThd::thd_func (arg=0x556f70546db8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f3d69807700 (LWP 225139)): #0 0x00007f3d7bf2e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f6cc0309a in LinuxAIOHandler::collect (this=0x7f3d69806b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f6cc035c3 in LinuxAIOHandler::poll (this=0x7f3d69806b00, m1=0x7f3d69806ba8, m2=0x7f3d69806bb0, request=0x7f3d69806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f6cc03726 in os_aio_linux_handler (global_segment=0, m1=0x7f3d69806ba8, m2=0x7f3d69806bb0, request=0x7f3d69806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f6cc0824a in os_aio_handler (segment=0, m1=0x7f3d69806ba8, m2=0x7f3d69806bb0, request=0x7f3d69806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f6ce5b4a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000556f6cd01a47 in io_handler_thread (arg=0x556f6de9c160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f3d4bfff700 (LWP 225142)): #0 0x00007f3d7bf2e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f6cc0309a in LinuxAIOHandler::collect (this=0x7f3d4bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f6cc035c3 in LinuxAIOHandler::poll (this=0x7f3d4bffeb00, m1=0x7f3d4bffeba8, m2=0x7f3d4bffebb0, request=0x7f3d4bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f6cc03726 in os_aio_linux_handler (global_segment=3, m1=0x7f3d4bffeba8, m2=0x7f3d4bffebb0, request=0x7f3d4bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f6cc0824a in os_aio_handler (segment=3, m1=0x7f3d4bffeba8, m2=0x7f3d4bffebb0, request=0x7f3d4bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f6ce5b4a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000556f6cd01a47 in io_handler_thread (arg=0x556f6de9c178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f3d4affd700 (LWP 225144)): #0 0x00007f3d7bf2e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f6cc0309a in LinuxAIOHandler::collect (this=0x7f3d4affcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f6cc035c3 in LinuxAIOHandler::poll (this=0x7f3d4affcb00, m1=0x7f3d4affcba8, m2=0x7f3d4affcbb0, request=0x7f3d4affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f6cc03726 in os_aio_linux_handler (global_segment=5, m1=0x7f3d4affcba8, m2=0x7f3d4affcbb0, request=0x7f3d4affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f6cc0824a in os_aio_handler (segment=5, m1=0x7f3d4affcba8, m2=0x7f3d4affcbb0, request=0x7f3d4affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f6ce5b4a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000556f6cd01a47 in io_handler_thread (arg=0x556f6de9c188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f3d43fff700 (LWP 225161)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0f698 in os_event::timed_wait (this=0x556f7068eb20, abstime=0x7f3d43ffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f6cc0f9af in os_event::wait_time_low (this=0x556f7068eb20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f6cc0fb4f in os_event_wait_time_low (event=0x556f7068eb20, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f6ccfdc08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f3d705a8700 (LWP 225132)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6d0ceb7f in safe_cond_wait (cond=0x7f3d54016c48, mp=0x7f3d54016b98, file=0x556f6d23d4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f6c81c4ba in inline_mysql_cond_wait (that=0x7f3d54016c48, mutex=0x7f3d54016b98, src_file=0x556f6d23d7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556f6c81f34e in Wsrep_thd_queue::pop_front (this=0x7f3d54016b40) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000556f6c81e0d0 in wsrep_post_rollback_process (post_rollbacker=0x7f3d54000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x0000556f6c80e587 in start_wsrep_THD (arg=0x556f705809f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f3d427fc700 (LWP 225172)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0fd7d in os_event::wait (this=0x556f7068e8a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f6cc0f803 in os_event::wait_low (this=0x556f7068e8a0, reset_sig_count=4226) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f6cc0fb74 in os_event_wait_low (event=0x556f7068e8a0, reset_sig_count=4226) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f6ccfbf21 in srv_resume_thread (slot=0x556f6dc5d948 , sig_count=4226, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f6ccfff32 in srv_purge_coordinator_suspend (slot=0x556f6dc5d948 , rseg_history_len=39) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000556f6cd001c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f3d495f5700 (LWP 225157)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0f698 in os_event::timed_wait (this=0x556f70866fd0, abstime=0x7f3d495f4d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f6cc0f9af in os_event::wait_time_low (this=0x556f70866fd0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f6cc0fb4f in os_event_wait_time_low (event=0x556f70866fd0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f6cbdc4f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f3d237fe700 (LWP 225181)): #0 0x00007f3d7c364e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000556f6cc1007f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000556f6cdb6b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f3d42ffd700 (LWP 225171)): #0 0x00007f3d7c364e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000556f6cc1007f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000556f6ccff489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000556f6ccff5b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f3d70df4700 (LWP 225128)): #0 0x00007f3d7a71c483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f3d7421fa68 in asio::detail::epoll_reactor::run (this=0x556f70587220, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f3d74224160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x556f705870c0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f3d74224412 in asio::detail::task_io_service::run (this=0x556f705870c0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f3d7423be6d in asio::io_service::run (this=0x556f705861c0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f3d7423c5c1 in gcomm::AsioProtonet::event_loop (this=0x556f70586120, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f3d7425a65d in GCommConn::run (this=0x556f705868a0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f3d7425d709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f3d48df4700 (LWP 225159)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0f698 in os_event::timed_wait (this=0x556f7068eaa0, abstime=0x7f3d48df3ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f6cc0f9af in os_event::wait_time_low (this=0x556f7068eaa0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f6cc0fb4f in os_event_wait_time_low (event=0x556f7068eaa0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f6ccfe12a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f3d69a07700 (LWP 225700)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0fd7d in os_event::wait (this=0x7f3cf4055d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f6cc0f803 in os_event::wait_low (this=0x7f3cf4055d20, reset_sig_count=546) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f6cc0fb74 in os_event_wait_low (event=0x7f3cf4055d20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f6cbdbd1d in lock_wait_suspend_thread (thr=0x7f3d0005f408) at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:355 #5 0x0000556f6cc9a7e1 in row_mysql_handle_errors (new_err=0x7f3d69a04478, trx=0x7f3d6a45b390, thr=0x7f3d0005f408, savept=0x7f3d69a04480) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:740 #6 0x0000556f6cc9c6bc in row_insert_for_mysql (mysql_rec=0x7f3d0005bd20 "\375\001", prebuilt=0x7f3d0005e8d8, ins_mode=ROW_INS_NORMAL) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:1474 #7 0x0000556f6cb38ba8 in ha_innobase::write_row (this=0x7f3d00057a38, record=0x7f3d0005bd20 "\375\001") at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:8086 #8 0x0000556f6c912c73 in handler::ha_write_row (this=0x7f3d00057a38, buf=0x7f3d0005bd20 "\375\001") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6663 #9 0x0000556f6c573acf in write_record (thd=0x7f3cf4000b00, table=0x7f3d0005da70, info=0x7f3d69a04e10) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #10 0x0000556f6c570e9d in mysql_insert (thd=0x7f3cf4000b00, table_list=0x7f3cf401fd50, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:1067 #11 0x0000556f6c5bc292 in mysql_execute_command (thd=0x7f3cf4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4791 #12 0x0000556f6c5c80f1 in mysql_parse (thd=0x7f3cf4000b00, rawbuf=0x7f3cf401fc58 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7f3d69a06040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8157 #13 0x0000556f6c5c7720 in wsrep_mysql_parse (thd=0x7f3cf4000b00, rawbuf=0x7f3cf401fc58 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7f3d69a06040, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7969 #14 0x0000556f6c5b346d in dispatch_command (command=COM_QUERY, thd=0x7f3cf4000b00, packet=0x7f3cf4017b21 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", packet_length=37, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1813 #15 0x0000556f6c5b1cbb in do_command (thd=0x7f3cf4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1358 #16 0x0000556f6c72cc9e in do_handle_one_connection (connect=0x556f709bcd80) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #17 0x0000556f6c72c9e2 in handle_one_connection (arg=0x556f709bcd80) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #18 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f70652b70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #19 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #20 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f3d700b8700 (LWP 225199)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6d0cee81 in safe_cond_timedwait (cond=0x556f6de204a0 , mp=0x556f6de22e40 , abstime=0x7f3d700b7c60, file=0x556f6d18ab08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556f6c47d015 in inline_mysql_cond_timedwait (that=0x556f6de204a0 , mutex=0x556f6de22e40 , abstime=0x7f3d700b7c60, src_file=0x556f6d18cb50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556f6c481b08 in cache_thread (thd=0x7f3d08000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x0000556f6c481db6 in one_thread_per_connection_end (thd=0x7f3d08000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x0000556f6c72cd28 in do_handle_one_connection (connect=0x556f709ff430) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000556f6c72c9e2 in handle_one_connection (arg=0x556f709ff430) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f70a24740) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f3d70103700 (LWP 225198)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6d0ceb7f in safe_cond_wait (cond=0x556f6de204e0 , mp=0x556f6de23800 , file=0x556f6d19a488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f6c4adfe2 in inline_mysql_cond_wait (that=0x556f6de204e0 , mutex=0x556f6de23800 , src_file=0x556f6d19aff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000556f6c4af874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f709b65e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f3d22ffd700 (LWP 225182)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0fd7d in os_event::wait (this=0x556f7068eca0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f6cc0f803 in os_event::wait_low (this=0x556f7068eca0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f6cc0fb74 in os_event_wait_low (event=0x556f7068eca0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f6cdc4cbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f3d23fff700 (LWP 225180)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0fd7d in os_event::wait (this=0x556f7068eba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f6cc0f803 in os_event::wait_low (this=0x556f7068eba0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f6cc0fb74 in os_event_wait_low (event=0x556f7068eba0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f6cddc19b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f3d7014e700 (LWP 225184)): #0 0x00007f3d7a65540a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000556f6c47c714 in my_sigwait (set=0x7f3d7014dcf0, sig=0x7f3d7014dcb4, code=0x7f3d7014dcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000556f6c482a71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f706520b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f3d417fa700 (LWP 225174)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0fd7d in os_event::wait (this=0x556f7068e9a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f6cc0f803 in os_event::wait_low (this=0x556f7068e9a0, reset_sig_count=16881) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f6cc0fb74 in os_event_wait_low (event=0x556f7068e9a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f6ccfbf21 in srv_resume_thread (slot=0x556f6dc5d9b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f6ccffb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f3d227fc700 (LWP 225183)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6d0ceb7f in safe_cond_wait (cond=0x7f3d227fbc70, mp=0x7f3d227fbcb0, file=0x556f6d3a5920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f6cb25cad in inline_mysql_cond_wait (that=0x7f3d227fbc70, mutex=0x7f3d227fbcb0, src_file=0x556f6d3a7348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556f6cb2a79b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f708cfba0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f3d49df6700 (LWP 225165)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0f698 in os_event::timed_wait (this=0x556f7082efa0, abstime=0x7f3d49df5b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f6cc0f9af in os_event::wait_time_low (this=0x556f7082efa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f6cc0fb4f in os_event_wait_time_low (event=0x556f7082efa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f6cd5e4c9 in ib_wqueue_timedwait (wq=0x556f7082ee70, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000556f6ce9f73a in fts_optimize_thread (arg=0x556f7082ee70) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f3d40ff9700 (LWP 225175)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0fd7d in os_event::wait (this=0x556f7068ea20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f6cc0f803 in os_event::wait_low (this=0x556f7068ea20, reset_sig_count=15177) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f6cc0fb74 in os_event_wait_low (event=0x556f7068ea20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f6ccfbf21 in srv_resume_thread (slot=0x556f6dc5d9f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f6ccffb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f3d437fe700 (LWP 225163)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0f698 in os_event::timed_wait (this=0x556f70899090, abstime=0x7f3d437fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f6cc0f9af in os_event::wait_time_low (this=0x556f70899090, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f6cc0fb4f in os_event_wait_time_low (event=0x556f70899090, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f6ce450e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f3d41ffb700 (LWP 225173)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0fd7d in os_event::wait (this=0x556f7068e920) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556f6cc0f803 in os_event::wait_low (this=0x556f7068e920, reset_sig_count=19579) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556f6cc0fb74 in os_event_wait_low (event=0x556f7068e920, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556f6ccfbf21 in srv_resume_thread (slot=0x556f6dc5d980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000556f6ccffb25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f3d4a7fc700 (LWP 225145)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6cc0f698 in os_event::timed_wait (this=0x556f7068ec20, abstime=0x7f3d4a7fbaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556f6cc0f9af in os_event::wait_time_low (this=0x556f7068ec20, time_in_usec=422000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556f6cc0fb4f in os_event_wait_time_low (event=0x556f7068ec20, time_in_usec=422000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556f6cde61fe in pc_sleep_if_needed (next_loop_time=1551281613456, sig_count=2, cur_time=1551281613034) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x0000556f6cde7137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f3d69006700 (LWP 225140)): #0 0x00007f3d7bf2e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f6cc0309a in LinuxAIOHandler::collect (this=0x7f3d69005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f6cc035c3 in LinuxAIOHandler::poll (this=0x7f3d69005b00, m1=0x7f3d69005ba8, m2=0x7f3d69005bb0, request=0x7f3d69005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f6cc03726 in os_aio_linux_handler (global_segment=1, m1=0x7f3d69005ba8, m2=0x7f3d69005bb0, request=0x7f3d69005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f6cc0824a in os_aio_handler (segment=1, m1=0x7f3d69005ba8, m2=0x7f3d69005bb0, request=0x7f3d69005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f6ce5b4a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000556f6cd01a47 in io_handler_thread (arg=0x556f6de9c168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f3d4b7fe700 (LWP 225143)): #0 0x00007f3d7bf2e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f6cc0309a in LinuxAIOHandler::collect (this=0x7f3d4b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f6cc035c3 in LinuxAIOHandler::poll (this=0x7f3d4b7fdb00, m1=0x7f3d4b7fdba8, m2=0x7f3d4b7fdbb0, request=0x7f3d4b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f6cc03726 in os_aio_linux_handler (global_segment=4, m1=0x7f3d4b7fdba8, m2=0x7f3d4b7fdbb0, request=0x7f3d4b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f6cc0824a in os_aio_handler (segment=4, m1=0x7f3d4b7fdba8, m2=0x7f3d4b7fdbb0, request=0x7f3d4b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f6ce5b4a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000556f6cd01a47 in io_handler_thread (arg=0x556f6de9c180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f3d7055d700 (LWP 225133)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3d74172272 in fifo_lock_get (q=q@entry=0x556f70574e00) at galerautils/src/gu_fifo.c:262 #2 0x00007f3d741728e1 in gu_fifo_get_head (q=0x556f70574e00, err=0x7f3d7055c06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f3d7425443c in gcs_recv (conn=0x556f7054e780, action=0x7f3d7055c0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f3d7428150c in galera::GcsActionSource::process (this=0x556f7054e370, recv_ctx=0x7f3d7055c4d0, exit_loop=@0x7f3d7055c29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f3d742a0433 in galera::ReplicatorSMM::async_recv (this=0x556f705466a0, recv_ctx=0x7f3d7055c4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000556f6d1838a8 in wsrep::wsrep_provider_v26::run_applier (this=0x556f704cd760, applier_ctx=0x7f3d7055c4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000556f6c81ca69 in wsrep_replication_process (thd=0x7f3d58000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000556f6c80e587 in start_wsrep_THD (arg=0x556f7059ede0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f3d68805700 (LWP 225141)): #0 0x00007f3d7bf2e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556f6cc0309a in LinuxAIOHandler::collect (this=0x7f3d68804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556f6cc035c3 in LinuxAIOHandler::poll (this=0x7f3d68804b00, m1=0x7f3d68804ba8, m2=0x7f3d68804bb0, request=0x7f3d68804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556f6cc03726 in os_aio_linux_handler (global_segment=2, m1=0x7f3d68804ba8, m2=0x7f3d68804bb0, request=0x7f3d68804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556f6cc0824a in os_aio_handler (segment=2, m1=0x7f3d68804ba8, m2=0x7f3d68804bb0, request=0x7f3d68804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000556f6ce5b4a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000556f6cd01a47 in io_handler_thread (arg=0x556f6de9c170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f3d6bfff700 (LWP 225130)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3d74258bd7 in gu::Lock::wait (cond=..., this=0x7f3d6bffe850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f3d7425b71b in RecvBuf::front (this=this@entry=0x556f70586a88, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f3d742592b6 in gcomm_recv (backend=, msg=0x556f7054ea60, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f3d7424d25c in core_msg_recv (backend=backend@entry=0x556f7054eba8, recv_msg=recv_msg@entry=0x556f7054ea60, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f3d742504d4 in gcs_core_recv (conn=0x556f7054ea00, recv_act=recv_act@entry=0x7f3d6bffed70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f3d74257748 in gcs_recv_thread (arg=0x556f7054e780) at gcs/src/gcs.cpp:1398 #7 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f3d6b059700 (LWP 225138)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6d0cee81 in safe_cond_timedwait (cond=0x556f6e6ada00 , mp=0x556f6e6ad940 , abstime=0x7f3d6b058cb0, file=0x556f6d51c630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556f6cf38309 in inline_mysql_cond_timedwait (that=0x556f6e6ada00 , mutex=0x556f6e6ad940 , abstime=0x7f3d6b058cb0, src_file=0x556f6d51c678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556f6cf3882d in my_service_thread_sleep (control=0x556f6dc61d00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000556f6cf2ccc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f70646a40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f3d74db6700 (LWP 225124)): #0 0x00007f3d7c361d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6d0cee81 in safe_cond_timedwait (cond=0x556f6e6c01c0 , mp=0x556f6e6c0100 , abstime=0x7f3d74db5d70, file=0x556f6d560e90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556f6d0cfe3b in inline_mysql_cond_timedwait (that=0x556f6e6c01c0 , mutex=0x556f6e6c0100 , abstime=0x7f3d74db5d70, src_file=0x556f6d560ed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556f6d0d099e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000556f6d060fb3 in pfs_spawn_thread (arg=0x556f7050fd40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f3d705f3700 (LWP 225131)): #0 0x00007f3d7c361965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556f6d0ceb7f in safe_cond_wait (cond=0x7f3d60016b88, mp=0x7f3d60016ad8, file=0x556f6d23d4c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556f6c81c4ba in inline_mysql_cond_wait (that=0x7f3d60016b88, mutex=0x7f3d60016ad8, src_file=0x556f6d23d7e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556f6c81f34e in Wsrep_thd_queue::pop_front (this=0x7f3d60016a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000556f6c81dc54 in wsrep_rollback_process (rollbacker=0x7f3d60001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x0000556f6c80e587 in start_wsrep_THD (arg=0x556f70596500) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f3d7c35ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3d7a71bead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f3d7c76e880 (LWP 225118)): #0 0x00007f3d7c3629d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556f6d0c9550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556f6c8feebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f3d7a71120d in poll () from /lib64/libc.so.6 #5 0x0000556f6c488f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000556f6c4881b4 in mysqld_main (argc=162, argv=0x556f703c8140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000556f6c47c695 in main (argc=23, argv=0x7ffdd7579ee8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.225232' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.GCF-1043B-innodb/mysqld.2/data/core.225232 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 225232] [New LWP 225248] [New LWP 225234] [New LWP 225247] [New LWP 225322] [New LWP 225323] [New LWP 225325] [New LWP 225635] [New LWP 225324] [New LWP 225640] [New LWP 225636] [New LWP 225634] [New LWP 225644] [New LWP 225637] [New LWP 225648] [New LWP 225646] [New LWP 225638] [New LWP 225651] [New LWP 225654] [New LWP 225641] [New LWP 225653] [New LWP 225657] [New LWP 225647] [New LWP 225676] [New LWP 225652] [New LWP 225655] [New LWP 225650] [New LWP 225656] [New LWP 225649] [New LWP 225673] [New LWP 225639] [New LWP 225645] [New LWP 225677] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f7ca2e229d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f7ca2e229d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000564c3f104550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000564c3e939ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f7ca11d120d in poll () from /lib64/libc.so.6 #5 0x0000564c3e4c3f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000564c3e4c31b4 in mysqld_main (argc=163, argv=0x564c40a41140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000564c3e4b7695 in main (argc=23, argv=0x7fff657ce888) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f7c88150700 (LWP 225677)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3f109e81 in safe_cond_timedwait (cond=0x564c3fe5b4a0 , mp=0x564c3fe5de40 , abstime=0x7f7c8814fc60, file=0x564c3f1c5b08 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564c3e4b8015 in inline_mysql_cond_timedwait (that=0x564c3fe5b4a0 , mutex=0x564c3fe5de40 , abstime=0x7f7c8814fc60, src_file=0x564c3f1c7b50 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2615) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564c3e4bcb08 in cache_thread (thd=0x7f7c6c000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2614 #4 0x0000564c3e4bcdb6 in one_thread_per_connection_end (thd=0x7f7c6c000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2701 #5 0x0000564c3e767d28 in do_handle_one_connection (connect=0x564c40cd7790) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000564c3e7679e2 in handle_one_connection (arg=0x564c40cd7790) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000564c3f09bfb3 in pfs_spawn_thread (arg=0x564c410990c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f7c5bfff700 (LWP 225645)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4a698 in os_event::timed_wait (this=0x564c40d10670, abstime=0x7f7c5bffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564c3ec4a9af in os_event::wait_time_low (this=0x564c40d10670, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564c3ec4ab4f in os_event_wait_time_low (event=0x564c40d10670, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564c3ed3912a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f7c61bfb700 (LWP 225639)): #0 0x00007f7ca29ee644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564c3ec3e09a in LinuxAIOHandler::collect (this=0x7f7c61bfab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564c3ec3e5c3 in LinuxAIOHandler::poll (this=0x7f7c61bfab00, m1=0x7f7c61bfaba8, m2=0x7f7c61bfabb0, request=0x7f7c61bfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564c3ec3e726 in os_aio_linux_handler (global_segment=4, m1=0x7f7c61bfaba8, m2=0x7f7c61bfabb0, request=0x7f7c61bfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564c3ec4324a in os_aio_handler (segment=4, m1=0x7f7c61bfaba8, m2=0x7f7c61bfabb0, request=0x7f7c61bfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564c3ee964a6 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000564c3ed3ca47 in io_handler_thread (arg=0x564c3fed7180 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f7c98078700 (LWP 225673)): #0 0x00007f7ca111540a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000564c3e4b7714 in my_sigwait (set=0x7f7c98077cf0, sig=0x7f7c98077cb4, code=0x7f7c98077cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000564c3e4bda71 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3173 #3 0x0000564c3f09bfb3 in pfs_spawn_thread (arg=0x564c40cd3e60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f7c597fa700 (LWP 225649)): #0 0x00007f7ca2e24e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000564c3ec4b07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000564c3ed3a489 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000564c3ed3a5b9 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f7c457fa700 (LWP 225656)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4ad7d in os_event::wait (this=0x564c40d10870) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564c3ec4a803 in os_event::wait_low (this=0x564c40d10870, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564c3ec4ab74 in os_event_wait_low (event=0x564c40d10870, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564c3edffcbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3152 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f7c58ff9700 (LWP 225650)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4ad7d in os_event::wait (this=0x564c40d10470) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564c3ec4a803 in os_event::wait_low (this=0x564c40d10470, reset_sig_count=4360) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564c3ec4ab74 in os_event_wait_low (event=0x564c40d10470, reset_sig_count=4360) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564c3ed36f21 in srv_resume_thread (slot=0x564c3fc98948 , sig_count=4360, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564c3ed3af32 in srv_purge_coordinator_suspend (slot=0x564c3fc98948 , rseg_history_len=9) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000564c3ed3b1c8 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f7c45ffb700 (LWP 225655)): #0 0x00007f7ca2e24e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000564c3ec4b07f in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000564c3edf1b55 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f7c477fe700 (LWP 225652)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4ad7d in os_event::wait (this=0x564c40d10570) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564c3ec4a803 in os_event::wait_low (this=0x564c40d10570, reset_sig_count=16255) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564c3ec4ab74 in os_event_wait_low (event=0x564c40d10570, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564c3ed36f21 in srv_resume_thread (slot=0x564c3fc989b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564c3ed3ab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f7c8819b700 (LWP 225676)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3f109b7f in safe_cond_wait (cond=0x564c3fe5b4e0 , mp=0x564c3fe5e800 , file=0x564c3f1d5488 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564c3e4e8fe2 in inline_mysql_cond_wait (that=0x564c3fe5b4e0 , mutex=0x564c3fe5e800 , src_file=0x564c3f1d5ff0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000564c3e4ea874 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000564c3f09bfb3 in pfs_spawn_thread (arg=0x564c41013870) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f7c5a7fc700 (LWP 225647)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4a698 in os_event::timed_wait (this=0x564c40f1ac60, abstime=0x7f7c5a7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564c3ec4a9af in os_event::wait_time_low (this=0x564c40f1ac60, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564c3ec4ab4f in os_event_wait_time_low (event=0x564c40f1ac60, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564c3ee800e7 in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f7c44ff9700 (LWP 225657)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3f109b7f in safe_cond_wait (cond=0x7f7c44ff8c70, mp=0x7f7c44ff8cb0, file=0x564c3f3e0920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564c3eb60cad in inline_mysql_cond_wait (that=0x7f7c44ff8c70, mutex=0x7f7c44ff8cb0, src_file=0x564c3f3e2348 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564c3eb6579b in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000564c3f09bfb3 in pfs_spawn_thread (arg=0x564c40f52770) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f7c46ffd700 (LWP 225653)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4ad7d in os_event::wait (this=0x564c40d105f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564c3ec4a803 in os_event::wait_low (this=0x564c40d105f0, reset_sig_count=13831) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564c3ec4ab74 in os_event_wait_low (event=0x564c40d105f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564c3ed36f21 in srv_resume_thread (slot=0x564c3fc989f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564c3ed3ab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f7c60bf9700 (LWP 225641)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4a698 in os_event::timed_wait (this=0x564c40d107f0, abstime=0x7f7c60bf8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564c3ec4a9af in os_event::wait_time_low (this=0x564c40d107f0, time_in_usec=377000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564c3ec4ab4f in os_event_wait_time_low (event=0x564c40d107f0, time_in_usec=377000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564c3ee211fe in pc_sleep_if_needed (next_loop_time=1551281619352, sig_count=2, cur_time=1551281618975) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2637 #5 0x0000564c3ee22137 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3096 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f7c467fc700 (LWP 225654)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4ad7d in os_event::wait (this=0x564c40d10770) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564c3ec4a803 in os_event::wait_low (this=0x564c40d10770, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564c3ec4ab74 in os_event_wait_low (event=0x564c40d10770, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564c3ee1719b in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f7c47fff700 (LWP 225651)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4ad7d in os_event::wait (this=0x564c40d104f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564c3ec4a803 in os_event::wait_low (this=0x564c40d104f0, reset_sig_count=18893) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564c3ec4ab74 in os_event_wait_low (event=0x564c40d104f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564c3ed36f21 in srv_resume_thread (slot=0x564c3fc98980 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564c3ed3ab25 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f7c623fc700 (LWP 225638)): #0 0x00007f7ca29ee644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564c3ec3e09a in LinuxAIOHandler::collect (this=0x7f7c623fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564c3ec3e5c3 in LinuxAIOHandler::poll (this=0x7f7c623fbb00, m1=0x7f7c623fbba8, m2=0x7f7c623fbbb0, request=0x7f7c623fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564c3ec3e726 in os_aio_linux_handler (global_segment=3, m1=0x7f7c623fbba8, m2=0x7f7c623fbbb0, request=0x7f7c623fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564c3ec4324a in os_aio_handler (segment=3, m1=0x7f7c623fbba8, m2=0x7f7c623fbbb0, request=0x7f7c623fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564c3ee964a6 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000564c3ed3ca47 in io_handler_thread (arg=0x564c3fed7178 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f7c5affd700 (LWP 225646)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4a698 in os_event::timed_wait (this=0x564c40d106f0, abstime=0x7f7c5affcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564c3ec4a9af in os_event::wait_time_low (this=0x564c40d106f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564c3ec4ab4f in os_event_wait_time_low (event=0x564c40d106f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564c3ed38c08 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f7c59ffb700 (LWP 225648)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4a698 in os_event::timed_wait (this=0x564c40eb0c30, abstime=0x7f7c59ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564c3ec4a9af in os_event::wait_time_low (this=0x564c40eb0c30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564c3ec4ab4f in os_event_wait_time_low (event=0x564c40eb0c30, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564c3ed994c9 in ib_wqueue_timedwait (wq=0x564c40eb0b00, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000564c3eeda73a in fts_optimize_thread (arg=0x564c40eb0b00) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f7c62bfd700 (LWP 225637)): #0 0x00007f7ca29ee644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564c3ec3e09a in LinuxAIOHandler::collect (this=0x7f7c62bfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564c3ec3e5c3 in LinuxAIOHandler::poll (this=0x7f7c62bfcb00, m1=0x7f7c62bfcba8, m2=0x7f7c62bfcbb0, request=0x7f7c62bfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564c3ec3e726 in os_aio_linux_handler (global_segment=2, m1=0x7f7c62bfcba8, m2=0x7f7c62bfcbb0, request=0x7f7c62bfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564c3ec4324a in os_aio_handler (segment=2, m1=0x7f7c62bfcba8, m2=0x7f7c62bfcbb0, request=0x7f7c62bfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564c3ee964a6 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000564c3ed3ca47 in io_handler_thread (arg=0x564c3fed7170 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f7c5b7fe700 (LWP 225644)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3ec4a698 in os_event::timed_wait (this=0x564c40ee8ba0, abstime=0x7f7c5b7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564c3ec4a9af in os_event::wait_time_low (this=0x564c40ee8ba0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564c3ec4ab4f in os_event_wait_time_low (event=0x564c40ee8ba0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564c3ec174f0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f7c895ed700 (LWP 225634)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3f109e81 in safe_cond_timedwait (cond=0x564c406e8a00 , mp=0x564c406e8940 , abstime=0x7f7c895eccb0, file=0x564c3f557630 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564c3ef73309 in inline_mysql_cond_timedwait (that=0x564c406e8a00 , mutex=0x564c406e8940 , abstime=0x7f7c895eccb0, src_file=0x564c3f557678 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564c3ef7382d in my_service_thread_sleep (control=0x564c3fc9cd00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000564c3ef67cc6 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000564c3f09bfb3 in pfs_spawn_thread (arg=0x564c40cc86e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f7c633fe700 (LWP 225636)): #0 0x00007f7ca29ee644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564c3ec3e09a in LinuxAIOHandler::collect (this=0x7f7c633fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564c3ec3e5c3 in LinuxAIOHandler::poll (this=0x7f7c633fdb00, m1=0x7f7c633fdba8, m2=0x7f7c633fdbb0, request=0x7f7c633fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564c3ec3e726 in os_aio_linux_handler (global_segment=1, m1=0x7f7c633fdba8, m2=0x7f7c633fdbb0, request=0x7f7c633fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564c3ec4324a in os_aio_handler (segment=1, m1=0x7f7c633fdba8, m2=0x7f7c633fdbb0, request=0x7f7c633fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564c3ee964a6 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000564c3ed3ca47 in io_handler_thread (arg=0x564c3fed7168 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f7c613fa700 (LWP 225640)): #0 0x00007f7ca29ee644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564c3ec3e09a in LinuxAIOHandler::collect (this=0x7f7c613f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564c3ec3e5c3 in LinuxAIOHandler::poll (this=0x7f7c613f9b00, m1=0x7f7c613f9ba8, m2=0x7f7c613f9bb0, request=0x7f7c613f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564c3ec3e726 in os_aio_linux_handler (global_segment=5, m1=0x7f7c613f9ba8, m2=0x7f7c613f9bb0, request=0x7f7c613f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564c3ec4324a in os_aio_handler (segment=5, m1=0x7f7c613f9ba8, m2=0x7f7c613f9bb0, request=0x7f7c613f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564c3ee964a6 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000564c3ed3ca47 in io_handler_thread (arg=0x564c3fed7188 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f7c9826a700 (LWP 225324)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3f109b7f in safe_cond_wait (cond=0x7f7c74015938, mp=0x7f7c74015888, file=0x564c3f2784c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564c3e8574ba in inline_mysql_cond_wait (that=0x7f7c74015938, mutex=0x7f7c74015888, src_file=0x564c3f2787e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564c3e85a34e in Wsrep_thd_queue::pop_front (this=0x7f7c74015830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000564c3e8590d0 in wsrep_post_rollback_process (post_rollbacker=0x7f7c74000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:288 #5 0x0000564c3e849587 in start_wsrep_THD (arg=0x564c40c17db0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f7c63bff700 (LWP 225635)): #0 0x00007f7ca29ee644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564c3ec3e09a in LinuxAIOHandler::collect (this=0x7f7c63bfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564c3ec3e5c3 in LinuxAIOHandler::poll (this=0x7f7c63bfeb00, m1=0x7f7c63bfeba8, m2=0x7f7c63bfebb0, request=0x7f7c63bfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564c3ec3e726 in os_aio_linux_handler (global_segment=0, m1=0x7f7c63bfeba8, m2=0x7f7c63bfebb0, request=0x7f7c63bfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564c3ec4324a in os_aio_handler (segment=0, m1=0x7f7c63bfeba8, m2=0x7f7c63bfebb0, request=0x7f7c63bfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564c3ee964a6 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4370 #6 0x0000564c3ed3ca47 in io_handler_thread (arg=0x564c3fed7160 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f7c9821f700 (LWP 225325)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7c9ac32272 in fifo_lock_get (q=q@entry=0x564c40bedc80) at galerautils/src/gu_fifo.c:262 #2 0x00007f7c9ac328e1 in gu_fifo_get_head (q=0x564c40bedc80, err=0x7f7c9821e06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f7c9ad1443c in gcs_recv (conn=0x564c40bc7600, action=0x7f7c9821e0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f7c9ad4150c in galera::GcsActionSource::process (this=0x564c40bf8020, recv_ctx=0x7f7c9821e4d0, exit_loop=@0x7f7c9821e29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f7c9ad60433 in galera::ReplicatorSMM::async_recv (this=0x564c40bbf3b0, recv_ctx=0x7f7c9821e4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000564c3f1be8a8 in wsrep::wsrep_provider_v26::run_applier (this=0x564c40b35250, applier_ctx=0x7f7c9821e4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000564c3e857a69 in wsrep_replication_process (thd=0x7f7c78000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000564c3e849587 in start_wsrep_THD (arg=0x564c40bdff30) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #9 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f7c982b5700 (LWP 225323)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3f109b7f in safe_cond_wait (cond=0x7f7c80017e98, mp=0x7f7c80017de8, file=0x564c3f2784c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564c3e8574ba in inline_mysql_cond_wait (that=0x7f7c80017e98, mutex=0x7f7c80017de8, src_file=0x564c3f2787e0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564c3e85a34e in Wsrep_thd_queue::pop_front (this=0x7f7c80017d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000564c3e858c54 in wsrep_rollback_process (rollbacker=0x7f7c80001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:145 #5 0x0000564c3e849587 in start_wsrep_THD (arg=0x564c40c21d00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2693 #6 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f7c8b7fd700 (LWP 225322)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7c9ad18bd7 in gu::Lock::wait (cond=..., this=0x7f7c8b7fc850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f7c9ad1b71b in RecvBuf::front (this=this@entry=0x564c40bff9d8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f7c9ad192b6 in gcomm_recv (backend=, msg=0x564c40bc78e0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f7c9ad0d25c in core_msg_recv (backend=backend@entry=0x564c40bc7a28, recv_msg=recv_msg@entry=0x564c40bc78e0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f7c9ad104d4 in gcs_core_recv (conn=0x564c40bc7880, recv_act=recv_act@entry=0x7f7c8b7fcd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f7c9ad17748 in gcs_recv_thread (arg=0x564c40bc7600) at gcs/src/gcs.cpp:1398 #7 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f7c9a739700 (LWP 225247)): #0 0x00007f7ca2e21965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7c9ad3e377 in gu::Lock::wait (cond=..., this=0x7f7c9a738bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f7c9ad3e409 in galera::ServiceThd::thd_func (arg=0x564c40bbfac8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f7c9b876700 (LWP 225234)): #0 0x00007f7ca2e21d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564c3f109e81 in safe_cond_timedwait (cond=0x564c406fb1c0 , mp=0x564c406fb100 , abstime=0x7f7c9b875d70, file=0x564c3f59be90 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564c3f10ae3b in inline_mysql_cond_timedwait (that=0x564c406fb1c0 , mutex=0x564c406fb100 , abstime=0x7f7c9b875d70, src_file=0x564c3f59bed8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564c3f10b99e in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000564c3f09bfb3 in pfs_spawn_thread (arg=0x564c40b88de0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f7c8bffe700 (LWP 225248)): #0 0x00007f7ca11dc483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f7c9acdfa68 in asio::detail::epoll_reactor::run (this=0x564c40c001c0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f7c9ace4160 in asio::detail::task_io_service::do_run_one (this=this@entry=0x564c40c00060, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f7c9ace4412 in asio::detail::task_io_service::run (this=0x564c40c00060, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f7c9acfbe6d in asio::io_service::run (this=0x564c40bfed80) at asio/asio/impl/io_service.ipp:58 #5 0x00007f7c9acfc5c1 in gcomm::AsioProtonet::event_loop (this=0x564c40bfece0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f7c9ad1a65d in GCommConn::run (this=0x564c40bff7f0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f7c9ad1d709 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f7ca2e1ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f7ca11dbead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f7ca322e880 (LWP 225232)): #0 0x00007f7ca2e229d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000564c3f104550 in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000564c3e939ebb in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f7ca11d120d in poll () from /lib64/libc.so.6 #5 0x0000564c3e4c3f19 in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6355 #6 0x0000564c3e4c31b4 in mysqld_main (argc=163, argv=0x564c40a41140) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5791 #7 0x0000564c3e4b7695 in main (argc=23, argv=0x7fff657ce888) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.GCF-1043B-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043B-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1043B Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-572 'innodb' w1 [ pass ] 4253 galera_sr.GCF-580 'innodb' w1 [ pass ] 4594 -------------------------------------------------------------------------- The servers were restarted 20 times Spent 2459.140 of 3478 seconds executing testcases Completed: Failed 11/83 tests, 86.75% were successful. Failing test(s): galera_sr.galera_sr_large_fragment galera_sr.galera-features#56 galera_sr.galera_sr_insert_select galera_sr.mysql-wsrep#215 galera_sr.galera_sr_kill_all_norecovery galera_sr.galera_sr_many_fragments galera_sr.galera_sr_mysqldump_sst galera_sr.GCF-1043A galera_sr.mysql-wsrep-features#8 galera_sr.galera_sr_small_gcache galera_sr.GCF-1043B The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html Errors/warnings were found in logfiles during server shutdown after running the following sequence(s) of tests: galera_sr.galera_sr_large_fragment galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera-features#56 galera_sr.GCF-851 galera_sr.GCF-845 galera_sr.GCF-867 galera_sr.galera_sr_create_drop galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_ddl_master galera_sr.galera_sr_insert_select galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery galera_sr.galera_sr_many_fragments galera_sr.mysql-wsrep-features#22 galera_sr.galera_sr_myisam galera_sr.galera_sr_kill_connection galera_sr.galera_sr_ws_size2 galera_sr.GCF-1008 galera_sr.mysql-wsrep-features#27 galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#148 galera_sr.galera_sr_load_data galera_sr.mysql-wsrep-features#15 galera_sr.GCF-1043A galera_sr.galera_sr_kill_query galera_sr.mysql-wsrep-features#9 galera_sr.mysql-wsrep-features#8 galera_sr.galera_sr_small_gcache galera_sr.GCF-1043B mysql-test-run: *** ERROR: there were failing test cases