Checking supported features... MariaDB Version 10.4.3-MariaDB-debug - SSL connections supported - binaries are debug compiled Using suites: galera_sr Collecting tests... Installing system database... Using parallel: 4 worker[2] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 worker[4] Using MTR_BUILD_THREAD 301, with reserved ports 16020..16039 worker[3] Using MTR_BUILD_THREAD 303, with reserved ports 16060..16079 worker[1] Using MTR_BUILD_THREAD 302, with reserved ports 16040..16059 worker[4] Trying to dump core for [mysqltest - pid: 29034, winpid: 29034] worker[4] Trying to dump core for [mysqld.1 - pid: 26368, winpid: 26368] worker[4] Trying to dump core for [mysqld.2 - pid: 26881, winpid: 26881] worker[3] Trying to dump core for [mysqltest - pid: 30994, winpid: 30994] worker[3] Trying to dump core for [mysqld.1 - pid: 29817, winpid: 29817] worker[3] Trying to dump core for [mysqld.2 - pid: 30359, winpid: 30359] ============================================================================== TEST WORKER RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- galera_sr.GCF-574 [ disabled ] CTAS is not supported together with SR galera_sr.galera_sr_sbr [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.mysql-wsrep-features#29 [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.galera_sr_gtid 'innodb' w1 [ pass ] 22607 galera_sr.mysql-wsrep-features#32 'innodb' w2 [ pass ] 23999 galera_sr.galera_sr_log_bin 'innodb' w3 [ pass ] 28704 galera_sr.mysql-wsrep-features#136 'innodb' w1 [ pass ] 5118 galera_sr.galera_sr_v1_row_events 'innodb' w2 [ pass ] 5475 galera_sr.GCF-623 'innodb' w1 [ pass ] 9713 galera_sr.galera_sr_conflict_with_rollback_master 'innodb' w2 [ pass ] 7980 galera_sr.GCF-627 'innodb' w1 [ pass ] 11170 galera_sr.GCF-845 'innodb' w1 [ pass ] 8847 galera_sr.galera_sr_ddl_master 'innodb' w2 [ pass ] 13814 galera_sr.GCF-851 'innodb' w1 [ pass ] 6056 galera_sr.galera_sr_ddl_schema 'innodb' w2 [ pass ] 9811 galera_sr.galera_sr_ddl_slave 'innodb' w2 [ pass ] 12338 galera_sr.galera_sr_ddl_unrelated 'innodb' w2 [ pass ] 15233 galera_sr.galera_sr_dupkey_error 'innodb' w2 [ pass ] 11714 galera_sr.GCF-867 'innodb' w1 [ pass ] 47132 galera_sr.GCF-889 'innodb' w1 [ pass ] 7011 galera_sr.GCF-900 'innodb' w1 [ pass ] 8051 galera_sr.galera_sr_fk_conflict 'innodb' w2 [ pass ] 22466 galera_sr.galera_sr_large_fragment 'innodb' w4 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 12:39:31 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr_large_fragment.log == connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1024 * 1024 * 10; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; connection node_2; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT COUNT(*) > 50000 FROM t1; COUNT(*) > 50000 1 connection node_1; ROLLBACK; == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) - found 'core.26372' (0/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.1/data/core.26372 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 26372] [New LWP 26401] [New LWP 26395] [New LWP 26414] [New LWP 26417] [New LWP 26474] [New LWP 26469] [New LWP 26379] [New LWP 26576] [New LWP 26573] [New LWP 26410] [New LWP 26583] [New LWP 26577] [New LWP 26415] [New LWP 26580] [New LWP 26584] [New LWP 26473] [New LWP 26574] [New LWP 26575] [New LWP 26582] [New LWP 26586] [New LWP 26587] [New LWP 26793] [New LWP 29109] [New LWP 26466] [New LWP 26468] [New LWP 26470] [New LWP 26471] [New LWP 26472] [New LWP 26578] [New LWP 26585] [New LWP 26792] [New LWP 26579] [New LWP 26581] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fdd12eef9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fdd12eef9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056354840b45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563547c495a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdd1129e20d in poll () from /lib64/libc.so.6 #5 0x00005635477e9a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005635477e8d25 in mysqld_main (argc=163, argv=0x563549b7ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005635477dcec5 in main (argc=25, argv=0x7ffeb0f3dfe8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fdcd37fe700 (LWP 26581)): #0 0x00007fdd12ef1d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000563547f47c27 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000563547f47f7d in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005635481904aa in pfs_os_file_flush_func (file=..., src_file=0x56354880ab50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000563548193225 in fil_flush_low (space=0x56354a057440, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056354819eaff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000056354811c7be in buf_dblwr_write_single_page (bpage=0x7fdd00860450, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1309 #7 0x0000563548123992 in buf_flush_write_block_low (bpage=0x7fdd00860450, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1014 #8 0x00005635481240be in buf_flush_page (buf_pool=0x563549f33130, bpage=0x7fdd00860450, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1165 #9 0x0000563548126ba3 in buf_flush_single_page_from_LRU (buf_pool=0x563549f33130) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2138 #10 0x0000563548131b0e in buf_LRU_get_free_block (buf_pool=0x563549f33130) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1194 #11 0x000056354810ee1b in buf_page_init_for_read (err=0x7fdcd37fc194, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5332 #12 0x00005635481381c6 in buf_read_page_low (err=0x7fdcd37fc194, sync=true, type=0, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #13 0x0000563548138da6 in buf_read_page (page_id=..., zip_size=0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:412 #14 0x000056354810bce2 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x5635487c2068 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=8113, mtr=0x7fdcd37fc620, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4340 #15 0x00005635480e021f in btr_free_externally_stored_field (index=0x56354a1e92f8, field_ref=0x7fdd00d7c1e4 "", rec=0x7fdd00d7c19f "6b622e7b-2b8b-11e9-b19b-fa5e78a09dc6\200", offsets=0x7fdcc0014988, page_zip=0x0, i=6, rollback=false, local_mtr=0x7fdcd37fd1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8113 #16 0x00005635480e0762 in btr_rec_free_externally_stored_fields (index=0x56354a1e92f8, rec=0x7fdd00d7c19f "6b622e7b-2b8b-11e9-b19b-fa5e78a09dc6\200", offsets=0x7fdcc0014988, page_zip=0x0, rollback=false, mtr=0x7fdcd37fd1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8215 #17 0x00005635480dadb9 in btr_cur_pessimistic_delete (err=0x7fdcd37fce5c, has_reserved_extents=0, cursor=0x56354a06f7c0, flags=0, rollback=false, mtr=0x7fdcd37fd1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5905 #18 0x0000563548000176 in row_purge_remove_clust_if_poss_low (node=0x56354a06f728, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:175 #19 0x00005635480002fb in row_purge_remove_clust_if_poss (node=0x56354a06f728) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #20 0x0000563548001b28 in row_purge_del_mark (node=0x56354a06f728) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #21 0x00005635480030b0 in row_purge_record_func (node=0x56354a06f728, undo_rec=0x56354a0702f8 "", thr=0x56354a06f280, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #22 0x00005635480033ed in row_purge (node=0x56354a06f728, undo_rec=0x56354a0702f8 "", thr=0x56354a06f280) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #23 0x0000563548003798 in row_purge_step (thr=0x56354a06f280) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #24 0x0000563547f85bf6 in que_thr_step (thr=0x56354a06f280) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #25 0x0000563547f85e26 in que_run_threads_low (thr=0x56354a06f280) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #26 0x0000563547f86016 in que_run_threads (thr=0x56354a06f280) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #27 0x0000563548042947 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #28 0x0000563548042aee in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #29 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fdcd8ff9700 (LWP 26579)): #0 0x00007fdd1128dd47 in sched_yield () from /lib64/libc.so.6 #1 0x0000563547f534ef in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x000056354806cccb in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x000056354806ced3 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x0000563548042dbc in srv_do_purge (n_total_purged=0x7fdcd8ff8d98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x00005635480431e9 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fdd0061d700 (LWP 26792)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563548410a8d in safe_cond_wait (cond=0x5635491501e0 , mp=0x563549153560 , file=0x5635484dbdc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056354780e4ee in inline_mysql_cond_wait (that=0x5635491501e0 , mutex=0x563549153560 , src_file=0x5635484dc930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000056354780fd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005635483a3555 in pfs_spawn_thread (arg=0x56354a1af8a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fdcd17fa700 (LWP 26585)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f5325b in os_event::wait (this=0x563549e4da30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563547f52ce1 in os_event::wait_low (this=0x563549e4da30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563547f53052 in os_event_wait_low (event=0x563549e4da30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563548107c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fdcd97fa700 (LWP 26578)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f5325b in os_event::wait (this=0x56354a063d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563547f52ce1 in os_event::wait_low (this=0x56354a063d20, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563547f53052 in os_event_wait_low (event=0x56354a063d20, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056354804ccee in sync_array_wait_event (arr=0x563549e248c0, cell=@0x7fdcd97f9b98: 0x563549e24af0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056354804fbd4 in rw_lock_x_lock_wait_func (lock=0x56354a058350, pass=0, threshold=0, file_name=0x563548790c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:459 #6 0x000056354804fd06 in rw_lock_x_lock_low (lock=0x56354a058350, pass=0, file_name=0x563548790c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:514 #7 0x0000563548050124 in rw_lock_x_lock_func (lock=0x56354a058350, pass=0, file_name=0x563548790c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:673 #8 0x000056354803e535 in pfs_rw_lock_x_lock_func (lock=0x56354a058350, pass=0, file_name=0x563548790c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:552 #9 0x0000563548041578 in srv_master_evict_from_table_cache (pct_check=50) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1999 #10 0x0000563548041bd8 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2162 #11 0x0000563548042643 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fdce1ffb700 (LWP 26472)): #0 0x00007fdd12abb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563547f46578 in LinuxAIOHandler::collect (this=0x7fdce1ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563547f46aa1 in LinuxAIOHandler::poll (this=0x7fdce1ffab00, m1=0x7fdce1ffaba8, m2=0x7fdce1ffabb0, request=0x7fdce1ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563547f46c04 in os_aio_linux_handler (global_segment=4, m1=0x7fdce1ffaba8, m2=0x7fdce1ffabb0, request=0x7fdce1ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563547f4b728 in os_aio_handler (segment=4, m1=0x7fdce1ffaba8, m2=0x7fdce1ffabb0, request=0x7fdce1ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056354819e5ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005635480449fb in io_handler_thread (arg=0x5635491cb640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fdce27fc700 (LWP 26471)): #0 0x00007fdd12abb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563547f46578 in LinuxAIOHandler::collect (this=0x7fdce27fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563547f46aa1 in LinuxAIOHandler::poll (this=0x7fdce27fbb00, m1=0x7fdce27fbba8, m2=0x7fdce27fbbb0, request=0x7fdce27fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563547f46c04 in os_aio_linux_handler (global_segment=3, m1=0x7fdce27fbba8, m2=0x7fdce27fbbb0, request=0x7fdce27fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563547f4b728 in os_aio_handler (segment=3, m1=0x7fdce27fbba8, m2=0x7fdce27fbbb0, request=0x7fdce27fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056354819e5ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005635480449fb in io_handler_thread (arg=0x5635491cb638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fdce2ffd700 (LWP 26470)): #0 0x00007fdd12abb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563547f46578 in LinuxAIOHandler::collect (this=0x7fdce2ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563547f46aa1 in LinuxAIOHandler::poll (this=0x7fdce2ffcb00, m1=0x7fdce2ffcba8, m2=0x7fdce2ffcbb0, request=0x7fdce2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563547f46c04 in os_aio_linux_handler (global_segment=2, m1=0x7fdce2ffcba8, m2=0x7fdce2ffcbb0, request=0x7fdce2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563547f4b728 in os_aio_handler (segment=2, m1=0x7fdce2ffcba8, m2=0x7fdce2ffcbb0, request=0x7fdce2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056354819e5ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005635480449fb in io_handler_thread (arg=0x5635491cb630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fdce3fff700 (LWP 26468)): #0 0x00007fdd12abb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563547f46578 in LinuxAIOHandler::collect (this=0x7fdce3ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563547f46aa1 in LinuxAIOHandler::poll (this=0x7fdce3ffeb00, m1=0x7fdce3ffeba8, m2=0x7fdce3ffebb0, request=0x7fdce3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563547f46c04 in os_aio_linux_handler (global_segment=0, m1=0x7fdce3ffeba8, m2=0x7fdce3ffebb0, request=0x7fdce3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563547f4b728 in os_aio_handler (segment=0, m1=0x7fdce3ffeba8, m2=0x7fdce3ffebb0, request=0x7fdce3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056354819e5ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005635480449fb in io_handler_thread (arg=0x5635491cb620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fdd01c6f700 (LWP 26466)): #0 0x00007fdd12eeed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563548410d8f in safe_cond_timedwait (cond=0x5635499dcea0 , mp=0x5635499dcde0 , abstime=0x7fdd01c6ecb0, file=0x5635488546f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056354827abc1 in inline_mysql_cond_timedwait (that=0x5635499dcea0 , mutex=0x5635499dcde0 , abstime=0x7fdd01c6ecb0, src_file=0x563548854738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056354827b0e5 in my_service_thread_sleep (control=0x563548f91b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056354826f57e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005635483a3555 in pfs_spawn_thread (arg=0x563549e056f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fdd00587700 (LWP 29109)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f5325b in os_event::wait (this=0x56354a063ca0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563547f52ce1 in os_event::wait_low (this=0x56354a063ca0, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563547f53052 in os_event_wait_low (event=0x56354a063ca0, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056354804ccee in sync_array_wait_event (arr=0x563549e248c0, cell=@0x7fdd00584198: 0x563549e24a30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056354804fa03 in rw_lock_s_lock_spin (lock=0x56354a058350, pass=0, file_name=0x5635487807f8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x0000563547fd8126 in rw_lock_s_lock_func (lock=0x56354a058350, pass=0, file_name=0x5635487807f8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x0000563547fd8597 in pfs_rw_lock_s_lock_func (lock=0x56354a058350, pass=0, file_name=0x5635487807f8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x0000563547fe1430 in row_mysql_freeze_data_dictionary_func (trx=0x7fdd01070268, file=0x5635487807f8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc", line=429) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:2123 #9 0x000056354802473d in row_undo (node=0x7fdc901fc8d8, thr=0x7fdc90c79cb8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc:429 #10 0x00005635480249aa in row_undo_step (thr=0x7fdc90c79cb8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0undo.cc:497 #11 0x0000563547f85bda in que_thr_step (thr=0x7fdc90c79cb8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1040 #12 0x0000563547f85e26 in que_run_threads_low (thr=0x7fdc90c79cb8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x0000563547f86016 in que_run_threads (thr=0x7fdc90c79cb8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x00005635480803af in trx_rollback_to_savepoint_low (trx=0x7fdd01070268, savept=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:137 #15 0x0000563548080868 in trx_rollback_for_mysql_low (trx=0x7fdd01070268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:211 #16 0x0000563548080ba7 in trx_rollback_for_mysql (trx=0x7fdd01070268) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0roll.cc:237 #17 0x0000563547e75815 in innobase_rollback (hton=0x563549e21a20, thd=0x7fdc90000b00, rollback_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4718 #18 0x0000563547c4f50c in ha_rollback_trans (thd=0x7fdc90000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1850 #19 0x0000563547a9441c in trans_rollback (thd=0x7fdc90000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:418 #20 0x000056354791db27 in mysql_execute_command (thd=0x7fdc90000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5866 #21 0x0000563547924935 in mysql_parse (thd=0x7fdc90000b00, rawbuf=0x7fdc90013e58 "ROLLBACK", length=8, parser_state=0x7fdd005860d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #22 0x0000563547923f68 in wsrep_mysql_parse (thd=0x7fdc90000b00, rawbuf=0x7fdc90013e58 "ROLLBACK", length=8, parser_state=0x7fdd005860d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #23 0x000056354790fe25 in dispatch_command (command=COM_QUERY, thd=0x7fdc90000b00, packet=0x7fdc9000a431 "ROLLBACK", packet_length=8, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #24 0x000056354790e80d in do_command (thd=0x7fdc90000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #25 0x0000563547a7f069 in do_handle_one_connection (connect=0x563549e11c30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #26 0x0000563547a7edad in handle_one_connection (arg=0x563549e11c30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #27 0x00005635483a3555 in pfs_spawn_thread (arg=0x563549cb9a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #28 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fdd005d2700 (LWP 26793)): #0 0x00007fdd12eeed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563548410d8f in safe_cond_timedwait (cond=0x5635491501a0 , mp=0x563549152ba0 , abstime=0x7fdd005d1c60, file=0x5635484cc5c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005635477dd845 in inline_mysql_cond_timedwait (that=0x5635491501a0 , mutex=0x563549152ba0 , abstime=0x7fdd005d1c60, src_file=0x5635484ce638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005635477e261b in cache_thread (thd=0x7fdca0000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005635477e28c9 in one_thread_per_connection_end (thd=0x7fdca0000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000563547a7f0f3 in do_handle_one_connection (connect=0x56354a1c1300) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000563547a7edad in handle_one_connection (arg=0x56354a1c1300) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005635483a3555 in pfs_spawn_thread (arg=0x56354a1e1590) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fdd08056700 (LWP 26587)): #0 0x00007fdd111e240a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005635477dcf44 in my_sigwait (set=0x7fdd08055cf0, sig=0x7fdd08055cb4, code=0x7fdd08055cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005635477e3584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005635483a3555 in pfs_spawn_thread (arg=0x563549e10fd0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fdcd0ff9700 (LWP 26586)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563548410a8d in safe_cond_wait (cond=0x7fdcd0ff8c70, mp=0x7fdcd0ff8cb0, file=0x5635486ded40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563547e6a525 in inline_mysql_cond_wait (that=0x7fdcd0ff8c70, mutex=0x7fdcd0ff8cb0, src_file=0x5635486e06d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563547e6f161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005635483a3555 in pfs_spawn_thread (arg=0x56354a08e9b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fdcd2ffd700 (LWP 26582)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f5325b in os_event::wait (this=0x56354a063ca0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563547f52ce1 in os_event::wait_low (this=0x56354a063ca0, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563547f53052 in os_event_wait_low (event=0x56354a063ca0, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056354804ccee in sync_array_wait_event (arr=0x563549e248c0, cell=@0x7fdcd2ffcb18: 0x563549e249f0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056354804fa03 in rw_lock_s_lock_spin (lock=0x56354a058350, pass=0, file_name=0x563548772e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x0000563547ffd328 in rw_lock_s_lock_func (lock=0x56354a058350, pass=0, file_name=0x563548772e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x0000563547ffd761 in pfs_rw_lock_s_lock_func (lock=0x56354a058350, pass=0, file_name=0x563548772e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x0000563548002b46 in row_purge_parse_undo_rec (node=0x56354a06ee68, undo_rec=0x56354a06f418 "", updated_extern=0x7fdcd2ffcc9e, thr=0x56354a06ec98) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1077 #9 0x00005635480034d1 in row_purge (node=0x56354a06ee68, undo_rec=0x56354a06f418 "", thr=0x56354a06ec98) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1258 #10 0x0000563548003798 in row_purge_step (thr=0x56354a06ec98) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #11 0x0000563547f85bf6 in que_thr_step (thr=0x56354a06ec98) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #12 0x0000563547f85e26 in que_run_threads_low (thr=0x56354a06ec98) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x0000563547f86016 in que_run_threads (thr=0x56354a06ec98) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x0000563548042947 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #15 0x0000563548042aee in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #16 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fdcdbfff700 (LWP 26575)): #0 0x00007fdd12eeed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f52b76 in os_event::timed_wait (this=0x563549e4d8b0, abstime=0x7fdcdbffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563547f52e8d in os_event::wait_time_low (this=0x563549e4d8b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563547f5302d in os_event_wait_time_low (event=0x563549e4d8b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563548040bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fdcdaffd700 (LWP 26574)): #0 0x00007fdd12ef1e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563547f5355d in os_thread_sleep (tm=30000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056354804e15a in sync_array_print_long_waits (waiter=0x7fdcdaffcc28, sema=0x7fdcdaffcc30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:1099 #3 0x0000563548040f7a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1801 #4 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fdce17fa700 (LWP 26473)): #0 0x00007fdd12abb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563547f46578 in LinuxAIOHandler::collect (this=0x7fdce17f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563547f46aa1 in LinuxAIOHandler::poll (this=0x7fdce17f9b00, m1=0x7fdce17f9ba8, m2=0x7fdce17f9bb0, request=0x7fdce17f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563547f46c04 in os_aio_linux_handler (global_segment=5, m1=0x7fdce17f9ba8, m2=0x7fdce17f9bb0, request=0x7fdce17f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563547f4b728 in os_aio_handler (segment=5, m1=0x7fdce17f9ba8, m2=0x7fdce17f9bb0, request=0x7fdce17f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056354819e5ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005635480449fb in io_handler_thread (arg=0x5635491cb648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fdcd1ffb700 (LWP 26584)): #0 0x00007fdd12ef1e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563547f5355d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005635480f9b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fdcd3fff700 (LWP 26580)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f5325b in os_event::wait (this=0x563549f74430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563547f52ce1 in os_event::wait_low (this=0x563549f74430, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563547f53052 in os_event_wait_low (event=0x563549f74430, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056354804ccee in sync_array_wait_event (arr=0x563549e248c0, cell=@0x7fdcd3ffc958: 0x563549e24ab0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005635480502ba in rw_lock_x_lock_func (lock=0x7fdd0084f688, pass=0, file_name=0x563548776430 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:735 #6 0x00005635480fb691 in pfs_rw_lock_x_lock_func (lock=0x7fdd0084f688, pass=0, file_name=0x563548776430 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:558 #7 0x000056354810d401 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x563548776430 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7fdcd3ffe1a0, err=0x7fdcd3ffccc0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4824 #8 0x00005635480ccd3c in btr_cur_search_to_nth_level_func (index=0x56354a1e92f8, level=0, tuple=0x7fdcc4011400, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x56354a06f168, ahi_latch=0x0, file=0x563548776430 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, mtr=0x7fdcd3ffe1a0, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1548 #9 0x00005635480086c4 in btr_pcur_open_low (index=0x56354a1e92f8, level=0, tuple=0x7fdcc4011400, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x56354a06f168, file=0x563548776430 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc", line=1219, autoinc=0, mtr=0x7fdcd3ffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:464 #10 0x000056354800b86c in row_search_on_row_ref (pcur=0x56354a06f168, mode=2, table=0x56354a1e5c08, ref=0x7fdcc4011400, mtr=0x7fdcd3ffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0row.cc:1219 #11 0x0000563547fffd13 in row_purge_reposition_pcur (mode=2, node=0x56354a06f0d0, mtr=0x7fdcd3ffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:104 #12 0x0000563547fffea5 in row_purge_remove_clust_if_poss_low (node=0x56354a06f0d0, mode=2) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:140 #13 0x00005635480002d5 in row_purge_remove_clust_if_poss (node=0x56354a06f0d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:219 #14 0x0000563548001b28 in row_purge_del_mark (node=0x56354a06f0d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #15 0x00005635480030b0 in row_purge_record_func (node=0x56354a06f0d0, undo_rec=0x7fdcc4011378 "", thr=0x56354a06f018, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #16 0x00005635480033ed in row_purge (node=0x56354a06f0d0, undo_rec=0x7fdcc4011378 "", thr=0x56354a06f018) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #17 0x0000563548003798 in row_purge_step (thr=0x56354a06f018) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #18 0x0000563547f85bf6 in que_thr_step (thr=0x56354a06f018) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #19 0x0000563547f85e26 in que_run_threads_low (thr=0x56354a06f018) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #20 0x0000563547f86016 in que_run_threads (thr=0x56354a06f018) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #21 0x0000563548042947 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #22 0x0000563548042aee in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #23 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fdd08248700 (LWP 26415)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563548410a8d in safe_cond_wait (cond=0x7fdcec015608, mp=0x7fdcec015558, file=0x56354857ac60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563547b68126 in inline_mysql_cond_wait (that=0x7fdcec015608, mutex=0x7fdcec015558, src_file=0x56354857af80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563547b6adbe in Wsrep_thd_queue::pop_front (this=0x7fdcec015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563547b69b9e in wsrep_post_rollback_process (post_rollbacker=0x7fdcec000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000563547b5a31f in start_wsrep_THD (arg=0x563549d3f810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fdcd9ffb700 (LWP 26577)): #0 0x00007fdd12eeed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f52b76 in os_event::timed_wait (this=0x563549fedca0, abstime=0x7fdcd9ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563547f52e8d in os_event::wait_time_low (this=0x563549fedca0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563547f5302d in os_event_wait_time_low (event=0x563549fedca0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005635480a1403 in ib_wqueue_timedwait (wq=0x563549fedb70, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005635481e1f9a in fts_optimize_thread (arg=0x563549fedb70) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fdcd27fc700 (LWP 26583)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f5325b in os_event::wait (this=0x563549e4d930) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563547f52ce1 in os_event::wait_low (this=0x563549e4d930, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563547f53052 in os_event_wait_low (event=0x563549e4d930, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056354811efb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fdd02e7d700 (LWP 26410)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd0add3867 in gu::Lock::wait (cond=..., this=0x7fdd02e7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fdd0add63ab in RecvBuf::front (this=this@entry=0x563549d458a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fdd0add3f46 in gcomm_recv (backend=, msg=0x563549d05830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fdd0adc7eec in core_msg_recv (backend=backend@entry=0x563549d05978, recv_msg=recv_msg@entry=0x563549d05830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fdd0adcb164 in gcs_core_recv (conn=0x563549d057d0, recv_act=recv_act@entry=0x7fdd02e7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fdd0add23d8 in gcs_recv_thread (arg=0x563549d05550) at gcs/src/gcs.cpp:1398 #7 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fdcdb7fe700 (LWP 26573)): #0 0x00007fdd12eeed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f52b76 in os_event::timed_wait (this=0x56354a025d60, abstime=0x7fdcdb7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563547f52e8d in os_event::wait_time_low (this=0x56354a025d60, time_in_usec=1000000, reset_sig_count=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563547f5302d in os_event_wait_time_low (event=0x56354a025d60, time_in_usec=1000000, reset_sig_count=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563547f1f9e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fdcda7fc700 (LWP 26576)): #0 0x00007fdd12eeed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f52b76 in os_event::timed_wait (this=0x56354a057e20, abstime=0x7fdcda7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563547f52e8d in os_event::wait_time_low (this=0x56354a057e20, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563547f5302d in os_event_wait_time_low (event=0x56354a057e20, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056354818824a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fdd0b943700 (LWP 26379)): #0 0x00007fdd12eeed12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563548410d8f in safe_cond_timedwait (cond=0x5635499ef640 , mp=0x5635499ef580 , abstime=0x7fdd0b942d70, file=0x563548898920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563548411d49 in inline_mysql_cond_timedwait (that=0x5635499ef640 , mutex=0x5635499ef580 , abstime=0x7fdd0b942d70, src_file=0x563548898968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005635484128ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005635483a3555 in pfs_spawn_thread (arg=0x563549cc6730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fdce37fe700 (LWP 26469)): #0 0x00007fdd12abb644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563547f46578 in LinuxAIOHandler::collect (this=0x7fdce37fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563547f46aa1 in LinuxAIOHandler::poll (this=0x7fdce37fdb00, m1=0x7fdce37fdba8, m2=0x7fdce37fdbb0, request=0x7fdce37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563547f46c04 in os_aio_linux_handler (global_segment=1, m1=0x7fdce37fdba8, m2=0x7fdce37fdbb0, request=0x7fdce37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563547f4b728 in os_aio_handler (segment=1, m1=0x7fdce37fdba8, m2=0x7fdce37fdbb0, request=0x7fdce37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056354819e5ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005635480449fb in io_handler_thread (arg=0x5635491cb628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fdce0ff9700 (LWP 26474)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563547f5325b in os_event::wait (this=0x563549f74430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563547f52ce1 in os_event::wait_low (this=0x563549f74430, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563547f53052 in os_event_wait_low (event=0x563549f74430, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056354804ccee in sync_array_wait_event (arr=0x563549e248c0, cell=@0x7fdce0ff8690: 0x563549e249b0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056354805056d in rw_lock_sx_lock_func (lock=0x7fdd0084f688, pass=2, file_name=0x5635487e1608 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x000056354811f93f in pfs_rw_lock_sx_lock_func (lock=0x7fdd0084f688, pass=2, file_name=0x5635487e1608 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x0000563548123fbc in buf_flush_page (buf_pool=0x563549f33130, bpage=0x7fdd0084f5c0, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1144 #8 0x0000563548124a06 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=4, n_to_flush=5) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1389 #9 0x0000563548124d89 in buf_flush_page_and_try_neighbors (bpage=0x7fdd0084f5c0, flush_type=BUF_FLUSH_LIST, n_to_flush=5, count=0x7fdce0ff88c8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1461 #10 0x00005635481259a2 in buf_do_flush_list_batch (buf_pool=0x563549f33130, min_n=5, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1719 #11 0x0000563548125f9a in buf_flush_batch (buf_pool=0x563549f33130, flush_type=BUF_FLUSH_LIST, min_n=5, lsn_limit=18446744073709551615, n=0x7fdce0ff8b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1789 #12 0x000056354812665a in buf_flush_do_batch (buf_pool=0x563549f33130, type=BUF_FLUSH_LIST, min_n=5, lsn_limit=18446744073709551615, n=0x7fdce0ff8b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1947 #13 0x0000563548129355 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #14 0x000056354812a2a3 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #15 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fdd081fd700 (LWP 26417)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd0aced132 in fifo_lock_get (q=q@entry=0x563549d2bbd0) at galerautils/src/gu_fifo.c:262 #2 0x00007fdd0aced7a1 in gu_fifo_get_head (q=0x563549d2bbd0, err=0x7fdd081fc06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fdd0adcf0cc in gcs_recv (conn=0x563549d05550, action=0x7fdd081fc0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fdd0adfc16c in galera::GcsActionSource::process (this=0x563549d04f30, recv_ctx=0x7fdd081fc4d0, exit_loop=@0x7fdd081fc29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fdd0ae1b033 in galera::ReplicatorSMM::async_recv (this=0x563549cfd280, recv_ctx=0x7fdd081fc4d0) at galera/src/replicator_smm.cpp:383 #6 0x00005635484c536c in wsrep::wsrep_provider_v26::run_applier (this=0x563549c84360, applier_ctx=0x7fdd081fc4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000563547b686d4 in wsrep_replication_process (thd=0x7fdcf0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000563547b5a31f in start_wsrep_THD (arg=0x563549d5f550) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fdd08293700 (LWP 26414)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563548410a8d in safe_cond_wait (cond=0x7fdcf8017b68, mp=0x7fdcf8017ab8, file=0x56354857ac60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563547b68126 in inline_mysql_cond_wait (that=0x7fdcf8017b68, mutex=0x7fdcf8017ab8, src_file=0x56354857af80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563547b6adbe in Wsrep_thd_queue::pop_front (this=0x7fdcf8017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563547b697de in wsrep_rollback_process (rollbacker=0x7fdcf8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000563547b5a31f in start_wsrep_THD (arg=0x563549d55340) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fdd09d96700 (LWP 26395)): #0 0x00007fdd12eee965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdd0adf8fd7 in gu::Lock::wait (cond=..., this=0x7fdd09d95bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fdd0adf9069 in galera::ServiceThd::thd_func (arg=0x563549cfd998) at galera/src/galera_service_thd.cpp:30 #3 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fdd0367e700 (LWP 26401)): #0 0x00007fdd112a9483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fdd0ad9a6f8 in asio::detail::epoll_reactor::run (this=0x563549d45fc0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fdd0ad9edf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x563549d45ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fdd0ad9f0a2 in asio::detail::task_io_service::run (this=0x563549d45ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fdd0adb6afd in asio::io_service::run (this=0x563549d44fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fdd0adb7251 in gcomm::AsioProtonet::event_loop (this=0x563549d44f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fdd0add52ed in GCommConn::run (this=0x563549d456c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fdd0add8399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fdd12eeadd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdd112a8ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fdd132fb880 (LWP 26372)): #0 0x00007fdd12eef9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056354840b45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563547c495a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fdd1129e20d in poll () from /lib64/libc.so.6 #5 0x00005635477e9a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005635477e8d25 in mysqld_main (argc=163, argv=0x563549b7ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005635477dcec5 in main (argc=25, argv=0x7ffeb0f3dfe8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.1/data/core.26372 - found 'core.26883' (1/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.2/data/core.26883 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 26883] [New LWP 27320] [New LWP 26955] [New LWP 26993] [New LWP 27317] [New LWP 27140] [New LWP 28975] [New LWP 27318] [New LWP 28987] [New LWP 27319] [New LWP 28988] [New LWP 28925] [New LWP 28996] [New LWP 28926] [New LWP 28927] [New LWP 28928] [New LWP 28929] [New LWP 28930] [New LWP 28931] [New LWP 28986] [New LWP 29027] [New LWP 28982] [New LWP 28980] [New LWP 28979] [New LWP 28978] [New LWP 28977] [New LWP 28976] [New LWP 28932] [New LWP 28995] [New LWP 28985] [New LWP 28981] [New LWP 28983] [New LWP 28984] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f75405519d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f75405519d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559488de245e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005594886205a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f753e90020d in poll () from /lib64/libc.so.6 #5 0x00005594881c0a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005594881bfd25 in mysqld_main (argc=164, argv=0x55948b32bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005594881b3ec5 in main (argc=25, argv=0x7fff709a6f48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f74e7fff700 (LWP 28984)): #0 0x00007f7540553d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055948891ec27 in os_file_fsync_posix (file=29) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055948891ef7d in os_file_flush_func (file=29) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000559488b674aa in pfs_os_file_flush_func (file=..., src_file=0x5594891e1b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559488b6a225 in fil_flush_low (space=0x55948b7a6c00, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000559488b75c19 in fil_flush (space=0x55948b7a6c00) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4468 #6 0x0000559488afaa21 in buf_flush_write_block_low (bpage=0x7f750d8ef140, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1027 #7 0x0000559488afb0be in buf_flush_page (buf_pool=0x55948b6e8e20, bpage=0x7f750d8ef140, flush_type=BUF_FLUSH_SINGLE_PAGE, sync=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1165 #8 0x0000559488afdba3 in buf_flush_single_page_from_LRU (buf_pool=0x55948b6e8e20) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2138 #9 0x0000559488b08b0e in buf_LRU_get_free_block (buf_pool=0x55948b6e8e20) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1194 #10 0x0000559488ae5e1b in buf_page_init_for_read (err=0x7f74e7ffd194, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5332 #11 0x0000559488b0f1c6 in buf_read_page_low (err=0x7f74e7ffd194, sync=true, type=0, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #12 0x0000559488b0fda6 in buf_read_page (page_id=..., zip_size=0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:412 #13 0x0000559488ae2ce2 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x559489199068 "/home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc", line=8113, mtr=0x7f74e7ffd620, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4340 #14 0x0000559488ab721f in btr_free_externally_stored_field (index=0x7f751402fb78, field_ref=0x7f750e0fc184 "", rec=0x7f750e0fc13f "6b622e7b-2b8b-11e9-b19b-fa5e78a09dc6\200", offsets=0x7f74d400a7a8, page_zip=0x0, i=6, rollback=false, local_mtr=0x7f74e7ffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8113 #15 0x0000559488ab7762 in btr_rec_free_externally_stored_fields (index=0x7f751402fb78, rec=0x7f750e0fc13f "6b622e7b-2b8b-11e9-b19b-fa5e78a09dc6\200", offsets=0x7f74d400a7a8, page_zip=0x0, rollback=false, mtr=0x7f74e7ffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:8215 #16 0x0000559488ab1db9 in btr_cur_pessimistic_delete (err=0x7f74e7ffde5c, has_reserved_extents=0, cursor=0x55948b827d10, flags=0, rollback=false, mtr=0x7f74e7ffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5905 #17 0x00005594889d7176 in row_purge_remove_clust_if_poss_low (node=0x55948b827c78, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:175 #18 0x00005594889d72fb in row_purge_remove_clust_if_poss (node=0x55948b827c78) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #19 0x00005594889d8b28 in row_purge_del_mark (node=0x55948b827c78) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #20 0x00005594889da0b0 in row_purge_record_func (node=0x55948b827c78, undo_rec=0x55948b828848 "", thr=0x55948b8277d0, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #21 0x00005594889da3ed in row_purge (node=0x55948b827c78, undo_rec=0x55948b828848 "", thr=0x55948b8277d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #22 0x00005594889da798 in row_purge_step (thr=0x55948b8277d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #23 0x000055948895cbf6 in que_thr_step (thr=0x55948b8277d0) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #24 0x000055948895ce26 in que_run_threads_low (thr=0x55948b8277d0) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #25 0x000055948895d016 in que_run_threads (thr=0x55948b8277d0) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #26 0x0000559488a19947 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #27 0x0000559488a19aee in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #28 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f74ecff9700 (LWP 28983)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055948892a25b in os_event::wait (this=0x55948b815380) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559488929ce1 in os_event::wait_low (this=0x55948b815380, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055948892a052 in os_event_wait_low (event=0x55948b815380, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559488a23cee in sync_array_wait_event (arr=0x55948b5da800, cell=@0x7f74ecff8b18: 0x55948b5da970) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000559488a26a03 in rw_lock_s_lock_spin (lock=0x55948b810940, pass=0, file_name=0x559489149e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x00005594889d4328 in rw_lock_s_lock_func (lock=0x55948b810940, pass=0, file_name=0x559489149e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x00005594889d4761 in pfs_rw_lock_s_lock_func (lock=0x55948b810940, pass=0, file_name=0x559489149e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x00005594889d9b46 in row_purge_parse_undo_rec (node=0x55948b827620, undo_rec=0x55948b827af8 "", updated_extern=0x7f74ecff8c9e, thr=0x55948b827568) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1077 #9 0x00005594889da4d1 in row_purge (node=0x55948b827620, undo_rec=0x55948b827af8 "", thr=0x55948b827568) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1258 #10 0x00005594889da798 in row_purge_step (thr=0x55948b827568) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #11 0x000055948895cbf6 in que_thr_step (thr=0x55948b827568) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #12 0x000055948895ce26 in que_run_threads_low (thr=0x55948b827568) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000055948895d016 in que_run_threads (thr=0x55948b827568) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x0000559488a19947 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #15 0x0000559488a19aee in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #16 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f74edffb700 (LWP 28981)): #0 0x00007f753e8efd47 in sched_yield () from /lib64/libc.so.6 #1 0x000055948892a4ef in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x0000559488a43ccb in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x0000559488a43ed3 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x0000559488a19dbc in srv_do_purge (n_total_purged=0x7f74edffad98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x0000559488a1a1e9 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f74e77fe700 (LWP 28985)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055948892a25b in os_event::wait (this=0x55948b603620) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559488929ce1 in os_event::wait_low (this=0x55948b603620, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055948892a052 in os_event_wait_low (event=0x55948b603620, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559488af5fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f750d6cf700 (LWP 28995)): #0 0x00007f753e84440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005594881b3f44 in my_sigwait (set=0x7f750d6cecf0, sig=0x7f750d6cecb4, code=0x7f750d6cecb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005594881ba584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000559488d7a555 in pfs_spawn_thread (arg=0x55948b5c6b80) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f7501ffb700 (LWP 28932)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055948892a25b in os_event::wait (this=0x55948b74dbb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559488929ce1 in os_event::wait_low (this=0x55948b74dbb0, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055948892a052 in os_event_wait_low (event=0x55948b74dbb0, reset_sig_count=5) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559488a23cee in sync_array_wait_event (arr=0x55948b5da800, cell=@0x7f7501ffa680: 0x55948b5daa30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000559488a2756d in rw_lock_sx_lock_func (lock=0x7f750d91cad8, pass=2, file_name=0x5594891b8608 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x0000559488af693f in pfs_rw_lock_sx_lock_func (lock=0x7f750d91cad8, pass=2, file_name=0x5594891b8608 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1144) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x0000559488afafbc in buf_flush_page (buf_pool=0x55948b6e8e20, bpage=0x7f750d91ca10, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1144 #8 0x0000559488afba06 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=62, n_to_flush=200) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1389 #9 0x0000559488afbd89 in buf_flush_page_and_try_neighbors (bpage=0x7f750d91ca10, flush_type=BUF_FLUSH_LIST, n_to_flush=200, count=0x7f7501ffa8b8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1461 #10 0x0000559488afc9a2 in buf_do_flush_list_batch (buf_pool=0x55948b6e8e20, min_n=200, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1719 #11 0x0000559488afcf9a in buf_flush_batch (buf_pool=0x55948b6e8e20, flush_type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7f7501ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1789 #12 0x0000559488afd65a in buf_flush_do_batch (buf_pool=0x55948b6e8e20, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7f7501ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1947 #13 0x0000559488afd92e in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7f7501ffabd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2053 #14 0x0000559488b01679 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3205 #15 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f75017fa700 (LWP 28976)): #0 0x00007f7540553e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055948892a55d in os_thread_sleep (tm=30000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559488a2515a in sync_array_print_long_waits (waiter=0x7f75017f9c28, sema=0x7f75017f9c30) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:1099 #3 0x0000559488a17f7a in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1801 #4 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f74effff700 (LWP 28977)): #0 0x00007f7540550d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488929b76 in os_event::timed_wait (this=0x55948b6035a0, abstime=0x7f74efffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559488929e8d in os_event::wait_time_low (this=0x55948b6035a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055948892a02d in os_event_wait_time_low (event=0x55948b6035a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559488a17bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f74ef7fe700 (LWP 28978)): #0 0x00007f7540550d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488929b76 in os_event::timed_wait (this=0x55948b80db10, abstime=0x7f74ef7fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559488929e8d in os_event::wait_time_low (this=0x55948b80db10, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055948892a02d in os_event_wait_time_low (event=0x55948b80db10, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559488b5f24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f74eeffd700 (LWP 28979)): #0 0x00007f7540550d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488929b76 in os_event::timed_wait (this=0x55948b7abb10, abstime=0x7f74eeffcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559488929e8d in os_event::wait_time_low (this=0x55948b7abb10, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055948892a02d in os_event_wait_time_low (event=0x55948b7abb10, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559488a78403 in ib_wqueue_timedwait (wq=0x55948b7ab9e0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559488bb8f9a in fts_optimize_thread (arg=0x55948b7ab9e0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f74ee7fc700 (LWP 28980)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055948892a25b in os_event::wait (this=0x55948b815400) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559488929ce1 in os_event::wait_low (this=0x55948b815400, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055948892a052 in os_event_wait_low (event=0x55948b815400, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559488a23cee in sync_array_wait_event (arr=0x55948b5da800, cell=@0x7f74ee7fbb98: 0x55948b5da9b0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000559488a26bd4 in rw_lock_x_lock_wait_func (lock=0x55948b810940, pass=0, threshold=0, file_name=0x559489167c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:459 #6 0x0000559488a26d06 in rw_lock_x_lock_low (lock=0x55948b810940, pass=0, file_name=0x559489167c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:514 #7 0x0000559488a27124 in rw_lock_x_lock_func (lock=0x55948b810940, pass=0, file_name=0x559489167c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:673 #8 0x0000559488a15535 in pfs_rw_lock_x_lock_func (lock=0x55948b810940, pass=0, file_name=0x559489167c28 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=1999) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:552 #9 0x0000559488a18578 in srv_master_evict_from_table_cache (pct_check=100) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1999 #10 0x0000559488a19074 in srv_master_do_idle_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2237 #11 0x0000559488a1964a in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2358 #12 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f74ed7fa700 (LWP 28982)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055948892a25b in os_event::wait (this=0x55948b815380) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559488929ce1 in os_event::wait_low (this=0x55948b815380, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055948892a052 in os_event_wait_low (event=0x55948b815380, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559488a23cee in sync_array_wait_event (arr=0x55948b5da800, cell=@0x7f74ed7f9b18: 0x55948b5da9f0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000559488a26a03 in rw_lock_s_lock_spin (lock=0x55948b810940, pass=0, file_name=0x559489149e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x00005594889d4328 in rw_lock_s_lock_func (lock=0x55948b810940, pass=0, file_name=0x559489149e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x00005594889d4761 in pfs_rw_lock_s_lock_func (lock=0x55948b810940, pass=0, file_name=0x559489149e60 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=1077) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x00005594889d9b46 in row_purge_parse_undo_rec (node=0x55948b8273b8, undo_rec=0x55948b827968 "", updated_extern=0x7f74ed7f9c9e, thr=0x55948b8271e8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1077 #9 0x00005594889da4d1 in row_purge (node=0x55948b8273b8, undo_rec=0x55948b827968 "", thr=0x55948b8271e8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1258 #10 0x00005594889da798 in row_purge_step (thr=0x55948b8271e8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #11 0x000055948895cbf6 in que_thr_step (thr=0x55948b8271e8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #12 0x000055948895ce26 in que_run_threads_low (thr=0x55948b8271e8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #13 0x000055948895d016 in que_run_threads (thr=0x55948b8271e8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #14 0x0000559488a19947 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #15 0x0000559488a19aee in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #16 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f750d639700 (LWP 29027)): #0 0x00007f7540550d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488de7d8f in safe_cond_timedwait (cond=0x559489b271a0 , mp=0x559489b29ba0 , abstime=0x7f750d638c60, file=0x559488ea35c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005594881b4845 in inline_mysql_cond_timedwait (that=0x559489b271a0 , mutex=0x559489b29ba0 , abstime=0x7f750d638c60, src_file=0x559488ea5638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005594881b961b in cache_thread (thd=0x7f7514000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005594881b98c9 in one_thread_per_connection_end (thd=0x7f7514000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005594884560f3 in do_handle_one_connection (connect=0x55948b964b00) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000559488455dad in handle_one_connection (arg=0x55948b964b00) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000559488d7a555 in pfs_spawn_thread (arg=0x55948b98a5d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f74e6ffd700 (LWP 28986)): #0 0x00007f7540553e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055948892a55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559488ad0b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f75027fc700 (LWP 28931)): #0 0x00007f754011d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055948891d578 in LinuxAIOHandler::collect (this=0x7f75027fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055948891daa1 in LinuxAIOHandler::poll (this=0x7f75027fbb00, m1=0x7f75027fbba8, m2=0x7f75027fbbb0, request=0x7f75027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055948891dc04 in os_aio_linux_handler (global_segment=5, m1=0x7f75027fbba8, m2=0x7f75027fbbb0, request=0x7f75027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559488922728 in os_aio_handler (segment=5, m1=0x7f75027fbba8, m2=0x7f75027fbbb0, request=0x7f75027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559488b755ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559488a1b9fb in io_handler_thread (arg=0x559489ba2648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f7502ffd700 (LWP 28930)): #0 0x00007f754011d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055948891d578 in LinuxAIOHandler::collect (this=0x7f7502ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055948891daa1 in LinuxAIOHandler::poll (this=0x7f7502ffcb00, m1=0x7f7502ffcba8, m2=0x7f7502ffcbb0, request=0x7f7502ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055948891dc04 in os_aio_linux_handler (global_segment=4, m1=0x7f7502ffcba8, m2=0x7f7502ffcbb0, request=0x7f7502ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559488922728 in os_aio_handler (segment=4, m1=0x7f7502ffcba8, m2=0x7f7502ffcbb0, request=0x7f7502ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559488b755ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559488a1b9fb in io_handler_thread (arg=0x559489ba2640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f75037fe700 (LWP 28929)): #0 0x00007f754011d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055948891d578 in LinuxAIOHandler::collect (this=0x7f75037fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055948891daa1 in LinuxAIOHandler::poll (this=0x7f75037fdb00, m1=0x7f75037fdba8, m2=0x7f75037fdbb0, request=0x7f75037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055948891dc04 in os_aio_linux_handler (global_segment=3, m1=0x7f75037fdba8, m2=0x7f75037fdbb0, request=0x7f75037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559488922728 in os_aio_handler (segment=3, m1=0x7f75037fdba8, m2=0x7f75037fdbb0, request=0x7f75037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559488b755ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559488a1b9fb in io_handler_thread (arg=0x559489ba2638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f7503fff700 (LWP 28928)): #0 0x00007f754011d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055948891d578 in LinuxAIOHandler::collect (this=0x7f7503ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055948891daa1 in LinuxAIOHandler::poll (this=0x7f7503ffeb00, m1=0x7f7503ffeba8, m2=0x7f7503ffebb0, request=0x7f7503ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055948891dc04 in os_aio_linux_handler (global_segment=2, m1=0x7f7503ffeba8, m2=0x7f7503ffebb0, request=0x7f7503ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559488922728 in os_aio_handler (segment=2, m1=0x7f7503ffeba8, m2=0x7f7503ffebb0, request=0x7f7503ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559488b755ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559488a1b9fb in io_handler_thread (arg=0x559489ba2630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f750ccce700 (LWP 28927)): #0 0x00007f754011d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055948891d578 in LinuxAIOHandler::collect (this=0x7f750cccdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055948891daa1 in LinuxAIOHandler::poll (this=0x7f750cccdb00, m1=0x7f750cccdba8, m2=0x7f750cccdbb0, request=0x7f750cccdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055948891dc04 in os_aio_linux_handler (global_segment=1, m1=0x7f750cccdba8, m2=0x7f750cccdbb0, request=0x7f750cccdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559488922728 in os_aio_handler (segment=1, m1=0x7f750cccdba8, m2=0x7f750cccdbb0, request=0x7f750cccdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559488b755ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559488a1b9fb in io_handler_thread (arg=0x559489ba2628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f750d4cf700 (LWP 28926)): #0 0x00007f754011d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055948891d578 in LinuxAIOHandler::collect (this=0x7f750d4ceb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055948891daa1 in LinuxAIOHandler::poll (this=0x7f750d4ceb00, m1=0x7f750d4ceba8, m2=0x7f750d4cebb0, request=0x7f750d4cebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055948891dc04 in os_aio_linux_handler (global_segment=0, m1=0x7f750d4ceba8, m2=0x7f750d4cebb0, request=0x7f750d4cebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559488922728 in os_aio_handler (segment=0, m1=0x7f750d4ceba8, m2=0x7f750d4cebb0, request=0x7f750d4cebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559488b755ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559488a1b9fb in io_handler_thread (arg=0x559489ba2620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f750d684700 (LWP 28996)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488de7a8d in safe_cond_wait (cond=0x559489b271e0 , mp=0x559489b2a560 , file=0x559488eb2dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005594881e54ee in inline_mysql_cond_wait (that=0x559489b271e0 , mutex=0x559489b2a560 , src_file=0x559488eb3930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005594881e6d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559488d7a555 in pfs_spawn_thread (arg=0x55948b965e20) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f750ed21700 (LWP 28925)): #0 0x00007f7540550d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488de7d8f in safe_cond_timedwait (cond=0x55948a3b3ea0 , mp=0x55948a3b3de0 , abstime=0x7f750ed20cb0, file=0x55948922b6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559488c51bc1 in inline_mysql_cond_timedwait (that=0x55948a3b3ea0 , mutex=0x55948a3b3de0 , abstime=0x7f750ed20cb0, src_file=0x55948922b738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559488c520e5 in my_service_thread_sleep (control=0x559489968b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559488c4657e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559488d7a555 in pfs_spawn_thread (arg=0x55948b5bb510) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f74e5ffb700 (LWP 28988)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488de7a8d in safe_cond_wait (cond=0x7f74e5ffac70, mp=0x7f74e5ffacb0, file=0x5594890b5d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559488841525 in inline_mysql_cond_wait (that=0x7f74e5ffac70, mutex=0x7f74e5ffacb0, src_file=0x5594890b76d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559488846161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000559488d7a555 in pfs_spawn_thread (arg=0x55948b7be6e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f7528925700 (LWP 27319)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488de7a8d in safe_cond_wait (cond=0x7f7518015608, mp=0x7f7518015558, file=0x559488f51c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055948853f126 in inline_mysql_cond_wait (that=0x7f7518015608, mutex=0x7f7518015558, src_file=0x559488f51f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559488541dbe in Wsrep_thd_queue::pop_front (this=0x7f7518015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559488540b9e in wsrep_post_rollback_process (post_rollbacker=0x7f7518000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055948853131f in start_wsrep_THD (arg=0x55948b50ab10) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f74e67fc700 (LWP 28987)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055948892a25b in os_event::wait (this=0x55948b603720) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559488929ce1 in os_event::wait_low (this=0x55948b603720, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055948892a052 in os_event_wait_low (event=0x55948b603720, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559488adec2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f7528970700 (LWP 27318)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488de7a8d in safe_cond_wait (cond=0x7f7520017b68, mp=0x7f7520017ab8, file=0x559488f51c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055948853f126 in inline_mysql_cond_wait (that=0x7f7520017b68, mutex=0x7f7520017ab8, src_file=0x559488f51f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559488541dbe in Wsrep_thd_queue::pop_front (this=0x7f7520017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005594885407de in wsrep_rollback_process (rollbacker=0x7f7520001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055948853131f in start_wsrep_THD (arg=0x55948b513ec0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f7500ff9700 (LWP 28975)): #0 0x00007f7540550d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488929b76 in os_event::timed_wait (this=0x55948b7dba50, abstime=0x7f7500ff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559488929e8d in os_event::wait_time_low (this=0x55948b7dba50, time_in_usec=1000000, reset_sig_count=14) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055948892a02d in os_event_wait_time_low (event=0x55948b7dba50, time_in_usec=1000000, reset_sig_count=14) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005594888f69e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f7529972700 (LWP 27140)): #0 0x00007f753e90b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f75383fc6f8 in asio::detail::epoll_reactor::run (this=0x55948b4f3030, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f7538400df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55948b4f2ed0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f75384010a2 in asio::detail::task_io_service::run (this=0x55948b4f2ed0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f7538418afd in asio::io_service::run (this=0x55948b4f1bf0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f7538419251 in gcomm::AsioProtonet::event_loop (this=0x55948b4f1b50, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f75384372ed in GCommConn::run (this=0x55948b4f2660) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f753843a399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f7529171700 (LWP 27317)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7538435867 in gu::Lock::wait (cond=..., this=0x7f7529170850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f75384383ab in RecvBuf::front (this=this@entry=0x55948b4f2848, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f7538435f46 in gcomm_recv (backend=, msg=0x55948b4b2750, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f7538429eec in core_msg_recv (backend=backend@entry=0x55948b4b2898, recv_msg=recv_msg@entry=0x55948b4b2750, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f753842d164 in gcs_core_recv (conn=0x55948b4b26f0, recv_act=recv_act@entry=0x7f7529170d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f75384343d8 in gcs_recv_thread (arg=0x55948b4b2470) at gcs/src/gcs.cpp:1398 #7 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f752bdf6700 (LWP 26993)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f753845afd7 in gu::Lock::wait (cond=..., this=0x7f752bdf5bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f753845b069 in galera::ServiceThd::thd_func (arg=0x55948b4aa958) at galera/src/galera_service_thd.cpp:30 #3 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f7538fa5700 (LWP 26955)): #0 0x00007f7540550d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559488de7d8f in safe_cond_timedwait (cond=0x55948a3c6640 , mp=0x55948a3c6580 , abstime=0x7f7538fa4d70, file=0x55948926f920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559488de8d49 in inline_mysql_cond_timedwait (that=0x55948a3c6640 , mutex=0x55948a3c6580 , abstime=0x7f7538fa4d70, src_file=0x55948926f968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559488de98ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559488d7a555 in pfs_spawn_thread (arg=0x55948b473860) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f75288da700 (LWP 27320)): #0 0x00007f7540550965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f753834f132 in fifo_lock_get (q=q@entry=0x55948b4d8af0) at galerautils/src/gu_fifo.c:262 #2 0x00007f753834f7a1 in gu_fifo_get_head (q=0x55948b4d8af0, err=0x7f75288d906c) at galerautils/src/gu_fifo.c:330 #3 0x00007f75384310cc in gcs_recv (conn=0x55948b4b2470, action=0x7f75288d90b0) at gcs/src/gcs.cpp:2024 #4 0x00007f753845e16c in galera::GcsActionSource::process (this=0x55948b4eae90, recv_ctx=0x7f75288d94d0, exit_loop=@0x7f75288d929e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f753847d033 in galera::ReplicatorSMM::async_recv (this=0x55948b4aa240, recv_ctx=0x7f75288d94d0) at galera/src/replicator_smm.cpp:383 #6 0x0000559488e9c36c in wsrep::wsrep_provider_v26::run_applier (this=0x55948b41fee0, applier_ctx=0x7f75288d94d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055948853f6d4 in wsrep_replication_process (thd=0x7f7510000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055948853131f in start_wsrep_THD (arg=0x55948b5136a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f754054cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f753e90aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f754095d880 (LWP 26883)): #0 0x00007f75405519d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559488de245e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005594886205a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f753e90020d in poll () from /lib64/libc.so.6 #5 0x00005594881c0a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005594881bfd25 in mysqld_main (argc=164, argv=0x55948b32bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005594881b3ec5 in main (argc=25, argv=0x7fff709a6f48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_large_fragment-innodb/mysqld.2/data/core.26883 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_large_fragment-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_large_fragment-innodb/' galera_sr.GCF-1018B 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 12:40:05 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/GCF-1018B.log == connection node_2; connection node_1; connection node_1; SET SESSION wsrep_sync_wait = 0; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 64; SET SESSION innodb_lock_wait_timeout = 1000; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 64; SET SESSION innodb_lock_wait_timeout = 1000; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.29818' (2/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.1/data/core.29818 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 29818] [New LWP 29940] [New LWP 29961] [New LWP 29850] [New LWP 29949] [New LWP 29884] [New LWP 29950] [New LWP 29886] [New LWP 29952] [New LWP 29936] [New LWP 29942] [New LWP 29953] [New LWP 29957] [New LWP 29959] [New LWP 29962] [New LWP 29963] [New LWP 29974] [New LWP 30257] [New LWP 29938] [New LWP 29866] [New LWP 31083] [New LWP 30256] [New LWP 29964] [New LWP 29960] [New LWP 29958] [New LWP 29956] [New LWP 29951] [New LWP 29941] [New LWP 29939] [New LWP 29935] [New LWP 29885] [New LWP 29854] [New LWP 29881] [New LWP 29937] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fbbefaa29d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fbbefaa29d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005639f738145e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005639f6bbf5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fbbede5120d in poll () from /lib64/libc.so.6 #5 0x00005639f675fa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005639f675ed25 in mysqld_main (argc=161, argv=0x5639f9ff6d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005639f6752ec5 in main (argc=23, argv=0x7fffb1068288) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fbbdc805700 (LWP 29937)): #0 0x00007fbbefaa4d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005639f6ebdc27 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005639f6ebdf7d in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005639f71064aa in pfs_os_file_flush_func (file=..., src_file=0x5639f7780b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005639f7109225 in fil_flush_low (space=0x5639fa4c7380, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x00005639f7114aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x00005639f71147af in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4390 #7 0x00005639f6fba9fb in io_handler_thread (arg=0x5639f8141628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #8 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fbbdf7fe700 (LWP 29881)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbbe7986867 in gu::Lock::wait (cond=..., this=0x7fbbdf7fd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fbbe79893ab in RecvBuf::front (this=this@entry=0x5639fa1b58a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fbbe7986f46 in gcomm_recv (backend=, msg=0x5639fa17d830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fbbe797aeec in core_msg_recv (backend=backend@entry=0x5639fa17d978, recv_msg=recv_msg@entry=0x5639fa17d830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fbbe797e164 in gcs_core_recv (conn=0x5639fa17d7d0, recv_act=recv_act@entry=0x7fbbdf7fdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fbbe79853d8 in gcs_recv_thread (arg=0x5639fa17d550) at gcs/src/gcs.cpp:1398 #7 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fbbe6949700 (LWP 29854)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbbe79abfd7 in gu::Lock::wait (cond=..., this=0x7fbbe6948bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fbbe79ac069 in galera::ServiceThd::thd_func (arg=0x5639fa175998) at galera/src/galera_service_thd.cpp:30 #3 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fbbe447a700 (LWP 29885)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386a8d in safe_cond_wait (cond=0x7fbbc8015608, mp=0x7fbbc8015558, file=0x5639f74f0c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005639f6ade126 in inline_mysql_cond_wait (that=0x7fbbc8015608, mutex=0x7fbbc8015558, src_file=0x5639f74f0f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005639f6ae0dbe in Wsrep_thd_queue::pop_front (this=0x7fbbc8015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005639f6adfb9e in wsrep_post_rollback_process (post_rollbacker=0x7fbbc8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005639f6ad031f in start_wsrep_THD (arg=0x5639fa1af810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fbbde858700 (LWP 29935)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386d8f in safe_cond_timedwait (cond=0x5639f8952ea0 , mp=0x5639f8952de0 , abstime=0x7fbbde857cb0, file=0x5639f77ca6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005639f71f0bc1 in inline_mysql_cond_timedwait (that=0x5639f8952ea0 , mutex=0x5639f8952de0 , abstime=0x7fbbde857cb0, src_file=0x5639f77ca738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005639f71f10e5 in my_service_thread_sleep (control=0x5639f7f07b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005639f71e557e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005639f7319555 in pfs_spawn_thread (arg=0x5639fa2757c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fbbbf7fe700 (LWP 29939)): #0 0x00007fbbef66e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005639f6ebc578 in LinuxAIOHandler::collect (this=0x7fbbbf7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005639f6ebcaa1 in LinuxAIOHandler::poll (this=0x7fbbbf7fdb00, m1=0x7fbbbf7fdba8, m2=0x7fbbbf7fdbb0, request=0x7fbbbf7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005639f6ebcc04 in os_aio_linux_handler (global_segment=3, m1=0x7fbbbf7fdba8, m2=0x7fbbbf7fdbb0, request=0x7fbbbf7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005639f6ec1728 in os_aio_handler (segment=3, m1=0x7fbbbf7fdba8, m2=0x7fbbbf7fdbb0, request=0x7fbbbf7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005639f71145ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005639f6fba9fb in io_handler_thread (arg=0x5639f8141638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fbbbe7fc700 (LWP 29941)): #0 0x00007fbbef66e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005639f6ebc578 in LinuxAIOHandler::collect (this=0x7fbbbe7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005639f6ebcaa1 in LinuxAIOHandler::poll (this=0x7fbbbe7fbb00, m1=0x7fbbbe7fbba8, m2=0x7fbbbe7fbbb0, request=0x7fbbbe7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005639f6ebcc04 in os_aio_linux_handler (global_segment=5, m1=0x7fbbbe7fbba8, m2=0x7fbbbe7fbbb0, request=0x7fbbbe7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005639f6ec1728 in os_aio_handler (segment=5, m1=0x7fbbbe7fbba8, m2=0x7fbbbe7fbbb0, request=0x7fbbbe7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005639f71145ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005639f6fba9fb in io_handler_thread (arg=0x5639f8141648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fbbbd5f5700 (LWP 29951)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec8b76 in os_event::timed_wait (this=0x5639fa2bd890, abstime=0x7fbbbd5f4d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005639f6ec8e8d in os_event::wait_time_low (this=0x5639fa2bd890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005639f6ec902d in os_event_wait_time_low (event=0x5639fa2bd890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005639f6fb6bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fbbb67fc700 (LWP 29956)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec925b in os_event::wait (this=0x5639fa455ad0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005639f6ec8ce1 in os_event::wait_low (this=0x5639fa455ad0, reset_sig_count=111) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005639f6ec9052 in os_event_wait_low (event=0x5639fa455ad0, reset_sig_count=111) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005639f6fc2cee in sync_array_wait_event (arr=0x5639fa294a10, cell=@0x7fbbb67fbbc8: 0x5639fa294c80) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005639f6fc5a03 in rw_lock_s_lock_spin (lock=0x5639f7f00540 , pass=0, file_name=0x5639f7689958 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:378 #6 0x00005639f6e963b1 in rw_lock_s_lock_func (lock=0x5639f7f00540 , pass=0, file_name=0x5639f7689958 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:285 #7 0x00005639f6e968e4 in pfs_rw_lock_s_lock_func (lock=0x5639f7f00540 , pass=0, file_name=0x5639f7689958 "/home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc", line=1385) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:645 #8 0x00005639f6e9adf3 in log_write_checkpoint_info (sync=true, end_lsn=2470223) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1385 #9 0x00005639f6e9b2fb in log_checkpoint (sync=true, write_always=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1534 #10 0x00005639f6fb7d7d in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2178 #11 0x00005639f6fb8643 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fbbb57fa700 (LWP 29958)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec925b in os_event::wait (this=0x5639fa2bd690) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005639f6ec8ce1 in os_event::wait_low (this=0x5639fa2bd690, reset_sig_count=8469) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005639f6ec9052 in os_event_wait_low (event=0x5639fa2bd690, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005639f6fb4ee5 in srv_resume_thread (slot=0x5639f7f03780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005639f6fb8ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fbba3fff700 (LWP 29960)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec925b in os_event::wait (this=0x5639fa2bd790) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005639f6ec8ce1 in os_event::wait_low (this=0x5639fa2bd790, reset_sig_count=7654) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005639f6ec9052 in os_event_wait_low (event=0x5639fa2bd790, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005639f6fb4ee5 in srv_resume_thread (slot=0x5639f7f037f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005639f6fb8ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fbba1ffb700 (LWP 29964)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386a8d in safe_cond_wait (cond=0x7fbba1ffac70, mp=0x7fbba1ffacb0, file=0x5639f7654d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005639f6de0525 in inline_mysql_cond_wait (that=0x7fbba1ffac70, mutex=0x7fbba1ffacb0, src_file=0x5639f76566d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005639f6de5161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005639f7319555 in pfs_spawn_thread (arg=0x5639fa4fe5d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fbbdd1bb700 (LWP 30256)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386a8d in safe_cond_wait (cond=0x5639f80c61e0 , mp=0x5639f80c9560 , file=0x5639f7451dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005639f67844ee in inline_mysql_cond_wait (that=0x5639f80c61e0 , mutex=0x5639f80c9560 , src_file=0x5639f7452930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005639f6785d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005639f7319555 in pfs_spawn_thread (arg=0x5639fa61ed70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fbbdd125700 (LWP 31083)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386d8f in safe_cond_timedwait (cond=0x5639f80c61a0 , mp=0x5639f80c8ba0 , abstime=0x7fbbdd124c60, file=0x5639f74425c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005639f6753845 in inline_mysql_cond_timedwait (that=0x5639f80c61a0 , mutex=0x5639f80c8ba0 , abstime=0x7fbbdd124c60, src_file=0x5639f7444638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005639f675861b in cache_thread (thd=0x7fbb70000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005639f67588c9 in one_thread_per_connection_end (thd=0x7fbb70000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005639f69f50f3 in do_handle_one_connection (connect=0x5639fa550d50) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005639f69f4dad in handle_one_connection (arg=0x5639fa550d50) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005639f7319555 in pfs_spawn_thread (arg=0x5639fa131a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fbbdffff700 (LWP 29866)): #0 0x00007fbbede5c483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fbbe794d6f8 in asio::detail::epoll_reactor::run (this=0x5639fa1b6040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fbbe7951df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5639fa1b5ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fbbe79520a2 in asio::detail::task_io_service::run (this=0x5639fa1b5ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fbbe7969afd in asio::io_service::run (this=0x5639fa1b4fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fbbe796a251 in gcomm::AsioProtonet::event_loop (this=0x5639fa1b4f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fbbe79882ed in GCommConn::run (this=0x5639fa1b56c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fbbe798b399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fbbbffff700 (LWP 29938)): #0 0x00007fbbef66e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005639f6ebc578 in LinuxAIOHandler::collect (this=0x7fbbbfffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005639f6ebcaa1 in LinuxAIOHandler::poll (this=0x7fbbbfffeb00, m1=0x7fbbbfffeba8, m2=0x7fbbbfffebb0, request=0x7fbbbfffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005639f6ebcc04 in os_aio_linux_handler (global_segment=2, m1=0x7fbbbfffeba8, m2=0x7fbbbfffebb0, request=0x7fbbbfffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005639f6ec1728 in os_aio_handler (segment=2, m1=0x7fbbbfffeba8, m2=0x7fbbbfffebb0, request=0x7fbbbfffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005639f71145ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005639f6fba9fb in io_handler_thread (arg=0x5639f8141630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fbbdd170700 (LWP 30257)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386d8f in safe_cond_timedwait (cond=0x5639f80c61a0 , mp=0x5639f80c8ba0 , abstime=0x7fbbdd16fc60, file=0x5639f74425c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005639f6753845 in inline_mysql_cond_timedwait (that=0x5639f80c61a0 , mutex=0x5639f80c8ba0 , abstime=0x7fbbdd16fc60, src_file=0x5639f7444638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005639f675861b in cache_thread (thd=0x7fbb7c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005639f67588c9 in one_thread_per_connection_end (thd=0x7fbb7c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005639f69f50f3 in do_handle_one_connection (connect=0x5639fa648890) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005639f69f4dad in handle_one_connection (arg=0x5639fa648890) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005639f7319555 in pfs_spawn_thread (arg=0x5639fa64dff0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fbbdd206700 (LWP 29974)): #0 0x00007fbbedd9540a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005639f6752f44 in my_sigwait (set=0x7fbbdd205cf0, sig=0x7fbbdd205cb4, code=0x7fbbdd205cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005639f6759584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005639f7319555 in pfs_spawn_thread (arg=0x5639fa280e30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fbba27fc700 (LWP 29963)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec925b in os_event::wait (this=0x5639fa2bda10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005639f6ec8ce1 in os_event::wait_low (this=0x5639fa2bda10, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005639f6ec9052 in os_event_wait_low (event=0x5639fa2bda10, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005639f707dc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fbba2ffd700 (LWP 29962)): #0 0x00007fbbefaa4e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005639f6ec955d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005639f706fb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fbbb4ff9700 (LWP 29959)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec925b in os_event::wait (this=0x5639fa2bd710) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005639f6ec8ce1 in os_event::wait_low (this=0x5639fa2bd710, reset_sig_count=8171) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005639f6ec9052 in os_event_wait_low (event=0x5639fa2bd710, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005639f6fb4ee5 in srv_resume_thread (slot=0x5639f7f037b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005639f6fb8ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fbbb5ffb700 (LWP 29957)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec925b in os_event::wait (this=0x5639fa2bd610) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005639f6ec8ce1 in os_event::wait_low (this=0x5639fa2bd610, reset_sig_count=2103) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005639f6ec9052 in os_event_wait_low (event=0x5639fa2bd610, reset_sig_count=2103) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005639f6fb4ee5 in srv_resume_thread (slot=0x5639f7f03748 , sig_count=2103, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005639f6fb8ef6 in srv_purge_coordinator_suspend (slot=0x5639f7f03748 , rseg_history_len=7) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005639f6fb918c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fbbb6ffd700 (LWP 29953)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec8b76 in os_event::timed_wait (this=0x5639fa45d760, abstime=0x7fbbb6ffcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005639f6ec8e8d in os_event::wait_time_low (this=0x5639fa45d760, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005639f6ec902d in os_event_wait_time_low (event=0x5639fa45d760, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005639f7017403 in ib_wqueue_timedwait (wq=0x5639fa45d630, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005639f7157f9a in fts_optimize_thread (arg=0x5639fa45d630) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fbbbdffb700 (LWP 29942)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec8b76 in os_event::timed_wait (this=0x5639fa2bd990, abstime=0x7fbbbdffaaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005639f6ec8e8d in os_event::wait_time_low (this=0x5639fa2bd990, time_in_usec=251000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005639f6ec902d in os_event_wait_time_low (event=0x5639fa2bd990, time_in_usec=251000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005639f709ee25 in pc_sleep_if_needed (next_loop_time=1549622327167, sig_count=2, cur_time=1549622326916) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x00005639f709fd5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fbbdd006700 (LWP 29936)): #0 0x00007fbbef66e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005639f6ebc578 in LinuxAIOHandler::collect (this=0x7fbbdd005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005639f6ebcaa1 in LinuxAIOHandler::poll (this=0x7fbbdd005b00, m1=0x7fbbdd005ba8, m2=0x7fbbdd005bb0, request=0x7fbbdd005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005639f6ebcc04 in os_aio_linux_handler (global_segment=0, m1=0x7fbbdd005ba8, m2=0x7fbbdd005bb0, request=0x7fbbdd005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005639f6ec1728 in os_aio_handler (segment=0, m1=0x7fbbdd005ba8, m2=0x7fbbdd005bb0, request=0x7fbbdd005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005639f71145ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005639f6fba9fb in io_handler_thread (arg=0x5639f8141620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fbbb77fe700 (LWP 29952)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec8b76 in os_event::timed_wait (this=0x5639fa4c79c0, abstime=0x7fbbb77fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005639f6ec8e8d in os_event::wait_time_low (this=0x5639fa4c79c0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005639f6ec902d in os_event_wait_time_low (event=0x5639fa4c79c0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005639f70fe24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fbbe442f700 (LWP 29886)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbbe78a0132 in fifo_lock_get (q=q@entry=0x5639fa1a3bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007fbbe78a07a1 in gu_fifo_get_head (q=0x5639fa1a3bd0, err=0x7fbbe442e06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fbbe79820cc in gcs_recv (conn=0x5639fa17d550, action=0x7fbbe442e0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fbbe79af16c in galera::GcsActionSource::process (this=0x5639fa17cf30, recv_ctx=0x7fbbe442e4d0, exit_loop=@0x7fbbe442e29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fbbe79ce033 in galera::ReplicatorSMM::async_recv (this=0x5639fa175280, recv_ctx=0x7fbbe442e4d0) at galera/src/replicator_smm.cpp:383 #6 0x00005639f743b36c in wsrep::wsrep_provider_v26::run_applier (this=0x5639fa0fc360, applier_ctx=0x7fbbe442e4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005639f6ade6d4 in wsrep_replication_process (thd=0x7fbbcc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005639f6ad031f in start_wsrep_THD (arg=0x5639fa1cdc00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fbbb7fff700 (LWP 29950)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec8b76 in os_event::timed_wait (this=0x5639fa2bd810, abstime=0x7fbbb7ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005639f6ec8e8d in os_event::wait_time_low (this=0x5639fa2bd810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005639f6ec902d in os_event_wait_time_low (event=0x5639fa2bd810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005639f6fb70ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fbbe44c5700 (LWP 29884)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386a8d in safe_cond_wait (cond=0x7fbbd4016858, mp=0x7fbbd40167a8, file=0x5639f74f0c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005639f6ade126 in inline_mysql_cond_wait (that=0x7fbbd4016858, mutex=0x7fbbd40167a8, src_file=0x5639f74f0f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005639f6ae0dbe in Wsrep_thd_queue::pop_front (this=0x7fbbd4016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005639f6adf7de in wsrep_rollback_process (rollbacker=0x7fbbd4001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005639f6ad031f in start_wsrep_THD (arg=0x5639fa1c5320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fbbbcdf4700 (LWP 29949)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec8b76 in os_event::timed_wait (this=0x5639fa495900, abstime=0x7fbbbcdf3d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005639f6ec8e8d in os_event::wait_time_low (this=0x5639fa495900, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005639f6ec902d in os_event_wait_time_low (event=0x5639fa495900, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005639f6e959e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fbbe84f6700 (LWP 29850)): #0 0x00007fbbefaa1d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f7386d8f in safe_cond_timedwait (cond=0x5639f8965640 , mp=0x5639f8965580 , abstime=0x7fbbe84f5d70, file=0x5639f780e920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005639f7387d49 in inline_mysql_cond_timedwait (that=0x5639f8965640 , mutex=0x5639f8965580 , abstime=0x7fbbe84f5d70, src_file=0x5639f780e968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005639f73888ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005639f7319555 in pfs_spawn_thread (arg=0x5639fa13e730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fbba37fe700 (LWP 29961)): #0 0x00007fbbefaa1965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005639f6ec925b in os_event::wait (this=0x5639fa2bd910) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005639f6ec8ce1 in os_event::wait_low (this=0x5639fa2bd910, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005639f6ec9052 in os_event_wait_low (event=0x5639fa2bd910, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005639f7094fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fbbbeffd700 (LWP 29940)): #0 0x00007fbbef66e644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005639f6ebc578 in LinuxAIOHandler::collect (this=0x7fbbbeffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005639f6ebcaa1 in LinuxAIOHandler::poll (this=0x7fbbbeffcb00, m1=0x7fbbbeffcba8, m2=0x7fbbbeffcbb0, request=0x7fbbbeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005639f6ebcc04 in os_aio_linux_handler (global_segment=4, m1=0x7fbbbeffcba8, m2=0x7fbbbeffcbb0, request=0x7fbbbeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005639f6ec1728 in os_aio_handler (segment=4, m1=0x7fbbbeffcba8, m2=0x7fbbbeffcbb0, request=0x7fbbbeffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005639f71145ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005639f6fba9fb in io_handler_thread (arg=0x5639f8141640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbbefa9ddd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbbede5bead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fbbefeae880 (LWP 29818)): #0 0x00007fbbefaa29d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005639f738145e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005639f6bbf5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fbbede5120d in poll () from /lib64/libc.so.6 #5 0x00005639f675fa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005639f675ed25 in mysqld_main (argc=161, argv=0x5639f9ff6d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005639f6752ec5 in main (argc=23, argv=0x7fffb1068288) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.1/data/core.29818 - found 'core.30361' (3/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.2/data/core.30361 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 30361] [New LWP 30469] [New LWP 30908] [New LWP 30471] [New LWP 30958] [New LWP 30966] [New LWP 30395] [New LWP 30968] [New LWP 30975] [New LWP 30468] [New LWP 30978] [New LWP 30973] [New LWP 30972] [New LWP 30969] [New LWP 30962] [New LWP 30960] [New LWP 30971] [New LWP 30974] [New LWP 30970] [New LWP 30967] [New LWP 30428] [New LWP 30470] [New LWP 30913] [New LWP 30959] [New LWP 30911] [New LWP 30909] [New LWP 30965] [New LWP 30961] [New LWP 30912] [New LWP 30910] [New LWP 30907] [New LWP 30431] [New LWP 30905] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fef4a7719d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fef4a7719d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055d650bd945e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055d6504175a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fef48b2020d in poll () from /lib64/libc.so.6 #5 0x000055d64ffb7a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055d64ffb6d25 in mysqld_main (argc=162, argv=0x55d653b42d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055d64ffaaec5 in main (argc=23, argv=0x7ffcdbba1148) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fef30ed4700 (LWP 30905)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650bded8f in safe_cond_timedwait (cond=0x55d6521aaea0 , mp=0x55d6521aade0 , abstime=0x7fef30ed3cb0, file=0x55d6510226f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d650a48bc1 in inline_mysql_cond_timedwait (that=0x55d6521aaea0 , mutex=0x55d6521aade0 , abstime=0x7fef30ed3cb0, src_file=0x55d651022738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d650a490e5 in my_service_thread_sleep (control=0x55d65175fb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055d650a3d57e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055d650b71555 in pfs_spawn_thread (arg=0x55d653dc66b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fef3367d700 (LWP 30431)): #0 0x00007fef48b2b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fef4261c6f8 in asio::detail::epoll_reactor::run (this=0x55d653cfe000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fef42620df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55d653cfdea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fef426210a2 in asio::detail::task_io_service::run (this=0x55d653cfdea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fef42638afd in asio::io_service::run (this=0x55d653cfcbc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fef42639251 in gcomm::AsioProtonet::event_loop (this=0x55d653cfcb20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fef426572ed in GCommConn::run (this=0x55d653cfd630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fef4265a399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fef0f5af700 (LWP 30907)): #0 0x00007fef4a33d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d650714578 in LinuxAIOHandler::collect (this=0x7fef0f5aeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d650714aa1 in LinuxAIOHandler::poll (this=0x7fef0f5aeb00, m1=0x7fef0f5aeba8, m2=0x7fef0f5aebb0, request=0x7fef0f5aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d650714c04 in os_aio_linux_handler (global_segment=0, m1=0x7fef0f5aeba8, m2=0x7fef0f5aebb0, request=0x7fef0f5aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d650719728 in os_aio_handler (segment=0, m1=0x7fef0f5aeba8, m2=0x7fef0f5aebb0, request=0x7fef0f5aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d65096c5ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d6508129fb in io_handler_thread (arg=0x55d651999620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fef0ddac700 (LWP 30910)): #0 0x00007fef4a33d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d650714578 in LinuxAIOHandler::collect (this=0x7fef0ddabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d650714aa1 in LinuxAIOHandler::poll (this=0x7fef0ddabb00, m1=0x7fef0ddabba8, m2=0x7fef0ddabbb0, request=0x7fef0ddabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d650714c04 in os_aio_linux_handler (global_segment=3, m1=0x7fef0ddabba8, m2=0x7fef0ddabbb0, request=0x7fef0ddabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d650719728 in os_aio_handler (segment=3, m1=0x7fef0ddabba8, m2=0x7fef0ddabbb0, request=0x7fef0ddabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d65096c5ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d6508129fb in io_handler_thread (arg=0x55d651999638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fef0cdaa700 (LWP 30912)): #0 0x00007fef4a33d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d650714578 in LinuxAIOHandler::collect (this=0x7fef0cda9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d650714aa1 in LinuxAIOHandler::poll (this=0x7fef0cda9b00, m1=0x7fef0cda9ba8, m2=0x7fef0cda9bb0, request=0x7fef0cda9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d650714c04 in os_aio_linux_handler (global_segment=5, m1=0x7fef0cda9ba8, m2=0x7fef0cda9bb0, request=0x7fef0cda9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d650719728 in os_aio_handler (segment=5, m1=0x7fef0cda9ba8, m2=0x7fef0cda9bb0, request=0x7fef0cda9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d65096c5ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d6508129fb in io_handler_thread (arg=0x55d651999648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fef05ffb700 (LWP 30961)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650720b76 in os_event::timed_wait (this=0x55d654018aa0, abstime=0x7fef05ffad60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d650720e8d in os_event::wait_time_low (this=0x55d654018aa0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d65072102d in os_event_wait_time_low (event=0x55d654018aa0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d65095624a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fef04ff9700 (LWP 30965)): #0 0x00007fef4a773e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055d65072155d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055d65081044d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055d65081057d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fef0e5ad700 (LWP 30909)): #0 0x00007fef4a33d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d650714578 in LinuxAIOHandler::collect (this=0x7fef0e5acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d650714aa1 in LinuxAIOHandler::poll (this=0x7fef0e5acb00, m1=0x7fef0e5acba8, m2=0x7fef0e5acbb0, request=0x7fef0e5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d650714c04 in os_aio_linux_handler (global_segment=2, m1=0x7fef0e5acba8, m2=0x7fef0e5acbb0, request=0x7fef0e5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d650719728 in os_aio_handler (segment=2, m1=0x7fef0e5acba8, m2=0x7fef0e5acbb0, request=0x7fef0e5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d65096c5ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d6508129fb in io_handler_thread (arg=0x55d651999630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fef0d5ab700 (LWP 30911)): #0 0x00007fef4a33d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d650714578 in LinuxAIOHandler::collect (this=0x7fef0d5aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d650714aa1 in LinuxAIOHandler::poll (this=0x7fef0d5aab00, m1=0x7fef0d5aaba8, m2=0x7fef0d5aabb0, request=0x7fef0d5aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d650714c04 in os_aio_linux_handler (global_segment=4, m1=0x7fef0d5aaba8, m2=0x7fef0d5aabb0, request=0x7fef0d5aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d650719728 in os_aio_handler (segment=4, m1=0x7fef0d5aaba8, m2=0x7fef0d5aabb0, request=0x7fef0d5aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d65096c5ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d6508129fb in io_handler_thread (arg=0x55d651999640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fef077fe700 (LWP 30959)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650720b76 in os_event::timed_wait (this=0x55d653e0e4b0, abstime=0x7fef077fdba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d650720e8d in os_event::wait_time_low (this=0x55d653e0e4b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d65072102d in os_event_wait_time_low (event=0x55d653e0e4b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d65080f0ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fef07fff700 (LWP 30913)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650720b76 in os_event::timed_wait (this=0x55d653e0e630, abstime=0x7fef07ffeaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d650720e8d in os_event::wait_time_low (this=0x55d653e0e630, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d65072102d in os_event_wait_time_low (event=0x55d653e0e630, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d6508f6e25 in pc_sleep_if_needed (next_loop_time=1549622334773, sig_count=2, cur_time=1549622333773) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055d6508f7d5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fef404cb700 (LWP 30470)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650bdea8d in safe_cond_wait (cond=0x7fef20015608, mp=0x7fef20015558, file=0x55d650d48c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d650336126 in inline_mysql_cond_wait (that=0x7fef20015608, mutex=0x7fef20015558, src_file=0x55d650d48f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d650338dbe in Wsrep_thd_queue::pop_front (this=0x7fef20015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055d650337b9e in wsrep_post_rollback_process (post_rollbacker=0x7fef20000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055d65032831f in start_wsrep_THD (arg=0x55d653d1e700) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fef42019700 (LWP 30428)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fef4267afd7 in gu::Lock::wait (cond=..., this=0x7fef42018bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fef4267b069 in galera::ServiceThd::thd_func (arg=0x55d653cc1908) at galera/src/galera_service_thd.cpp:30 #3 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7feef77fe700 (LWP 30967)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d65072125b in os_event::wait (this=0x55d653e0e430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d650720ce1 in os_event::wait_low (this=0x55d653e0e430, reset_sig_count=7571) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d650721052 in os_event_wait_low (event=0x55d653e0e430, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d65080cee5 in srv_resume_thread (slot=0x55d65175b7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d650810ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7feef5ffb700 (LWP 30970)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d65072125b in os_event::wait (this=0x55d653e0e5b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d650720ce1 in os_event::wait_low (this=0x55d653e0e5b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d650721052 in os_event_wait_low (event=0x55d653e0e5b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d6508ecfb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fef40071700 (LWP 30974)): #0 0x00007fef48a6440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055d64ffaaf44 in my_sigwait (set=0x7fef40070cf0, sig=0x7fef40070cb4, code=0x7fef40070cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055d64ffb1584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055d650b71555 in pfs_spawn_thread (arg=0x55d653dd1d20) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7feef57fa700 (LWP 30971)): #0 0x00007fef4a773e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055d65072155d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055d6508c7b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fef067fc700 (LWP 30960)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650720b76 in os_event::timed_wait (this=0x55d653e0e530, abstime=0x7fef067fbd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d650720e8d in os_event::wait_time_low (this=0x55d653e0e530, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d65072102d in os_event_wait_time_low (event=0x55d653e0e530, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d65080ebcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fef057fa700 (LWP 30962)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650720b76 in os_event::timed_wait (this=0x55d653fb63d0, abstime=0x7fef057f9b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d650720e8d in os_event::wait_time_low (this=0x55d653fb63d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d65072102d in os_event_wait_time_low (event=0x55d653fb63d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d65086f403 in ib_wqueue_timedwait (wq=0x55d653fb6320, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055d6509aff9a in fts_optimize_thread (arg=0x55d653fb6320) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7feef67fc700 (LWP 30969)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d65072125b in os_event::wait (this=0x55d653e0e3b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d650720ce1 in os_event::wait_low (this=0x55d653e0e3b0, reset_sig_count=8109) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d650721052 in os_event_wait_low (event=0x55d653e0e3b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d65080cee5 in srv_resume_thread (slot=0x55d65175b7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d650810ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7feef4ff9700 (LWP 30972)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d65072125b in os_event::wait (this=0x55d653e0e6b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d650720ce1 in os_event::wait_low (this=0x55d653e0e6b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d650721052 in os_event_wait_low (event=0x55d653e0e6b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d6508d5c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7feed3fff700 (LWP 30973)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650bdea8d in safe_cond_wait (cond=0x7feed3ffec70, mp=0x7feed3ffecb0, file=0x55d650eacd40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d650638525 in inline_mysql_cond_wait (that=0x7feed3ffec70, mutex=0x7feed3ffecb0, src_file=0x55d650eae6d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d65063d161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055d650b71555 in pfs_spawn_thread (arg=0x55d653fc91d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fef30087700 (LWP 30978)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fef426527f5 in gcs_replv (conn=0x55d653cc9440, act_in=0x7fef30083c50, act=act@entry=0x7fef30083c30, scheduled=) at gcs/src/gcs.cpp:1843 #2 0x00007fef426a4b7e in galera::Gcs::replv (this=this@entry=0x55d653cc18f8, actv=..., act=..., scheduled=scheduled@entry=true) at galera/src/galera_gcs.hpp:138 #3 0x00007fef4269a1f1 in galera::ReplicatorSMM::replicate (this=0x55d653cc11f0, trx=..., meta=0x7fef300842d8) at galera/src/replicator_smm.cpp:661 #4 0x00007fef426b6932 in galera_certify (gh=, conn_id=, trx_handle=, flags=, meta=0x7fef300842d8) at galera/src/wsrep_provider.cpp:535 #5 0x000055d650c936de in wsrep::wsrep_provider_v26::certify (this=0x55d653c36e50, client_id=..., ws_handle=..., flags=1, ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:737 #6 0x000055d650c89444 in wsrep::transaction::certify_fragment (this=0x7fef1c006b10, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1140 #7 0x000055d650c88d2d in wsrep::transaction::streaming_step (this=0x7fef1c006b10, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #8 0x000055d650c860e8 in wsrep::transaction::after_row (this=0x7fef1c006b10) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #9 0x000055d65042f446 in wsrep::client_state::after_row (this=0x7fef1c006aa0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #10 0x000055d6504186c3 in wsrep_after_row (thd=0x7fef1c000c40) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #11 0x000055d65042a5c8 in wsrep_after_row (thd=0x7fef1c000c40) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #12 0x000055d65042b278 in handler::ha_delete_row (this=0x7fef18044ed8, buf=0x7fef1802cdd0 "\375\367\005") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6575 #13 0x000055d6505cfa89 in TABLE::delete_row (this=0x7fef18041ad0) at /home/stepan/galera/git/10.4/server/sql/sql_delete.cc:253 #14 0x000055d6505cca8b in mysql_delete (thd=0x7fef1c000c40, table_list=0x7fef1c031440, conds=0x0, order_list=0x7fef1c0054c0, limit=2, options=0, result=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_delete.cc:751 #15 0x000055d6500e798d in mysql_execute_command (thd=0x7fef1c000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4948 #16 0x000055d6500f2935 in mysql_parse (thd=0x7fef1c000c40, rawbuf=0x7fef1c0305d8 "DELETE FROM t1 ORDER BY f1 DESC LIMIT 2", length=39, parser_state=0x7fef300860d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #17 0x000055d6500f1f68 in wsrep_mysql_parse (thd=0x7fef1c000c40, rawbuf=0x7fef1c0305d8 "DELETE FROM t1 ORDER BY f1 DESC LIMIT 2", length=39, parser_state=0x7fef300860d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #18 0x000055d6500dde25 in dispatch_command (command=COM_QUERY, thd=0x7fef1c000c40, packet=0x7fef1c0411f1 "DELETE FROM t1 ORDER BY f1 DESC LIMIT 2;", packet_length=40, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #19 0x000055d6500dc80d in do_command (thd=0x7fef1c000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #20 0x000055d65024d069 in do_handle_one_connection (connect=0x55d65410e440) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #21 0x000055d65024cdad in handle_one_connection (arg=0x55d65410e440) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #22 0x000055d650b71555 in pfs_spawn_thread (arg=0x55d65410c5b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #23 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fef32e7c700 (LWP 30468)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fef42655867 in gu::Lock::wait (cond=..., this=0x7fef32e7b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fef426583ab in RecvBuf::front (this=this@entry=0x55d653cfd818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fef42655f46 in gcomm_recv (backend=, msg=0x55d653cc9720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fef42649eec in core_msg_recv (backend=backend@entry=0x55d653cc9868, recv_msg=recv_msg@entry=0x55d653cc9720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fef4264d164 in gcs_core_recv (conn=0x55d653cc96c0, recv_act=recv_act@entry=0x7fef32e7bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fef426543d8 in gcs_recv_thread (arg=0x55d653cc9440) at gcs/src/gcs.cpp:1398 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fef300d2700 (LWP 30975)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650bdea8d in safe_cond_wait (cond=0x55d65191e1e0 , mp=0x55d651921560 , file=0x55d650ca9dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d64ffdc4ee in inline_mysql_cond_wait (that=0x55d65191e1e0 , mutex=0x55d651921560 , src_file=0x55d650caa930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055d64ffddd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055d650b71555 in pfs_spawn_thread (arg=0x55d653dd2210) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7feef6ffd700 (LWP 30968)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d65072125b in os_event::wait (this=0x55d653e0e330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d650720ce1 in os_event::wait_low (this=0x55d653e0e330, reset_sig_count=8358) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d650721052 in os_event_wait_low (event=0x55d653e0e330, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d65080cee5 in srv_resume_thread (slot=0x55d65175b780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d650810ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fef431c5700 (LWP 30395)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650bded8f in safe_cond_timedwait (cond=0x55d6521bd640 , mp=0x55d6521bd580 , abstime=0x7fef431c4d70, file=0x55d651066920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d650bdfd49 in inline_mysql_cond_timedwait (that=0x55d6521bd640 , mutex=0x55d6521bd580 , abstime=0x7fef431c4d70, src_file=0x55d651066968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d650be08ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055d650b71555 in pfs_spawn_thread (arg=0x55d653c8a7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7feef7fff700 (LWP 30966)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d65072125b in os_event::wait (this=0x55d653e0e2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d650720ce1 in os_event::wait_low (this=0x55d653e0e2b0, reset_sig_count=2345) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d650721052 in os_event_wait_low (event=0x55d653e0e2b0, reset_sig_count=2345) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d65080cee5 in srv_resume_thread (slot=0x55d65175b748 , sig_count=2345, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d650810ef6 in srv_purge_coordinator_suspend (slot=0x55d65175b748 , rseg_history_len=10) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055d65081118c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fef06ffd700 (LWP 30958)): #0 0x00007fef4a770d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650720b76 in os_event::timed_wait (this=0x55d653fe69e0, abstime=0x7fef06ffcd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d650720e8d in os_event::wait_time_low (this=0x55d653fe69e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d65072102d in os_event_wait_time_low (event=0x55d653fe69e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d6506ed9e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fef40480700 (LWP 30471)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fef4256f132 in fifo_lock_get (q=q@entry=0x55d653cefac0) at galerautils/src/gu_fifo.c:262 #2 0x00007fef4256f7a1 in gu_fifo_get_head (q=0x55d653cefac0, err=0x7fef4047f06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fef426510cc in gcs_recv (conn=0x55d653cc9440, action=0x7fef4047f0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fef4267e16c in galera::GcsActionSource::process (this=0x55d653cf5e60, recv_ctx=0x7fef4047f4d0, exit_loop=@0x7fef4047f29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fef4269d033 in galera::ReplicatorSMM::async_recv (this=0x55d653cc11f0, recv_ctx=0x7fef4047f4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055d650c9336c in wsrep::wsrep_provider_v26::run_applier (this=0x55d653c36e50, applier_ctx=0x7fef4047f4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055d6503366d4 in wsrep_replication_process (thd=0x7fef18000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055d65032831f in start_wsrep_THD (arg=0x55d653d1fb90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fef0edae700 (LWP 30908)): #0 0x00007fef4a33d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d650714578 in LinuxAIOHandler::collect (this=0x7fef0edadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d650714aa1 in LinuxAIOHandler::poll (this=0x7fef0edadb00, m1=0x7fef0edadba8, m2=0x7fef0edadbb0, request=0x7fef0edadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d650714c04 in os_aio_linux_handler (global_segment=1, m1=0x7fef0edadba8, m2=0x7fef0edadbb0, request=0x7fef0edadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d650719728 in os_aio_handler (segment=1, m1=0x7fef0edadba8, m2=0x7fef0edadbb0, request=0x7fef0edadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d65096c5ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d6508129fb in io_handler_thread (arg=0x55d651999628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fef40516700 (LWP 30469)): #0 0x00007fef4a770965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d650bdea8d in safe_cond_wait (cond=0x7fef28017b68, mp=0x7fef28017ab8, file=0x55d650d48c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d650336126 in inline_mysql_cond_wait (that=0x7fef28017b68, mutex=0x7fef28017ab8, src_file=0x55d650d48f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d650338dbe in Wsrep_thd_queue::pop_front (this=0x7fef28017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055d6503377de in wsrep_rollback_process (rollbacker=0x7fef28001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055d65032831f in start_wsrep_THD (arg=0x55d653d1f980) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fef4a76cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef48b2aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fef4ab7d880 (LWP 30361)): #0 0x00007fef4a7719d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055d650bd945e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055d6504175a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fef48b2020d in poll () from /lib64/libc.so.6 #5 0x000055d64ffb7a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055d64ffb6d25 in mysqld_main (argc=162, argv=0x55d653b42d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055d64ffaaec5 in main (argc=23, argv=0x7ffcdbba1148) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 worker[4] mysql-test-run: WARNING: Test reserved for w3 picked up by w4 worker[1] Trying to dump core for [mysqltest - pid: 36266, winpid: 36266] worker[1] Trying to dump core for [mysqld.1 - pid: 31153, winpid: 31153] worker[1] Trying to dump core for [mysqld.2 - pid: 31374, winpid: 31374] worker[2] Trying to dump core for [mysqltest - pid: 36449, winpid: 36449] worker[2] Trying to dump core for [mysqld.1 - pid: 31583, winpid: 31583] worker[2] Trying to dump core for [mysqld.2 - pid: 32034, winpid: 32034] Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/mysqld.2/data/core.30361 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1018B-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1018B-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_large_fragment 2019-02-08 12:31:12 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:31:12 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:31:43 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:31:43 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:14 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:14 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:47 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:47 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:34:18 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:34:18 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:34:49 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:34:49 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:20 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:20 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:51 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:51 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:22 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:22 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:53 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:24 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:24 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:55 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:55 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:55 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:55 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:55 0 [Warning] InnoDB: A long semaphore wait: Attempting backtrace. You can use the following information to find out 2019-02-08 12:32:21 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:32:52 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:23 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:33:54 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:34:25 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:34:56 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:27 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:58 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:58 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:58 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:35:58 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:29 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:29 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:29 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:36:29 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:00 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:00 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:31 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:37:31 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:38:02 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:38:02 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:38:33 0 [Warning] InnoDB: A long semaphore wait: 2019-02-08 12:38:33 0 [Warning] InnoDB: A long semaphore wait: Attempting backtrace. You can use the following information to find out ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1018B Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_parallel_apply 'innodb' w4 [ pass ] 9121 galera_sr.galera_sr_kill_all_nobootstrap 'innodb' w4 [ pass ] 43974 galera_sr.galera_sr_rollback 'innodb' w4 [ pass ] 8042 galera_sr.galera_sr_rollback_retry 'innodb' w4 [ pass ] 5476 galera_sr.galera_sr_rollback_savepoint 'innodb' w4 [ pass ] 8122 galera_sr.galera-features#56 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 12:42:01 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera-features#56.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); CREATE TABLE t1 (f1 INTEGER AUTO_INCREMENT PRIMARY KEY, f2 INTEGER) Engine=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; connection node_2; SET GLOBAL wsrep_slave_threads = 4; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1a; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_2; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.31154' (4/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera-features#56-innodb/mysqld.1/data/core.31154 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 31154] [New LWP 31199] [New LWP 31189] [New LWP 31201] [New LWP 31198] [New LWP 31252] [New LWP 31225] [New LWP 31226] [New LWP 31251] [New LWP 31255] [New LWP 31257] [New LWP 31264] [New LWP 31265] [New LWP 31196] [New LWP 31197] [New LWP 31200] [New LWP 31258] [New LWP 31259] [New LWP 31261] [New LWP 31262] [New LWP 33909] [New LWP 31362] [New LWP 31263] [New LWP 31254] [New LWP 31256] [New LWP 31260] [New LWP 33903] [New LWP 31253] [New LWP 31229] [New LWP 31228] [New LWP 31224] [New LWP 31223] [New LWP 31222] [New LWP 33908] [New LWP 31227] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fb4a8e0c9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fb4a8e0c9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000562ec997245e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000562ec91b05a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb4a71bb20d in poll () from /lib64/libc.so.6 #5 0x0000562ec8d50a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000562ec8d4fd25 in mysqld_main (argc=161, argv=0x562ecc9f3d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000562ec8d43ec5 in main (argc=23, argv=0x7fff6d23a0a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7fb477fff700 (LWP 31227)): #0 0x00007fb4a8e0ed1d in fsync () from /lib64/libpthread.so.0 #1 0x0000562ec94aec27 in os_file_fsync_posix (file=51) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000562ec94aef7d in os_file_flush_func (file=51) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000562ec96f74aa in pfs_os_file_flush_func (file=..., src_file=0x562ec9d71b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000562ec96fa225 in fil_flush_low (space=0x562ecd050710, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000562ec9705aff in fil_flush (space_id=7) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000562ec9705d86 in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x0000562ec968210a in buf_dblwr_update (bpage=0x7fb48244bba0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x0000562ec9689af7 in buf_flush_write_complete (bpage=0x7fb48244bba0, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x0000562ec96787b1 in buf_page_io_complete (bpage=0x7fb48244bba0, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x0000562ec9705859 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x0000562ec95ab9fb in io_handler_thread (arg=0x562eca732640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fb4742fd700 (LWP 33908)): #0 0x00007fb4a8e0ed1d in fsync () from /lib64/libpthread.so.0 #1 0x0000562ec94aec27 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000562ec94aef7d in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000562ec96f74aa in pfs_os_file_flush_func (file=..., src_file=0x562ec9d71b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000562ec96fa225 in fil_flush_low (space=0x562eccec07c0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000562ec9705aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000562ec948a321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000562ec948ad83 in log_write_up_to (lsn=5668142, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000562ec95f5e99 in trx_flush_log_if_needed_low (lsn=5668142) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000562ec95f5ee2 in trx_flush_log_if_needed (lsn=5668142, trx=0x7fb482c5f708) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000562ec95f7d0b in trx_commit_complete_for_mysql (trx=0x7fb482c5f708) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000562ec93dc5b2 in innobase_commit (hton=0x562eccc8aae0, thd=0x7fb458024ca0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x0000562ec91b60d7 in commit_one_phase_2 (thd=0x7fb458024ca0, all=true, trans=0x7fb4580281f0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000562ec91b5fd9 in ha_commit_one_phase (thd=0x7fb458024ca0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000562ec91b5355 in ha_commit_trans (thd=0x7fb458024ca0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000562ec8ffae8e in trans_commit (thd=0x7fb458024ca0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000562ec93ba22a in Wsrep_storage_service::commit (this=0x7fb458036f40, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x0000562ec9a225c9 in wsrep::transaction::certify_fragment (this=0x7fb4580107a0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x0000562ec9a21d2d in wsrep::transaction::streaming_step (this=0x7fb4580107a0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x0000562ec9a1f0e8 in wsrep::transaction::after_row (this=0x7fb4580107a0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x0000562ec91c8446 in wsrep::client_state::after_row (this=0x7fb458010730) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x0000562ec91b16c3 in wsrep_after_row (thd=0x7fb45800a8d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x0000562ec91c35c8 in wsrep_after_row (thd=0x7fb45800a8d0) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x0000562ec91c39b9 in handler::ha_write_row (this=0x7fb458041338, buf=0x7fb458013220 "\375\363\021") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x0000562ec8e39443 in write_record (thd=0x7fb45800a8d0, table=0x7fb4580125e0, info=0x7fb458020aa0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x0000562ec8e3e742 in select_insert::send_data (this=0x7fb458020a58, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x0000562ec8ef6e07 in end_send (join=0x7fb458020b08, join_tab=0x7fb4580874c8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x0000562ec9033fa7 in JOIN_CACHE::generate_full_extensions (this=0x7fb4580239d8, rec_ptr=0x7fb45810fec4 "\b\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x0000562ec9033cc2 in JOIN_CACHE::join_matching_records (this=0x7fb4580239d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x0000562ec903368e in JOIN_CACHE::join_records (this=0x7fb4580239d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x0000562ec9033828 in JOIN_CACHE::join_records (this=0x7fb4580238a8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x0000562ec9033828 in JOIN_CACHE::join_records (this=0x7fb458023778, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x0000562ec8ef3abb in sub_select_cache (join=0x7fb458020b08, join_tab=0x7fb4580869b8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x0000562ec8ef3cc8 in sub_select (join=0x7fb458020b08, join_tab=0x7fb458086608, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x0000562ec8ef3493 in do_select (join=0x7fb458020b08, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x0000562ec8ecbfb1 in JOIN::exec_inner (this=0x7fb458020b08) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x0000562ec8ecb3fa in JOIN::exec (this=0x7fb458020b08) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x0000562ec8ecc6ec in mysql_select (thd=0x7fb45800a8d0, tables=0x7fb45801e8b0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7fb458020a58, unit=0x7fb45800e790, select_lex=0x7fb45801e390) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x0000562ec8ebdfed in handle_select (thd=0x7fb45800a8d0, lex=0x7fb45800e6c8, result=0x7fb458020a58, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x0000562ec8e803fc in mysql_execute_command (thd=0x7fb45800a8d0) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #40 0x0000562ec8e8b935 in mysql_parse (thd=0x7fb45800a8d0, rawbuf=0x7fb45801db08 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fb4742fc0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #41 0x0000562ec8e8af68 in wsrep_mysql_parse (thd=0x7fb45800a8d0, rawbuf=0x7fb45801db08 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fb4742fc0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #42 0x0000562ec8e76e25 in dispatch_command (command=COM_QUERY, thd=0x7fb45800a8d0, packet=0x7fb45802e1f1 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #43 0x0000562ec8e7580d in do_command (thd=0x7fb45800a8d0) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x0000562ec8fe6069 in do_handle_one_connection (connect=0x562eccc7ab40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #45 0x0000562ec8fe5dad in handle_one_connection (arg=0x562eccc7ab40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #46 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccfe2060) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fb48385a700 (LWP 31222)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec9977d8f in safe_cond_timedwait (cond=0x562ecaf43ea0 , mp=0x562ecaf43de0 , abstime=0x7fb483859cb0, file=0x562ec9dbb6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562ec97e1bc1 in inline_mysql_cond_timedwait (that=0x562ecaf43ea0 , mutex=0x562ecaf43de0 , abstime=0x7fb483859cb0, src_file=0x562ec9dbb738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562ec97e20e5 in my_service_thread_sleep (control=0x562eca4f8b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000562ec97d657e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccc6e7b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fb482208700 (LWP 31223)): #0 0x00007fb4a89d8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ec94ad578 in LinuxAIOHandler::collect (this=0x7fb482207b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ec94adaa1 in LinuxAIOHandler::poll (this=0x7fb482207b00, m1=0x7fb482207ba8, m2=0x7fb482207bb0, request=0x7fb482207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ec94adc04 in os_aio_linux_handler (global_segment=0, m1=0x7fb482207ba8, m2=0x7fb482207bb0, request=0x7fb482207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ec94b2728 in os_aio_handler (segment=0, m1=0x7fb482207ba8, m2=0x7fb482207bb0, request=0x7fb482207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ec97055ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ec95ab9fb in io_handler_thread (arg=0x562eca732620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fb481a07700 (LWP 31224)): #0 0x00007fb4a89d8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ec94ad578 in LinuxAIOHandler::collect (this=0x7fb481a06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ec94adaa1 in LinuxAIOHandler::poll (this=0x7fb481a06b00, m1=0x7fb481a06ba8, m2=0x7fb481a06bb0, request=0x7fb481a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ec94adc04 in os_aio_linux_handler (global_segment=1, m1=0x7fb481a06ba8, m2=0x7fb481a06bb0, request=0x7fb481a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ec94b2728 in os_aio_handler (segment=1, m1=0x7fb481a06ba8, m2=0x7fb481a06bb0, request=0x7fb481a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ec97055ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ec95ab9fb in io_handler_thread (arg=0x562eca732628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fb4777fe700 (LWP 31228)): #0 0x00007fb4a89d8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ec94ad578 in LinuxAIOHandler::collect (this=0x7fb4777fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ec94adaa1 in LinuxAIOHandler::poll (this=0x7fb4777fdb00, m1=0x7fb4777fdba8, m2=0x7fb4777fdbb0, request=0x7fb4777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ec94adc04 in os_aio_linux_handler (global_segment=5, m1=0x7fb4777fdba8, m2=0x7fb4777fdbb0, request=0x7fb4777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ec94b2728 in os_aio_handler (segment=5, m1=0x7fb4777fdba8, m2=0x7fb4777fdbb0, request=0x7fb4777fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ec97055ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ec95ab9fb in io_handler_thread (arg=0x562eca732648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fb476ffd700 (LWP 31229)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94b9b76 in os_event::timed_wait (this=0x562ecccb6990, abstime=0x7fb476ffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ec94b9e8d in os_event::wait_time_low (this=0x562ecccb6990, time_in_usec=231000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ec94ba02d in os_event_wait_time_low (event=0x562ecccb6990, time_in_usec=231000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ec968fe25 in pc_sleep_if_needed (next_loop_time=1549622449956, sig_count=2, cur_time=1549622449725) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000562ec9690d5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fb4757fa700 (LWP 31253)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94b9b76 in os_event::timed_wait (this=0x562ecccb6890, abstime=0x7fb4757f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ec94b9e8d in os_event::wait_time_low (this=0x562ecccb6890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ec94ba02d in os_event_wait_time_low (event=0x562ecccb6890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ec95a7bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fb474474700 (LWP 33903)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec9977d8f in safe_cond_timedwait (cond=0x562eca6b71a0 , mp=0x562eca6b9ba0 , abstime=0x7fb474473c60, file=0x562ec9a335c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562ec8d44845 in inline_mysql_cond_timedwait (that=0x562eca6b71a0 , mutex=0x562eca6b9ba0 , abstime=0x7fb474473c60, src_file=0x562ec9a35638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562ec8d4961b in cache_thread (thd=0x7fb43800b7f0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000562ec8d498c9 in one_thread_per_connection_end (thd=0x7fb43800b7f0, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000562ec8fe60f3 in do_handle_one_connection (connect=0x562eccc7ab40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000562ec8fe5dad in handle_one_connection (arg=0x562eccc7ab40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccfcb7a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fb46d7fa700 (LWP 31260)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94ba25b in os_event::wait (this=0x562ecccb6690) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ec94b9ce1 in os_event::wait_low (this=0x562ecccb6690, reset_sig_count=11582) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ec94ba052 in os_event_wait_low (event=0x562ecccb6690, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ec95a5ee5 in srv_resume_thread (slot=0x562eca4f4780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ec95a9ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fb46f7fe700 (LWP 31256)): #0 0x00007fb4a8e0ee3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000562ec94ba55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000562ec95a944d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000562ec95a957d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fb474ff9700 (LWP 31254)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94b9b76 in os_event::timed_wait (this=0x562eccec0e00, abstime=0x7fb474ff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ec94b9e8d in os_event::wait_time_low (this=0x562eccec0e00, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ec94ba02d in os_event_wait_time_low (event=0x562eccec0e00, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ec96ef24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fb4477fe700 (LWP 31263)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94ba25b in os_event::wait (this=0x562ecccb6a10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ec94b9ce1 in os_event::wait_low (this=0x562ecccb6a10, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ec94ba052 in os_event_wait_low (event=0x562ecccb6a10, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ec966ec2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fb49812d700 (LWP 31362)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec9977a8d in safe_cond_wait (cond=0x562eca6b71e0 , mp=0x562eca6ba560 , file=0x562ec9a42dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ec8d754ee in inline_mysql_cond_wait (that=0x562eca6b71e0 , mutex=0x562eca6ba560 , src_file=0x562ec9a43930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000562ec8d76d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccc7a3d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fb4742b2700 (LWP 33909)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb4a0d2e067 in gu::Lock::wait (cond=..., this=0x7fb4742ae5f0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb4a0d3ec4c in galera::Monitor::enter (this=this@entry=0x562eccb732b0, obj=...) at galera/src/monitor.hpp:156 #3 0x00007fb4a0d33717 in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x562eccb72250, trx=...) at galera/src/replicator_smm.cpp:1282 #4 0x00007fb4a0d52063 in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:652 #5 0x0000562ec9a2c8cf in wsrep::wsrep_provider_v26::commit_order_enter (this=0x562eccaf9360, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:769 #6 0x0000562ec9a1fbd0 in wsrep::transaction::before_commit (this=0x7fb46004ded0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:440 #7 0x0000562ec90b0ac6 in wsrep::client_state::before_commit (this=0x7fb46004de60) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:391 #8 0x0000562ec93b8f8d in wsrep_before_commit (thd=0x7fb460048000, all=true) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:195 #9 0x0000562ec93ba208 in Wsrep_storage_service::commit (this=0x7fb46005fbd0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:171 #10 0x0000562ec9a225c9 in wsrep::transaction::certify_fragment (this=0x7fb460010630, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #11 0x0000562ec9a21d2d in wsrep::transaction::streaming_step (this=0x7fb460010630, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #12 0x0000562ec9a1f0e8 in wsrep::transaction::after_row (this=0x7fb460010630) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #13 0x0000562ec91c8446 in wsrep::client_state::after_row (this=0x7fb4600105c0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #14 0x0000562ec91b16c3 in wsrep_after_row (thd=0x7fb46000a760) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #15 0x0000562ec91c35c8 in wsrep_after_row (thd=0x7fb46000a760) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #16 0x0000562ec91c39b9 in handler::ha_write_row (this=0x7fb46007ff78, buf=0x7fb46007fb20 "\375\345\031") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #17 0x0000562ec8e39443 in write_record (thd=0x7fb46000a760, table=0x7fb46007eee0, info=0x7fb4600209b0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #18 0x0000562ec8e3e742 in select_insert::send_data (this=0x7fb460020968, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #19 0x0000562ec8ef6e07 in end_send (join=0x7fb460020a18, join_tab=0x7fb460076728, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #20 0x0000562ec9033fa7 in JOIN_CACHE::generate_full_extensions (this=0x7fb460076ad8, rec_ptr=0x7fb460114868 "\354") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #21 0x0000562ec9033cc2 in JOIN_CACHE::join_matching_records (this=0x7fb460076ad8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #22 0x0000562ec903368e in JOIN_CACHE::join_records (this=0x7fb460076ad8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #23 0x0000562ec9033828 in JOIN_CACHE::join_records (this=0x7fb4600237b8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #24 0x0000562ec9033828 in JOIN_CACHE::join_records (this=0x7fb460023688, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #25 0x0000562ec8ef3abb in sub_select_cache (join=0x7fb460020a18, join_tab=0x7fb460075c18, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #26 0x0000562ec8ef3cc8 in sub_select (join=0x7fb460020a18, join_tab=0x7fb460075868, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #27 0x0000562ec8ef3493 in do_select (join=0x7fb460020a18, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #28 0x0000562ec8ecbfb1 in JOIN::exec_inner (this=0x7fb460020a18) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #29 0x0000562ec8ecb3fa in JOIN::exec (this=0x7fb460020a18) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #30 0x0000562ec8ecc6ec in mysql_select (thd=0x7fb46000a760, tables=0x7fb46001e740, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7fb460020968, unit=0x7fb46000e620, select_lex=0x7fb46001e220) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #31 0x0000562ec8ebdfed in handle_select (thd=0x7fb46000a760, lex=0x7fb46000e558, result=0x7fb460020968, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #32 0x0000562ec8e803fc in mysql_execute_command (thd=0x7fb46000a760) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #33 0x0000562ec8e8b935 in mysql_parse (thd=0x7fb46000a760, rawbuf=0x7fb46001d998 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fb4742b10d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #34 0x0000562ec8e8af68 in wsrep_mysql_parse (thd=0x7fb46000a760, rawbuf=0x7fb46001d998 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fb4742b10d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #35 0x0000562ec8e76e25 in dispatch_command (command=COM_QUERY, thd=0x7fb46000a760, packet=0x7fb460013f71 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #36 0x0000562ec8e7580d in do_command (thd=0x7fb46000a760) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #37 0x0000562ec8fe6069 in do_handle_one_connection (connect=0x562eccc7ab40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #38 0x0000562ec8fe5dad in handle_one_connection (arg=0x562eccc7ab40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #39 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccfb5bc0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #40 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #41 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fb44ffff700 (LWP 31262)): #0 0x00007fb4a8e0ee3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000562ec94ba55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000562ec9660b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fb46cff9700 (LWP 31261)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94ba25b in os_event::wait (this=0x562ecccb6910) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ec94b9ce1 in os_event::wait_low (this=0x562ecccb6910, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ec94ba052 in os_event_wait_low (event=0x562ecccb6910, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ec9685fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fb46dffb700 (LWP 31259)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94ba25b in os_event::wait (this=0x562ecccb6710) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ec94b9ce1 in os_event::wait_low (this=0x562ecccb6710, reset_sig_count=11033) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ec94ba052 in os_event_wait_low (event=0x562ecccb6710, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ec95a5ee5 in srv_resume_thread (slot=0x562eca4f47b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ec95a9ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fb46e7fc700 (LWP 31258)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94ba25b in os_event::wait (this=0x562ecccb6790) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ec94b9ce1 in os_event::wait_low (this=0x562ecccb6790, reset_sig_count=10766) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ec94ba052 in os_event_wait_low (event=0x562ecccb6790, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ec95a5ee5 in srv_resume_thread (slot=0x562eca4f47f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ec95a9ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fb4a0669700 (LWP 31200)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec9977a8d in safe_cond_wait (cond=0x7fb494017b68, mp=0x7fb494017ab8, file=0x562ec9ae1c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ec90cf126 in inline_mysql_cond_wait (that=0x7fb494017b68, mutex=0x7fb494017ab8, src_file=0x562ec9ae1f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562ec90d1dbe in Wsrep_thd_queue::pop_front (this=0x7fb494017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000562ec90d0b9e in wsrep_post_rollback_process (post_rollbacker=0x7fb494001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000562ec90c131f in start_wsrep_THD (arg=0x562eccbc6bd0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fb49917a700 (LWP 31197)): #0 0x00007fb4a71c6483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb4a0cb76f8 in asio::detail::epoll_reactor::run (this=0x562eccbaf010, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fb4a0cbbdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x562eccbaeeb0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fb4a0cbc0a2 in asio::detail::task_io_service::run (this=0x562eccbaeeb0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fb4a0cd3afd in asio::io_service::run (this=0x562eccbadfb0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fb4a0cd4251 in gcomm::AsioProtonet::event_loop (this=0x562eccbadf10, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fb4a0cf22ed in GCommConn::run (this=0x562eccbae690) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fb4a0cf5399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fb49b5fe700 (LWP 31196)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb4a0d15fd7 in gu::Lock::wait (cond=..., this=0x7fb49b5fdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb4a0d16069 in galera::ServiceThd::thd_func (arg=0x562eccb72968) at galera/src/galera_service_thd.cpp:30 #3 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fb498178700 (LWP 31265)): #0 0x00007fb4a70ff40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000562ec8d43f44 in my_sigwait (set=0x7fb498177cf0, sig=0x7fb498177cb4, code=0x7fb498177cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000562ec8d4a584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccc79ee0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fb44f7fe700 (LWP 31264)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec9977a8d in safe_cond_wait (cond=0x7fb44f7fdc70, mp=0x7fb44f7fdcb0, file=0x562ec9c45d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ec93d1525 in inline_mysql_cond_wait (that=0x7fb44f7fdc70, mutex=0x7fb44f7fdcb0, src_file=0x562ec9c476d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562ec93d6161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccef7910) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fb46effd700 (LWP 31257)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94ba25b in os_event::wait (this=0x562ecccb6610) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562ec94b9ce1 in os_event::wait_low (this=0x562ecccb6610, reset_sig_count=2862) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562ec94ba052 in os_event_wait_low (event=0x562ecccb6610, reset_sig_count=2862) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562ec95a5ee5 in srv_resume_thread (slot=0x562eca4f4748 , sig_count=2862, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562ec95a9ef6 in srv_purge_coordinator_suspend (slot=0x562eca4f4748 , rseg_history_len=26) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000562ec95aa18c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fb46ffff700 (LWP 31255)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94b9b76 in os_event::timed_wait (this=0x562ecce56a90, abstime=0x7fb46fffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ec94b9e8d in os_event::wait_time_low (this=0x562ecce56a90, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ec94ba02d in os_event_wait_time_low (event=0x562ecce56a90, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ec9608403 in ib_wqueue_timedwait (wq=0x562ecce56960, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000562ec9748f9a in fts_optimize_thread (arg=0x562ecce56960) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fb475ffb700 (LWP 31251)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94b9b76 in os_event::timed_wait (this=0x562ecce8ed40, abstime=0x7fb475ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ec94b9e8d in os_event::wait_time_low (this=0x562ecce8ed40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ec94ba02d in os_event_wait_time_low (event=0x562ecce8ed40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ec94869e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fb480a05700 (LWP 31226)): #0 0x00007fb4a89d8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ec94ad578 in LinuxAIOHandler::collect (this=0x7fb480a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ec94adaa1 in LinuxAIOHandler::poll (this=0x7fb480a04b00, m1=0x7fb480a04ba8, m2=0x7fb480a04bb0, request=0x7fb480a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ec94adc04 in os_aio_linux_handler (global_segment=3, m1=0x7fb480a04ba8, m2=0x7fb480a04bb0, request=0x7fb480a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ec94b2728 in os_aio_handler (segment=3, m1=0x7fb480a04ba8, m2=0x7fb480a04bb0, request=0x7fb480a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ec97055ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ec95ab9fb in io_handler_thread (arg=0x562eca732638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb481206700 (LWP 31225)): #0 0x00007fb4a89d8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562ec94ad578 in LinuxAIOHandler::collect (this=0x7fb481205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562ec94adaa1 in LinuxAIOHandler::poll (this=0x7fb481205b00, m1=0x7fb481205ba8, m2=0x7fb481205bb0, request=0x7fb481205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562ec94adc04 in os_aio_linux_handler (global_segment=2, m1=0x7fb481205ba8, m2=0x7fb481205bb0, request=0x7fb481205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562ec94b2728 in os_aio_handler (segment=2, m1=0x7fb481205ba8, m2=0x7fb481205bb0, request=0x7fb481205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562ec97055ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562ec95ab9fb in io_handler_thread (arg=0x562eca732630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb4767fc700 (LWP 31252)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec94b9b76 in os_event::timed_wait (this=0x562ecccb6810, abstime=0x7fb4767fbba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562ec94b9e8d in os_event::wait_time_low (this=0x562ecccb6810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562ec94ba02d in os_event_wait_time_low (event=0x562ecccb6810, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562ec95a80ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb498979700 (LWP 31198)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb4a0cf0867 in gu::Lock::wait (cond=..., this=0x7fb498978850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb4a0cf33ab in RecvBuf::front (this=this@entry=0x562eccbae878, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fb4a0cf0f46 in gcomm_recv (backend=, msg=0x562eccb7a800, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fb4a0ce4eec in core_msg_recv (backend=backend@entry=0x562eccb7a948, recv_msg=recv_msg@entry=0x562eccb7a800, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fb4a0ce8164 in gcs_core_recv (conn=0x562eccb7a7a0, recv_act=recv_act@entry=0x7fb498978d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fb4a0cef3d8 in gcs_recv_thread (arg=0x562eccb7a520) at gcs/src/gcs.cpp:1398 #7 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fb4a061e700 (LWP 31201)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb4a0c0a132 in fifo_lock_get (q=q@entry=0x562eccba0ba0) at galerautils/src/gu_fifo.c:262 #2 0x00007fb4a0c0a7a1 in gu_fifo_get_head (q=0x562eccba0ba0, err=0x7fb4a061d06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fb4a0cec0cc in gcs_recv (conn=0x562eccb7a520, action=0x7fb4a061d0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fb4a0d1916c in galera::GcsActionSource::process (this=0x562eccb79f00, recv_ctx=0x7fb4a061d4d0, exit_loop=@0x7fb4a061d29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fb4a0d38033 in galera::ReplicatorSMM::async_recv (this=0x562eccb72250, recv_ctx=0x7fb4a061d4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000562ec9a2c36c in wsrep::wsrep_provider_v26::run_applier (this=0x562eccaf9360, applier_ctx=0x7fb4a061d4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000562ec90cf6d4 in wsrep_replication_process (thd=0x7fb48c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000562ec90c131f in start_wsrep_THD (arg=0x562eccbc7270) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fb4a1860700 (LWP 31189)): #0 0x00007fb4a8e0bd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec9977d8f in safe_cond_timedwait (cond=0x562ecaf56640 , mp=0x562ecaf56580 , abstime=0x7fb4a185fd70, file=0x562ec9dff920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562ec9978d49 in inline_mysql_cond_timedwait (that=0x562ecaf56640 , mutex=0x562ecaf56580 , abstime=0x7fb4a185fd70, src_file=0x562ec9dff968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562ec99798ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000562ec990a555 in pfs_spawn_thread (arg=0x562eccb3b730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb4a06b4700 (LWP 31199)): #0 0x00007fb4a8e0b965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562ec9977a8d in safe_cond_wait (cond=0x7fb47c015608, mp=0x7fb47c015558, file=0x562ec9ae1c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562ec90cf126 in inline_mysql_cond_wait (that=0x7fb47c015608, mutex=0x7fb47c015558, src_file=0x562ec9ae1f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562ec90d1dbe in Wsrep_thd_queue::pop_front (this=0x7fb47c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000562ec90d07de in wsrep_rollback_process (rollbacker=0x7fb47c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000562ec90c131f in start_wsrep_THD (arg=0x562eccbbe2f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb4a8e07dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb4a71c5ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb4a9218880 (LWP 31154)): #0 0x00007fb4a8e0c9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000562ec997245e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000562ec91b05a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb4a71bb20d in poll () from /lib64/libc.so.6 #5 0x0000562ec8d50a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000562ec8d4fd25 in mysqld_main (argc=161, argv=0x562ecc9f3d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000562ec8d43ec5 in main (argc=23, argv=0x7fff6d23a0a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera-features#56-innodb/mysqld.1/data/core.31154 - found 'core.31376' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera-features#56-innodb/mysqld.2/data/core.31376 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 31376] [New LWP 31886] [New LWP 31391] [New LWP 31888] [New LWP 31392] [New LWP 31889] [New LWP 31434] [New LWP 31935] [New LWP 31884] [New LWP 31938] [New LWP 31943] [New LWP 31887] [New LWP 31883] [New LWP 31387] [New LWP 31934] [New LWP 31936] [New LWP 31939] [New LWP 36558] [New LWP 31942] [New LWP 31946] [New LWP 36556] [New LWP 31937] [New LWP 31931] [New LWP 36557] [New LWP 31930] [New LWP 31941] [New LWP 31929] [New LWP 31940] [New LWP 31933] [New LWP 31928] [New LWP 31885] [New LWP 31436] [New LWP 31927] [New LWP 31890] [New LWP 31433] [New LWP 31435] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f944a1659d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f944a1659d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055edd9d8c45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055edd95ca5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f944851420d in poll () from /lib64/libc.so.6 #5 0x000055edd916aa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055edd9169d25 in mysqld_main (argc=162, argv=0x55eddb7aed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055edd915dec5 in main (argc=23, argv=0x7ffce9eb7cb8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 36 (Thread 0x7f94380be700 (LWP 31435)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd9d91a8d in safe_cond_wait (cond=0x7f941c017b68, mp=0x7f941c017ab8, file=0x55edd9efbc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055edd94e9126 in inline_mysql_cond_wait (that=0x7f941c017b68, mutex=0x7f941c017ab8, src_file=0x55edd9efbf80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055edd94ebdbe in Wsrep_thd_queue::pop_front (this=0x7f941c017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055edd94eab9e in wsrep_post_rollback_process (post_rollbacker=0x7f941c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055edd94db31f in start_wsrep_THD (arg=0x55eddb98a6f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f942ac7d700 (LWP 31433)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9442049867 in gu::Lock::wait (cond=..., this=0x7f942ac7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f944204c3ab in RecvBuf::front (this=this@entry=0x55eddb969808, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f9442049f46 in gcomm_recv (backend=, msg=0x55eddb935780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f944203deec in core_msg_recv (backend=backend@entry=0x55eddb9358c8, recv_msg=recv_msg@entry=0x55eddb935780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f9442041164 in gcs_core_recv (conn=0x55eddb935720, recv_act=recv_act@entry=0x7f942ac7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f94420483d8 in gcs_recv_thread (arg=0x55eddb9354a0) at gcs/src/gcs.cpp:1398 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f94237fe700 (LWP 31890)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d3b76 in os_event::timed_wait (this=0x55eddba7a5e0, abstime=0x7f94237fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055edd98d3e8d in os_event::wait_time_low (this=0x55eddba7a5e0, time_in_usec=487000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055edd98d402d in os_event_wait_time_low (event=0x55eddba7a5e0, time_in_usec=487000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055edd9aa9e25 in pc_sleep_if_needed (next_loop_time=1549622458424, sig_count=2, cur_time=1549622457937) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055edd9aaad5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f94227fc700 (LWP 31927)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d3b76 in os_event::timed_wait (this=0x55eddbc52990, abstime=0x7f94227fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055edd98d3e8d in os_event::wait_time_low (this=0x55eddbc52990, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055edd98d402d in os_event_wait_time_low (event=0x55eddbc52990, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055edd98a09e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f9438073700 (LWP 31436)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9441f63132 in fifo_lock_get (q=q@entry=0x55eddb95bb20) at galerautils/src/gu_fifo.c:262 #2 0x00007f9441f637a1 in gu_fifo_get_head (q=0x55eddb95bb20, err=0x7f943807206c) at galerautils/src/gu_fifo.c:330 #3 0x00007f94420450cc in gcs_recv (conn=0x55eddb9354a0, action=0x7f94380720b0) at gcs/src/gcs.cpp:2024 #4 0x00007f944207216c in galera::GcsActionSource::process (this=0x55eddb934d50, recv_ctx=0x7f94380724d0, exit_loop=@0x7f943807229e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f9442091033 in galera::ReplicatorSMM::async_recv (this=0x55eddb92d480, recv_ctx=0x7f94380724d0) at galera/src/replicator_smm.cpp:383 #6 0x000055edd9e4636c in wsrep::wsrep_provider_v26::run_applier (this=0x55eddb8a2e50, applier_ctx=0x7f94380724d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055edd94e96d4 in wsrep_replication_process (thd=0x7f9418000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055edd94db31f in start_wsrep_THD (arg=0x55eddb98b470) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f942a47c700 (LWP 31885)): #0 0x00007f9449d31644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055edd98c7578 in LinuxAIOHandler::collect (this=0x7f942a47bb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055edd98c7aa1 in LinuxAIOHandler::poll (this=0x7f942a47bb00, m1=0x7f942a47bba8, m2=0x7f942a47bbb0, request=0x7f942a47bbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055edd98c7c04 in os_aio_linux_handler (global_segment=1, m1=0x7f942a47bba8, m2=0x7f942a47bbb0, request=0x7f942a47bbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055edd98cc728 in os_aio_handler (segment=1, m1=0x7f942a47bba8, m2=0x7f942a47bbb0, request=0x7f942a47bbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055edd9b1f5ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055edd99c59fb in io_handler_thread (arg=0x55eddab4c628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f9421ffb700 (LWP 31928)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d3b76 in os_event::timed_wait (this=0x55eddba7a460, abstime=0x7f9421ffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055edd98d3e8d in os_event::wait_time_low (this=0x55eddba7a460, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055edd98d402d in os_event_wait_time_low (event=0x55eddba7a460, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055edd99c20ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f9422ffd700 (LWP 31933)): #0 0x00007f944a167e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055edd98d455d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055edd99c344d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055edd99c357d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f93dbfff700 (LWP 31940)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d425b in os_event::wait (this=0x55eddba7a660) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055edd98d3ce1 in os_event::wait_low (this=0x55eddba7a660, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055edd98d4052 in os_event_wait_low (event=0x55eddba7a660, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055edd9a88c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f94217fa700 (LWP 31929)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d3b76 in os_event::timed_wait (this=0x55eddba7a4e0, abstime=0x7f94217f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055edd98d3e8d in os_event::wait_time_low (this=0x55eddba7a4e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055edd98d402d in os_event_wait_time_low (event=0x55eddba7a4e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055edd99c1bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f93db7fe700 (LWP 31941)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd9d91a8d in safe_cond_wait (cond=0x7f93db7fdc70, mp=0x7f93db7fdcb0, file=0x55edda05fd40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055edd97eb525 in inline_mysql_cond_wait (that=0x7f93db7fdc70, mutex=0x7f93db7fdcb0, src_file=0x55edda0616d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055edd97f0161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055edd9d24555 in pfs_spawn_thread (arg=0x55eddbc35670) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f9420ff9700 (LWP 31930)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d3b76 in os_event::timed_wait (this=0x55eddbc84a50, abstime=0x7f9420ff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055edd98d3e8d in os_event::wait_time_low (this=0x55eddbc84a50, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055edd98d402d in os_event_wait_time_low (event=0x55eddbc84a50, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055edd9b0924a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f94301e4700 (LWP 36557)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9441f63132 in fifo_lock_get (q=q@entry=0x55eddb95bb20) at galerautils/src/gu_fifo.c:262 #2 0x00007f9441f637a1 in gu_fifo_get_head (q=0x55eddb95bb20, err=0x7f94301e306c) at galerautils/src/gu_fifo.c:330 #3 0x00007f94420450cc in gcs_recv (conn=0x55eddb9354a0, action=0x7f94301e30b0) at gcs/src/gcs.cpp:2024 #4 0x00007f944207216c in galera::GcsActionSource::process (this=0x55eddb934d50, recv_ctx=0x7f94301e34d0, exit_loop=@0x7f94301e329e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f9442091033 in galera::ReplicatorSMM::async_recv (this=0x55eddb92d480, recv_ctx=0x7f94301e34d0) at galera/src/replicator_smm.cpp:383 #6 0x000055edd9e4636c in wsrep::wsrep_provider_v26::run_applier (this=0x55eddb8a2e50, applier_ctx=0x7f94301e34d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055edd94e96d4 in wsrep_replication_process (thd=0x7f93b8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055edd94db31f in start_wsrep_THD (arg=0x7f940c1ef9e0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f93fbfff700 (LWP 31931)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d3b76 in os_event::timed_wait (this=0x55eddbc22c60, abstime=0x7f93fbffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055edd98d3e8d in os_event::wait_time_low (this=0x55eddbc22c60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055edd98d402d in os_event_wait_time_low (event=0x55eddbc22c60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055edd9a22403 in ib_wqueue_timedwait (wq=0x55eddbc87510, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055edd9b62f9a in fts_optimize_thread (arg=0x55eddbc87510) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f93f9ffb700 (LWP 31937)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d425b in os_event::wait (this=0x55eddba7a360) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055edd98d3ce1 in os_event::wait_low (this=0x55eddba7a360, reset_sig_count=11705) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055edd98d4052 in os_event_wait_low (event=0x55eddba7a360, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055edd99bfee5 in srv_resume_thread (slot=0x55edda90e7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055edd99c3ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f943022f700 (LWP 36556)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9441f63132 in fifo_lock_get (q=q@entry=0x55eddb95bb20) at galerautils/src/gu_fifo.c:262 #2 0x00007f9441f637a1 in gu_fifo_get_head (q=0x55eddb95bb20, err=0x7f943022e06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f94420450cc in gcs_recv (conn=0x55eddb9354a0, action=0x7f943022e0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f944207216c in galera::GcsActionSource::process (this=0x55eddb934d50, recv_ctx=0x7f943022e4d0, exit_loop=@0x7f943022e29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f9442091033 in galera::ReplicatorSMM::async_recv (this=0x55eddb92d480, recv_ctx=0x7f943022e4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055edd9e4636c in wsrep::wsrep_provider_v26::run_applier (this=0x55eddb8a2e50, applier_ctx=0x7f943022e4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055edd94e96d4 in wsrep_replication_process (thd=0x7f93bc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055edd94db31f in start_wsrep_THD (arg=0x7f940c167e60) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f943027a700 (LWP 31946)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f94420467f5 in gcs_replv (conn=0x55eddb9354a0, act_in=0x7f9430276d60, act=act@entry=0x7f9430276d40, scheduled=) at gcs/src/gcs.cpp:1843 #2 0x00007f9442098b7e in galera::Gcs::replv (this=this@entry=0x55eddb92db88, actv=..., act=..., scheduled=scheduled@entry=true) at galera/src/galera_gcs.hpp:138 #3 0x00007f944208e1f1 in galera::ReplicatorSMM::replicate (this=0x55eddb92d480, trx=..., meta=0x7f94302773e8) at galera/src/replicator_smm.cpp:661 #4 0x00007f94420aa932 in galera_certify (gh=, conn_id=, trx_handle=, flags=, meta=0x7f94302773e8) at galera/src/wsrep_provider.cpp:535 #5 0x000055edd9e466de in wsrep::wsrep_provider_v26::certify (this=0x55eddb8a2e50, client_id=..., ws_handle=..., flags=0, ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:737 #6 0x000055edd9e3c444 in wsrep::transaction::certify_fragment (this=0x7f940c006b10, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1140 #7 0x000055edd9e3bd2d in wsrep::transaction::streaming_step (this=0x7f940c006b10, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #8 0x000055edd9e390e8 in wsrep::transaction::after_row (this=0x7f940c006b10) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #9 0x000055edd95e2446 in wsrep::client_state::after_row (this=0x7f940c006aa0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #10 0x000055edd95cb6c3 in wsrep_after_row (thd=0x7f940c000c40) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #11 0x000055edd95dd5c8 in wsrep_after_row (thd=0x7f940c000c40) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #12 0x000055edd95dd9b9 in handler::ha_write_row (this=0x7f940c1c9f38, buf=0x7f940c214d20 "\375\360\031") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #13 0x000055edd9253443 in write_record (thd=0x7f940c000c40, table=0x7f940c215760, info=0x7f940c0335f0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #14 0x000055edd9258742 in select_insert::send_data (this=0x7f940c0335a8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #15 0x000055edd9310e07 in end_send (join=0x7f940c033658, join_tab=0x7f940c1cbf08, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #16 0x000055edd944dfa7 in JOIN_CACHE::generate_full_extensions (this=0x7f940c1cc2b8, rec_ptr=0x7f940c2b2f74 "h") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #17 0x000055edd944dcc2 in JOIN_CACHE::join_matching_records (this=0x7f940c1cc2b8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #18 0x000055edd944d68e in JOIN_CACHE::join_records (this=0x7f940c1cc2b8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #19 0x000055edd944d828 in JOIN_CACHE::join_records (this=0x7f940c0363f8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #20 0x000055edd944d828 in JOIN_CACHE::join_records (this=0x7f940c0362c8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #21 0x000055edd930dabb in sub_select_cache (join=0x7f940c033658, join_tab=0x7f940c1cb3f8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #22 0x000055edd930dcc8 in sub_select (join=0x7f940c033658, join_tab=0x7f940c1cb048, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #23 0x000055edd930d493 in do_select (join=0x7f940c033658, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #24 0x000055edd92e5fb1 in JOIN::exec_inner (this=0x7f940c033658) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #25 0x000055edd92e53fa in JOIN::exec (this=0x7f940c033658) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #26 0x000055edd92e66ec in mysql_select (thd=0x7f940c000c40, tables=0x7f940c031380, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f940c0335a8, unit=0x7f940c004b00, select_lex=0x7f940c030e60) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #27 0x000055edd92d7fed in handle_select (thd=0x7f940c000c40, lex=0x7f940c004a38, result=0x7f940c0335a8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #28 0x000055edd929a3fc in mysql_execute_command (thd=0x7f940c000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #29 0x000055edd92a5935 in mysql_parse (thd=0x7f940c000c40, rawbuf=0x7f940c0305d8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f94302790d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #30 0x000055edd92a4f68 in wsrep_mysql_parse (thd=0x7f940c000c40, rawbuf=0x7f940c0305d8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f94302790d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #31 0x000055edd9290e25 in dispatch_command (command=COM_QUERY, thd=0x7f940c000c40, packet=0x7f940c0411f1 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #32 0x000055edd928f80d in do_command (thd=0x7f940c000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #33 0x000055edd9400069 in do_handle_one_connection (connect=0x55eddbcf7110) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #34 0x000055edd93ffdad in handle_one_connection (arg=0x55eddbcf7110) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #35 0x000055edd9d24555 in pfs_spawn_thread (arg=0x55eddba40890) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #36 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #37 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f9430310700 (LWP 31942)): #0 0x00007f944845840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055edd915df44 in my_sigwait (set=0x7f943030fcf0, sig=0x7f943030fcb4, code=0x7f943030fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055edd9164584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055edd9d24555 in pfs_spawn_thread (arg=0x55eddba3cbb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f9430199700 (LWP 36558)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9441f63132 in fifo_lock_get (q=q@entry=0x55eddb95bb20) at galerautils/src/gu_fifo.c:262 #2 0x00007f9441f637a1 in gu_fifo_get_head (q=0x55eddb95bb20, err=0x7f943019806c) at galerautils/src/gu_fifo.c:330 #3 0x00007f94420450cc in gcs_recv (conn=0x55eddb9354a0, action=0x7f94301980b0) at gcs/src/gcs.cpp:2024 #4 0x00007f944207216c in galera::GcsActionSource::process (this=0x55eddb934d50, recv_ctx=0x7f94301984d0, exit_loop=@0x7f943019829e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f9442091033 in galera::ReplicatorSMM::async_recv (this=0x55eddb92d480, recv_ctx=0x7f94301984d0) at galera/src/replicator_smm.cpp:383 #6 0x000055edd9e4636c in wsrep::wsrep_provider_v26::run_applier (this=0x55eddb8a2e50, applier_ctx=0x7f94301984d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055edd94e96d4 in wsrep_replication_process (thd=0x7f93b0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055edd94db31f in start_wsrep_THD (arg=0x7f940c1f29d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f93f8ff9700 (LWP 31939)): #0 0x00007f944a167e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055edd98d455d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055edd9a7ab44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f93fa7fc700 (LWP 31936)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d425b in os_event::wait (this=0x55eddba7a3e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055edd98d3ce1 in os_event::wait_low (this=0x55eddba7a3e0, reset_sig_count=11057) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055edd98d4052 in os_event_wait_low (event=0x55eddba7a3e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055edd99bfee5 in srv_resume_thread (slot=0x55edda90e7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055edd99c3ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f93fb7fe700 (LWP 31934)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d425b in os_event::wait (this=0x55eddba7a260) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055edd98d3ce1 in os_event::wait_low (this=0x55eddba7a260, reset_sig_count=3112) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055edd98d4052 in os_event_wait_low (event=0x55eddba7a260, reset_sig_count=3112) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055edd99bfee5 in srv_resume_thread (slot=0x55edda90e748 , sig_count=3112, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055edd99c3ef6 in srv_purge_coordinator_suspend (slot=0x55edda90e748 , rseg_history_len=14) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055edd99c418c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f9442bb9700 (LWP 31387)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd9d91d8f in safe_cond_timedwait (cond=0x55eddb370640 , mp=0x55eddb370580 , abstime=0x7f9442bb8d70, file=0x55edda219920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055edd9d92d49 in inline_mysql_cond_timedwait (that=0x55eddb370640 , mutex=0x55eddb370580 , abstime=0x7f9442bb8d70, src_file=0x55edda219968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055edd9d938ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055edd9d24555 in pfs_spawn_thread (arg=0x55eddb8f67d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f9431112700 (LWP 31883)): #0 0x00007f944a164d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd9d91d8f in safe_cond_timedwait (cond=0x55eddb35dea0 , mp=0x55eddb35dde0 , abstime=0x7f9431111cb0, file=0x55edda1d56f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055edd9bfbbc1 in inline_mysql_cond_timedwait (that=0x55eddb35dea0 , mutex=0x55eddb35dde0 , abstime=0x7f9431111cb0, src_file=0x55edda1d5738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055edd9bfc0e5 in my_service_thread_sleep (control=0x55edda912b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055edd9bf057e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055edd9d24555 in pfs_spawn_thread (arg=0x55eddba32640) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f942947a700 (LWP 31887)): #0 0x00007f9449d31644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055edd98c7578 in LinuxAIOHandler::collect (this=0x7f9429479b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055edd98c7aa1 in LinuxAIOHandler::poll (this=0x7f9429479b00, m1=0x7f9429479ba8, m2=0x7f9429479bb0, request=0x7f9429479be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055edd98c7c04 in os_aio_linux_handler (global_segment=3, m1=0x7f9429479ba8, m2=0x7f9429479bb0, request=0x7f9429479be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055edd98cc728 in os_aio_handler (segment=3, m1=0x7f9429479ba8, m2=0x7f9429479bb0, request=0x7f9429479be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055edd9b1f5ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055edd99c59fb in io_handler_thread (arg=0x55eddab4c638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f94302c5700 (LWP 31943)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd9d91a8d in safe_cond_wait (cond=0x55eddaad11e0 , mp=0x55eddaad4560 , file=0x55edd9e5cdc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055edd918f4ee in inline_mysql_cond_wait (that=0x55eddaad11e0 , mutex=0x55eddaad4560 , src_file=0x55edd9e5d930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055edd9190d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055edd9d24555 in pfs_spawn_thread (arg=0x55eddbddd380) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f93f97fa700 (LWP 31938)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d425b in os_event::wait (this=0x55eddba7a560) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055edd98d3ce1 in os_event::wait_low (this=0x55eddba7a560, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055edd98d4052 in os_event_wait_low (event=0x55eddba7a560, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055edd9a9ffb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f942b7af700 (LWP 31884)): #0 0x00007f9449d31644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055edd98c7578 in LinuxAIOHandler::collect (this=0x7f942b7aeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055edd98c7aa1 in LinuxAIOHandler::poll (this=0x7f942b7aeb00, m1=0x7f942b7aeba8, m2=0x7f942b7aebb0, request=0x7f942b7aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055edd98c7c04 in os_aio_linux_handler (global_segment=0, m1=0x7f942b7aeba8, m2=0x7f942b7aebb0, request=0x7f942b7aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055edd98cc728 in os_aio_handler (segment=0, m1=0x7f942b7aeba8, m2=0x7f942b7aebb0, request=0x7f942b7aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055edd9b1f5ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055edd99c59fb in io_handler_thread (arg=0x55eddab4c620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f93faffd700 (LWP 31935)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd98d425b in os_event::wait (this=0x55eddba7a2e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055edd98d3ce1 in os_event::wait_low (this=0x55eddba7a2e0, reset_sig_count=12090) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055edd98d4052 in os_event_wait_low (event=0x55eddba7a2e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055edd99bfee5 in srv_resume_thread (slot=0x55edda90e780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055edd99c3ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f9438109700 (LWP 31434)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055edd9d91a8d in safe_cond_wait (cond=0x7f9414015608, mp=0x7f9414015558, file=0x55edd9efbc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055edd94e9126 in inline_mysql_cond_wait (that=0x7f9414015608, mutex=0x7f9414015558, src_file=0x55edd9efbf80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055edd94ebdbe in Wsrep_thd_queue::pop_front (this=0x7f9414015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055edd94ea7de in wsrep_rollback_process (rollbacker=0x7f9414000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055edd94db31f in start_wsrep_THD (arg=0x55eddb98bb80) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f9423fff700 (LWP 31889)): #0 0x00007f9449d31644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055edd98c7578 in LinuxAIOHandler::collect (this=0x7f9423ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055edd98c7aa1 in LinuxAIOHandler::poll (this=0x7f9423ffeb00, m1=0x7f9423ffeba8, m2=0x7f9423ffebb0, request=0x7f9423ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055edd98c7c04 in os_aio_linux_handler (global_segment=5, m1=0x7f9423ffeba8, m2=0x7f9423ffebb0, request=0x7f9423ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055edd98cc728 in os_aio_handler (segment=5, m1=0x7f9423ffeba8, m2=0x7f9423ffebb0, request=0x7f9423ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055edd9b1f5ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055edd99c59fb in io_handler_thread (arg=0x55eddab4c648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f943890a700 (LWP 31392)): #0 0x00007f944851f483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f94420106f8 in asio::detail::epoll_reactor::run (this=0x55eddb969ff0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f9442014df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55eddb969e90, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f94420150a2 in asio::detail::task_io_service::run (this=0x55eddb969e90, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f944202cafd in asio::io_service::run (this=0x55eddb968bb0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f944202d251 in gcomm::AsioProtonet::event_loop (this=0x55eddb968b10, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f944204b2ed in GCommConn::run (this=0x55eddb969620) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f944204e399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f9428c79700 (LWP 31888)): #0 0x00007f9449d31644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055edd98c7578 in LinuxAIOHandler::collect (this=0x7f9428c78b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055edd98c7aa1 in LinuxAIOHandler::poll (this=0x7f9428c78b00, m1=0x7f9428c78ba8, m2=0x7f9428c78bb0, request=0x7f9428c78be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055edd98c7c04 in os_aio_linux_handler (global_segment=4, m1=0x7f9428c78ba8, m2=0x7f9428c78bb0, request=0x7f9428c78be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055edd98cc728 in os_aio_handler (segment=4, m1=0x7f9428c78ba8, m2=0x7f9428c78bb0, request=0x7f9428c78be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055edd9b1f5ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055edd99c59fb in io_handler_thread (arg=0x55eddab4c640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f9439a0c700 (LWP 31391)): #0 0x00007f944a164965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f944206efd7 in gu::Lock::wait (cond=..., this=0x7f9439a0bbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f944206f069 in galera::ServiceThd::thd_func (arg=0x55eddb92db98) at galera/src/galera_service_thd.cpp:30 #3 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f9429c7b700 (LWP 31886)): #0 0x00007f9449d31644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055edd98c7578 in LinuxAIOHandler::collect (this=0x7f9429c7ab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055edd98c7aa1 in LinuxAIOHandler::poll (this=0x7f9429c7ab00, m1=0x7f9429c7aba8, m2=0x7f9429c7abb0, request=0x7f9429c7abe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055edd98c7c04 in os_aio_linux_handler (global_segment=2, m1=0x7f9429c7aba8, m2=0x7f9429c7abb0, request=0x7f9429c7abe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055edd98cc728 in os_aio_handler (segment=2, m1=0x7f9429c7aba8, m2=0x7f9429c7abb0, request=0x7f9429c7abe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055edd9b1f5ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055edd99c59fb in io_handler_thread (arg=0x55eddab4c630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f944a160dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f944851eead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f944a571880 (LWP 31376)): #0 0x00007f944a1659d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055edd9d8c45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055edd95ca5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f944851420d in poll () from /lib64/libc.so.6 #5 0x000055edd916aa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055edd9169d25 in mysqld_main (argc=162, argv=0x55eddb7aed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055edd915dec5 in main (argc=23, argv=0x7ffce9eb7cb8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera-features#56-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera-features#56-innodb/' galera_sr.galera_sr_insert_select 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 12:42:21 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr_insert_select.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.31584' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_insert_select-innodb/mysqld.1/data/core.31584 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 31584] [New LWP 31725] [New LWP 31693] [New LWP 31797] [New LWP 31730] [New LWP 31731] [New LWP 31800] [New LWP 31732] [New LWP 31807] [New LWP 31802] [New LWP 31810] [New LWP 31808] [New LWP 31811] [New LWP 31815] [New LWP 31812] [New LWP 31816] [New LWP 31821] [New LWP 31994] [New LWP 31993] [New LWP 31818] [New LWP 31814] [New LWP 31813] [New LWP 31820] [New LWP 31819] [New LWP 31817] [New LWP 31801] [New LWP 31809] [New LWP 31794] [New LWP 31799] [New LWP 31587] [New LWP 31798] [New LWP 31729] [New LWP 32682] [New LWP 31803] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fb938d359d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fb938d359d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005598c79cb45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005598c72095a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb9370e420d in poll () from /lib64/libc.so.6 #5 0x00005598c6da9a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005598c6da8d25 in mysqld_main (argc=161, argv=0x5598ca747d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005598c6d9cec5 in main (argc=23, argv=0x7fff7b3a69f8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fb906ffd700 (LWP 31803)): #0 0x00007fb938d37d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005598c7507c27 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005598c7507f7d in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005598c77504aa in pfs_os_file_flush_func (file=..., src_file=0x5598c7dcab50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005598c7753225 in fil_flush_low (space=0x5598cac143f0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x00005598c775eaff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x00005598c76dbdc5 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x00005598c76e64ad in buf_flush_end (buf_pool=0x5598caaf00e0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x00005598c76e666b in buf_flush_do_batch (buf_pool=0x5598caaf00e0, type=BUF_FLUSH_LIST, min_n=3, lsn_limit=18446744073709551615, n=0x7fb906ffcb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x00005598c76e9355 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x00005598c76ea2a3 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fb930057700 (LWP 32682)): #0 0x00007fb938d37d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005598c7507c27 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005598c7507f7d in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005598c77504aa in pfs_os_file_flush_func (file=..., src_file=0x5598c7dcab50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005598c7753225 in fil_flush_low (space=0x5598cac14790, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x00005598c775eaff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x00005598c74e3321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005598c74e3d83 in log_write_up_to (lsn=5503427, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005598c764ee99 in trx_flush_log_if_needed_low (lsn=5503427) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005598c764eee2 in trx_flush_log_if_needed (lsn=5503427, trx=0x7fb912c5c390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005598c7650d0b in trx_commit_complete_for_mysql (trx=0x7fb912c5c390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005598c74355b2 in innobase_commit (hton=0x5598ca9deb10, thd=0x7fb8b40be110, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x00005598c720f0d7 in commit_one_phase_2 (thd=0x7fb8b40be110, all=true, trans=0x7fb8b40c1660, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x00005598c720efd9 in ha_commit_one_phase (thd=0x7fb8b40be110, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x00005598c720e355 in ha_commit_trans (thd=0x7fb8b40be110, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x00005598c7053e8e in trans_commit (thd=0x7fb8b40be110) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005598c741322a in Wsrep_storage_service::commit (this=0x7fb8b4096e20, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x00005598c7a7b5c9 in wsrep::transaction::certify_fragment (this=0x7fb8b40069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x00005598c7a7ad2d in wsrep::transaction::streaming_step (this=0x7fb8b40069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x00005598c7a780e8 in wsrep::transaction::after_row (this=0x7fb8b40069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x00005598c7221446 in wsrep::client_state::after_row (this=0x7fb8b4006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x00005598c720a6c3 in wsrep_after_row (thd=0x7fb8b4000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x00005598c721c5c8 in wsrep_after_row (thd=0x7fb8b4000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x00005598c721c9b9 in handler::ha_write_row (this=0x7fb8b406a2c8, buf=0x7fb8b4067320 "\375I\022") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x00005598c6e92443 in write_record (thd=0x7fb8b4000b00, table=0x7fb8b4063d10, info=0x7fb8b4017040) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x00005598c6e97742 in select_insert::send_data (this=0x7fb8b4016ff8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x00005598c6f4fe07 in end_send (join=0x7fb8b40170a8, join_tab=0x7fb8b40b71f8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x00005598c708cfa7 in JOIN_CACHE::generate_full_extensions (this=0x7fb8b40b7808, rec_ptr=0x7fb8b418d700 "\244") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x00005598c708ccc2 in JOIN_CACHE::join_matching_records (this=0x7fb8b40b7808, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x00005598c708c68e in JOIN_CACHE::join_records (this=0x7fb8b40b7808, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x00005598c708c828 in JOIN_CACHE::join_records (this=0x7fb8b40b76d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x00005598c708c828 in JOIN_CACHE::join_records (this=0x7fb8b40b75a8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x00005598c6f4cabb in sub_select_cache (join=0x7fb8b40170a8, join_tab=0x7fb8b40b66e8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x00005598c6f4ccc8 in sub_select (join=0x7fb8b40170a8, join_tab=0x7fb8b40b6338, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x00005598c6f4c493 in do_select (join=0x7fb8b40170a8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x00005598c6f24fb1 in JOIN::exec_inner (this=0x7fb8b40170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x00005598c6f243fa in JOIN::exec (this=0x7fb8b40170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x00005598c6f256ec in mysql_select (thd=0x7fb8b4000b00, tables=0x7fb8b4014dd0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7fb8b4016ff8, unit=0x7fb8b40049c0, select_lex=0x7fb8b4014700) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x00005598c6f16fed in handle_select (thd=0x7fb8b4000b00, lex=0x7fb8b40048f8, result=0x7fb8b4016ff8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x00005598c6ed93fc in mysql_execute_command (thd=0x7fb8b4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #40 0x00005598c6ee4935 in mysql_parse (thd=0x7fb8b4000b00, rawbuf=0x7fb8b4013e58 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fb9300560d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #41 0x00005598c6ee3f68 in wsrep_mysql_parse (thd=0x7fb8b4000b00, rawbuf=0x7fb8b4013e58 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fb9300560d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #42 0x00005598c6ecfe25 in dispatch_command (command=COM_QUERY, thd=0x7fb8b4000b00, packet=0x7fb8b400a431 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #43 0x00005598c6ece80d in do_command (thd=0x7fb8b4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x00005598c703f069 in do_handle_one_connection (connect=0x5598cad5f240) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #45 0x00005598c703edad in handle_one_connection (arg=0x5598cad5f240) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #46 0x00005598c7963555 in pfs_spawn_thread (arg=0x5598ca882a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fb928979700 (LWP 31729)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb930c19867 in gu::Lock::wait (cond=..., this=0x7fb928978850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb930c1c3ab in RecvBuf::front (this=this@entry=0x5598ca9028a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fb930c19f46 in gcomm_recv (backend=, msg=0x5598ca8ce830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fb930c0deec in core_msg_recv (backend=backend@entry=0x5598ca8ce978, recv_msg=recv_msg@entry=0x5598ca8ce830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fb930c11164 in gcs_core_recv (conn=0x5598ca8ce7d0, recv_act=recv_act@entry=0x7fb928978d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fb930c183d8 in gcs_recv_thread (arg=0x5598ca8ce550) at gcs/src/gcs.cpp:1398 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fb911807700 (LWP 31798)): #0 0x00007fb938901644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005598c7506578 in LinuxAIOHandler::collect (this=0x7fb911806b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005598c7506aa1 in LinuxAIOHandler::poll (this=0x7fb911806b00, m1=0x7fb911806ba8, m2=0x7fb911806bb0, request=0x7fb911806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005598c7506c04 in os_aio_linux_handler (global_segment=1, m1=0x7fb911806ba8, m2=0x7fb911806bb0, request=0x7fb911806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005598c750b728 in os_aio_handler (segment=1, m1=0x7fb911806ba8, m2=0x7fb911806bb0, request=0x7fb911806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005598c775e5ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005598c76049fb in io_handler_thread (arg=0x5598c878b628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fb931789700 (LWP 31587)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c79d0d8f in safe_cond_timedwait (cond=0x5598c8faf640 , mp=0x5598c8faf580 , abstime=0x7fb931788d70, file=0x5598c7e58920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005598c79d1d49 in inline_mysql_cond_timedwait (that=0x5598c8faf640 , mutex=0x5598c8faf580 , abstime=0x7fb931788d70, src_file=0x5598c7e58968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005598c79d28ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005598c7963555 in pfs_spawn_thread (arg=0x5598ca88f730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fb911006700 (LWP 31799)): #0 0x00007fb938901644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005598c7506578 in LinuxAIOHandler::collect (this=0x7fb911005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005598c7506aa1 in LinuxAIOHandler::poll (this=0x7fb911005b00, m1=0x7fb911005ba8, m2=0x7fb911005bb0, request=0x7fb911005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005598c7506c04 in os_aio_linux_handler (global_segment=2, m1=0x7fb911005ba8, m2=0x7fb911005bb0, request=0x7fb911005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005598c750b728 in os_aio_handler (segment=2, m1=0x7fb911005ba8, m2=0x7fb911005bb0, request=0x7fb911005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005598c775e5ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005598c76049fb in io_handler_thread (arg=0x5598c878b630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fb91385a700 (LWP 31794)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c79d0d8f in safe_cond_timedwait (cond=0x5598c8f9cea0 , mp=0x5598c8f9cde0 , abstime=0x7fb913859cb0, file=0x5598c7e146f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005598c783abc1 in inline_mysql_cond_timedwait (that=0x5598c8f9cea0 , mutex=0x5598c8f9cde0 , abstime=0x7fb913859cb0, src_file=0x5598c7e14738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005598c783b0e5 in my_service_thread_sleep (control=0x5598c8551b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005598c782f57e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005598c7963555 in pfs_spawn_thread (arg=0x5598ca9c27e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fb904df4700 (LWP 31809)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c7512b76 in os_event::timed_wait (this=0x5598caa0a860, abstime=0x7fb904df3d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005598c7512e8d in os_event::wait_time_low (this=0x5598caa0a860, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005598c751302d in os_event_wait_time_low (event=0x5598caa0a860, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005598c7600bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fb907fff700 (LWP 31801)): #0 0x00007fb938901644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005598c7506578 in LinuxAIOHandler::collect (this=0x7fb907ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005598c7506aa1 in LinuxAIOHandler::poll (this=0x7fb907ffeb00, m1=0x7fb907ffeba8, m2=0x7fb907ffebb0, request=0x7fb907ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005598c7506c04 in os_aio_linux_handler (global_segment=4, m1=0x7fb907ffeba8, m2=0x7fb907ffebb0, request=0x7fb907ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005598c750b728 in os_aio_handler (segment=4, m1=0x7fb907ffeba8, m2=0x7fb907ffebb0, request=0x7fb907ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005598c775e5ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005598c76049fb in io_handler_thread (arg=0x5598c878b640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fb8fcff9700 (LWP 31817)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c751325b in os_event::wait (this=0x5598caa0a8e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005598c7512ce1 in os_event::wait_low (this=0x5598caa0a8e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005598c7513052 in os_event_wait_low (event=0x5598caa0a8e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005598c76defb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fb8e37fe700 (LWP 31819)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c751325b in os_event::wait (this=0x5598caa0a9e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005598c7512ce1 in os_event::wait_low (this=0x5598caa0a9e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005598c7513052 in os_event_wait_low (event=0x5598caa0a9e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005598c76c7c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fb8e2ffd700 (LWP 31820)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c79d0a8d in safe_cond_wait (cond=0x7fb8e2ffcc70, mp=0x7fb8e2ffccb0, file=0x5598c7c9ed40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005598c742a525 in inline_mysql_cond_wait (that=0x7fb8e2ffcc70, mutex=0x7fb8e2ffccb0, src_file=0x5598c7ca06d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005598c742f161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005598c7963555 in pfs_spawn_thread (arg=0x5598cac4b9e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fb8feffd700 (LWP 31813)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c751325b in os_event::wait (this=0x5598caa0a5e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005598c7512ce1 in os_event::wait_low (this=0x5598caa0a5e0, reset_sig_count=2310) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005598c7513052 in os_event_wait_low (event=0x5598caa0a5e0, reset_sig_count=2310) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005598c75feee5 in srv_resume_thread (slot=0x5598c854d748 , sig_count=2310, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005598c7602ef6 in srv_purge_coordinator_suspend (slot=0x5598c854d748 , rseg_history_len=39) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005598c760318c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fb8fe7fc700 (LWP 31814)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c751325b in os_event::wait (this=0x5598caa0a660) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005598c7512ce1 in os_event::wait_low (this=0x5598caa0a660, reset_sig_count=9564) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005598c7513052 in os_event_wait_low (event=0x5598caa0a660, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005598c75feee5 in srv_resume_thread (slot=0x5598c854d780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005598c7602ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fb8e3fff700 (LWP 31818)): #0 0x00007fb938d37e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005598c751355d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005598c76b9b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fb9300ed700 (LWP 31993)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c79d0a8d in safe_cond_wait (cond=0x5598c87101e0 , mp=0x5598c8713560 , file=0x5598c7a9bdc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005598c6dce4ee in inline_mysql_cond_wait (that=0x5598c87101e0 , mutex=0x5598c8713560 , src_file=0x5598c7a9c930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005598c6dcfd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005598c7963555 in pfs_spawn_thread (arg=0x5598ca9ce400) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fb9300a2700 (LWP 31994)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c79d0d8f in safe_cond_timedwait (cond=0x5598c87101a0 , mp=0x5598c8712ba0 , abstime=0x7fb9300a1c60, file=0x5598c7a8c5c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005598c6d9d845 in inline_mysql_cond_timedwait (that=0x5598c87101a0 , mutex=0x5598c8712ba0 , abstime=0x7fb9300a1c60, src_file=0x5598c7a8e638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005598c6da261b in cache_thread (thd=0x7fb8bc000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005598c6da28c9 in one_thread_per_connection_end (thd=0x7fb8bc000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005598c703f0f3 in do_handle_one_connection (connect=0x5598cad79870) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005598c703edad in handle_one_connection (arg=0x5598cad79870) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005598c7963555 in pfs_spawn_thread (arg=0x5598cad9b680) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fb930138700 (LWP 31821)): #0 0x00007fb93702840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005598c6d9cf44 in my_sigwait (set=0x7fb930137cf0, sig=0x7fb930137cb4, code=0x7fb930137cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005598c6da3584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005598c7963555 in pfs_spawn_thread (arg=0x5598ca9cdf10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fb8fd7fa700 (LWP 31816)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c751325b in os_event::wait (this=0x5598caa0a760) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005598c7512ce1 in os_event::wait_low (this=0x5598caa0a760, reset_sig_count=8469) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005598c7513052 in os_event_wait_low (event=0x5598caa0a760, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005598c75feee5 in srv_resume_thread (slot=0x5598c854d7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005598c7602ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fb8ff7fe700 (LWP 31812)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c751325b in os_event::wait (this=0x5598caba2d50) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005598c7512ce1 in os_event::wait_low (this=0x5598caba2d50, reset_sig_count=3307) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005598c7513052 in os_event_wait_low (event=0x5598caba2d50, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005598c74e38b3 in log_write_up_to (lsn=5503470, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x00005598c74e3eb3 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #6 0x00005598c7601522 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #7 0x00005598c7601ae5 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #8 0x00005598c7602643 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #9 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fb8fdffb700 (LWP 31815)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c751325b in os_event::wait (this=0x5598caa0a6e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005598c7512ce1 in os_event::wait_low (this=0x5598caa0a6e0, reset_sig_count=8593) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005598c7513052 in os_event_wait_low (event=0x5598caa0a6e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005598c75feee5 in srv_resume_thread (slot=0x5598c854d7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005598c7602ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fb8fffff700 (LWP 31811)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c7512b76 in os_event::timed_wait (this=0x5598cabaaa60, abstime=0x7fb8ffffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005598c7512e8d in os_event::wait_time_low (this=0x5598cabaaa60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005598c751302d in os_event_wait_time_low (event=0x5598cabaaa60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005598c7661403 in ib_wqueue_timedwait (wq=0x5598cabaa930, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005598c77a1f9a in fts_optimize_thread (arg=0x5598cabaa930) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fb9055f5700 (LWP 31808)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c7512b76 in os_event::timed_wait (this=0x5598caa0a7e0, abstime=0x7fb9055f4ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005598c7512e8d in os_event::wait_time_low (this=0x5598caa0a7e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005598c751302d in os_event_wait_time_low (event=0x5598caa0a7e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005598c76010ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fb9065f7700 (LWP 31810)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c7512b76 in os_event::timed_wait (this=0x5598cac14dd0, abstime=0x7fb9065f6d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005598c7512e8d in os_event::wait_time_low (this=0x5598cac14dd0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005598c751302d in os_event_wait_time_low (event=0x5598cac14dd0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005598c774824a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fb9077fe700 (LWP 31802)): #0 0x00007fb938901644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005598c7506578 in LinuxAIOHandler::collect (this=0x7fb9077fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005598c7506aa1 in LinuxAIOHandler::poll (this=0x7fb9077fdb00, m1=0x7fb9077fdba8, m2=0x7fb9077fdbb0, request=0x7fb9077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005598c7506c04 in os_aio_linux_handler (global_segment=5, m1=0x7fb9077fdba8, m2=0x7fb9077fdbb0, request=0x7fb9077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005598c750b728 in os_aio_handler (segment=5, m1=0x7fb9077fdba8, m2=0x7fb9077fdbb0, request=0x7fb9077fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005598c775e5ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005598c76049fb in io_handler_thread (arg=0x5598c878b648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fb905df6700 (LWP 31807)): #0 0x00007fb938d34d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c7512b76 in os_event::timed_wait (this=0x5598cabe2d10, abstime=0x7fb905df5d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005598c7512e8d in os_event::wait_time_low (this=0x5598cabe2d10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005598c751302d in os_event_wait_time_low (event=0x5598cabe2d10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005598c74df9e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fb930547700 (LWP 31732)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb930b33132 in fifo_lock_get (q=q@entry=0x5598ca8f4bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007fb930b337a1 in gu_fifo_get_head (q=0x5598ca8f4bd0, err=0x7fb93054606c) at galerautils/src/gu_fifo.c:330 #3 0x00007fb930c150cc in gcs_recv (conn=0x5598ca8ce550, action=0x7fb9305460b0) at gcs/src/gcs.cpp:2024 #4 0x00007fb930c4216c in galera::GcsActionSource::process (this=0x5598ca8cdf30, recv_ctx=0x7fb9305464d0, exit_loop=@0x7fb93054629e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fb930c61033 in galera::ReplicatorSMM::async_recv (this=0x5598ca8c6280, recv_ctx=0x7fb9305464d0) at galera/src/replicator_smm.cpp:383 #6 0x00005598c7a8536c in wsrep::wsrep_provider_v26::run_applier (this=0x5598ca84d360, applier_ctx=0x7fb9305464d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005598c71286d4 in wsrep_replication_process (thd=0x7fb918000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005598c711a31f in start_wsrep_THD (arg=0x5598ca91b2a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb910805700 (LWP 31800)): #0 0x00007fb938901644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005598c7506578 in LinuxAIOHandler::collect (this=0x7fb910804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005598c7506aa1 in LinuxAIOHandler::poll (this=0x7fb910804b00, m1=0x7fb910804ba8, m2=0x7fb910804bb0, request=0x7fb910804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005598c7506c04 in os_aio_linux_handler (global_segment=3, m1=0x7fb910804ba8, m2=0x7fb910804bb0, request=0x7fb910804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005598c750b728 in os_aio_handler (segment=3, m1=0x7fb910804ba8, m2=0x7fb910804bb0, request=0x7fb910804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005598c775e5ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005598c76049fb in io_handler_thread (arg=0x5598c878b638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb930592700 (LWP 31731)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c79d0a8d in safe_cond_wait (cond=0x7fb920015608, mp=0x7fb920015558, file=0x5598c7b3ac60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005598c7128126 in inline_mysql_cond_wait (that=0x7fb920015608, mutex=0x7fb920015558, src_file=0x5598c7b3af80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005598c712adbe in Wsrep_thd_queue::pop_front (this=0x7fb920015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005598c7129b9e in wsrep_post_rollback_process (post_rollbacker=0x7fb920000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005598c711a31f in start_wsrep_THD (arg=0x5598ca91ac00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb9305dd700 (LWP 31730)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005598c79d0a8d in safe_cond_wait (cond=0x7fb91c017b68, mp=0x7fb91c017ab8, file=0x5598c7b3ac60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005598c7128126 in inline_mysql_cond_wait (that=0x7fb91c017b68, mutex=0x7fb91c017ab8, src_file=0x5598c7b3af80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005598c712adbe in Wsrep_thd_queue::pop_front (this=0x7fb91c017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005598c71297de in wsrep_rollback_process (rollbacker=0x7fb91c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005598c711a31f in start_wsrep_THD (arg=0x5598ca912320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fb912008700 (LWP 31797)): #0 0x00007fb938901644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005598c7506578 in LinuxAIOHandler::collect (this=0x7fb912007b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005598c7506aa1 in LinuxAIOHandler::poll (this=0x7fb912007b00, m1=0x7fb912007ba8, m2=0x7fb912007bb0, request=0x7fb912007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005598c7506c04 in os_aio_linux_handler (global_segment=0, m1=0x7fb912007ba8, m2=0x7fb912007bb0, request=0x7fb912007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005598c750b728 in os_aio_handler (segment=0, m1=0x7fb912007ba8, m2=0x7fb912007bb0, request=0x7fb912007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005598c775e5ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005598c76049fb in io_handler_thread (arg=0x5598c878b620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fb92b5fe700 (LWP 31693)): #0 0x00007fb938d34965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb930c3efd7 in gu::Lock::wait (cond=..., this=0x7fb92b5fdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb930c3f069 in galera::ServiceThd::thd_func (arg=0x5598ca8c6998) at galera/src/galera_service_thd.cpp:30 #3 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb92917a700 (LWP 31725)): #0 0x00007fb9370ef483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb930be06f8 in asio::detail::epoll_reactor::run (this=0x5598ca903040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fb930be4df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5598ca902ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fb930be50a2 in asio::detail::task_io_service::run (this=0x5598ca902ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fb930bfcafd in asio::io_service::run (this=0x5598ca901fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fb930bfd251 in gcomm::AsioProtonet::event_loop (this=0x5598ca901f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fb930c1b2ed in GCommConn::run (this=0x5598ca9026c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fb930c1e399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fb938d30dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb9370eeead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb939141880 (LWP 31584)): #0 0x00007fb938d359d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005598c79cb45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005598c72095a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb9370e420d in poll () from /lib64/libc.so.6 #5 0x00005598c6da9a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005598c6da8d25 in mysqld_main (argc=161, argv=0x5598ca747d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005598c6d9cec5 in main (argc=23, argv=0x7fff7b3a69f8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.32036' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_insert_select-innodb/mysqld.2/data/core.32036 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 32036] [New LWP 32043] [New LWP 32044] [New LWP 32098] [New LWP 32479] [New LWP 32100] [New LWP 32480] [New LWP 32482] [New LWP 32522] [New LWP 32523] [New LWP 32524] [New LWP 32525] [New LWP 32532] [New LWP 32534] [New LWP 32533] [New LWP 32099] [New LWP 32541] [New LWP 36219] [New LWP 32481] [New LWP 32586] [New LWP 32057] [New LWP 32483] [New LWP 32535] [New LWP 32478] [New LWP 32582] [New LWP 32521] [New LWP 32540] [New LWP 32538] [New LWP 32539] [New LWP 32536] [New LWP 32473] [New LWP 32477] [New LWP 32101] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fb7f55ee9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fb7f55ee9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056299cfcf45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056299c80d5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb7f399d20d in poll () from /lib64/libc.so.6 #5 0x000056299c3ada8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056299c3acd25 in mysqld_main (argc=162, argv=0x56299f1a9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056299c3a0ec5 in main (argc=23, argv=0x7fff8c0df568) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fb7ec5ff700 (LWP 32101)): #0 0x00007fb7f55f0d1d in fsync () from /lib64/libpthread.so.0 #1 0x000056299cb0bc27 in os_file_fsync_posix (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056299cb0bf7d in os_file_flush_func (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056299cd544aa in pfs_os_file_flush_func (file=..., src_file=0x56299d3ceb50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056299cd57225 in fil_flush_low (space=0x56299f67d3d0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056299cd62aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000056299cae7321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000056299cae7d83 in log_write_up_to (lsn=3373012, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000056299cc52e99 in trx_flush_log_if_needed_low (lsn=3373012) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000056299cc52ee2 in trx_flush_log_if_needed (lsn=3373012, trx=0x7fb7d6bfe140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000056299cc54d0b in trx_commit_complete_for_mysql (trx=0x7fb7d6bfe140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000056299ca395b2 in innobase_commit (hton=0x56299f4477a0, thd=0x7fb7c0000b00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x000056299c8130d7 in commit_one_phase_2 (thd=0x7fb7c0000b00, all=true, trans=0x7fb7c0004050, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000056299c812fd9 in ha_commit_one_phase (thd=0x7fb7c0000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000056299c812355 in ha_commit_trans (thd=0x7fb7c0000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000056299c657e8e in trans_commit (thd=0x7fb7c0000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000056299c70b3d6 in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7fb7ec5fe4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:264 #17 0x000056299d071b47 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:67 #18 0x000056299d0725a7 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:241 #19 0x000056299d0758fd in wsrep::server_state::on_apply (this=0x56299f2f2740, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:944 #20 0x000056299d08afc7 in wsrep::high_priority_service::apply (this=0x7fb7ec5fe4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x000056299d0886ae in (anonymous namespace)::apply_cb (ctx=0x7fb7ec5fe4d0, wsh=0x7fb7ec5fd3e0, flags=0, buf=0x7fb7ec5fd420, meta=0x7fb7ec5fd6b0, exit_loop=0x7fb7ec5fd5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007fb7ed4e13b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7fb7c008d0b0, recv_ctx=recv_ctx@entry=0x7fb7ec5fe4d0, apply_cb=0x56299d08848b <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7fb7ec5fd5ef: false) at galera/src/trx_handle.cpp:418 #23 0x00007fb7ed513d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x56299f3281f0, recv_ctx=recv_ctx@entry=0x7fb7ec5fe4d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007fb7ed519a40 in galera::ReplicatorSMM::process_trx (this=0x56299f3281f0, recv_ctx=0x7fb7ec5fe4d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #25 0x00007fb7ed4fac6b in galera::GcsActionSource::process_writeset (this=this@entry=0x56299f35ae60, recv_ctx=0x7fb7ec5fe4d0, act=..., exit_loop=@0x7fb7ec5fe29e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007fb7ed4fae45 in galera::GcsActionSource::dispatch (this=this@entry=0x56299f35ae60, recv_ctx=recv_ctx@entry=0x7fb7ec5fe4d0, act=..., exit_loop=@0x7fb7ec5fe29e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007fb7ed4fb21d in galera::GcsActionSource::process (this=0x56299f35ae60, recv_ctx=0x7fb7ec5fe4d0, exit_loop=@0x7fb7ec5fe29e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007fb7ed51a033 in galera::ReplicatorSMM::async_recv (this=0x56299f3281f0, recv_ctx=0x7fb7ec5fe4d0) at galera/src/replicator_smm.cpp:383 #29 0x000056299d08936c in wsrep::wsrep_provider_v26::run_applier (this=0x56299f29de50, applier_ctx=0x7fb7ec5fe4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x000056299c72c6d4 in wsrep_replication_process (thd=0x7fb7c0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x000056299c71e31f in start_wsrep_THD (arg=0x56299f348d70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #32 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fb7d61ac700 (LWP 32477)): #0 0x00007fb7f51ba644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056299cb0a578 in LinuxAIOHandler::collect (this=0x7fb7d61abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056299cb0aaa1 in LinuxAIOHandler::poll (this=0x7fb7d61abb00, m1=0x7fb7d61abba8, m2=0x7fb7d61abbb0, request=0x7fb7d61abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056299cb0ac04 in os_aio_linux_handler (global_segment=0, m1=0x7fb7d61abba8, m2=0x7fb7d61abbb0, request=0x7fb7d61abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056299cb0f728 in os_aio_handler (segment=0, m1=0x7fb7d61abba8, m2=0x7fb7d61abbb0, request=0x7fb7d61abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056299cd625ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056299cc089fb in io_handler_thread (arg=0x56299dd8f620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fb7d77fe700 (LWP 32473)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cfd4d8f in safe_cond_timedwait (cond=0x56299e5a0ea0 , mp=0x56299e5a0de0 , abstime=0x7fb7d77fdcb0, file=0x56299d4186f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056299ce3ebc1 in inline_mysql_cond_timedwait (that=0x56299e5a0ea0 , mutex=0x56299e5a0de0 , abstime=0x7fb7d77fdcb0, src_file=0x56299d418738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056299ce3f0e5 in my_service_thread_sleep (control=0x56299db55b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056299ce3357e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056299cf67555 in pfs_spawn_thread (arg=0x56299f42b470) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fb7a17fa700 (LWP 32536)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb1725b in os_event::wait (this=0x56299f473320) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056299cb16ce1 in os_event::wait_low (this=0x56299f473320, reset_sig_count=8809) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056299cb17052 in os_event_wait_low (event=0x56299f473320, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056299cc02ee5 in srv_resume_thread (slot=0x56299db517b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056299cc06ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fb793fff700 (LWP 32539)): #0 0x00007fb7f55f0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056299cb1755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056299ccbdb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fb7a0ff9700 (LWP 32538)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb1725b in os_event::wait (this=0x56299f473520) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056299cb16ce1 in os_event::wait_low (this=0x56299f473520, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056299cb17052 in os_event_wait_low (event=0x56299f473520, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056299cce2fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fb7937fe700 (LWP 32540)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb1725b in os_event::wait (this=0x56299f473620) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056299cb16ce1 in os_event::wait_low (this=0x56299f473620, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056299cb17052 in os_event_wait_low (event=0x56299f473620, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056299cccbc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fb7b5df6700 (LWP 32521)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb16b76 in os_event::timed_wait (this=0x56299f64b950, abstime=0x7fb7b5df5d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056299cb16e8d in os_event::wait_time_low (this=0x56299f64b950, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056299cb1702d in os_event_wait_time_low (event=0x56299f64b950, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056299cae39e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fb7ec1f0700 (LWP 32582)): #0 0x00007fb7f38e140a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000056299c3a0f44 in my_sigwait (set=0x7fb7ec1efcf0, sig=0x7fb7ec1efcb4, code=0x7fb7ec1efcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000056299c3a7584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000056299cf67555 in pfs_spawn_thread (arg=0x56299f4359e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fb7d59ab700 (LWP 32478)): #0 0x00007fb7f51ba644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056299cb0a578 in LinuxAIOHandler::collect (this=0x7fb7d59aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056299cb0aaa1 in LinuxAIOHandler::poll (this=0x7fb7d59aab00, m1=0x7fb7d59aaba8, m2=0x7fb7d59aabb0, request=0x7fb7d59aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056299cb0ac04 in os_aio_linux_handler (global_segment=1, m1=0x7fb7d59aaba8, m2=0x7fb7d59aabb0, request=0x7fb7d59aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056299cb0f728 in os_aio_handler (segment=1, m1=0x7fb7d59aaba8, m2=0x7fb7d59aabb0, request=0x7fb7d59aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056299cd625ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056299cc089fb in io_handler_thread (arg=0x56299dd8f628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fb7a1ffb700 (LWP 32535)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb1725b in os_event::wait (this=0x56299f4732a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056299cb16ce1 in os_event::wait_low (this=0x56299f4732a0, reset_sig_count=9366) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056299cb17052 in os_event_wait_low (event=0x56299f4732a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056299cc02ee5 in srv_resume_thread (slot=0x56299db51780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056299cc06ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fb7b6ffd700 (LWP 32483)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb1725b in os_event::wait (this=0x56299f60b990) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056299cb16ce1 in os_event::wait_low (this=0x56299f60b990, reset_sig_count=3127) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056299cb17052 in os_event_wait_low (event=0x56299f60b990, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056299cae78b3 in log_write_up_to (lsn=3372164, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000056299cce76fa in buf_flush_write_block_low (bpage=0x7fb7d63b22e0, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:962 #6 0x000056299cce80be in buf_flush_page (buf_pool=0x56299f558d20, bpage=0x7fb7d63b22e0, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1165 #7 0x000056299cce8a06 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=0, n_to_flush=5) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1389 #8 0x000056299cce8d89 in buf_flush_page_and_try_neighbors (bpage=0x7fb7d63b22e0, flush_type=BUF_FLUSH_LIST, n_to_flush=5, count=0x7fb7b6ffc8c8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1461 #9 0x000056299cce99a2 in buf_do_flush_list_batch (buf_pool=0x56299f558d20, min_n=5, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1719 #10 0x000056299cce9f9a in buf_flush_batch (buf_pool=0x56299f558d20, flush_type=BUF_FLUSH_LIST, min_n=5, lsn_limit=18446744073709551615, n=0x7fb7b6ffcb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1789 #11 0x000056299ccea65a in buf_flush_do_batch (buf_pool=0x56299f558d20, type=BUF_FLUSH_LIST, min_n=5, lsn_limit=18446744073709551615, n=0x7fb7b6ffcb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1947 #12 0x000056299cced355 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #13 0x000056299ccee2a3 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #14 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #15 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fb7de37b700 (LWP 32057)): #0 0x00007fb7f39a8483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb7ed4996f8 in asio::detail::epoll_reactor::run (this=0x56299f363000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fb7ed49ddf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x56299f362ea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fb7ed49e0a2 in asio::detail::task_io_service::run (this=0x56299f362ea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fb7ed4b5afd in asio::io_service::run (this=0x56299f361bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fb7ed4b6251 in gcomm::AsioProtonet::event_loop (this=0x56299f361b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fb7ed4d42ed in GCommConn::run (this=0x56299f362630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fb7ed4d7399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fb7ec1a5700 (LWP 32586)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cfd4a8d in safe_cond_wait (cond=0x56299dd141e0 , mp=0x56299dd17560 , file=0x56299d09fdc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056299c3d24ee in inline_mysql_cond_wait (that=0x56299dd141e0 , mutex=0x56299dd17560 , src_file=0x56299d0a0930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000056299c3d3d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056299cf67555 in pfs_spawn_thread (arg=0x56299f7e7a10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fb7b7fff700 (LWP 32481)): #0 0x00007fb7f51ba644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056299cb0a578 in LinuxAIOHandler::collect (this=0x7fb7b7ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056299cb0aaa1 in LinuxAIOHandler::poll (this=0x7fb7b7ffeb00, m1=0x7fb7b7ffeba8, m2=0x7fb7b7ffebb0, request=0x7fb7b7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056299cb0ac04 in os_aio_linux_handler (global_segment=4, m1=0x7fb7b7ffeba8, m2=0x7fb7b7ffebb0, request=0x7fb7b7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056299cb0f728 in os_aio_handler (segment=4, m1=0x7fb7b7ffeba8, m2=0x7fb7b7ffebb0, request=0x7fb7b7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056299cd625ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056299cc089fb in io_handler_thread (arg=0x56299dd8f640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fb7ec10f700 (LWP 36219)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cfd4d8f in safe_cond_timedwait (cond=0x56299dd141a0 , mp=0x56299dd16ba0 , abstime=0x7fb7ec10ec60, file=0x56299d0905c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056299c3a1845 in inline_mysql_cond_timedwait (that=0x56299dd141a0 , mutex=0x56299dd16ba0 , abstime=0x7fb7ec10ec60, src_file=0x56299d092638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056299c3a661b in cache_thread (thd=0x7fb764000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000056299c3a68c9 in one_thread_per_connection_end (thd=0x7fb764000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000056299c6430f3 in do_handle_one_connection (connect=0x56299f439de0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000056299c642dad in handle_one_connection (arg=0x56299f439de0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000056299cf67555 in pfs_spawn_thread (arg=0x56299f7fbdf0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fb792ffd700 (LWP 32541)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cfd4a8d in safe_cond_wait (cond=0x7fb792ffcc70, mp=0x7fb792ffccb0, file=0x56299d2a2d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056299ca2e525 in inline_mysql_cond_wait (that=0x7fb792ffcc70, mutex=0x7fb792ffccb0, src_file=0x56299d2a46d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056299ca33161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000056299cf67555 in pfs_spawn_thread (arg=0x56299f62e1c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fb7ec695700 (LWP 32099)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cfd4a8d in safe_cond_wait (cond=0x7fb7c8017b68, mp=0x7fb7c8017ab8, file=0x56299d13ec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056299c72c126 in inline_mysql_cond_wait (that=0x7fb7c8017b68, mutex=0x7fb7c8017ab8, src_file=0x56299d13ef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056299c72edbe in Wsrep_thd_queue::pop_front (this=0x7fb7c8017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056299c72d7de in wsrep_rollback_process (rollbacker=0x7fb7c8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000056299c71e31f in start_wsrep_THD (arg=0x56299f384b40) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fb7a2ffd700 (LWP 32533)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb1725b in os_event::wait (this=0x56299f473220) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056299cb16ce1 in os_event::wait_low (this=0x56299f473220, reset_sig_count=2435) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056299cb17052 in os_event_wait_low (event=0x56299f473220, reset_sig_count=2435) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056299cc02ee5 in srv_resume_thread (slot=0x56299db51748 , sig_count=2435, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056299cc06ef6 in srv_purge_coordinator_suspend (slot=0x56299db51748 , rseg_history_len=14) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000056299cc0718c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fb7a27fc700 (LWP 32534)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb1725b in os_event::wait (this=0x56299f4733a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056299cb16ce1 in os_event::wait_low (this=0x56299f4733a0, reset_sig_count=8334) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056299cb17052 in os_event_wait_low (event=0x56299f4733a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056299cc02ee5 in srv_resume_thread (slot=0x56299db517f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056299cc06ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fb7a37fe700 (LWP 32532)): #0 0x00007fb7f55f0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056299cb1755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056299cc0644d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000056299cc0657d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fb7a3fff700 (LWP 32525)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb16b76 in os_event::timed_wait (this=0x56299f61b710, abstime=0x7fb7a3ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056299cb16e8d in os_event::wait_time_low (this=0x56299f61b710, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056299cb1702d in os_event_wait_time_low (event=0x56299f61b710, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056299cc65403 in ib_wqueue_timedwait (wq=0x56299f61b5e0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056299cda5f9a in fts_optimize_thread (arg=0x56299f61b5e0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fb7b4df4700 (LWP 32524)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb16b76 in os_event::timed_wait (this=0x56299f67da10, abstime=0x7fb7b4df3d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056299cb16e8d in os_event::wait_time_low (this=0x56299f67da10, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056299cb1702d in os_event_wait_time_low (event=0x56299f67da10, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056299cd4c24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fb7b55f5700 (LWP 32523)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb16b76 in os_event::timed_wait (this=0x56299f4734a0, abstime=0x7fb7b55f4d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056299cb16e8d in os_event::wait_time_low (this=0x56299f4734a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056299cb1702d in os_event_wait_time_low (event=0x56299f4734a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056299cc04bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fb7b65f7700 (LWP 32522)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cb16b76 in os_event::timed_wait (this=0x56299f473420, abstime=0x7fb7b65f6ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056299cb16e8d in os_event::wait_time_low (this=0x56299f473420, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056299cb1702d in os_event_wait_time_low (event=0x56299f473420, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056299cc050ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fb7b77fe700 (LWP 32482)): #0 0x00007fb7f51ba644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056299cb0a578 in LinuxAIOHandler::collect (this=0x7fb7b77fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056299cb0aaa1 in LinuxAIOHandler::poll (this=0x7fb7b77fdb00, m1=0x7fb7b77fdba8, m2=0x7fb7b77fdbb0, request=0x7fb7b77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056299cb0ac04 in os_aio_linux_handler (global_segment=5, m1=0x7fb7b77fdba8, m2=0x7fb7b77fdbb0, request=0x7fb7b77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056299cb0f728 in os_aio_handler (segment=5, m1=0x7fb7b77fdba8, m2=0x7fb7b77fdbb0, request=0x7fb7b77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056299cd625ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056299cc089fb in io_handler_thread (arg=0x56299dd8f648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb7d49a9700 (LWP 32480)): #0 0x00007fb7f51ba644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056299cb0a578 in LinuxAIOHandler::collect (this=0x7fb7d49a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056299cb0aaa1 in LinuxAIOHandler::poll (this=0x7fb7d49a8b00, m1=0x7fb7d49a8ba8, m2=0x7fb7d49a8bb0, request=0x7fb7d49a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056299cb0ac04 in os_aio_linux_handler (global_segment=3, m1=0x7fb7d49a8ba8, m2=0x7fb7d49a8bb0, request=0x7fb7d49a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056299cb0f728 in os_aio_handler (segment=3, m1=0x7fb7d49a8ba8, m2=0x7fb7d49a8bb0, request=0x7fb7d49a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056299cd625ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056299cc089fb in io_handler_thread (arg=0x56299dd8f638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb7ec64a700 (LWP 32100)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cfd4a8d in safe_cond_wait (cond=0x7fb7cc015608, mp=0x7fb7cc015558, file=0x56299d13ec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056299c72c126 in inline_mysql_cond_wait (that=0x7fb7cc015608, mutex=0x7fb7cc015558, src_file=0x56299d13ef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056299c72edbe in Wsrep_thd_queue::pop_front (this=0x7fb7cc015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056299c72db9e in wsrep_post_rollback_process (post_rollbacker=0x7fb7cc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000056299c71e31f in start_wsrep_THD (arg=0x56299f37abf0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb7d51aa700 (LWP 32479)): #0 0x00007fb7f51ba644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056299cb0a578 in LinuxAIOHandler::collect (this=0x7fb7d51a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056299cb0aaa1 in LinuxAIOHandler::poll (this=0x7fb7d51a9b00, m1=0x7fb7d51a9ba8, m2=0x7fb7d51a9bb0, request=0x7fb7d51a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056299cb0ac04 in os_aio_linux_handler (global_segment=2, m1=0x7fb7d51a9ba8, m2=0x7fb7d51a9bb0, request=0x7fb7d51a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056299cb0f728 in os_aio_handler (segment=2, m1=0x7fb7d51a9ba8, m2=0x7fb7d51a9bb0, request=0x7fb7d51a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056299cd625ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056299cc089fb in io_handler_thread (arg=0x56299dd8f630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fb7d7fff700 (LWP 32098)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb7ed4d2867 in gu::Lock::wait (cond=..., this=0x7fb7d7ffe850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb7ed4d53ab in RecvBuf::front (this=this@entry=0x56299f362818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fb7ed4d2f46 in gcomm_recv (backend=, msg=0x56299f330720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fb7ed4c6eec in core_msg_recv (backend=backend@entry=0x56299f330868, recv_msg=recv_msg@entry=0x56299f330720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fb7ed4ca164 in gcs_core_recv (conn=0x56299f3306c0, recv_act=recv_act@entry=0x7fb7d7ffed70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fb7ed4d13d8 in gcs_recv_thread (arg=0x56299f330440) at gcs/src/gcs.cpp:1398 #7 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fb7ece96700 (LWP 32044)): #0 0x00007fb7f55ed965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb7ed4f7fd7 in gu::Lock::wait (cond=..., this=0x7fb7ece95bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb7ed4f8069 in galera::ServiceThd::thd_func (arg=0x56299f328908) at galera/src/galera_service_thd.cpp:30 #3 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb7ee042700 (LWP 32043)): #0 0x00007fb7f55edd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056299cfd4d8f in safe_cond_timedwait (cond=0x56299e5b3worker[1] mysql-test-run: WARNING: Test reserved for w2 picked up by w1 640 , mp=0x56299e5b3580 , abstime=0x7fb7ee041d70, file=0x56299d45c920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056299cfd5d49 in inline_mysql_cond_timedwait (that=0x56299e5b3640 , mutex=0x56299e5b3580 , abstime=0x7fb7ee041d70, src_file=0x56299d45c968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056299cfd68ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056299cf67555 in pfs_spawn_thread (arg=0x56299f2f17d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb7f55e9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb7f39a7ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb7f59fa880 (LWP 32036)): #0 0x00007fb7f55ee9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056299cfcf45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056299c80d5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb7f399d20d in poll () from /lib64/libc.so.6 #5 0x000056299c3ada8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056299c3acd25 in mysqld_main (argc=162, argv=0x56299f1a9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056299c3a0ec5 in main (argc=23, argv=0x7fff8c0df568) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_insert_select-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_insert_select-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.GCF-623 galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera-features#56 galera_sr.GCF-845 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_conflict_with_rollback_master galera_sr.galera_sr_ddl_master galera_sr.galera_sr_insert_select Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_kill_connection 'innodb' w2 [ pass ] 9930 galera_sr.galera_sr_bf_abort 'innodb' w1 [ pass ] 66636 galera_sr.galera_sr_kill_query 'innodb' w1 [ pass ] 19375 galera_sr.galera_sr_blob 'innodb' w1 [ pass ] 10437 galera_sr.GCF-437 'innodb' w3 [ fail ] Test ended at 2019-02-08 12:44:24 CURRENT_TEST: galera_sr.GCF-1018B mysqltest got signal 6 read_command_buf (0x55dbfecc7c60): } conn->name (0x55dbfed3fa40): node_2 Attempting backtrace... stack_bottom = 0x0 thread_stack 0x3c000 mysys/stacktrace.c:269(my_print_stacktrace)[0x55dbfe3aa1e0] client/mysqltest.cc:9058(dump_backtrace())[0x55dbfe35dae2] client/mysqltest.cc:9074(signal_handler)[0x55dbfe35db15] sigaction.c:0(__restore_rt)[0x7f4aa1e305d0] :0(__poll_nocancel)[0x7f4aa083b1f0] pvio/pvio_socket.c:526(pvio_socket_wait_io_or_timeout)[0x55dbfe38c32b] pvio/pvio_socket.c:312(pvio_socket_read)[0x55dbfe38bfa8] libmariadb/ma_pvio.c:255(ma_pvio_read)[0x55dbfe36fbf2] libmariadb/ma_pvio.c:297(ma_pvio_cache_read)[0x55dbfe36fda4] libmariadb/ma_net.c:373(ma_real_read)[0x55dbfe390bb2] libmariadb/ma_net.c:427(ma_net_read)[0x55dbfe390de2] libmariadb/mariadb_lib.c:191(ma_net_safe_read)[0x55dbfe3643aa] libmariadb/mariadb_lib.c:2115(mthd_my_read_query_result)[0x55dbfe369888] libmariadb/mariadb_lib.c:2173(mysql_read_query_result)[0x55dbfe369b98] tests/nonblock-wrappers.h:324(wrap_mysql_read_query_result(st_mysql*))[0x55dbfe34acea] client/mysqltest.cc:8006(run_query_normal(st_connection*, st_command*, int, char*, unsigned long, st_dynamic_string*, st_dynamic_string*))[0x55dbfe35b78d] client/mysqltest.cc:8786(run_query(st_connection*, st_command*, int))[0x55dbfe35d13d] client/mysqltest.cc:9569(main)[0x55dbfe35ef61] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4aa076a3d5] /home/stepan/galera/git/10.4/server/client/mysqltest(+0x59169)[0x55dbfe34a169] Writing a core file... CURRENT_TEST: galera_sr.GCF-437 mysqltest: At line 13: query 'INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5' succeeded - should have failed with errno 1534... The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=MyISAM; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 2 * 1024 * 1024; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-437-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-437-innodb/' galera_sr.galera_sr_cc_master 'innodb' w1 [ fail ] Test ended at 2019-02-08 12:44:57 CURRENT_TEST: galera_sr.galera_sr_cc_master Server [mysqld.2 - pid: 72020, winpid: 72020, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2019-02-08 12:44:22 50 [Note] WSREP: Stop replication 2019-02-08 12:44:22 50 [Note] WSREP: Server status change synced -> disconnecting 2019-02-08 12:44:22 50 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-08 12:44:22 50 [Note] WSREP: Closing send monitor... 2019-02-08 12:44:22 50 [Note] WSREP: Closed send monitor. 2019-02-08 12:44:22 50 [Note] WSREP: gcomm: terminating thread 2019-02-08 12:44:22 50 [Note] WSREP: gcomm: joining thread 2019-02-08 12:44:22 50 [Note] WSREP: gcomm: closing backend 2019-02-08 12:44:25 50 [Note] WSREP: (3b0d69a0, 'tcp://0.0.0.0:16045') connection to peer 374d145d with addr tcp://127.0.0.1:16042 timed out, no messages seen in PT3S 2019-02-08 12:44:25 50 [Note] WSREP: (3b0d69a0, 'tcp://0.0.0.0:16045') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16042 2019-02-08 12:44:27 50 [Note] WSREP: (3b0d69a0, 'tcp://0.0.0.0:16045') reconnecting to 374d145d (tcp://127.0.0.1:16042), attempt 0 2019-02-08 12:44:30 50 [Note] WSREP: (3b0d69a0, 'tcp://0.0.0.0:16045') connection to peer 374d145d with addr tcp://127.0.0.1:16042 timed out, no messages seen in PT3S 2019-02-08 12:44:32 50 [Note] WSREP: (3b0d69a0, 'tcp://0.0.0.0:16045') connection established to 374d145d tcp://127.0.0.1:16042 2019-02-08 12:44:32 50 [Note] WSREP: evs::proto(3b0d69a0, LEAVING, view_id(REG,374d145d,2)) suspecting node: 374d145d 2019-02-08 12:44:32 50 [Note] WSREP: evs::proto(3b0d69a0, LEAVING, view_id(REG,374d145d,2)) suspected node without join message, declaring inactive 2019-02-08 12:44:32 50 [Note] WSREP: view(view_id(NON_PRIM,374d145d,2) memb { 3b0d69a0,0 } joined { } left { } partitioned { 374d145d,0 }) 2019-02-08 12:44:32 50 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-08 12:44:32 50 [Note] WSREP: view((empty)) 2019-02-08 12:44:32 50 [Note] WSREP: gcomm: closed 2019-02-08 12:44:32 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-08 12:44:32 0 [Note] WSREP: Writing down CC checksum: 0f87aaf7 62d56430 6c1ca95c 3177561d at offset 144 2019-02-08 12:44:32 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-08 12:44:32 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-08 12:44:32 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-08 12:44:32 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 82) 2019-02-08 12:44:32 0 [Note] WSREP: Received self-leave message. 2019-02-08 12:44:32 0 [Note] WSREP: Writing down CC checksum: 4c5c6da9 8947ed8d 248b92e6 1503f905 at offset 64 2019-02-08 12:44:32 1 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-08 12:44:32 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-08 12:44:32 1 [Note] WSREP: ####### drain monitors upto 82 2019-02-08 12:44:32 1 [Note] WSREP: ####### My UUID: 3b0d69a0-2b8e-11e9-9b47-fbb5262d5f86 2019-02-08 12:44:32 1 [Note] WSREP: ####### ST not required 2019-02-08 12:44:32 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-08 12:44:32 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-08 12:44:32 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-08 12:44:32 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-08 12:44:32 50 [Note] WSREP: recv_thread() joined. 2019-02-08 12:44:32 50 [Note] WSREP: Closing replication queue. 2019-02-08 12:44:32 50 [Note] WSREP: Closing slave action queue. 2019-02-08 12:44:32 1 [Note] WSREP: ================================================ View: id: 377097cd-2b8e-11e9-9974-2773cf2bd981:-1 seqno: -1 status: 1 prococol_version: 4 own_index: 0 final: 0 members 2019-02-08 12:44:32 1 [Note] WSREP: id: 3b0d69a0-2b8e-11e9-9b47-fbb5262d5f86 name: cnts7gl1.localdomain 2019-02-08 12:44:32 1 [Note] WSREP: ================================================= 2019-02-08 12:44:32 1 [Note] WSREP: Non-primary view 2019-02-08 12:44:32 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-08 12:44:32 1 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-08 12:44:32 1 [Note] WSREP: ####### drain monitors upto 82 2019-02-08 12:44:32 1 [Note] WSREP: ####### My UUID: 3b0d69a0-2b8e-11e9-9b47-fbb5262d5f86 2019-02-08 12:44:32 1 [Note] WSREP: ####### ST not required 2019-02-08 12:44:32 1 [Note] WSREP: ================================================ View: id: 377097cd-2b8e-11e9-9974-2773cf2bd981:-1 seqno: -1 status: 1 prococol_version: 4 own_index: -1 final: 1 members 2019-02-08 12:44:32 1 [Note] WSREP: ================================================= 2019-02-08 12:44:32 1 [Note] WSREP: Non-primary view 2019-02-08 12:44:32 1 [Note] WSREP: Server status change disconnecting -> disconnected 2019-02-08 12:44:32 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-08 12:44:32 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-08 12:44:33 1 [Note] WSREP: Applier thread exiting 0 2019-02-08 12:44:33 44 [Warning] WSREP: Failed to replicate rollback fragment for 361: 3 2019-02-08 12:44:35 2 [Note] WSREP: rollbacker thread exiting 2019-02-08 12:44:35 50 [Note] WSREP: Stop replication 2019-02-08 12:44:37 52 [Note] WSREP: rollbacker thread exiting 2019-02-08 12:44:37 50 [Note] WSREP: Start replication 2019-02-08 12:44:37 50 [Note] WSREP: Connecting with bootstrap option: 0 2019-02-08 12:44:37 50 [Note] WSREP: Setting GCS initial position to 377097cd-2b8e-11e9-9974-2773cf2bd981:82 2019-02-08 12:44:37 50 [Note] WSREP: protonet asio version 0 2019-02-08 12:44:37 50 [Note] WSREP: Using CRC-32C for message checksums. 2019-02-08 12:44:37 50 [Note] WSREP: backend: asio 2019-02-08 12:44:37 50 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-02-08 12:44:37 50 [Warning] WSREP: access file(/home/stepan/galera/git/10.4/server/mysql-test/var/1/mysqld.2/data//gvwstate.dat) failed(No such file or directory) 2019-02-08 12:44:37 50 [Note] WSREP: restore pc from disk failed 2019-02-08 12:44:37 50 [Note] WSREP: GMCast version 0 2019-02-08 12:44:37 50 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') listening at tcp://0.0.0.0:16045 2019-02-08 12:44:37 50 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') multicast: , ttl: 1 2019-02-08 12:44:37 50 [Note] WSREP: EVS version 1 2019-02-08 12:44:37 50 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '127.0.0.1:16042' 2019-02-08 12:44:37 50 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') connection established to 374d145d tcp://127.0.0.1:16042 2019-02-08 12:44:37 50 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') turning message relay requesting on, nonlive peers: 2019-02-08 12:44:38 50 [Note] WSREP: EVS version upgrade 0 -> 1 2019-02-08 12:44:38 50 [Note] WSREP: declaring 374d145d at tcp://127.0.0.1:16042 stable 2019-02-08 12:44:38 50 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-02-08 12:44:38 50 [Note] WSREP: Node 374d145d state prim 2019-02-08 12:44:38 50 [Note] WSREP: view(view_id(PRIM,374d145d,4) memb { 374d145d,0 87e8446f,0 } joined { } left { } partitioned { }) 2019-02-08 12:44:38 50 [Note] WSREP: save pc into disk 2019-02-08 12:44:38 50 [Note] WSREP: gcomm: connected 2019-02-08 12:44:38 50 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-02-08 12:44:38 50 [Note] WSREP: Shifting CLOSED -> OPEN (TO: -1) 2019-02-08 12:44:38 50 [Note] WSREP: Opened channel 'my_wsrep_cluster' 2019-02-08 12:44:38 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2019-02-08 12:44:38 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-02-08 12:44:38 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 88516d5c-2b8e-11e9-8949-ff44de1e5f16 2019-02-08 12:44:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 88516d5c-2b8e-11e9-8949-ff44de1e5f16 from 0 (cnts7gl1.localdomain) 2019-02-08 12:44:38 54 [Note] WSREP: Starting applier thread 54 2019-02-08 12:44:38 0 [Note] WSREP: STATE EXCHANGE: got state msg: 88516d5c-2b8e-11e9-8949-ff44de1e5f16 from 1 (cnts7gl1.localdomain) 2019-02-08 12:44:38 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 3, members = 1/2 (joined/total), act_id = 83, last_appl. = 0, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 377097cd-2b8e-11e9-9974-2773cf2bd981 2019-02-08 12:44:38 0 [Note] WSREP: Writing down CC checksum: 2bf70eae 3715e82d 8cb29c12 8ae69ab0 at offset 224 2019-02-08 12:44:38 0 [Note] WSREP: Flow-control interval: [23, 23] 2019-02-08 12:44:38 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-08 12:44:38 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 84) 2019-02-08 12:44:38 54 [Note] WSREP: ####### processing CC 84, local, ordered 2019-02-08 12:44:38 54 [Note] WSREP: ####### drain monitors upto 82 2019-02-08 12:44:38 54 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-02-08 12:44:38 54 [Note] WSREP: ####### My UUID: 87e8446f-2b8e-11e9-bcda-cea9d6fde668 2019-02-08 12:44:38 54 [Note] WSREP: Server cnts7gl1.localdomain connected to cluster at position 377097cd-2b8e-11e9-9974-2773cf2bd981:84 with ID 87e8446f-2b8e-11e9-bcda-cea9d6fde668 2019-02-08 12:44:38 54 [Note] WSREP: Server status change disconnected -> connected 2019-02-08 12:44:38 54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-08 12:44:38 54 [Note] WSREP: State transfer required: Group state: 377097cd-2b8e-11e9-9974-2773cf2bd981:84 Local state: 377097cd-2b8e-11e9-9974-2773cf2bd981:82 2019-02-08 12:44:38 54 [Note] WSREP: Server status change connected -> joiner 2019-02-08 12:44:38 54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-08 12:44:38 54 [Note] WSREP: You have configured 'rsync' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2019-02-08 12:44:38 54 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-02-08 12:44:38 0 [Note] WSREP: Service thread queue flushed. 2019-02-08 12:44:38 54 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-02-08 12:44:38 54 [Note] WSREP: ####### IST uuid:377097cd-2b8e-11e9-9974-2773cf2bd981 f: 83, l: 84, p: 10 2019-02-08 12:44:38 54 [Note] WSREP: IST receiver addr using tcp://127.0.0.1:16046 2019-02-08 12:44:38 54 [Note] WSREP: Prepared IST receiver for 83-84, listening at: tcp://127.0.0.1:16046 2019-02-08 12:44:38 0 [Note] WSREP: Member 1.0 (cnts7gl1.localdomain) requested state transfer from '*any*'. Selected 0.0 (cnts7gl1.localdomain)(SYNCED) as donor. 2019-02-08 12:44:38 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 84) 2019-02-08 12:44:38 54 [Note] WSREP: Requesting state transfer: success, donor: 0 2019-02-08 12:44:39 54 [Note] WSREP: Receiving IST: 2 writesets, seqnos 83-84 2019-02-08 12:44:40 0 [Note] WSREP: ####### IST applying starts with 83 2019-02-08 12:44:40 0 [Note] WSREP: ####### IST current seqno initialized to 3 2019-02-08 12:44:40 0 [Note] WSREP: Receiving IST... 0.0% ( 0/82 events) complete. 2019-02-08 12:44:40 0 [Note] WSREP: IST preload starting at 3 2019-02-08 12:44:40 0 [Note] WSREP: 0.0 (cnts7gl1.localdomain): State transfer to 1.0 (cnts7gl1.localdomain) complete. 2019-02-08 12:44:40 0 [Note] WSREP: Service thread queue flushed. 2019-02-08 12:44:40 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:2, protocol version: 5 2019-02-08 12:44:40 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-08 12:44:40 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 84 last received: 5 2019-02-08 12:44:40 54 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) at galera/src/replicator_smm.hpp:pop_front():320. Null event. 2019-02-08 12:44:40 0 [Note] WSREP: Member 0.0 (cnts7gl1.localdomain) synced with group. 2019-02-08 12:44:40 54 [Note] WSREP: Closing send monitor... 2019-02-08 12:44:40 54 [Note] WSREP: Closed send monitor. 2019-02-08 12:44:40 54 [Note] WSREP: gcomm: terminating thread 2019-02-08 12:44:40 54 [Note] WSREP: gcomm: joining thread 2019-02-08 12:44:40 54 [Note] WSREP: gcomm: closing backend 2019-02-08 12:44:40 54 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') turning message relay requesting off 2019-02-08 12:44:43 54 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') connection to peer 374d145d with addr tcp://127.0.0.1:16042 timed out, no messages seen in PT3S 2019-02-08 12:44:43 54 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16042 2019-02-08 12:44:45 54 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') reconnecting to 374d145d (tcp://127.0.0.1:16042), attempt 0 2019-02-08 12:44:48 54 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') connection to peer 374d145d with addr tcp://127.0.0.1:16042 timed out, no messages seen in PT3S 2019-02-08 12:44:50 54 [Note] WSREP: (87e8446f, 'tcp://0.0.0.0:16045') connection established to 374d145d tcp://127.0.0.1:16042 2019-02-08 12:44:50 54 [Note] WSREP: evs::proto(87e8446f, LEAVING, view_id(REG,374d145d,4)) suspecting node: 374d145d 2019-02-08 12:44:50 54 [Note] WSREP: evs::proto(87e8446f, LEAVING, view_id(REG,374d145d,4)) suspected node without join message, declaring inactive 2019-02-08 12:44:50 54 [Note] WSREP: view(view_id(NON_PRIM,374d145d,4) memb { 87e8446f,0 } joined { } left { } partitioned { 374d145d,0 }) 2019-02-08 12:44:50 54 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-08 12:44:50 54 [Note] WSREP: view((empty)) 2019-02-08 12:44:50 54 [Note] WSREP: gcomm: closed 2019-02-08 12:44:50 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-08 12:44:50 0 [Note] WSREP: Writing down CC checksum: ccf97269 46238840 0fafc5b8 c6713bc7 at offset 144 2019-02-08 12:44:50 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-08 12:44:50 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-08 12:44:50 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-08 12:44:50 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 84) 2019-02-08 12:44:50 0 [Note] WSREP: Received self-leave message. 2019-02-08 12:44:50 0 [Note] WSREP: Writing down CC checksum: 4c5c6da9 8947ed8d 248b92e6 1503f905 at offset 64 2019-02-08 12:44:50 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-08 12:44:50 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-08 12:44:50 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-08 12:44:50 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-08 12:44:50 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-08 12:44:50 54 [Note] WSREP: recv_thread() joined. 2019-02-08 12:44:50 54 [Note] WSREP: Closing replication queue. 2019-02-08 12:44:50 54 [Note] WSREP: Closing slave action queue. mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. 190208 12:44:50 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.3-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63454 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fb2b0001d50 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fb2cc3109a0 thread_stack 0x49000 /home/stepan/galera/git/10.4/server/sql/mysqld(my_print_stacktrace+0x40)[0x563e6f1b8370] mysys/stacktrace.c:269(my_print_stacktrace)[0x563e6e9f6175] sigaction.c:0(__restore_rt)[0x7fb2d5bb95d0] :0(__GI_raise)[0x7fb2d3ea8207] :0(__GI_abort)[0x7fb2d3ea98f8] :0(__assert_fail_base)[0x7fb2d3ea1026] :0(__GI___assert_fail)[0x7fb2d3ea10d2] src/replicator_str.cpp:945(galera::ReplicatorSMM::request_state_transfer(void*, wsrep_uuid const&, long, void const*, long))[0x7fb2cdaf3dc7] src/replicator_smm.cpp:2733(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7fb2cdae2e78] src/gcs_action_source.cpp:154(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7fb2cdac2ec9] src/gcs_action_source.cpp:188(galera::GcsActionSource::process(void*, bool&))[0x7fb2cdac321d] src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7fb2cdae2033] /home/stepan/galera/git/10.4/server/sql/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x30)[0x563e6f27236c] src/wsrep_provider_v26.cpp:690(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x563e6e9156d4] sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x563e6e90731f] pthread_create.c:0(start_thread)[0x7fb2d5bb1dd5] /lib64/libc.so.6(clone+0x6d)[0x7fb2d3f6fead] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 54 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file at /home/stepan/galera/git/10.4/server/mysql-test/var/1/mysqld.2/data/ ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > connection node_1; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2a; SET SESSION wsrep_sync_wait=0; SET GLOBAL wsrep_cluster_address = ''; SET SESSION wsrep_sync_wait = DEFAULT; connection node_2; INSERT INTO t1 VALUES (6); ERROR HY000: Lost connection to MySQL server during query connection node_1; SELECT COUNT(*) = 0 FROM mysql.wsrep_streaming_log; COUNT(*) = 0 1 connection node_2a; connection node_1; connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2b; - found 'core.72022' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.72022 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 77716] [New LWP 77781] [New LWP 72487] [New LWP 72048] [New LWP 72022] [New LWP 72482] [New LWP 72047] [New LWP 72492] [New LWP 72488] [New LWP 72486] [New LWP 72524] [New LWP 72489] [New LWP 72491] [New LWP 72530] [New LWP 72490] [New LWP 72525] [New LWP 72531] [New LWP 72523] [New LWP 72532] [New LWP 72543] [New LWP 72529] [New LWP 72542] [New LWP 72563] [New LWP 72541] [New LWP 77714] [New LWP 72562] [New LWP 76802] [New LWP 77715] [New LWP 72522] [New LWP 72544] [New LWP 72526] [New LWP 72533] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fb2d5bb69d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fb2d5bb69d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563e6f1b845e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563e6e9f65a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb2d3ea8207 in raise () from /lib64/libc.so.6 #5 0x00007fb2d3ea98f8 in abort () from /lib64/libc.so.6 #6 0x00007fb2d3ea1026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007fb2d3ea10d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007fb2cdaf3dc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x563e717e81f0, recv_ctx=recv_ctx@entry=0x7fb2cc3104d0, group_uuid=..., cc_seqno=cc_seqno@entry=84, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007fb2cdae2e78 in galera::ReplicatorSMM::process_conf_change (this=0x563e717e81f0, recv_ctx=0x7fb2cc3104d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007fb2cdac2ec9 in galera::GcsActionSource::dispatch (this=this@entry=0x563e71820e60, recv_ctx=recv_ctx@entry=0x7fb2cc3104d0, act=..., exit_loop=@0x7fb2cc31029e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007fb2cdac321d in galera::GcsActionSource::process (this=0x563e71820e60, recv_ctx=0x7fb2cc3104d0, exit_loop=@0x7fb2cc31029e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007fb2cdae2033 in galera::ReplicatorSMM::async_recv (this=0x563e717e81f0, recv_ctx=0x7fb2cc3104d0) at galera/src/replicator_smm.cpp:383 #13 0x0000563e6f27236c in wsrep::wsrep_provider_v26::run_applier (this=0x563e7175de50, applier_ctx=0x7fb2cc3104d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #14 0x0000563e6e9156d4 in wsrep_replication_process (thd=0x7fb2b0001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x0000563e6e90731f in start_wsrep_THD (arg=0x7fb2440401b0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #16 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fb281ffb700 (LWP 72533)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ed0025b in os_event::wait (this=0x563e71939390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563e6ecffce1 in os_event::wait_low (this=0x563e71939390, reset_sig_count=371) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563e6ed00052 in os_event_wait_low (event=0x563e71939390, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563e6edebee5 in srv_resume_thread (slot=0x563e6fd3a7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563e6edefae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fb283fff700 (LWP 72526)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ecffb76 in os_event::timed_wait (this=0x563e71ae1f40, abstime=0x7fb283ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563e6ecffe8d in os_event::wait_time_low (this=0x563e71ae1f40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563e6ed0002d in os_event_wait_time_low (event=0x563e71ae1f40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563e6ee4e403 in ib_wqueue_timedwait (wq=0x563e71ae1e10, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000563e6ef8ef9a in fts_optimize_thread (arg=0x563e71ae1e10) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fb2637fe700 (LWP 72544)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6f1bda8d in safe_cond_wait (cond=0x7fb2637fdc70, mp=0x7fb2637fdcb0, file=0x563e6f48bd40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563e6ec17525 in inline_mysql_cond_wait (that=0x7fb2637fdc70, mutex=0x7fb2637fdcb0, src_file=0x563e6f48d6d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563e6ec1c161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000563e6f150555 in pfs_spawn_thread (arg=0x563e71af4620) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fb2965f7700 (LWP 72522)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ecffb76 in os_event::timed_wait (this=0x563e71b11940, abstime=0x7fb2965f6d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563e6ecffe8d in os_event::wait_time_low (this=0x563e71b11940, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563e6ed0002d in os_event_wait_time_low (event=0x563e71b11940, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563e6eccc9e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fb2bc0ec700 (LWP 77715)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6f1bda8d in safe_cond_wait (cond=0x7fb2b8024648, mp=0x7fb2b8024598, file=0x563e6f327c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563e6e915126 in inline_mysql_cond_wait (that=0x7fb2b8024648, mutex=0x7fb2b8024598, src_file=0x563e6f327f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563e6e917dbe in Wsrep_thd_queue::pop_front (this=0x7fb2b8024540) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563e6e916b9e in wsrep_post_rollback_process (post_rollbacker=0x7fb2b800ffa0, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000563e6e90731f in start_wsrep_THD (arg=0x7fb244040090) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fb2bc0a1700 (LWP 76802)): #0 0x00007fb2d3f6520d in poll () from /lib64/libc.so.6 #1 0x0000563e6f229702 in vio_io_wait (vio=0x563e718ff840, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:976 #2 0x0000563e6f227d22 in vio_socket_io_wait (vio=0x563e718ff840, event=VIO_IO_EVENT_READ) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:118 #3 0x0000563e6f227f08 in vio_read (vio=0x563e718ff840, buf=0x7fb24400a430 "\a", size=4) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:195 #4 0x0000563e6eb5f15e in my_real_read (net=0x7fb244000df8, complen=0x7fb2bc0a0810, header=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:962 #5 0x0000563e6eb5f9e5 in my_net_read_packet_reallen (net=0x7fb244000df8, read_from_server=1 '\001', reallen=0x7fb2bc0a0878) at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1243 #6 0x0000563e6eb5f998 in my_net_read_packet (net=0x7fb244000df8, read_from_server=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1227 #7 0x0000563e6e6bb146 in do_command (thd=0x7fb244000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1229 #8 0x0000563e6e82c069 in do_handle_one_connection (connect=0x563e71bb60c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #9 0x0000563e6e82bdad in handle_one_connection (arg=0x563e71bb60c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #10 0x0000563e6f150555 in pfs_spawn_thread (arg=0x563e718ffd30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #11 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fb2cc09e700 (LWP 72562)): #0 0x00007fb2d3ea940a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000563e6e589f44 in my_sigwait (set=0x7fb2cc09dcf0, sig=0x7fb2cc09dcb4, code=0x7fb2cc09dcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000563e6e590584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000563e6f150555 in pfs_spawn_thread (arg=0x563e718fbb60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fb2cc2c6700 (LWP 77714)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6f1bda8d in safe_cond_wait (cond=0x7fb2a8015698, mp=0x7fb2a80155e8, file=0x563e6f327c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563e6e915126 in inline_mysql_cond_wait (that=0x7fb2a8015698, mutex=0x7fb2a80155e8, src_file=0x563e6f327f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563e6e917dbe in Wsrep_thd_queue::pop_front (this=0x7fb2a8015590) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563e6e9167de in wsrep_rollback_process (rollbacker=0x7fb2a8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000563e6e90731f in start_wsrep_THD (arg=0x7fb244040070) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fb2817fa700 (LWP 72541)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ed0025b in os_event::wait (this=0x563e71939510) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563e6ecffce1 in os_event::wait_low (this=0x563e71939510, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563e6ed00052 in os_event_wait_low (event=0x563e71939510, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563e6eecbfb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fb2cc053700 (LWP 72563)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6f1bda8d in safe_cond_wait (cond=0x563e6fefd1e0 , mp=0x563e6ff00560 , file=0x563e6f288dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563e6e5bb4ee in inline_mysql_cond_wait (that=0x563e6fefd1e0 , mutex=0x563e6ff00560 , src_file=0x563e6f289930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000563e6e5bcd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000563e6f150555 in pfs_spawn_thread (arg=0x563e71c9c330) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fb280ff9700 (LWP 72542)): #0 0x00007fb2d5bb8e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563e6ed0055d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563e6eea6b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fb296df8700 (LWP 72529)): #0 0x00007fb2d5bb8e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563e6ed0055d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563e6edef44d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000563e6edef57d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fb263fff700 (LWP 72543)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ed0025b in os_event::wait (this=0x563e71939610) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563e6ecffce1 in os_event::wait_low (this=0x563e71939610, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563e6ed00052 in os_event_wait_low (event=0x563e71939610, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563e6eeb4c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fb2827fc700 (LWP 72532)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ed0025b in os_event::wait (this=0x563e71939310) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563e6ecffce1 in os_event::wait_low (this=0x563e71939310, reset_sig_count=390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563e6ed00052 in os_event_wait_low (event=0x563e71939310, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563e6edebee5 in srv_resume_thread (slot=0x563e6fd3a7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563e6edefae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fb295df6700 (LWP 72523)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ecffb76 in os_event::timed_wait (this=0x563e71939410, abstime=0x7fb295df5ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563e6ecffe8d in os_event::wait_time_low (this=0x563e71939410, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563e6ed0002d in os_event_wait_time_low (event=0x563e71939410, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563e6edee0ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fb282ffd700 (LWP 72531)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ed0025b in os_event::wait (this=0x563e71939290) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563e6ecffce1 in os_event::wait_low (this=0x563e71939290, reset_sig_count=392) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563e6ed00052 in os_event_wait_low (event=0x563e71939290, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563e6edebee5 in srv_resume_thread (slot=0x563e6fd3a780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563e6edefae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fb294df4700 (LWP 72525)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ecffb76 in os_event::timed_wait (this=0x563e71b43a00, abstime=0x7fb294df3d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563e6ecffe8d in os_event::wait_time_low (this=0x563e71b43a00, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563e6ed0002d in os_event_wait_time_low (event=0x563e71b43a00, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563e6ef3524a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fb2ac9a9700 (LWP 72490)): #0 0x00007fb2d5782644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563e6ecf3578 in LinuxAIOHandler::collect (this=0x7fb2ac9a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563e6ecf3aa1 in LinuxAIOHandler::poll (this=0x7fb2ac9a8b00, m1=0x7fb2ac9a8ba8, m2=0x7fb2ac9a8bb0, request=0x7fb2ac9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563e6ecf3c04 in os_aio_linux_handler (global_segment=4, m1=0x7fb2ac9a8ba8, m2=0x7fb2ac9a8bb0, request=0x7fb2ac9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563e6ecf8728 in os_aio_handler (segment=4, m1=0x7fb2ac9a8ba8, m2=0x7fb2ac9a8bb0, request=0x7fb2ac9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563e6ef4b5ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563e6edf19fb in io_handler_thread (arg=0x563e6ff78640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fb2837fe700 (LWP 72530)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ed0025b in os_event::wait (this=0x563e71939210) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563e6ecffce1 in os_event::wait_low (this=0x563e71939210, reset_sig_count=129) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563e6ed00052 in os_event_wait_low (event=0x563e71939210, reset_sig_count=129) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563e6edebee5 in srv_resume_thread (slot=0x563e6fd3a748 , sig_count=129, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563e6edefef6 in srv_purge_coordinator_suspend (slot=0x563e6fd3a748 , rseg_history_len=13) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000563e6edf018c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fb297fff700 (LWP 72491)): #0 0x00007fb2d5782644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563e6ecf3578 in LinuxAIOHandler::collect (this=0x7fb297ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563e6ecf3aa1 in LinuxAIOHandler::poll (this=0x7fb297ffeb00, m1=0x7fb297ffeba8, m2=0x7fb297ffebb0, request=0x7fb297ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563e6ecf3c04 in os_aio_linux_handler (global_segment=5, m1=0x7fb297ffeba8, m2=0x7fb297ffebb0, request=0x7fb297ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563e6ecf8728 in os_aio_handler (segment=5, m1=0x7fb297ffeba8, m2=0x7fb297ffebb0, request=0x7fb297ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563e6ef4b5ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563e6edf19fb in io_handler_thread (arg=0x563e6ff78648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fb2ad1aa700 (LWP 72489)): #0 0x00007fb2d5782644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563e6ecf3578 in LinuxAIOHandler::collect (this=0x7fb2ad1a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563e6ecf3aa1 in LinuxAIOHandler::poll (this=0x7fb2ad1a9b00, m1=0x7fb2ad1a9ba8, m2=0x7fb2ad1a9bb0, request=0x7fb2ad1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563e6ecf3c04 in os_aio_linux_handler (global_segment=3, m1=0x7fb2ad1a9ba8, m2=0x7fb2ad1a9bb0, request=0x7fb2ad1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563e6ecf8728 in os_aio_handler (segment=3, m1=0x7fb2ad1a9ba8, m2=0x7fb2ad1a9bb0, request=0x7fb2ad1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563e6ef4b5ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563e6edf19fb in io_handler_thread (arg=0x563e6ff78638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fb2955f5700 (LWP 72524)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ecffb76 in os_event::timed_wait (this=0x563e71939490, abstime=0x7fb2955f4d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563e6ecffe8d in os_event::wait_time_low (this=0x563e71939490, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563e6ed0002d in os_event_wait_time_low (event=0x563e71939490, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563e6ededbcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fb2ae9ad700 (LWP 72486)): #0 0x00007fb2d5782644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563e6ecf3578 in LinuxAIOHandler::collect (this=0x7fb2ae9acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563e6ecf3aa1 in LinuxAIOHandler::poll (this=0x7fb2ae9acb00, m1=0x7fb2ae9acba8, m2=0x7fb2ae9acbb0, request=0x7fb2ae9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563e6ecf3c04 in os_aio_linux_handler (global_segment=0, m1=0x7fb2ae9acba8, m2=0x7fb2ae9acbb0, request=0x7fb2ae9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563e6ecf8728 in os_aio_handler (segment=0, m1=0x7fb2ae9acba8, m2=0x7fb2ae9acbb0, request=0x7fb2ae9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563e6ef4b5ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563e6edf19fb in io_handler_thread (arg=0x563e6ff78620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fb2ad9ab700 (LWP 72488)): #0 0x00007fb2d5782644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563e6ecf3578 in LinuxAIOHandler::collect (this=0x7fb2ad9aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563e6ecf3aa1 in LinuxAIOHandler::poll (this=0x7fb2ad9aab00, m1=0x7fb2ad9aaba8, m2=0x7fb2ad9aabb0, request=0x7fb2ad9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563e6ecf3c04 in os_aio_linux_handler (global_segment=2, m1=0x7fb2ad9aaba8, m2=0x7fb2ad9aabb0, request=0x7fb2ad9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563e6ecf8728 in os_aio_handler (segment=2, m1=0x7fb2ad9aaba8, m2=0x7fb2ad9aabb0, request=0x7fb2ad9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563e6ef4b5ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563e6edf19fb in io_handler_thread (arg=0x563e6ff78630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fb2977fe700 (LWP 72492)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6ecffb76 in os_event::timed_wait (this=0x563e71939590, abstime=0x7fb2977fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563e6ecffe8d in os_event::wait_time_low (this=0x563e71939590, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563e6ed0002d in os_event_wait_time_low (event=0x563e71939590, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563e6eed5e25 in pc_sleep_if_needed (next_loop_time=1549622690887, sig_count=2, cur_time=1549622689887) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000563e6eed6d5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb2ce60a700 (LWP 72047)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6f1bdd8f in safe_cond_timedwait (cond=0x563e7079c640 , mp=0x563e7079c580 , abstime=0x7fb2ce609d70, file=0x563e6f645920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563e6f1bed49 in inline_mysql_cond_timedwait (that=0x563e7079c640 , mutex=0x563e7079c580 , abstime=0x7fb2ce609d70, src_file=0x563e6f645968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563e6f1bf8ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000563e6f150555 in pfs_spawn_thread (arg=0x563e717b17d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb2affff700 (LWP 72482)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563e6f1bdd8f in safe_cond_timedwait (cond=0x563e70789ea0 , mp=0x563e70789de0 , abstime=0x7fb2afffecb0, file=0x563e6f6016f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563e6f027bc1 in inline_mysql_cond_timedwait (that=0x563e70789ea0 , mutex=0x563e70789de0 , abstime=0x7fb2afffecb0, src_file=0x563e6f601738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563e6f0280e5 in my_service_thread_sleep (control=0x563e6fd3eb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000563e6f01c57e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000563e6f150555 in pfs_spawn_thread (arg=0x563e718f15f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb2d5fc2880 (LWP 72022)): #0 0x00007fb2d3f6520d in poll () from /lib64/libc.so.6 #1 0x0000563e6e596a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #2 0x0000563e6e595d25 in mysqld_main (argc=162, argv=0x563e71669d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #3 0x0000563e6e589ec5 in main (argc=23, argv=0x7ffeef4636c8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 4 (Thread 0x7fb2cd45e700 (LWP 72048)): #0 0x00007fb2d5bb5965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb2cdabffd7 in gu::Lock::wait (cond=..., this=0x7fb2cd45dbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb2cdac0069 in galera::ServiceThd::thd_func (arg=0x563e717e8908) at galera/src/galera_service_thd.cpp:30 #3 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fb2ae1ac700 (LWP 72487)): #0 0x00007fb2d5782644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563e6ecf3578 in LinuxAIOHandler::collect (this=0x7fb2ae1abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563e6ecf3aa1 in LinuxAIOHandler::poll (this=0x7fb2ae1abb00, m1=0x7fb2ae1abba8, m2=0x7fb2ae1abbb0, request=0x7fb2ae1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563e6ecf3c04 in os_aio_linux_handler (global_segment=1, m1=0x7fb2ae1abba8, m2=0x7fb2ae1abbb0, request=0x7fb2ae1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563e6ecf8728 in os_aio_handler (segment=1, m1=0x7fb2ae1abba8, m2=0x7fb2ae1abbb0, request=0x7fb2ae1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563e6ef4b5ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563e6edf19fb in io_handler_thread (arg=0x563e6ff78628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb2cc35c700 (LWP 77781)): #0 0x00007fb2d5bb5d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb2cda9cbdb in gu::Lock::wait (this=this@entry=0x7fb2cc3596f0, cond=..., date=...) at galerautils/src/gu_lock.hpp:62 #2 0x00007fb2cdae7976 in galera::Monitor::wait (this=this@entry=0x563e717e9190, gtid=..., wait_until=...) at galera/src/monitor.hpp:355 #3 0x00007fb2cdad957f in galera::ReplicatorSMM::sync_wait (this=0x563e717e81f0, upto=, tout=, gtid=0x7fb2cc359b30) at galera/src/replicator_smm.cpp:1661 #4 0x00007fb2cdaf8643 in galera_sync_wait (wsrep=, upto=, tout=, gtid=) at galera/src/wsrep_provider.cpp:998 #5 0x0000563e6f273023 in wsrep::wsrep_provider_v26::causal_read (this=0x563e7175de50, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:845 #6 0x0000563e6f25d577 in wsrep::server_state::causal_read (this=0x563e717b1760, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:648 #7 0x0000563e6f257e01 in wsrep::client_state::sync_wait (this=0x7fb2a0006980, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/client_state.cpp:417 #8 0x0000563e6e8ffb2a in wsrep_sync_wait (thd=0x7fb2a0000b20, mask=1) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:1022 #9 0x0000563e6e6c29f4 in mysql_execute_command (thd=0x7fb2a0000b20) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:3779 #10 0x0000563e6e6d1935 in mysql_parse (thd=0x7fb2a0000b20, rawbuf=0x7fb2a0016c08 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7fb2cc35b0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #11 0x0000563e6e6d0f68 in wsrep_mysql_parse (thd=0x7fb2a0000b20, rawbuf=0x7fb2a0016c08 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7fb2cc35b0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #12 0x0000563e6e6bce25 in dispatch_command (command=COM_QUERY, thd=0x7fb2a0000b20, packet=0x7fb2a000a2e1 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", packet_length=102, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #13 0x0000563e6e6bb80d in do_command (thd=0x7fb2a0000b20) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #14 0x0000563e6e82c069 in do_handle_one_connection (connect=0x563e71bb60c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #15 0x0000563e6e82bdad in handle_one_connection (arg=0x563e71bb60c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #16 0x0000563e6f150555 in pfs_spawn_thread (arg=0x563e7182af60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #17 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb2cc311700 (LWP 77716)): #0 0x00007fb2d5bb69d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563e6f1b845e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563e6e9f65a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb2d3ea8207 in raise () from /lib64/libc.so.6 #5 0x00007fb2d3ea98f8 in abort () from /lib64/libc.so.6 #6 0x00007fb2d3ea1026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007fb2d3ea10d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007fb2cdaf3dc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x563e717e81f0, recv_ctx=recv_ctx@entry=0x7fb2cc3104d0, group_uuid=..., cc_seqno=cc_seqno@entry=84, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007fb2cdae2e78 in galera::ReplicatorSMM::process_conf_change (this=0x563e717e81f0, recv_ctx=0x7fb2cc3104d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007fb2cdac2ec9 in galera::GcsActionSource::dispatch (this=this@entry=0x563e71820e60, recv_ctx=recv_ctx@entry=0x7fb2cc3104d0, act=..., exit_loop=@0x7fb2cc31029e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007fb2cdac321d in galera::GcsActionSource::process (this=0x563e71820e60, recv_ctx=0x7fb2cc3104d0, exit_loop=@0x7fb2cc31029e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007fb2cdae2033 in galera::ReplicatorSMM::async_recv (this=0x563e717e81f0, recv_ctx=0x7fb2cc3104d0) at galera/src/replicator_smm.cpp:383 #13 0x0000563e6f27236c in wsrep::wsrep_provider_v26::run_applier (this=0x563e7175de50, applier_ctx=0x7fb2cc3104d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #14 0x0000563e6e9156d4 in wsrep_replication_process (thd=0x7fb2b0001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x0000563e6e90731f in start_wsrep_THD (arg=0x7fb2440401b0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #16 0x00007fb2d5bb1dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fb2d3f6fead in clone () from /lib64/libc.so.6 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_cc_master-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_cc_master galera_sr.galera_sr_bf_abort galera_sr.galera_sr_blob galera_sr.galera_sr_kill_query 2019-02-08 12:44:33 44 [Warning] WSREP: Failed to replicate rollback fragment for 361: 3 2019-02-08 12:44:40 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-08 12:44:40 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 84 last received: 5 2019-02-08 12:44:40 54 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_kill_all_pcrecovery 'innodb' w3 [ pass ] 32136 galera_sr.galera_sr_cc_slave 'innodb' w1 [ pass ] 34508 galera_sr.galera_sr_concurrent 'innodb' w1 [ pass ] 13207 galera_sr.galera_sr_conflict 'innodb' w1 [ pass ] 7463 galera_sr.galera_sr_conflict_on_commit 'innodb' w1 [ pass ] 9786 galera_sr.galera_sr_conflict_on_commit2 'innodb' w1 [ pass ] 8252 galera_sr.galera_sr_unit_statements 'innodb' w1 [ pass ] 6536 galera_sr.galera_sr_ws_size 'innodb' w1 [ pass ] 29276 galera_sr.galera_sr_ws_size2 'innodb' w1 [ pass ] 10777 galera_sr.galera_var_ignore_apply_errors_sr 'innodb' w1 [ pass ] 9467 galera_sr.mysql-wsrep#215 'innodb' w1 [ pass ] 19161 galera_sr.mysql-wsrep-features#138 'innodb' w1 [ pass ] 8946 galera_sr.mysql-wsrep-features#14 'innodb' w1 [ pass ] 6781 galera_sr.galera_sr_rollback_statement 'innodb' w4 [ fail ] Test ended at 2019-02-08 12:48:31 CURRENT_TEST: galera_sr.galera_sr_rollback_statement mysqltest: At line 34: query 'INSERT INTO t2 SELECT * FROM t1' succeeded - should have failed with errno 1062... The result from queries just before the failure was: connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 CHAR(255)) ENGINE=InnoDB; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3; ALTER TABLE t1 CHANGE f1 f1 INTEGER; ALTER TABLE t1 DROP PRIMARY KEY; INSERT INTO t1 VALUES (1, 'abc'); SET SESSION wsrep_trx_fragment_size = 1; INSERT INTO t2 SELECT * FROM t1; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_rollback_statement-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_rollback_statement-innodb/' galera_sr.mysql-wsrep-features#148 'innodb' w1 [ pass ] 18979 MTR's internal check of the test case 'galera_sr.mysql-wsrep-features#148' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16041 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.result 2019-02-08 12:48:04.159178000 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.reject 2019-02-08 12:48:23.436178000 +0200 @@ -1099,4 +1099,4 @@ user_variables 1.0 DISABLED INFORMATION SCHEMA 100403.0 NULL NULL Sergey Vojtovich User-defined variables GPL OFF Gamma 1.0 partition 1.0 DISABLED STORAGE ENGINE 100403.0 NULL NULL Mikael Ronstrom, MySQL AB Partition Storage Engine Helper GPL OFF Stable 1.0 VARIABLE_NAME VARIABLE_VALUE -DEBUG_SYNC ON - current signal: '' +DEBUG_SYNC ON - current signal: 'signal.wsrep_apply_cb' mysqltest: Result length mismatch not ok galera_sr.mysql-wsrep-features#15 'innodb' w1 [ pass ] 5469 galera_sr.mysql-wsrep-features#165 'innodb' w1 [ fail ] Test ended at 2019-02-08 12:49:13 CURRENT_TEST: galera_sr.mysql-wsrep-features#165 mysqltest: In included file "/home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.test at line 27: At line 101: query 'SELECT * FROM t1' failed: 1205: Lock wait timeout exceeded; try restarting transaction The result from queries just before the failure was: < snip > connection node_1; SET DEBUG_SYNC = 'now WAIT_FOR wait'; connection node_1; SET DEBUG_SYNC = 'now SIGNAL continue'; connection node_1c; UPDATE t1 SET f2 = 'x' WHERE f1 = 3; connection node_1a; COMMIT; connection node_1b; ERROR 40001: Deadlock found when trying to get lock; try restarting transaction connection node_1; SELECT * FROM t1; f1 f2 1 a 2 a 3 x 4 a 5 a connection node_2; SELECT * FROM t1; More results from queries before failure can be found in /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/mysql-wsrep-features#165.log - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.mysql-wsrep-features#165-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#165-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 2019-02-08 12:49:02 3 [Warning] WSREP: Event 3 Write_rows_v1 apply failed: 121, seqno 12 galera_sr.galera_sr_shutdown_master 'innodb' w4 [ pass ] 28309 galera_sr.mysql-wsrep-features#22 'innodb' w1 [ pass ] 9672 galera_sr.galera_sr_shutdown_slave 'innodb' w4 [ pass ] 40518 galera_sr.mysql-wsrep-features#27 'innodb' w1 [ pass ] 11059 galera_sr.galera_sr_table_contents 'innodb' w4 [ fail ] Test ended at 2019-02-08 12:50:02 CURRENT_TEST: galera_sr.galera_sr_table_contents cp: cannot stat 'std_data/binlog-header.log': No such file or directory mysqltest: In included file "./suite/galera/include/galera_dump_sr_table.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/galera_sr_table_contents.test at line 21: At line 20: exec of 'cp std_data/binlog-header.log /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/sr_binlog.log' failed, error: 256, status: 1, errno: 11 Output from before failure: Warnings: Warning 1287 ' INTO FROM...' instead The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t1 VALUES (1),(2),(3); CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t2 VALUES (1),(2),(3); SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; Start of Simple Insert INSERT INTO t1 VALUES (4); Warnings: Warning 1287 ' INTO FROM...' instead - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_table_contents-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_table_contents-innodb/' galera_sr.mysql-wsrep-features#35 'innodb' w1 [ pass ] 9369 galera_sr.mysql-wsrep-features#93 'innodb' w1 [ pass ] 7060 galera_sr.mysql-wsrep-features#96 'innodb' w1 [ pass ] 11967 galera_sr.galera_sr_transaction_replay 'innodb' w4 [ pass ] 13774 galera_sr.galera_sr_kill_all_norecovery 'innodb' w1 [ pass ] 29852 MTR's internal check of the test case 'galera_sr.galera_sr_kill_all_norecovery' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16041 (socket /home/stepan/galera/git/10.4/server/mysqworker[2] Trying to dump core for [mysqltest - pid: 74013, winpid: 74013] worker[2] Trying to dump core for [mysqld.1 - pid: 72659, winpid: 72659] worker[2] Trying to dump core for [mysqld.2 - pid: 73074, winpid: 73074] l-test/var/tmp/1/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.result 2019-02-08 12:50:53.685178000 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/check-mysqld_2.reject 2019-02-08 12:51:23.739178000 +0200 @@ -77,7 +77,7 @@ ARIA_USED_FOR_TEMP_TABLES ON AUTOCOMMIT ON AUTOMATIC_SP_PRIVILEGES ON -AUTO_INCREMENT_OFFSET 2 +AUTO_INCREMENT_OFFSET 1 BACK_LOG 80 BASEDIR /home/stepan/galera/git/10.4/server BIG_TABLES OFF mysqltest: Result content mismatch not ok galera_sr.galera_sr_kill_slave 'innodb' w4 [ pass ] 37274 galera_sr.mysql-wsrep-features#9 'innodb' w4 [ pass ] 25824 galera_sr.GCF-1018 'innodb' w4 [ pass ] 8630 galera_sr.galera_sr_load_data 'innodb' w2 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 12:59:11 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr_load_data.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 512; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; connection node_2; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/2/mysqld.' (111) - found 'core.72660' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_load_data-innodb/mysqld.1/data/core.72660 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 72660] [New LWP 72691] [New LWP 72661] [New LWP 72662] [New LWP 72728] [New LWP 72729] [New LWP 72692] [New LWP 72742] [New LWP 72693] [New LWP 72731] [New LWP 72753] [New LWP 72695] [New LWP 72732] [New LWP 72734] [New LWP 72733] [New LWP 72789] [New LWP 72751] [New LWP 72741] [New LWP 73873] [New LWP 72754] [New LWP 72730] [New LWP 72743] [New LWP 72750] [New LWP 73016] [New LWP 72752] [New LWP 72739] [New LWP 72766] [New LWP 72763] [New LWP 72694] [New LWP 72764] [New LWP 72765] [New LWP 72740] [New LWP 72735] [New LWP 73017] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fe4af6959d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fe4af6959d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000558bc818145e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000558bc79bf5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fe4ada4420d in poll () from /lib64/libc.so.6 #5 0x0000558bc755fa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000558bc755ed25 in mysqld_main (argc=161, argv=0x558bc989fd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000558bc7552ec5 in main (argc=23, argv=0x7fffa182aea8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fe49cd61700 (LWP 73017)): #0 0x00007fe4af697d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000558bc7cbdc27 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000558bc7cbdf7d in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000558bc7f064aa in pfs_os_file_flush_func (file=..., src_file=0x558bc8580b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000558bc7f09225 in fil_flush_low (space=0x558bc9d78790, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000558bc7f14aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000558bc7c99321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000558bc7c99d83 in log_write_up_to (lsn=5840062, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000558bc7e04e99 in trx_flush_log_if_needed_low (lsn=5840062) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000558bc7e04ee2 in trx_flush_log_if_needed (lsn=5840062, trx=0x7fe49d84b390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000558bc7e06d0b in trx_commit_complete_for_mysql (trx=0x7fe49d84b390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000558bc7beb5b2 in innobase_commit (hton=0x558bc9b429f0, thd=0x7fe43408af00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x0000558bc79c50d7 in commit_one_phase_2 (thd=0x7fe43408af00, all=true, trans=0x7fe43408e450, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000558bc79c4fd9 in ha_commit_one_phase (thd=0x7fe43408af00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000558bc79c4355 in ha_commit_trans (thd=0x7fe43408af00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000558bc7809e8e in trans_commit (thd=0x7fe43408af00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000558bc7bc922a in Wsrep_storage_service::commit (this=0x7fe434271ec0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x0000558bc82315c9 in wsrep::transaction::certify_fragment (this=0x7fe4340069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x0000558bc8230d2d in wsrep::transaction::streaming_step (this=0x7fe4340069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x0000558bc822e0e8 in wsrep::transaction::after_row (this=0x7fe4340069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x0000558bc79d7446 in wsrep::client_state::after_row (this=0x7fe434006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x0000558bc79c06c3 in wsrep_after_row (thd=0x7fe434000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x0000558bc79d25c8 in wsrep_after_row (thd=0x7fe434000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x0000558bc79d29b9 in handler::ha_write_row (this=0x7fe434168c98, buf=0x7fe434164f10 "\377\220\033") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x0000558bc7648443 in write_record (thd=0x7fe434000b00, table=0x7fe434168060, info=0x7fe49cd5eb10) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x0000558bc767b3c3 in read_sep_field (thd=0x7fe434000b00, info=..., table_list=0x7fe4340153a0, fields_vars=..., set_fields=..., set_values=..., read_info=..., enclosed=..., skip_lines=0, ignore_check_option_errors=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:1152 #26 0x0000558bc767997e in mysql_load (thd=0x7fe434000b00, ex=0x7fe434015318, table_list=0x7fe4340153a0, fields_vars=..., set_fields=..., set_values=..., handle_duplicates=DUP_ERROR, ignore=false, read_file_from_client=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:660 #27 0x0000558bc769075d in mysql_execute_command (thd=0x7fe434000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5148 #28 0x0000558bc769a935 in mysql_parse (thd=0x7fe434000b00, rawbuf=0x7fe434015168 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7fe49cd600d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #29 0x0000558bc7699f68 in wsrep_mysql_parse (thd=0x7fe434000b00, rawbuf=0x7fe434015168 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/2/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7fe49cd600d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #30 0x0000558bc7685e25 in dispatch_command (command=COM_QUERY, thd=0x7fe434000b00, packet=0x7fe434028201 "", packet_length=114, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #31 0x0000558bc768480d in do_command (thd=0x7fe434000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #32 0x0000558bc77f5069 in do_handle_one_connection (connect=0x558bc9edd830) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #33 0x0000558bc77f4dad in handle_one_connection (arg=0x558bc9edd830) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #34 0x0000558bc8119555 in pfs_spawn_thread (arg=0x558bc9eff640) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #35 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #36 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fe48d7fa700 (LWP 72735)): #0 0x00007fe4af697d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000558bc7cbdc27 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000558bc7cbdf7d in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000558bc7f064aa in pfs_os_file_flush_func (file=..., src_file=0x558bc8580b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000558bc7f09225 in fil_flush_low (space=0x558bc9d783f0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000558bc7f14aff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000558bc7e91dc5 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x0000558bc7e9c4ad in buf_flush_end (buf_pool=0x558bc9c540e0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x0000558bc7e9c66b in buf_flush_do_batch (buf_pool=0x558bc9c540e0, type=BUF_FLUSH_LIST, min_n=2, lsn_limit=18446744073709551615, n=0x7fe48d7f9b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x0000558bc7e9f355 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x0000558bc7ea02a3 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fe4777fe700 (LWP 72740)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc8b76 in os_event::timed_wait (this=0x558bc9b6e820, abstime=0x7fe4777fdba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558bc7cc8e8d in os_event::wait_time_low (this=0x558bc9b6e820, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558bc7cc902d in os_event_wait_time_low (event=0x558bc9b6e820, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558bc7db70ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fe461ffb700 (LWP 72765)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc925b in os_event::wait (this=0x558bc9b6ea20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558bc7cc8ce1 in os_event::wait_low (this=0x558bc9b6ea20, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558bc7cc9052 in os_event_wait_low (event=0x558bc9b6ea20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558bc7e7dc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fe4627fc700 (LWP 72764)): #0 0x00007fe4af697e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000558bc7cc955d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000558bc7e6fb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fe4a406d700 (LWP 72694)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc8186a8d in safe_cond_wait (cond=0x7fe494015608, mp=0x7fe494015558, file=0x558bc82f0c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558bc78de126 in inline_mysql_cond_wait (that=0x7fe494015608, mutex=0x7fe494015558, src_file=0x558bc82f0f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000558bc78e0dbe in Wsrep_thd_queue::pop_front (this=0x7fe494015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000558bc78dfb9e in wsrep_post_rollback_process (post_rollbacker=0x7fe494000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000558bc78d031f in start_wsrep_THD (arg=0x558bc9a60810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fe462ffd700 (LWP 72763)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc925b in os_event::wait (this=0x558bc9b6e920) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558bc7cc8ce1 in os_event::wait_low (this=0x558bc9b6e920, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558bc7cc9052 in os_event_wait_low (event=0x558bc9b6e920, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558bc7e94fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fe4617fa700 (LWP 72766)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc8186a8d in safe_cond_wait (cond=0x7fe4617f9c70, mp=0x7fe4617f9cb0, file=0x558bc8454d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558bc7be0525 in inline_mysql_cond_wait (that=0x7fe4617f9c70, mutex=0x7fe4617f9cb0, src_file=0x558bc84566d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000558bc7be5161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000558bc8119555 in pfs_spawn_thread (arg=0x558bc9daf8e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fe477fff700 (LWP 72739)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc8b76 in os_event::timed_wait (this=0x558bc9d46d10, abstime=0x7fe477ffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558bc7cc8e8d in os_event::wait_time_low (this=0x558bc9d46d10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558bc7cc902d in os_event_wait_time_low (event=0x558bc9d46d10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558bc7c959e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fe474ff9700 (LWP 72752)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc925b in os_event::wait (this=0x558bc9b6e6a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558bc7cc8ce1 in os_event::wait_low (this=0x558bc9b6e6a0, reset_sig_count=9963) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558bc7cc9052 in os_event_wait_low (event=0x558bc9b6e6a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558bc7db4ee5 in srv_resume_thread (slot=0x558bc8d03780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000558bc7db8ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fe49cdac700 (LWP 73016)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc8186a8d in safe_cond_wait (cond=0x558bc8ec61e0 , mp=0x558bc8ec9560 , file=0x558bc8251dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558bc75844ee in inline_mysql_cond_wait (that=0x558bc8ec61e0 , mutex=0x558bc8ec9560 , src_file=0x558bc8252930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000558bc7585d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000558bc8119555 in pfs_spawn_thread (arg=0x558bc9b322e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fe475ffb700 (LWP 72750)): #0 0x00007fe4af697e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000558bc7cc955d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000558bc7db844d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000558bc7db857d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fe4767fc700 (LWP 72743)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc8b76 in os_event::timed_wait (this=0x558bc9d0ea60, abstime=0x7fe4767fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558bc7cc8e8d in os_event::wait_time_low (this=0x558bc9d0ea60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558bc7cc902d in os_event_wait_time_low (event=0x558bc9d0ea60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558bc7e17403 in ib_wqueue_timedwait (wq=0x558bc9d0e930, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000558bc7f57f9a in fts_optimize_thread (arg=0x558bc9d0e930) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fe48ffff700 (LWP 72730)): #0 0x00007fe4af261644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558bc7cbc578 in LinuxAIOHandler::collect (this=0x7fe48fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558bc7cbcaa1 in LinuxAIOHandler::poll (this=0x7fe48fffeb00, m1=0x7fe48fffeba8, m2=0x7fe48fffebb0, request=0x7fe48fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558bc7cbcc04 in os_aio_linux_handler (global_segment=1, m1=0x7fe48fffeba8, m2=0x7fe48fffebb0, request=0x7fe48fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558bc7cc1728 in os_aio_handler (segment=1, m1=0x7fe48fffeba8, m2=0x7fe48fffebb0, request=0x7fe48fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000558bc7f145ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000558bc7dba9fb in io_handler_thread (arg=0x558bc8f41628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fe4637fe700 (LWP 72754)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc925b in os_event::wait (this=0x558bc9b6e7a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558bc7cc8ce1 in os_event::wait_low (this=0x558bc9b6e7a0, reset_sig_count=8367) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558bc7cc9052 in os_event_wait_low (event=0x558bc9b6e7a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558bc7db4ee5 in srv_resume_thread (slot=0x558bc8d037f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000558bc7db8ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fe49cccb700 (LWP 73873)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc8186d8f in safe_cond_timedwait (cond=0x558bc8ec61a0 , mp=0x558bc8ec8ba0 , abstime=0x7fe49cccac60, file=0x558bc82425c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558bc7553845 in inline_mysql_cond_timedwait (that=0x558bc8ec61a0 , mutex=0x558bc8ec8ba0 , abstime=0x7fe49cccac60, src_file=0x558bc8244638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000558bc755861b in cache_thread (thd=0x7fe42c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000558bc75588c9 in one_thread_per_connection_end (thd=0x7fe42c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000558bc77f50f3 in do_handle_one_connection (connect=0x558bc9e05c60) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000558bc77f4dad in handle_one_connection (arg=0x558bc9e05c60) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000558bc8119555 in pfs_spawn_thread (arg=0x558bc9e65000) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fe476ffd700 (LWP 72741)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc8b76 in os_event::timed_wait (this=0x558bc9b6e8a0, abstime=0x7fe476ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558bc7cc8e8d in os_event::wait_time_low (this=0x558bc9b6e8a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558bc7cc902d in os_event_wait_time_low (event=0x558bc9b6e8a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558bc7db6bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fe4757fa700 (LWP 72751)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc925b in os_event::wait (this=0x558bc9b6e620) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558bc7cc8ce1 in os_event::wait_low (this=0x558bc9b6e620, reset_sig_count=2291) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558bc7cc9052 in os_event_wait_low (event=0x558bc9b6e620, reset_sig_count=2291) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558bc7db4ee5 in srv_resume_thread (slot=0x558bc8d03748 , sig_count=2291, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000558bc7db8ef6 in srv_purge_coordinator_suspend (slot=0x558bc8d03748 , rseg_history_len=18) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000558bc7db918c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fe49cdf7700 (LWP 72789)): #0 0x00007fe4ad98840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000558bc7552f44 in my_sigwait (set=0x7fe49cdf6cf0, sig=0x7fe49cdf6cb4, code=0x7fe49cdf6cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000558bc7559584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000558bc8119555 in pfs_spawn_thread (arg=0x558bc9b31df0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fe48e7fc700 (LWP 72733)): #0 0x00007fe4af261644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558bc7cbc578 in LinuxAIOHandler::collect (this=0x7fe48e7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558bc7cbcaa1 in LinuxAIOHandler::poll (this=0x7fe48e7fbb00, m1=0x7fe48e7fbba8, m2=0x7fe48e7fbbb0, request=0x7fe48e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558bc7cbcc04 in os_aio_linux_handler (global_segment=4, m1=0x7fe48e7fbba8, m2=0x7fe48e7fbbb0, request=0x7fe48e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558bc7cc1728 in os_aio_handler (segment=4, m1=0x7fe48e7fbba8, m2=0x7fe48e7fbbb0, request=0x7fe48e7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000558bc7f145ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000558bc7dba9fb in io_handler_thread (arg=0x558bc8f41640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fe48dffb700 (LWP 72734)): #0 0x00007fe4af261644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558bc7cbc578 in LinuxAIOHandler::collect (this=0x7fe48dffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558bc7cbcaa1 in LinuxAIOHandler::poll (this=0x7fe48dffab00, m1=0x7fe48dffaba8, m2=0x7fe48dffabb0, request=0x7fe48dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558bc7cbcc04 in os_aio_linux_handler (global_segment=5, m1=0x7fe48dffaba8, m2=0x7fe48dffabb0, request=0x7fe48dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558bc7cc1728 in os_aio_handler (segment=5, m1=0x7fe48dffaba8, m2=0x7fe48dffabb0, request=0x7fe48dffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000558bc7f145ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000558bc7dba9fb in io_handler_thread (arg=0x558bc8f41648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fe48effd700 (LWP 72732)): #0 0x00007fe4af261644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558bc7cbc578 in LinuxAIOHandler::collect (this=0x7fe48effcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558bc7cbcaa1 in LinuxAIOHandler::poll (this=0x7fe48effcb00, m1=0x7fe48effcba8, m2=0x7fe48effcbb0, request=0x7fe48effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558bc7cbcc04 in os_aio_linux_handler (global_segment=3, m1=0x7fe48effcba8, m2=0x7fe48effcbb0, request=0x7fe48effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558bc7cc1728 in os_aio_handler (segment=3, m1=0x7fe48effcba8, m2=0x7fe48effcbb0, request=0x7fe48effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000558bc7f145ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000558bc7dba9fb in io_handler_thread (arg=0x558bc8f41638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fe49effd700 (LWP 72695)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe4a7493132 in fifo_lock_get (q=q@entry=0x558bc9a4cbd0) at galerautils/src/gu_fifo.c:262 #2 0x00007fe4a74937a1 in gu_fifo_get_head (q=0x558bc9a4cbd0, err=0x7fe49effc06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fe4a75750cc in gcs_recv (conn=0x558bc9a26550, action=0x7fe49effc0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fe4a75a216c in galera::GcsActionSource::process (this=0x558bc9a25f30, recv_ctx=0x7fe49effc4d0, exit_loop=@0x7fe49effc29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fe4a75c1033 in galera::ReplicatorSMM::async_recv (this=0x558bc9a1e280, recv_ctx=0x7fe49effc4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000558bc823b36c in wsrep::wsrep_provider_v26::run_applier (this=0x558bc99a5360, applier_ctx=0x7fe49effc4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000558bc78de6d4 in wsrep_replication_process (thd=0x7fe490001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000558bc78d031f in start_wsrep_THD (arg=0x558bc9a80550) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fe463fff700 (LWP 72753)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc925b in os_event::wait (this=0x558bc9b6e720) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558bc7cc8ce1 in os_event::wait_low (this=0x558bc9b6e720, reset_sig_count=8663) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558bc7cc9052 in os_event_wait_low (event=0x558bc9b6e720, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558bc7db4ee5 in srv_resume_thread (slot=0x558bc8d037b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000558bc7db8ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fe48f7fe700 (LWP 72731)): #0 0x00007fe4af261644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558bc7cbc578 in LinuxAIOHandler::collect (this=0x7fe48f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558bc7cbcaa1 in LinuxAIOHandler::poll (this=0x7fe48f7fdb00, m1=0x7fe48f7fdba8, m2=0x7fe48f7fdbb0, request=0x7fe48f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558bc7cbcc04 in os_aio_linux_handler (global_segment=2, m1=0x7fe48f7fdba8, m2=0x7fe48f7fdbb0, request=0x7fe48f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558bc7cc1728 in os_aio_handler (segment=2, m1=0x7fe48f7fdba8, m2=0x7fe48f7fdbb0, request=0x7fe48f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000558bc7f145ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000558bc7dba9fb in io_handler_thread (arg=0x558bc8f41630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fe4a40b8700 (LWP 72693)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc8186a8d in safe_cond_wait (cond=0x7fe488015608, mp=0x7fe488015558, file=0x558bc82f0c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558bc78de126 in inline_mysql_cond_wait (that=0x7fe488015608, mutex=0x7fe488015558, src_file=0x558bc82f0f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000558bc78e0dbe in Wsrep_thd_queue::pop_front (this=0x7fe488015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000558bc78df7de in wsrep_rollback_process (rollbacker=0x7fe488000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000558bc78d031f in start_wsrep_THD (arg=0x558bc9a76340) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fe48cff9700 (LWP 72742)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc7cc8b76 in os_event::timed_wait (this=0x558bc9d78dd0, abstime=0x7fe48cff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558bc7cc8e8d in os_event::wait_time_low (this=0x558bc9d78dd0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558bc7cc902d in os_event_wait_time_low (event=0x558bc9d78dd0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558bc7efe24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fe49f7fe700 (LWP 72692)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe4a7579867 in gu::Lock::wait (cond=..., this=0x7fe49f7fd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fe4a757c3ab in RecvBuf::front (this=this@entry=0x558bc9a668a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fe4a7579f46 in gcomm_recv (backend=, msg=0x558bc9a26830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fe4a756deec in core_msg_recv (backend=backend@entry=0x558bc9a26978, recv_msg=recv_msg@entry=0x558bc9a26830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fe4a7571164 in gcs_core_recv (conn=0x558bc9a267d0, recv_act=recv_act@entry=0x7fe49f7fdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fe4a75783d8 in gcs_recv_thread (arg=0x558bc9a26550) at gcs/src/gcs.cpp:1398 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fe49cbf7700 (LWP 72729)): #0 0x00007fe4af261644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558bc7cbc578 in LinuxAIOHandler::collect (this=0x7fe49cbf6b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558bc7cbcaa1 in LinuxAIOHandler::poll (this=0x7fe49cbf6b00, m1=0x7fe49cbf6ba8, m2=0x7fe49cbf6bb0, request=0x7fe49cbf6be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558bc7cbcc04 in os_aio_linux_handler (global_segment=0, m1=0x7fe49cbf6ba8, m2=0x7fe49cbf6bb0, request=0x7fe49cbf6be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558bc7cc1728 in os_aio_handler (segment=0, m1=0x7fe49cbf6ba8, m2=0x7fe49cbf6bb0, request=0x7fe49cbf6be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000558bc7f145ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000558bc7dba9fb in io_handler_thread (arg=0x558bc8f41620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fe49e449700 (LWP 72728)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc8186d8f in safe_cond_timedwait (cond=0x558bc9752ea0 , mp=0x558bc9752de0 , abstime=0x7fe49e448cb0, file=0x558bc85ca6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558bc7ff0bc1 in inline_mysql_cond_timedwait (that=0x558bc9752ea0 , mutex=0x558bc9752de0 , abstime=0x7fe49e448cb0, src_file=0x558bc85ca738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000558bc7ff10e5 in my_service_thread_sleep (control=0x558bc8d07b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000558bc7fe557e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000558bc8119555 in pfs_spawn_thread (arg=0x558bc9b266c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fe4a653c700 (LWP 72662)): #0 0x00007fe4af694965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe4a759efd7 in gu::Lock::wait (cond=..., this=0x7fe4a653bbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fe4a759f069 in galera::ServiceThd::thd_func (arg=0x558bc9a1e998) at galera/src/galera_service_thd.cpp:30 #3 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fe4a80e9700 (LWP 72661)): #0 0x00007fe4af694d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558bc8186d8f in safe_cond_timedwait (cond=0x558bc9765640 , mp=0x558bc9765580 , abstime=0x7fe4a80e8d70, file=0x558bc860e920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558bc8187d49 in inline_mysql_cond_timedwait (that=0x558bc9765640 , mutex=0x558bc9765580 , abstime=0x7fe4a80e8d70, src_file=0x558bc860e968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000558bc81888ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000558bc8119555 in pfs_spawn_thread (arg=0x558bc99e7730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fe49ffff700 (LWP 72691)): #0 0x00007fe4ada4f483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe4a75406f8 in asio::detail::epoll_reactor::run (this=0x558bc9a66fc0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fe4a7544df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x558bc9a66ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fe4a75450a2 in asio::detail::task_io_service::run (this=0x558bc9a66ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fe4a755cafd in asio::io_service::run (this=0x558bc9a65fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fe4a755d251 in gcomm::AsioProtonet::event_loop (this=0x558bc9a65f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fe4a757b2ed in GCommConn::run (this=0x558bc9a666c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fe4a757e399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fe4af690dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fe4ada4eead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fe4afaa1880 (LWP 72660)): #0 0x00007fe4af6959d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000558bc818145e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000558bc79bf5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fe4ada4420d in poll () from /lib64/libc.so.6 #5 0x0000558bc755fa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000558bc755ed25 in mysqld_main (argc=161, argv=0x558bc989fd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000558bc7552ec5 in main (argc=23, argv=0x7fffa182aea8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.73076' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_load_data-innodb/mysqld.2/data/core.73076 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 73076] [New LWP 73105] [New LWP 73557] [New LWP 73149] [New LWP 73078] [New LWP 73569] [New LWP 73556] [New LWP 73554] [New LWP 73576] [New LWP 73571] [New LWP 73577] [New LWP 73558] [New LWP 73611] [New LWP 73578] [New LWP 73560] [New LWP 73579] [New LWP 73568] [New LWP 73586] [New LWP 73572] [New LWP 73612] [New LWP 73585] [New LWP 73152] [New LWP 73587] [New LWP 73559] [New LWP 73608] [New LWP 73120] [New LWP 73584] [New LWP 73570] [New LWP 73555] [New LWP 73151] [New LWP 73150] [New LWP 73561] [New LWP 73575] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007faaf198d9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007faaf198d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056202a73745e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000562029f755a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007faaefd3c20d in poll () from /lib64/libc.so.6 #5 0x0000562029b15a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000562029b14d25 in mysqld_main (argc=162, argv=0x56202d66bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000562029b08ec5 in main (argc=23, argv=0x7ffc842300c8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7faaa77fe700 (LWP 73575)): #0 0x00007faaf198fd1d in fsync () from /lib64/libpthread.so.0 #1 0x000056202a273c27 in os_file_fsync_posix (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056202a273f7d in os_file_flush_func (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056202a4bc4aa in pfs_os_file_flush_func (file=..., src_file=0x56202ab36b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056202a4bf225 in fil_flush_low (space=0x56202db45460, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056202a4caaff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000056202a24f321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000056202a24fd83 in log_write_up_to (lsn=3258699, flush_to_disk=true, rotate_key=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000056202a251116 in log_checkpoint (sync=true, write_always=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1494 #9 0x000056202a36dd7d in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2178 #10 0x000056202a36e643 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #11 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7faaaeffd700 (LWP 73561)): #0 0x00007faaf198fd1d in fsync () from /lib64/libpthread.so.0 #1 0x000056202a273c27 in os_file_fsync_posix (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056202a273f7d in os_file_flush_func (file=17) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056202a4bc4aa in pfs_os_file_flush_func (file=..., src_file=0x56202ab36b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056202a4bf225 in fil_flush_low (space=0x56202db450c0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000056202a4caaff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000056202a447dc5 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x000056202a4524ad in buf_flush_end (buf_pool=0x56202da20db0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #8 0x000056202a45266b in buf_flush_do_batch (buf_pool=0x56202da20db0, type=BUF_FLUSH_LIST, min_n=5, lsn_limit=18446744073709551615, n=0x7faaaeffcb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #9 0x000056202a455355 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #10 0x000056202a4562a3 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #11 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7faae8133700 (LWP 73150)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a73ca8d in safe_cond_wait (cond=0x7faad0017b68, mp=0x7faad0017ab8, file=0x56202a8a6c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562029e94126 in inline_mysql_cond_wait (that=0x7faad0017b68, mutex=0x7faad0017ab8, src_file=0x56202a8a6f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562029e96dbe in Wsrep_thd_queue::pop_front (this=0x7faad0017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000562029e957de in wsrep_rollback_process (rollbacker=0x7faad0001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000562029e8631f in start_wsrep_THD (arg=0x56202d84cb90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7faae80e8700 (LWP 73151)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a73ca8d in safe_cond_wait (cond=0x7faacc015608, mp=0x7faacc015558, file=0x56202a8a6c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562029e94126 in inline_mysql_cond_wait (that=0x7faacc015608, mutex=0x7faacc015558, src_file=0x56202a8a6f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000562029e96dbe in Wsrep_thd_queue::pop_front (this=0x7faacc015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000562029e95b9e in wsrep_post_rollback_process (post_rollbacker=0x7faacc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000562029e8631f in start_wsrep_THD (arg=0x56202d84c460) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7faaba7ad700 (LWP 73555)): #0 0x00007faaf1559644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056202a272578 in LinuxAIOHandler::collect (this=0x7faaba7acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056202a272aa1 in LinuxAIOHandler::poll (this=0x7faaba7acb00, m1=0x7faaba7acba8, m2=0x7faaba7acbb0, request=0x7faaba7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056202a272c04 in os_aio_linux_handler (global_segment=0, m1=0x7faaba7acba8, m2=0x7faaba7acbb0, request=0x7faaba7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056202a277728 in os_aio_handler (segment=0, m1=0x7faaba7acba8, m2=0x7faaba7acbb0, request=0x7faaba7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056202a4ca5ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056202a3709fb in io_handler_thread (arg=0x56202b4f7620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7faaad7fa700 (LWP 73570)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27eb76 in os_event::timed_wait (this=0x56202d93b530, abstime=0x7faaad7f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056202a27ee8d in os_event::wait_time_low (this=0x56202d93b530, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056202a27f02d in os_event_wait_time_low (event=0x56202d93b530, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056202a36cbcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7faaa4ff9700 (LWP 73584)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27f25b in os_event::wait (this=0x56202d93b5b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056202a27ece1 in os_event::wait_low (this=0x56202d93b5b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056202a27f052 in os_event_wait_low (event=0x56202d93b5b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056202a44afb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7faadac7c700 (LWP 73120)): #0 0x00007faaefd47483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007faae98386f8 in asio::detail::epoll_reactor::run (this=0x56202d82b000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007faae983cdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x56202d82aea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007faae983d0a2 in asio::detail::task_io_service::run (this=0x56202d82aea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007faae9854afd in asio::io_service::run (this=0x56202d829bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007faae9855251 in gcomm::AsioProtonet::event_loop (this=0x56202d829b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007faae98732ed in GCommConn::run (this=0x56202d82a630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007faae9876399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7faad8140700 (LWP 73608)): #0 0x00007faaefc8040a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000562029b08f44 in my_sigwait (set=0x7faad813fcf0, sig=0x7faad813fcb4, code=0x7faad813fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000562029b0f584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000056202a6cf555 in pfs_spawn_thread (arg=0x56202d8fed30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7faaaffff700 (LWP 73559)): #0 0x00007faaf1559644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056202a272578 in LinuxAIOHandler::collect (this=0x7faaafffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056202a272aa1 in LinuxAIOHandler::poll (this=0x7faaafffeb00, m1=0x7faaafffeba8, m2=0x7faaafffebb0, request=0x7faaafffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056202a272c04 in os_aio_linux_handler (global_segment=4, m1=0x7faaafffeba8, m2=0x7faaafffebb0, request=0x7faaafffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056202a277728 in os_aio_handler (segment=4, m1=0x7faaafffeba8, m2=0x7faaafffebb0, request=0x7faaafffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056202a4ca5ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056202a3709fb in io_handler_thread (arg=0x56202b4f7640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7faa877fe700 (LWP 73587)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a73ca8d in safe_cond_wait (cond=0x7faa877fdc70, mp=0x7faa877fdcb0, file=0x56202aa0ad40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056202a196525 in inline_mysql_cond_wait (that=0x7faa877fdc70, mutex=0x7faa877fdcb0, src_file=0x56202aa0c6d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056202a19b161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000056202a6cf555 in pfs_spawn_thread (arg=0x56202db7d5b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7faae809d700 (LWP 73152)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27f25b in os_event::wait (this=0x56202dad3a20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056202a27ece1 in os_event::wait_low (this=0x56202dad3a20, reset_sig_count=3033) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056202a27f052 in os_event_wait_low (event=0x56202dad3a20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056202a24f8b3 in log_write_up_to (lsn=3258612, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000056202a3bae99 in trx_flush_log_if_needed_low (lsn=3258612) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #6 0x000056202a3baee2 in trx_flush_log_if_needed (lsn=3258612, trx=0x7faabb3ff140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #7 0x000056202a3bcd0b in trx_commit_complete_for_mysql (trx=0x7faabb3ff140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #8 0x000056202a1a15b2 in innobase_commit (hton=0x56202d90f8d0, thd=0x7faac8000b00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #9 0x0000562029f7b0d7 in commit_one_phase_2 (thd=0x7faac8000b00, all=true, trans=0x7faac8004050, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #10 0x0000562029f7afd9 in ha_commit_one_phase (thd=0x7faac8000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #11 0x0000562029f7a355 in ha_commit_trans (thd=0x7faac8000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #12 0x0000562029dbfe8e in trans_commit (thd=0x7faac8000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #13 0x0000562029e733d6 in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7faae809c4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:264 #14 0x000056202a7d9b47 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:67 #15 0x000056202a7da5a7 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:241 #16 0x000056202a7dd8fd in wsrep::server_state::on_apply (this=0x56202d7b4950, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:944 #17 0x000056202a7f2fc7 in wsrep::high_priority_service::apply (this=0x7faae809c4d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #18 0x000056202a7f06ae in (anonymous namespace)::apply_cb (ctx=0x7faae809c4d0, wsh=0x7faae809b3e0, flags=0, buf=0x7faae809b420, meta=0x7faae809b6b0, exit_loop=0x7faae809b5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #19 0x00007faae98803b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7faac804acd0, recv_ctx=recv_ctx@entry=0x7faae809c4d0, apply_cb=0x56202a7f048b <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7faae809b5ef: false) at galera/src/trx_handle.cpp:418 #20 0x00007faae98b2d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x56202d7ea1f0, recv_ctx=recv_ctx@entry=0x7faae809c4d0, ts=...) at galera/src/replicator_smm.cpp:489 #21 0x00007faae98b8a40 in galera::ReplicatorSMM::process_trx (this=0x56202d7ea1f0, recv_ctx=0x7faae809c4d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #22 0x00007faae9899c6b in galera::GcsActionSource::process_writeset (this=this@entry=0x56202d822e60, recv_ctx=0x7faae809c4d0, act=..., exit_loop=@0x7faae809c29e: false) at galera/src/gcs_action_source.cpp:62 #23 0x00007faae9899e45 in galera::GcsActionSource::dispatch (this=this@entry=0x56202d822e60, recv_ctx=recv_ctx@entry=0x7faae809c4d0, act=..., exit_loop=@0x7faae809c29e: false) at galera/src/gcs_action_source.cpp:109 #24 0x00007faae989a21d in galera::GcsActionSource::process (this=0x56202d822e60, recv_ctx=0x7faae809c4d0, exit_loop=@0x7faae809c29e: false) at galera/src/gcs_action_source.cpp:182 #25 0x00007faae98b9033 in galera::ReplicatorSMM::async_recv (this=0x56202d7ea1f0, recv_ctx=0x7faae809c4d0) at galera/src/replicator_smm.cpp:383 #26 0x000056202a7f136c in wsrep::wsrep_provider_v26::run_applier (this=0x56202d75fe50, applier_ctx=0x7faae809c4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #27 0x0000562029e946d4 in wsrep_replication_process (thd=0x7faac8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #28 0x0000562029e8631f in start_wsrep_THD (arg=0x56202d842bf0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #29 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7faa87fff700 (LWP 73585)): #0 0x00007faaf198fe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056202a27f55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056202a425b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7faad80aa700 (LWP 73612)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a73cd8f in safe_cond_timedwait (cond=0x56202b47c1a0 , mp=0x56202b47eba0 , abstime=0x7faad80a9c60, file=0x56202a7f85c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562029b09845 in inline_mysql_cond_timedwait (that=0x56202b47c1a0 , mutex=0x56202b47eba0 , abstime=0x7faad80a9c60, src_file=0x56202a7fa638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562029b0e61b in cache_thread (thd=0x7faa6c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000562029b0e8c9 in one_thread_per_connection_end (thd=0x7faa6c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000562029dab0f3 in do_handle_one_connection (connect=0x56202d901f60) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000562029daadad in handle_one_connection (arg=0x56202d901f60) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000056202a6cf555 in pfs_spawn_thread (arg=0x56202dcc2460) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7faaa7fff700 (LWP 73572)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27eb76 in os_event::timed_wait (this=0x56202dadba70, abstime=0x7faaa7ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056202a27ee8d in os_event::wait_time_low (this=0x56202dadba70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056202a27f02d in os_event_wait_time_low (event=0x56202dadba70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056202a3cd403 in ib_wqueue_timedwait (wq=0x56202dadb940, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056202a50df9a in fts_optimize_thread (arg=0x56202dadb940) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7faa7ffff700 (LWP 73586)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27f25b in os_event::wait (this=0x56202d93b6b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056202a27ece1 in os_event::wait_low (this=0x56202d93b6b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056202a27f052 in os_event_wait_low (event=0x56202d93b6b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056202a433c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7faaadffb700 (LWP 73568)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27eb76 in os_event::timed_wait (this=0x56202db139e0, abstime=0x7faaadffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056202a27ee8d in os_event::wait_time_low (this=0x56202db139e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056202a27f02d in os_event_wait_time_low (event=0x56202db139e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056202a24b9e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7faaa57fa700 (LWP 73579)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27f25b in os_event::wait (this=0x56202d93b3b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056202a27ece1 in os_event::wait_low (this=0x56202d93b3b0, reset_sig_count=8728) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056202a27f052 in os_event_wait_low (event=0x56202d93b3b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056202a36aee5 in srv_resume_thread (slot=0x56202b2b97b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056202a36eae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7faaaf7fe700 (LWP 73560)): #0 0x00007faaf1559644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056202a272578 in LinuxAIOHandler::collect (this=0x7faaaf7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056202a272aa1 in LinuxAIOHandler::poll (this=0x7faaaf7fdb00, m1=0x7faaaf7fdba8, m2=0x7faaaf7fdbb0, request=0x7faaaf7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056202a272c04 in os_aio_linux_handler (global_segment=5, m1=0x7faaaf7fdba8, m2=0x7faaaf7fdbb0, request=0x7faaaf7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056202a277728 in os_aio_handler (segment=5, m1=0x7faaaf7fdba8, m2=0x7faaaf7fdbb0, request=0x7faaaf7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056202a4ca5ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056202a3709fb in io_handler_thread (arg=0x56202b4f7648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7faaa5ffb700 (LWP 73578)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27f25b in os_event::wait (this=0x56202d93b430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056202a27ece1 in os_event::wait_low (this=0x56202d93b430, reset_sig_count=8124) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056202a27f052 in os_event_wait_low (event=0x56202d93b430, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056202a36aee5 in srv_resume_thread (slot=0x56202b2b97f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056202a36eae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7faad80f5700 (LWP 73611)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a73ca8d in safe_cond_wait (cond=0x56202b47c1e0 , mp=0x56202b47f560 , file=0x56202a807dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000562029b3a4ee in inline_mysql_cond_wait (that=0x56202b47c1e0 , mutex=0x56202b47f560 , src_file=0x56202a808930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000562029b3bd80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056202a6cf555 in pfs_spawn_thread (arg=0x56202dcaf9b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7faab8faa700 (LWP 73558)): #0 0x00007faaf1559644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056202a272578 in LinuxAIOHandler::collect (this=0x7faab8fa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056202a272aa1 in LinuxAIOHandler::poll (this=0x7faab8fa9b00, m1=0x7faab8fa9ba8, m2=0x7faab8fa9bb0, request=0x7faab8fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056202a272c04 in os_aio_linux_handler (global_segment=3, m1=0x7faab8fa9ba8, m2=0x7faab8fa9bb0, request=0x7faab8fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056202a277728 in os_aio_handler (segment=3, m1=0x7faab8fa9ba8, m2=0x7faab8fa9bb0, request=0x7faab8fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056202a4ca5ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056202a3709fb in io_handler_thread (arg=0x56202b4f7638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7faaa67fc700 (LWP 73577)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27f25b in os_event::wait (this=0x56202d93b330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056202a27ece1 in os_event::wait_low (this=0x56202d93b330, reset_sig_count=9798) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056202a27f052 in os_event_wait_low (event=0x56202d93b330, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056202a36aee5 in srv_resume_thread (slot=0x56202b2b9780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056202a36eae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7faaacff9700 (LWP 73571)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27eb76 in os_event::timed_wait (this=0x56202db45aa0, abstime=0x7faaacff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056202a27ee8d in os_event::wait_time_low (this=0x56202db45aa0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056202a27f02d in os_event_wait_time_low (event=0x56202db45aa0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056202a4b424a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7faaa6ffd700 (LWP 73576)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27f25b in os_event::wait (this=0x56202d93b2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056202a27ece1 in os_event::wait_low (this=0x56202d93b2b0, reset_sig_count=2330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056202a27f052 in os_event_wait_low (event=0x56202d93b2b0, reset_sig_count=2330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056202a36aee5 in srv_resume_thread (slot=0x56202b2b9748 , sig_count=2330, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000056202a36eef6 in srv_purge_coordinator_suspend (slot=0x56202b2b9748 , rseg_history_len=18) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000056202a36f18c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7faabbfff700 (LWP 73554)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a73cd8f in safe_cond_timedwait (cond=0x56202bd08ea0 , mp=0x56202bd08de0 , abstime=0x7faabbffecb0, file=0x56202ab806f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056202a5a6bc1 in inline_mysql_cond_timedwait (that=0x56202bd08ea0 , mutex=0x56202bd08de0 , abstime=0x7faabbffecb0, src_file=0x56202ab80738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056202a5a70e5 in my_service_thread_sleep (control=0x56202b2bdb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056202a59b57e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056202a6cf555 in pfs_spawn_thread (arg=0x56202d8f35a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7faab9fac700 (LWP 73556)): #0 0x00007faaf1559644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056202a272578 in LinuxAIOHandler::collect (this=0x7faab9fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056202a272aa1 in LinuxAIOHandler::poll (this=0x7faab9fabb00, m1=0x7faab9fabba8, m2=0x7faab9fabbb0, request=0x7faab9fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056202a272c04 in os_aio_linux_handler (global_segment=1, m1=0x7faab9fabba8, m2=0x7faab9fabbb0, request=0x7faab9fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056202a277728 in os_aio_handler (segment=1, m1=0x7faab9fabba8, m2=0x7faab9fabbb0, request=0x7faab9fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056202a4ca5ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056202a3709fb in io_handler_thread (arg=0x56202b4f7628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7faaae7fc700 (LWP 73569)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a27eb76 in os_event::timed_wait (this=0x56202d93b4b0, abstime=0x7faaae7fbba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056202a27ee8d in os_event::wait_time_low (this=0x56202d93b4b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056202a27f02d in os_event_wait_time_low (event=0x56202d93b4b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056202a36d0ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7faaea3e1700 (LWP 73078)): #0 0x00007faaf198cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056202a73cd8f in safe_cond_timedwait (cond=0x56202bd1b640 , mp=0x56202bd1b580 , abstime=0x7faaea3e0d70, file=0x56202abc4920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056202a73dd49 in inline_mysql_cond_timedwait (that=0x56202bd1b640 , mutex=0x56202bd1b580 , abstime=0x7faaea3e0d70, src_file=0x56202abc4968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056202a73e8ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056202a6cf555 in pfs_spawn_thread (arg=0x56202d7b37d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7faada47b700 (LWP 73149)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faae9871867 in gu::Lock::wait (cond=..., this=0x7faada47a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007faae98743ab in RecvBuf::front (this=this@entry=0x56202d82a818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007faae9871f46 in gcomm_recv (backend=, msg=0x56202d7f2720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007faae9865eec in core_msg_recv (backend=backend@entry=0x56202d7f2868, recv_msg=recv_msg@entry=0x56202d7f2720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007faae9869164 in gcs_core_recv (conn=0x56202d7f26c0, recv_act=recv_act@entry=0x7faada47ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007faae98703d8 in gcs_recv_thread (arg=0x56202d7f2440) at gcs/src/gcs.cpp:1398 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7faab97ab700 (LWP 73557)): #0 0x00007faaf1559644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056202a272578 in LinuxAIOHandler::collect (this=0x7faab97aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056202a272aa1 in LinuxAIOHandler::poll (this=0x7faab97aab00, m1=0x7faab97aaba8, m2=0x7faab97aabb0, request=0x7faab97aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056202a272c04 in os_aio_linux_handler (global_segment=2, m1=0x7faab97aaba8, m2=0x7faab97aabb0, request=0x7faab97aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056202a277728 in os_aio_handler (segment=2, m1=0x7faab97aaba8, m2=0x7faab97aabb0, request=0x7faab97aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000056202a4ca5ad in fil_aio_wait (segment=2) at /home/stepan/galworker[3] Trying to dump core for [mysqltest - pid: 79475, winpid: 79475] worker[3] Trying to dump core for [mysqld.1 - pid: 78164, winpid: 78164] worker[3] Trying to dump core for [mysqld.2 - pid: 78661, winpid: 78661] era/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000056202a3709fb in io_handler_thread (arg=0x56202b4f7630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7faae9235700 (LWP 73105)): #0 0x00007faaf198c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faae9896fd7 in gu::Lock::wait (cond=..., this=0x7faae9234bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007faae9897069 in galera::ServiceThd::thd_func (arg=0x56202d7ea908) at galera/src/galera_service_thd.cpp:30 #3 0x00007faaf1988dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007faaefd46ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7faaf1d99880 (LWP 73076)): #0 0x00007faaf198d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056202a73745e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000562029f755a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007faaefd3c20d in poll () from /lib64/libc.so.6 #5 0x0000562029b15a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000562029b14d25 in mysqld_main (argc=162, argv=0x56202d66bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000562029b08ec5 in main (argc=23, argv=0x7ffc842300c8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_load_data-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_load_data galera_sr.galera_sr_kill_connection Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_myisam 'innodb' w2 [ pass ] 4874 galera_sr.GCF-1051 'innodb' w2 [ pass ] 6889 galera_sr.galera_sr_load_data_splitting 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 13:01:15 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr_load_data_splitting.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 512; SET GLOBAL wsrep_load_data_splitting = TRUE; Warnings: Warning 1287 '@@wsrep_load_data_splitting' is deprecated and will be removed in a future release CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; connection node_2; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.78165' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.1/data/core.78165 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 78165] [New LWP 78193] [New LWP 78194] [New LWP 78180] [New LWP 78203] [New LWP 78247] [New LWP 78206] [New LWP 78285] [New LWP 78246] [New LWP 78286] [New LWP 78248] [New LWP 78293] [New LWP 78294] [New LWP 78249] [New LWP 78251] [New LWP 78288] [New LWP 78289] [New LWP 78202] [New LWP 78291] [New LWP 78295] [New LWP 78290] [New LWP 78245] [New LWP 78287] [New LWP 78292] [New LWP 78250] [New LWP 78457] [New LWP 78298] [New LWP 78456] [New LWP 78363] [New LWP 78297] [New LWP 78296] [New LWP 78204] [New LWP 78252] [New LWP 79599] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fa7b73819d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fa7b73819d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563d6729d45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563d66adb5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa7b573020d in poll () from /lib64/libc.so.6 #5 0x0000563d6667ba8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000563d6667ad25 in mysqld_main (argc=161, argv=0x563d6a5f9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000563d6666eec5 in main (argc=23, argv=0x7ffcfe6228a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fa7a49ef700 (LWP 79599)): #0 0x00007fa7b7383d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000563d66dd9c27 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000563d66dd9f7d in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000563d670224aa in pfs_os_file_flush_func (file=..., src_file=0x563d6769cb50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000563d67025225 in fil_flush_low (space=0x563d6aaca880, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000563d67030aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000563d66db5321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000563d66db5d83 in log_write_up_to (lsn=5844603, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000563d66f20e99 in trx_flush_log_if_needed_low (lsn=5844603) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000563d66f20ee2 in trx_flush_log_if_needed (lsn=5844603, trx=0x7fa7a52d9390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000563d66f22d0b in trx_commit_complete_for_mysql (trx=0x7fa7a52d9390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000563d66d075b2 in innobase_commit (hton=0x563d6a894b00, thd=0x7fa72805a970, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x0000563d66ae10d7 in commit_one_phase_2 (thd=0x7fa72805a970, all=true, trans=0x7fa72805dec0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000563d66ae0fd9 in ha_commit_one_phase (thd=0x7fa72805a970, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000563d66ae0355 in ha_commit_trans (thd=0x7fa72805a970, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000563d66925e8e in trans_commit (thd=0x7fa72805a970) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000563d66ce522a in Wsrep_storage_service::commit (this=0x7fa7280a7730, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x0000563d6734d5c9 in wsrep::transaction::certify_fragment (this=0x7fa7280069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x0000563d6734cd2d in wsrep::transaction::streaming_step (this=0x7fa7280069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x0000563d6734a0e8 in wsrep::transaction::after_row (this=0x7fa7280069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x0000563d66af3446 in wsrep::client_state::after_row (this=0x7fa728006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x0000563d66adc6c3 in wsrep_after_row (thd=0x7fa728000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x0000563d66aee5c8 in wsrep_after_row (thd=0x7fa728000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x0000563d66aee9b9 in handler::ha_write_row (this=0x7fa728026998, buf=0x7fa7280217e0 "\377]\033") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x0000563d66764443 in write_record (thd=0x7fa728000b00, table=0x7fa728025d60, info=0x7fa7a49ecb10) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x0000563d667973c3 in read_sep_field (thd=0x7fa728000b00, info=..., table_list=0x7fa728014090, fields_vars=..., set_fields=..., set_values=..., read_info=..., enclosed=..., skip_lines=0, ignore_check_option_errors=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:1152 #26 0x0000563d6679597e in mysql_load (thd=0x7fa728000b00, ex=0x7fa728014008, table_list=0x7fa728014090, fields_vars=..., set_fields=..., set_values=..., handle_duplicates=DUP_ERROR, ignore=false, read_file_from_client=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:660 #27 0x0000563d667ac75d in mysql_execute_command (thd=0x7fa728000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5148 #28 0x0000563d667b6935 in mysql_parse (thd=0x7fa728000b00, rawbuf=0x7fa728013e58 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7fa7a49ee0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #29 0x0000563d667b5f68 in wsrep_mysql_parse (thd=0x7fa728000b00, rawbuf=0x7fa728013e58 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=113, parser_state=0x7fa7a49ee0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #30 0x0000563d667a1e25 in dispatch_command (command=COM_QUERY, thd=0x7fa728000b00, packet=0x7fa72800a431 "", packet_length=114, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #31 0x0000563d667a080d in do_command (thd=0x7fa728000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #32 0x0000563d66911069 in do_handle_one_connection (connect=0x563d6a887360) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #33 0x0000563d66910dad in handle_one_connection (arg=0x563d6a887360) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #34 0x0000563d67235555 in pfs_spawn_thread (arg=0x563d6ac499f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #35 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #36 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fa7957fa700 (LWP 78252)): #0 0x00007fa7b7383d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000563d66dd9c27 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000563d66dd9f7d in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000563d670224aa in pfs_os_file_flush_func (file=..., src_file=0x563d6769cb50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000563d67025225 in fil_flush_low (space=0x563d6aaca4e0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000563d67030aff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000563d66faddc5 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1072 #7 0x0000563d66fb5f98 in buf_flush_page (buf_pool=0x563d6a9a61d0, bpage=0x7fa7a4abbc70, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1139 #8 0x0000563d66fb6a06 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=5, n_to_flush=10) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1389 #9 0x0000563d66fb6d89 in buf_flush_page_and_try_neighbors (bpage=0x7fa7a4abbc70, flush_type=BUF_FLUSH_LIST, n_to_flush=10, count=0x7fa7957f98c8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1461 #10 0x0000563d66fb79a2 in buf_do_flush_list_batch (buf_pool=0x563d6a9a61d0, min_n=10, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1719 #11 0x0000563d66fb7f9a in buf_flush_batch (buf_pool=0x563d6a9a61d0, flush_type=BUF_FLUSH_LIST, min_n=10, lsn_limit=18446744073709551615, n=0x7fa7957f9b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1789 #12 0x0000563d66fb865a in buf_flush_do_batch (buf_pool=0x563d6a9a61d0, type=BUF_FLUSH_LIST, min_n=10, lsn_limit=18446744073709551615, n=0x7fa7957f9b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1947 #13 0x0000563d66fbb355 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2750 #14 0x0000563d66fbc2a3 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #15 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fa7ac6da700 (LWP 78204)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d672a2a8d in safe_cond_wait (cond=0x7fa79c015608, mp=0x7fa79c015558, file=0x563d6740cc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563d669fa126 in inline_mysql_cond_wait (that=0x7fa79c015608, mutex=0x7fa79c015558, src_file=0x563d6740cf80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563d669fcdbe in Wsrep_thd_queue::pop_front (this=0x7fa79c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563d669fbb9e in wsrep_post_rollback_process (post_rollbacker=0x7fa79c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000563d669ec31f in start_wsrep_THD (arg=0x563d6a7d0d70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fa761ffb700 (LWP 78296)): #0 0x00007fa7b7383e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563d66de555d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563d66f8bb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fa7617fa700 (LWP 78297)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de525b in os_event::wait (this=0x563d6a8c0ad0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563d66de4ce1 in os_event::wait_low (this=0x563d6a8c0ad0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563d66de5052 in os_event_wait_low (event=0x563d6a8c0ad0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563d66f99c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fa7ac080700 (LWP 78363)): #0 0x00007fa7b567440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000563d6666ef44 in my_sigwait (set=0x7fa7ac07fcf0, sig=0x7fa7ac07fcb4, code=0x7fa7ac07fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000563d66675584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000563d67235555 in pfs_spawn_thread (arg=0x563d6a883e40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fa7a4a85700 (LWP 78456)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d672a2a8d in safe_cond_wait (cond=0x563d67fe21e0 , mp=0x563d67fe5560 , file=0x563d6736ddc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563d666a04ee in inline_mysql_cond_wait (that=0x563d67fe21e0 , mutex=0x563d67fe5560 , src_file=0x563d6736e930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000563d666a1d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000563d67235555 in pfs_spawn_thread (arg=0x563d6ac36f40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fa760ff9700 (LWP 78298)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d672a2a8d in safe_cond_wait (cond=0x7fa760ff8c70, mp=0x7fa760ff8cb0, file=0x563d67570d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563d66cfc525 in inline_mysql_cond_wait (that=0x7fa760ff8c70, mutex=0x7fa760ff8cb0, src_file=0x563d675726d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563d66d01161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000563d67235555 in pfs_spawn_thread (arg=0x563d6aa66690) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fa7a4a3a700 (LWP 78457)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d672a2d8f in safe_cond_timedwait (cond=0x563d67fe21a0 , mp=0x563d67fe4ba0 , abstime=0x7fa7a4a39c60, file=0x563d6735e5c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563d6666f845 in inline_mysql_cond_timedwait (that=0x563d67fe21a0 , mutex=0x563d67fe4ba0 , abstime=0x7fa7a4a39c60, src_file=0x563d67360638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563d6667461b in cache_thread (thd=0x7fa730000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000563d666748c9 in one_thread_per_connection_end (thd=0x7fa730000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000563d669110f3 in do_handle_one_connection (connect=0x563d6ac25820) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000563d66910dad in handle_one_connection (arg=0x563d6ac25820) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000563d67235555 in pfs_spawn_thread (arg=0x563d6ac499f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fa7967fc700 (LWP 78250)): #0 0x00007fa7b6f4d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563d66dd8578 in LinuxAIOHandler::collect (this=0x7fa7967fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563d66dd8aa1 in LinuxAIOHandler::poll (this=0x7fa7967fbb00, m1=0x7fa7967fbba8, m2=0x7fa7967fbbb0, request=0x7fa7967fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563d66dd8c04 in os_aio_linux_handler (global_segment=4, m1=0x7fa7967fbba8, m2=0x7fa7967fbbb0, request=0x7fa7967fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563d66ddd728 in os_aio_handler (segment=4, m1=0x7fa7967fbba8, m2=0x7fa7967fbbb0, request=0x7fa7967fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563d670305ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563d66ed69fb in io_handler_thread (arg=0x563d6805d640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fa763fff700 (LWP 78292)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de525b in os_event::wait (this=0x563d6a8c0750) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563d66de4ce1 in os_event::wait_low (this=0x563d6a8c0750, reset_sig_count=9587) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563d66de5052 in os_event_wait_low (event=0x563d6a8c0750, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563d66ed0ee5 in srv_resume_thread (slot=0x563d67e1f780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563d66ed4ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fa776ffd700 (LWP 78287)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de4b76 in os_event::timed_wait (this=0x563d6a8c0950, abstime=0x7fa776ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563d66de4e8d in os_event::wait_time_low (this=0x563d6a8c0950, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563d66de502d in os_event_wait_time_low (event=0x563d6a8c0950, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563d66ed2bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fa7a5ed7700 (LWP 78245)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d672a2d8f in safe_cond_timedwait (cond=0x563d6886eea0 , mp=0x563d6886ede0 , abstime=0x7fa7a5ed6cb0, file=0x563d676e66f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563d6710cbc1 in inline_mysql_cond_timedwait (that=0x563d6886eea0 , mutex=0x563d6886ede0 , abstime=0x7fa7a5ed6cb0, src_file=0x563d676e6738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563d6710d0e5 in my_service_thread_sleep (control=0x563d67e23b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000563d6710157e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000563d67235555 in pfs_spawn_thread (arg=0x563d6a8787d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fa7757fa700 (LWP 78290)): #0 0x00007fa7b7383e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563d66de555d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563d66ed444d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000563d66ed457d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fa7627fc700 (LWP 78295)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de525b in os_event::wait (this=0x563d6a8c09d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563d66de4ce1 in os_event::wait_low (this=0x563d6a8c09d0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563d66de5052 in os_event_wait_low (event=0x563d6a8c09d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563d66fb0fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fa774ff9700 (LWP 78291)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de4b76 in os_event::timed_wait (this=0x563d6a8c06d0, abstime=0x7fa774ff8cb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563d66de4e8d in os_event::wait_time_low (this=0x563d6a8c06d0, time_in_usec=10000, reset_sig_count=2239) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563d66de502d in os_event_wait_time_low (event=0x563d6a8c06d0, time_in_usec=10000, reset_sig_count=2239) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563d66ed0ebe in srv_resume_thread (slot=0x563d67e1f748 , sig_count=2239, wait=true, timeout_usec=10000) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:898 #5 0x0000563d66ed4ef6 in srv_purge_coordinator_suspend (slot=0x563d67e1f748 , rseg_history_len=14) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000563d66ed518c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fa7a6e7d700 (LWP 78202)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa7af265867 in gu::Lock::wait (cond=..., this=0x7fa7a6e7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fa7af2683ab in RecvBuf::front (this=this@entry=0x563d6a7b8878, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fa7af265f46 in gcomm_recv (backend=, msg=0x563d6a780800, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fa7af259eec in core_msg_recv (backend=backend@entry=0x563d6a780948, recv_msg=recv_msg@entry=0x563d6a780800, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fa7af25d164 in gcs_core_recv (conn=0x563d6a7807a0, recv_act=recv_act@entry=0x7fa7a6e7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fa7af2643d8 in gcs_recv_thread (arg=0x563d6a780520) at gcs/src/gcs.cpp:1398 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fa775ffb700 (LWP 78289)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de4b76 in os_event::timed_wait (this=0x563d6ab11c00, abstime=0x7fa775ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563d66de4e8d in os_event::wait_time_low (this=0x563d6ab11c00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563d66de502d in os_event_wait_time_low (event=0x563d6ab11c00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563d66f33403 in ib_wqueue_timedwait (wq=0x563d6ab11ad0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000563d67073f9a in fts_optimize_thread (arg=0x563d6ab11ad0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fa7767fc700 (LWP 78288)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de4b76 in os_event::timed_wait (this=0x563d6aacaec0, abstime=0x7fa7767fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563d66de4e8d in os_event::wait_time_low (this=0x563d6aacaec0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563d66de502d in os_event_wait_time_low (event=0x563d6aacaec0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563d6701a24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fa795ffb700 (LWP 78251)): #0 0x00007fa7b6f4d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563d66dd8578 in LinuxAIOHandler::collect (this=0x7fa795ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563d66dd8aa1 in LinuxAIOHandler::poll (this=0x7fa795ffab00, m1=0x7fa795ffaba8, m2=0x7fa795ffabb0, request=0x7fa795ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563d66dd8c04 in os_aio_linux_handler (global_segment=5, m1=0x7fa795ffaba8, m2=0x7fa795ffabb0, request=0x7fa795ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563d66ddd728 in os_aio_handler (segment=5, m1=0x7fa795ffaba8, m2=0x7fa795ffabb0, request=0x7fa795ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563d670305ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563d66ed69fb in io_handler_thread (arg=0x563d6805d648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fa796ffd700 (LWP 78249)): #0 0x00007fa7b6f4d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563d66dd8578 in LinuxAIOHandler::collect (this=0x7fa796ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563d66dd8aa1 in LinuxAIOHandler::poll (this=0x7fa796ffcb00, m1=0x7fa796ffcba8, m2=0x7fa796ffcbb0, request=0x7fa796ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563d66dd8c04 in os_aio_linux_handler (global_segment=3, m1=0x7fa796ffcba8, m2=0x7fa796ffcbb0, request=0x7fa796ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563d66ddd728 in os_aio_handler (segment=3, m1=0x7fa796ffcba8, m2=0x7fa796ffcbb0, request=0x7fa796ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563d670305ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563d66ed69fb in io_handler_thread (arg=0x563d6805d638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fa762ffd700 (LWP 78294)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de525b in os_event::wait (this=0x563d6a8c07d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563d66de4ce1 in os_event::wait_low (this=0x563d6a8c07d0, reset_sig_count=8564) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563d66de5052 in os_event_wait_low (event=0x563d6a8c07d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563d66ed0ee5 in srv_resume_thread (slot=0x563d67e1f7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563d66ed4ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fa7637fe700 (LWP 78293)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de525b in os_event::wait (this=0x563d6a8c0850) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563d66de4ce1 in os_event::wait_low (this=0x563d6a8c0850, reset_sig_count=8235) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563d66de5052 in os_event_wait_low (event=0x563d6a8c0850, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563d66ed0ee5 in srv_resume_thread (slot=0x563d67e1f7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563d66ed4ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fa7977fe700 (LWP 78248)): #0 0x00007fa7b6f4d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563d66dd8578 in LinuxAIOHandler::collect (this=0x7fa7977fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563d66dd8aa1 in LinuxAIOHandler::poll (this=0x7fa7977fdb00, m1=0x7fa7977fdba8, m2=0x7fa7977fdbb0, request=0x7fa7977fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563d66dd8c04 in os_aio_linux_handler (global_segment=2, m1=0x7fa7977fdba8, m2=0x7fa7977fdbb0, request=0x7fa7977fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563d66ddd728 in os_aio_handler (segment=2, m1=0x7fa7977fdba8, m2=0x7fa7977fdbb0, request=0x7fa7977fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563d670305ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563d66ed69fb in io_handler_thread (arg=0x563d6805d630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fa7777fe700 (LWP 78286)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de4b76 in os_event::timed_wait (this=0x563d6a8c08d0, abstime=0x7fa7777fdba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563d66de4e8d in os_event::wait_time_low (this=0x563d6a8c08d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563d66de502d in os_event_wait_time_low (event=0x563d6a8c08d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563d66ed30ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fa7a4885700 (LWP 78246)): #0 0x00007fa7b6f4d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563d66dd8578 in LinuxAIOHandler::collect (this=0x7fa7a4884b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563d66dd8aa1 in LinuxAIOHandler::poll (this=0x7fa7a4884b00, m1=0x7fa7a4884ba8, m2=0x7fa7a4884bb0, request=0x7fa7a4884be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563d66dd8c04 in os_aio_linux_handler (global_segment=0, m1=0x7fa7a4884ba8, m2=0x7fa7a4884bb0, request=0x7fa7a4884be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563d66ddd728 in os_aio_handler (segment=0, m1=0x7fa7a4884ba8, m2=0x7fa7a4884bb0, request=0x7fa7a4884be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563d670305ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563d66ed69fb in io_handler_thread (arg=0x563d6805d620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fa777fff700 (LWP 78285)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d66de4b76 in os_event::timed_wait (this=0x563d6aa98e00, abstime=0x7fa777ffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563d66de4e8d in os_event::wait_time_low (this=0x563d6aa98e00, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563d66de502d in os_event_wait_time_low (event=0x563d6aa98e00, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563d66db19e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fa7ac68f700 (LWP 78206)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa7af17f132 in fifo_lock_get (q=q@entry=0x563d6a7a6ba0) at galerautils/src/gu_fifo.c:262 #2 0x00007fa7af17f7a1 in gu_fifo_get_head (q=0x563d6a7a6ba0, err=0x7fa7ac68e06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fa7af2610cc in gcs_recv (conn=0x563d6a780520, action=0x7fa7ac68e0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fa7af28e16c in galera::GcsActionSource::process (this=0x563d6a77ff00, recv_ctx=0x7fa7ac68e4d0, exit_loop=@0x7fa7ac68e29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fa7af2ad033 in galera::ReplicatorSMM::async_recv (this=0x563d6a778250, recv_ctx=0x7fa7ac68e4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000563d6735736c in wsrep::wsrep_provider_v26::run_applier (this=0x563d6a6ff360, applier_ctx=0x7fa7ac68e4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000563d669fa6d4 in wsrep_replication_process (thd=0x7fa788000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000563d669ec31f in start_wsrep_THD (arg=0x563d6a7d1410) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fa797fff700 (LWP 78247)): #0 0x00007fa7b6f4d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563d66dd8578 in LinuxAIOHandler::collect (this=0x7fa797ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563d66dd8aa1 in LinuxAIOHandler::poll (this=0x7fa797ffeb00, m1=0x7fa797ffeba8, m2=0x7fa797ffebb0, request=0x7fa797ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563d66dd8c04 in os_aio_linux_handler (global_segment=1, m1=0x7fa797ffeba8, m2=0x7fa797ffebb0, request=0x7fa797ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563d66ddd728 in os_aio_handler (segment=1, m1=0x7fa797ffeba8, m2=0x7fa797ffebb0, request=0x7fa797ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563d670305ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563d66ed69fb in io_handler_thread (arg=0x563d6805d628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fa7ac725700 (LWP 78203)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d672a2a8d in safe_cond_wait (cond=0x7fa798017b68, mp=0x7fa798017ab8, file=0x563d6740cc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563d669fa126 in inline_mysql_cond_wait (that=0x7fa798017b68, mutex=0x7fa798017ab8, src_file=0x563d6740cf80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563d669fcdbe in Wsrep_thd_queue::pop_front (this=0x7fa798017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563d669fb7de in wsrep_rollback_process (rollbacker=0x7fa798001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000563d669ec31f in start_wsrep_THD (arg=0x563d6a7c8490) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fa7afdd5700 (LWP 78180)): #0 0x00007fa7b7380d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563d672a2d8f in safe_cond_timedwait (cond=0x563d68881640 , mp=0x563d68881580 , abstime=0x7fa7afdd4d70, file=0x563d6772a920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563d672a3d49 in inline_mysql_cond_timedwait (that=0x563d68881640 , mutex=0x563d68881580 , abstime=0x7fa7afdd4d70, src_file=0x563d6772a968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563d672a48ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000563d67235555 in pfs_spawn_thread (arg=0x563d6a741730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fa7a767e700 (LWP 78194)): #0 0x00007fa7b573b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa7af22c6f8 in asio::detail::epoll_reactor::run (this=0x563d6a7b9010, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fa7af230df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x563d6a7b8eb0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fa7af2310a2 in asio::detail::task_io_service::run (this=0x563d6a7b8eb0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fa7af248afd in asio::io_service::run (this=0x563d6a7b7fb0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fa7af249251 in gcomm::AsioProtonet::event_loop (this=0x563d6a7b7f10, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fa7af2672ed in GCommConn::run (this=0x563d6a7b8690) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fa7af26a399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fa7ae228700 (LWP 78193)): #0 0x00007fa7b7380965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa7af28afd7 in gu::Lock::wait (cond=..., this=0x7fa7ae227bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fa7af28b069 in galera::ServiceThd::thd_func (arg=0x563d6a778968) at galera/src/galera_service_thd.cpp:30 #3 0x00007fa7b737cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa7b573aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fa7b778d880 (LWP 78165)): #0 0x00007fa7b73819d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563d6729d45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563d66adb5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa7b573020d in poll () from /lib64/libc.so.6 #5 0x0000563d6667ba8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000563d6667ad25 in mysqld_main (argc=161, argv=0x563d6a5f9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000563d6666eec5 in main (argc=23, argv=0x7ffcfe6228a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.78662' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.2/data/core.78662 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 78662] [New LWP 78664] [New LWP 78730] [New LWP 78732] [New LWP 79236] [New LWP 78733] [New LWP 79241] [New LWP 79235] [New LWP 79266] [New LWP 79238] [New LWP 79291] [New LWP 79265] [New LWP 79268] [New LWP 79292] [New LWP 79310] [New LWP 78703] [New LWP 79234] [New LWP 79237] [New LWP 79239] [New LWP 79240] [New LWP 79267] [New LWP 79275] [New LWP 79277] [New LWP 79280] [New LWP 79293] [New LWP 79311] [New LWP 79309] [New LWP 79294] [New LWP 79278] [New LWP 79276] [New LWP 79264] [New LWP 78663] [New LWP 78731] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007ff340f499d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007ff340f499d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000564a4f51e45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000564a4ed5c5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff33f2f820d in poll () from /lib64/libc.so.6 #5 0x0000564a4e8fca8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000564a4e8fbd25 in mysqld_main (argc=162, argv=0x564a516ddd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000564a4e8efec5 in main (argc=23, argv=0x7ffd1804c8a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7ff3387f1700 (LWP 78731)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f523a8d in safe_cond_wait (cond=0x7ff320015608, mp=0x7ff320015558, file=0x564a4f68dc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564a4ec7b126 in inline_mysql_cond_wait (that=0x7ff320015608, mutex=0x7ff320015558, src_file=0x564a4f68df80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564a4ec7ddbe in Wsrep_thd_queue::pop_front (this=0x7ff320015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000564a4ec7c7de in wsrep_rollback_process (rollbacker=0x7ff320000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000564a4ec6d31f in start_wsrep_THD (arg=0x564a518b96c0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff33999d700 (LWP 78663)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f523d8f in safe_cond_timedwait (cond=0x564a50b02640 , mp=0x564a50b02580 , abstime=0x7ff33999cd70, file=0x564a4f9ab920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564a4f524d49 in inline_mysql_cond_timedwait (that=0x564a50b02640 , mutex=0x564a50b02580 , abstime=0x7ff33999cd70, src_file=0x564a4f9ab968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564a4f5258ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000564a4f4b6555 in pfs_spawn_thread (arg=0x564a518257d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff3017fa700 (LWP 79264)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f065b76 in os_event::timed_wait (this=0x564a51b81260, abstime=0x7ff3017f9d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564a4f065e8d in os_event::wait_time_low (this=0x564a51b81260, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564a4f06602d in os_event_wait_time_low (event=0x564a51b81260, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564a4f0329e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff2f67fc700 (LWP 79276)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f06625b in os_event::wait (this=0x564a519a8b30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564a4f065ce1 in os_event::wait_low (this=0x564a519a8b30, reset_sig_count=2312) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564a4f066052 in os_event_wait_low (event=0x564a519a8b30, reset_sig_count=2312) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564a4f151ee5 in srv_resume_thread (slot=0x564a500a0748 , sig_count=2312, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564a4f155ef6 in srv_purge_coordinator_suspend (slot=0x564a500a0748 , rseg_history_len=5) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000564a4f15618c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff2f57fa700 (LWP 79278)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f06625b in os_event::wait (this=0x564a519a8c30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564a4f065ce1 in os_event::wait_low (this=0x564a519a8c30, reset_sig_count=8689) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564a4f066052 in os_event_wait_low (event=0x564a519a8c30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564a4f151ee5 in srv_resume_thread (slot=0x564a500a07b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564a4f155ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff2d67fc700 (LWP 79294)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f523a8d in safe_cond_wait (cond=0x7ff2d67fbc70, mp=0x7ff2d67fbcb0, file=0x564a4f7f1d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564a4ef7d525 in inline_mysql_cond_wait (that=0x7ff2d67fbc70, mutex=0x7ff2d67fbcb0, src_file=0x564a4f7f36d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564a4ef82161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000564a4f4b6555 in pfs_spawn_thread (arg=0x564a51beb430) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff33814c700 (LWP 79309)): #0 0x00007ff33f23c40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000564a4e8eff44 in my_sigwait (set=0x7ff33814bcf0, sig=0x7ff33814bcb4, code=0x7ff33814bcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000564a4e8f6584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000564a4f4b6555 in pfs_spawn_thread (arg=0x564a5196b3e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff3380b6700 (LWP 79311)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f523d8f in safe_cond_timedwait (cond=0x564a502631a0 , mp=0x564a50265ba0 , abstime=0x7ff3380b5c60, file=0x564a4f5df5c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564a4e8f0845 in inline_mysql_cond_timedwait (that=0x564a502631a0 , mutex=0x564a50265ba0 , abstime=0x7ff3380b5c60, src_file=0x564a4f5e1638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564a4e8f561b in cache_thread (thd=0x7ff30c01ad00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000564a4e8f58c9 in one_thread_per_connection_end (thd=0x7ff30c01ad00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000564a4eb920f3 in do_handle_one_connection (connect=0x564a51c26070) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000564a4eb91dad in handle_one_connection (arg=0x564a51c26070) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000564a4f4b6555 in pfs_spawn_thread (arg=0x564a5196f0c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff2d6ffd700 (LWP 79293)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f06625b in os_event::wait (this=0x564a519a8f30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564a4f065ce1 in os_event::wait_low (this=0x564a519a8f30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564a4f066052 in os_event_wait_low (event=0x564a519a8f30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564a4f21ac2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff2f4ff9700 (LWP 79280)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f06625b in os_event::wait (this=0x564a519a8bb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564a4f065ce1 in os_event::wait_low (this=0x564a519a8bb0, reset_sig_count=9405) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564a4f066052 in os_event_wait_low (event=0x564a519a8bb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564a4f151ee5 in srv_resume_thread (slot=0x564a500a0780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564a4f155ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff2f5ffb700 (LWP 79277)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f06625b in os_event::wait (this=0x564a519a8cb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564a4f065ce1 in os_event::wait_low (this=0x564a519a8cb0, reset_sig_count=8142) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564a4f066052 in os_event_wait_low (event=0x564a519a8cb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564a4f151ee5 in srv_resume_thread (slot=0x564a500a07f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000564a4f155ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff301ffb700 (LWP 79275)): #0 0x00007ff340f4be3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000564a4f06655d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000564a4f15544d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000564a4f15557d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff2f77fe700 (LWP 79267)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f065b76 in os_event::timed_wait (this=0x564a51bb3320, abstime=0x7ff2f77fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564a4f065e8d in os_event::wait_time_low (this=0x564a51bb3320, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564a4f06602d in os_event_wait_time_low (event=0x564a51bb3320, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564a4f29b24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff302ffd700 (LWP 79240)): #0 0x00007ff340b15644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564a4f059578 in LinuxAIOHandler::collect (this=0x7ff302ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564a4f059aa1 in LinuxAIOHandler::poll (this=0x7ff302ffcb00, m1=0x7ff302ffcba8, m2=0x7ff302ffcbb0, request=0x7ff302ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564a4f059c04 in os_aio_linux_handler (global_segment=5, m1=0x7ff302ffcba8, m2=0x7ff302ffcbb0, request=0x7ff302ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564a4f05e728 in os_aio_handler (segment=5, m1=0x7ff302ffcba8, m2=0x7ff302ffcbb0, request=0x7ff302ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564a4f2b15ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000564a4f1579fb in io_handler_thread (arg=0x564a502de648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff3037fe700 (LWP 79239)): #0 0x00007ff340b15644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564a4f059578 in LinuxAIOHandler::collect (this=0x7ff3037fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564a4f059aa1 in LinuxAIOHandler::poll (this=0x7ff3037fdb00, m1=0x7ff3037fdba8, m2=0x7ff3037fdbb0, request=0x7ff3037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564a4f059c04 in os_aio_linux_handler (global_segment=4, m1=0x7ff3037fdba8, m2=0x7ff3037fdbb0, request=0x7ff3037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564a4f05e728 in os_aio_handler (segment=4, m1=0x7ff3037fdba8, m2=0x7ff3037fdbb0, request=0x7ff3037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564a4f2b15ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000564a4f1579fb in io_handler_thread (arg=0x564a502de640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff308c05700 (LWP 79237)): #0 0x00007ff340b15644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564a4f059578 in LinuxAIOHandler::collect (this=0x7ff308c04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564a4f059aa1 in LinuxAIOHandler::poll (this=0x7ff308c04b00, m1=0x7ff308c04ba8, m2=0x7ff308c04bb0, request=0x7ff308c04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564a4f059c04 in os_aio_linux_handler (global_segment=2, m1=0x7ff308c04ba8, m2=0x7ff308c04bb0, request=0x7ff308c04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564a4f05e728 in os_aio_handler (segment=2, m1=0x7ff308c04ba8, m2=0x7ff308c04bb0, request=0x7ff308c04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564a4f2b15ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000564a4f1579fb in io_handler_thread (arg=0x564a502de630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff30b059700 (LWP 79234)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f523d8f in safe_cond_timedwait (cond=0x564a50aefea0 , mp=0x564a50aefde0 , abstime=0x7ff30b058cb0, file=0x564a4f9676f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564a4f38dbc1 in inline_mysql_cond_timedwait (that=0x564a50aefea0 , mutex=0x564a50aefde0 , abstime=0x7ff30b058cb0, src_file=0x564a4f967738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564a4f38e0e5 in my_service_thread_sleep (control=0x564a500a4b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000564a4f38257e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000564a4f4b6555 in pfs_spawn_thread (arg=0x564a51960e70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff329b7a700 (LWP 78703)): #0 0x00007ff33f303483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff338df46f8 in asio::detail::epoll_reactor::run (this=0x564a51899010, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007ff338df8df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x564a51897f70, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007ff338df90a2 in asio::detail::task_io_service::run (this=0x564a51897f70, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007ff338e10afd in asio::io_service::run (this=0x564a51895f70) at asio/asio/impl/io_service.ipp:58 #5 0x00007ff338e11251 in gcomm::AsioProtonet::event_loop (this=0x564a51895ed0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007ff338e2f2ed in GCommConn::run (this=0x564a51898770) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007ff338e32399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff338101700 (LWP 79310)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f523a8d in safe_cond_wait (cond=0x564a502631e0 , mp=0x564a50266560 , file=0x564a4f5eedc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564a4e9214ee in inline_mysql_cond_wait (that=0x564a502631e0 , mutex=0x564a50266560 , src_file=0x564a4f5ef930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000564a4e922d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000564a4f4b6555 in pfs_spawn_thread (arg=0x564a51d0c1a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff2d77fe700 (LWP 79292)): #0 0x00007ff340f4be3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000564a4f06655d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000564a4f20cb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff2f6ffd700 (LWP 79268)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f065b76 in os_event::timed_wait (this=0x564a51b49550, abstime=0x7ff2f6ffcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564a4f065e8d in os_event::wait_time_low (this=0x564a51b49550, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564a4f06602d in os_event_wait_time_low (event=0x564a51b49550, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564a4f1b4403 in ib_wqueue_timedwait (wq=0x564a51b49420, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000564a4f2f4f9a in fts_optimize_thread (arg=0x564a51b49420) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff300ff9700 (LWP 79265)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f065b76 in os_event::timed_wait (this=0x564a519a8d30, abstime=0x7ff300ff8ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564a4f065e8d in os_event::wait_time_low (this=0x564a519a8d30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564a4f06602d in os_event_wait_time_low (event=0x564a519a8d30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564a4f1540ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff2d7fff700 (LWP 79291)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f06625b in os_event::wait (this=0x564a519a8e30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000564a4f065ce1 in os_event::wait_low (this=0x564a519a8e30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000564a4f066052 in os_event_wait_low (event=0x564a519a8e30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000564a4f231fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff303fff700 (LWP 79238)): #0 0x00007ff340b15644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564a4f059578 in LinuxAIOHandler::collect (this=0x7ff303ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564a4f059aa1 in LinuxAIOHandler::poll (this=0x7ff303ffeb00, m1=0x7ff303ffeba8, m2=0x7ff303ffebb0, request=0x7ff303ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564a4f059c04 in os_aio_linux_handler (global_segment=3, m1=0x7ff303ffeba8, m2=0x7ff303ffebb0, request=0x7ff303ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564a4f05e728 in os_aio_handler (segment=3, m1=0x7ff303ffeba8, m2=0x7ff303ffebb0, request=0x7ff303ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564a4f2b15ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000564a4f1579fb in io_handler_thread (arg=0x564a502de638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff2f7fff700 (LWP 79266)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f065b76 in os_event::timed_wait (this=0x564a519a8db0, abstime=0x7ff2f7ffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564a4f065e8d in os_event::wait_time_low (this=0x564a519a8db0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564a4f06602d in os_event_wait_time_low (event=0x564a519a8db0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564a4f153bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff309c07700 (LWP 79235)): #0 0x00007ff340b15644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564a4f059578 in LinuxAIOHandler::collect (this=0x7ff309c06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564a4f059aa1 in LinuxAIOHandler::poll (this=0x7ff309c06b00, m1=0x7ff309c06ba8, m2=0x7ff309c06bb0, request=0x7ff309c06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564a4f059c04 in os_aio_linux_handler (global_segment=0, m1=0x7ff309c06ba8, m2=0x7ff309c06bb0, request=0x7ff309c06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564a4f05e728 in os_aio_handler (segment=0, m1=0x7ff309c06ba8, m2=0x7ff309c06bb0, request=0x7ff309c06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564a4f2b15ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000564a4f1579fb in io_handler_thread (arg=0x564a502de620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff3027fc700 (LWP 79241)): #0 0x00007ff340f48d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f065b76 in os_event::timed_wait (this=0x564a519a8eb0, abstime=0x7ff3027fbaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000564a4f065e8d in os_event::wait_time_low (this=0x564a519a8eb0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000564a4f06602d in os_event_wait_time_low (event=0x564a519a8eb0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564a4f23be25 in pc_sleep_if_needed (next_loop_time=1549623621480, sig_count=2, cur_time=1549623620480) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000564a4f23cd5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff33875b700 (LWP 78733)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff338d47132 in fifo_lock_get (q=q@entry=0x564a5188e100) at galerautils/src/gu_fifo.c:262 #2 0x00007ff338d477a1 in gu_fifo_get_head (q=0x564a5188e100, err=0x7ff33875a06c) at galerautils/src/gu_fifo.c:330 #3 0x00007ff338e290cc in gcs_recv (conn=0x564a51867b60, action=0x7ff33875a0b0) at gcs/src/gcs.cpp:2024 #4 0x00007ff338e5616c in galera::GcsActionSource::process (this=0x564a51867840, recv_ctx=0x7ff33875a4d0, exit_loop=@0x7ff33875a29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007ff338e75033 in galera::ReplicatorSMM::async_recv (this=0x564a5185c1f0, recv_ctx=0x7ff33875a4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000564a4f5d836c in wsrep::wsrep_provider_v26::run_applier (this=0x564a517d1e50, applier_ctx=0x7ff33875a4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000564a4ec7b6d4 in wsrep_replication_process (thd=0x7ff314000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000564a4ec6d31f in start_wsrep_THD (arg=0x564a51880860) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff309406700 (LWP 79236)): #0 0x00007ff340b15644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000564a4f059578 in LinuxAIOHandler::collect (this=0x7ff309405b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000564a4f059aa1 in LinuxAIOHandler::poll (this=0x7ff309405b00, m1=0x7ff309405ba8, m2=0x7ff309405bb0, request=0x7ff309405be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000564a4f059c04 in os_aio_linux_handler (global_segment=1, m1=0x7ff309405ba8, m2=0x7ff309405bb0, request=0x7ff309405be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000564a4f05e728 in os_aio_handler (segment=1, m1=0x7ff309405ba8, m2=0x7ff309405bb0,worker[4] Trying to dump core for [mysqltest - pid: 99514, winpid: 99514] worker[4] Trying to dump core for [mysqld.1 - pid: 97100, winpid: 97100] worker[4] Trying to dump core for [mysqld.2 - pid: 98614, winpid: 98614] request=0x7ff309405be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000564a4f2b15ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000564a4f1579fb in io_handler_thread (arg=0x564a502de628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff3387a6700 (LWP 78732)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000564a4f523a8d in safe_cond_wait (cond=0x7ff31c017b68, mp=0x7ff31c017ab8, file=0x564a4f68dc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564a4ec7b126 in inline_mysql_cond_wait (that=0x7ff31c017b68, mutex=0x7ff31c017ab8, src_file=0x564a4f68df80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564a4ec7ddbe in Wsrep_thd_queue::pop_front (this=0x7ff31c017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000564a4ec7cb9e in wsrep_post_rollback_process (post_rollbacker=0x7ff31c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000564a4ec6d31f in start_wsrep_THD (arg=0x564a518b98c0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff329379700 (LWP 78730)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff338e2d867 in gu::Lock::wait (cond=..., this=0x7ff329378850) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff338e303ab in RecvBuf::front (this=this@entry=0x564a51898958, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007ff338e2df46 in gcomm_recv (backend=, msg=0x564a51867e40, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007ff338e21eec in core_msg_recv (backend=backend@entry=0x564a51867f88, recv_msg=recv_msg@entry=0x564a51867e40, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007ff338e25164 in gcs_core_recv (conn=0x564a51867de0, recv_act=recv_act@entry=0x7ff329378d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007ff338e2c3d8 in gcs_recv_thread (arg=0x564a51867b60) at gcs/src/gcs.cpp:1398 #7 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff32bffe700 (LWP 78664)): #0 0x00007ff340f48965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff338e52fd7 in gu::Lock::wait (cond=..., this=0x7ff32bffdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff338e53069 in galera::ServiceThd::thd_func (arg=0x564a5185c908) at galera/src/galera_service_thd.cpp:30 #3 0x00007ff340f44dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff33f302ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff341355880 (LWP 78662)): #0 0x00007ff340f499d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000564a4f51e45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000564a4ed5c5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff33f2f820d in poll () from /lib64/libc.so.6 #5 0x0000564a4e8fca8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000564a4e8fbd25 in mysqld_main (argc=162, argv=0x564a516ddd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000564a4e8efec5 in main (argc=23, argv=0x7ffd1804c8a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_load_data_splitting-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_mysqldump_sst 'innodb' w1 [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-08 13:02:33 line 2019-02-08 12:58:45 13 [Warning] WSREP: no THD for trx: 51 2019-02-08 12:58:45 13 [Warning] WSREP: no THD for trx: 51 ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/mysqld.2.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_mysqldump_sst-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' galera_sr.GCF-1043A 'innodb' w4 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 13:08:39 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/GCF-1043A.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 REPLACE INTO t1 VALUES (1,'y'),(2,'x') REPLACE INTO t1 VALUES (1,'y'),(2,'y'),(3,'y') connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) - found 'core.97101' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.GCF-1043A-innodb/mysqld.1/data/core.97101 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 97101] [New LWP 97197] [New LWP 97148] [New LWP 97152] [New LWP 97265] [New LWP 97224] [New LWP 97196] [New LWP 97272] [New LWP 97266] [New LWP 97217] [New LWP 97221] [New LWP 97223] [New LWP 97267] [New LWP 97263] [New LWP 97268] [New LWP 97270] [New LWP 98090] [New LWP 97271] [New LWP 97374] [New LWP 97218] [New LWP 97276] [New LWP 97273] [New LWP 97220] [New LWP 97219] [New LWP 99594] [New LWP 97262] [New LWP 97275] [New LWP 97269] [New LWP 97264] [New LWP 97274] [New LWP 97198] [New LWP 97188] [New LWP 97195] [New LWP 97222] [New LWP 97375] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fbf3a84b9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fbf3a84b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000561fadeea45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000561fad7285a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fbf38bfa20d in poll () from /lib64/libc.so.6 #5 0x0000561fad2c8a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000561fad2c7d25 in mysqld_main (argc=161, argv=0x561fb05a2d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000561fad2bbec5 in main (argc=23, argv=0x7ffc33d48bb8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7fbf300b5700 (LWP 97375)): #0 0x00007fbf3a84dd1d in fsync () from /lib64/libpthread.so.0 #1 0x0000561fada26c27 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000561fada26f7d in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000561fadc6f4aa in pfs_os_file_flush_func (file=..., src_file=0x561fae2e9b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000561fadc72225 in fil_flush_low (space=0x561fb0a737d0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000561fadc7daff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000561fada02321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000561fada02d83 in log_write_up_to (lsn=2509818, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000561fadb6de99 in trx_flush_log_if_needed_low (lsn=2509818) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000561fadb6dee2 in trx_flush_log_if_needed (lsn=2509818, trx=0x7fbf288d94b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000561fadb6fd0b in trx_commit_complete_for_mysql (trx=0x7fbf288d94b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000561fad9545b2 in innobase_commit (hton=0x561fb083db70, thd=0x7fbec00e6770, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x0000561fad72e0d7 in commit_one_phase_2 (thd=0x7fbec00e6770, all=true, trans=0x7fbec00e9cc0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000561fad72dfd9 in ha_commit_one_phase (thd=0x7fbec00e6770, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000561fad72d355 in ha_commit_trans (thd=0x7fbec00e6770, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000561fad572e8e in trans_commit (thd=0x7fbec00e6770) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000561fad93222a in Wsrep_storage_service::commit (this=0x7fbec011f9d0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x0000561fadf9a5c9 in wsrep::transaction::certify_fragment (this=0x7fbec00069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x0000561fadf99d2d in wsrep::transaction::streaming_step (this=0x7fbec00069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x0000561fadf970e8 in wsrep::transaction::after_row (this=0x7fbec00069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x0000561fad740446 in wsrep::client_state::after_row (this=0x7fbec0006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x0000561fad7296c3 in wsrep_after_row (thd=0x7fbec0000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x0000561fad73b5c8 in wsrep_after_row (thd=0x7fbec0000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x0000561fad73c278 in handler::ha_delete_row (this=0x7fbeb00276c8, buf=0x7fbeb0017270 "\375\002") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6575 #24 0x0000561fad8e0a89 in TABLE::delete_row (this=0x7fbeb0026a90) at /home/stepan/galera/git/10.4/server/sql/sql_delete.cc:253 #25 0x0000561fad8dda8b in mysql_delete (thd=0x7fbec0000b00, table_list=0x7fbec0015cd8, conds=0x0, order_list=0x7fbec0005380, limit=18446744073709551614, options=0, result=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_delete.cc:751 #26 0x0000561fad3f898d in mysql_execute_command (thd=0x7fbec0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4948 #27 0x0000561fad403935 in mysql_parse (thd=0x7fbec0000b00, rawbuf=0x7fbec0015168 "DELETE FROM t1", length=14, parser_state=0x7fbf300b40d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #28 0x0000561fad402f68 in wsrep_mysql_parse (thd=0x7fbec0000b00, rawbuf=0x7fbec0015168 "DELETE FROM t1", length=14, parser_state=0x7fbf300b40d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #29 0x0000561fad3eee25 in dispatch_command (command=COM_QUERY, thd=0x7fbec0000b00, packet=0x7fbec00ed951 "DELETE FROM t1", packet_length=14, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #30 0x0000561fad3ed80d in do_command (thd=0x7fbec0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #31 0x0000561fad55e069 in do_handle_one_connection (connect=0x561fb0bf4c20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #32 0x0000561fad55ddad in handle_one_connection (arg=0x561fb0bf4c20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #33 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb0bfa380) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #34 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #35 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fbf09bfb700 (LWP 97222)): #0 0x00007fbf3a84dd1d in fsync () from /lib64/libpthread.so.0 #1 0x0000561fada26c27 in os_file_fsync_posix (file=83) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000561fada26f7d in os_file_flush_func (file=83) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000561fadc6f4aa in pfs_os_file_flush_func (file=..., src_file=0x561fae2e9b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000561fadc72225 in fil_flush_low (space=0x7fbec0267760, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000561fadc7daff in fil_flush (space_id=10) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000561fadc7dd86 in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x0000561fadbfa10a in buf_dblwr_update (bpage=0x7fbf280c28c0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x0000561fadc01af7 in buf_flush_write_complete (bpage=0x7fbf280c28c0, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x0000561fadbf07b1 in buf_page_io_complete (bpage=0x7fbf280c28c0, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x0000561fadc7d859 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x0000561fadb239fb in io_handler_thread (arg=0x561faecaa640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fbf2a47c700 (LWP 97195)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbf3272f867 in gu::Lock::wait (cond=..., this=0x7fbf2a47b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fbf327323ab in RecvBuf::front (this=this@entry=0x561fb07618a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fbf3272ff46 in gcomm_recv (backend=, msg=0x561fb0729830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fbf32723eec in core_msg_recv (backend=backend@entry=0x561fb0729978, recv_msg=recv_msg@entry=0x561fb0729830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fbf32727164 in gcs_core_recv (conn=0x561fb07297d0, recv_act=recv_act@entry=0x7fbf2a47bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fbf3272e3d8 in gcs_recv_thread (arg=0x561fb0729550) at gcs/src/gcs.cpp:1398 #7 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fbf2ac7d700 (LWP 97188)): #0 0x00007fbf38c05483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fbf326f66f8 in asio::detail::epoll_reactor::run (this=0x561fb0762040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fbf326fadf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x561fb0761ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fbf326fb0a2 in asio::detail::task_io_service::run (this=0x561fb0761ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fbf32712afd in asio::io_service::run (this=0x561fb0760fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fbf32713251 in gcomm::AsioProtonet::event_loop (this=0x561fb0760f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fbf327312ed in GCommConn::run (this=0x561fb07616c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fbf32734399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fbf3055a700 (LWP 97198)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbf32649132 in fifo_lock_get (q=q@entry=0x561fb074fbd0) at galerautils/src/gu_fifo.c:262 #2 0x00007fbf326497a1 in gu_fifo_get_head (q=0x561fb074fbd0, err=0x7fbf3055906c) at galerautils/src/gu_fifo.c:330 #3 0x00007fbf3272b0cc in gcs_recv (conn=0x561fb0729550, action=0x7fbf305590b0) at gcs/src/gcs.cpp:2024 #4 0x00007fbf3275816c in galera::GcsActionSource::process (this=0x561fb0728f30, recv_ctx=0x7fbf305594d0, exit_loop=@0x7fbf3055929e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fbf32777033 in galera::ReplicatorSMM::async_recv (this=0x561fb0721280, recv_ctx=0x7fbf305594d0) at galera/src/replicator_smm.cpp:383 #6 0x0000561fadfa436c in wsrep::wsrep_provider_v26::run_applier (this=0x561fb06a8360, applier_ctx=0x7fbf305594d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000561fad6476d4 in wsrep_replication_process (thd=0x7fbf18000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000561fad63931f in start_wsrep_THD (arg=0x561fb0771110) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fbef97fa700 (LWP 97274)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada3225b in os_event::wait (this=0x561fb0869a20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561fada31ce1 in os_event::wait_low (this=0x561fb0869a20, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561fada32052 in os_event_wait_low (event=0x561fb0869a20, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561fadbe6c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fbf03fff700 (LWP 97264)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada31b76 in os_event::timed_wait (this=0x561fb08698a0, abstime=0x7fbf03ffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561fada31e8d in os_event::wait_time_low (this=0x561fb08698a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561fada3202d in os_event_wait_time_low (event=0x561fb08698a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561fadb1fbcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fbefbfff700 (LWP 97269)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada3225b in os_event::wait (this=0x561fb0869720) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561fada31ce1 in os_event::wait_low (this=0x561fb0869720, reset_sig_count=8088) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561fada32052 in os_event_wait_low (event=0x561fb0869720, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561fadb1dee5 in srv_resume_thread (slot=0x561faea6c7b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000561fadb21ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fbef8ff9700 (LWP 97275)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefa8d in safe_cond_wait (cond=0x7fbef8ff8c70, mp=0x7fbef8ff8cb0, file=0x561fae1bdd40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561fad949525 in inline_mysql_cond_wait (that=0x7fbef8ff8c70, mutex=0x7fbef8ff8cb0, src_file=0x561fae1bf6d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000561fad94e161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb0aaa960) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fbf037fe700 (LWP 97262)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada31b76 in os_event::timed_wait (this=0x561fb0a41d50, abstime=0x7fbf037fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561fada31e8d in os_event::wait_time_low (this=0x561fb0a41d50, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561fada3202d in os_event_wait_time_low (event=0x561fb0a41d50, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561fad9fe9e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fbf28084700 (LWP 99594)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefd8f in safe_cond_timedwait (cond=0x561faec2f1a0 , mp=0x561faec31ba0 , abstime=0x7fbf28083c60, file=0x561fadfab5c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000561fad2bc845 in inline_mysql_cond_timedwait (that=0x561faec2f1a0 , mutex=0x561faec31ba0 , abstime=0x7fbf28083c60, src_file=0x561fadfad638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000561fad2c161b in cache_thread (thd=0x7fbeb0000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000561fad2c18c9 in one_thread_per_connection_end (thd=0x7fbeb0000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000561fad55e0f3 in do_handle_one_connection (connect=0x561fb0b46a70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000561fad55ddad in handle_one_connection (arg=0x561fb0b46a70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb0b5ff50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fbf0b3fe700 (LWP 97219)): #0 0x00007fbf3a417644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561fada25578 in LinuxAIOHandler::collect (this=0x7fbf0b3fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561fada25aa1 in LinuxAIOHandler::poll (this=0x7fbf0b3fdb00, m1=0x7fbf0b3fdba8, m2=0x7fbf0b3fdbb0, request=0x7fbf0b3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561fada25c04 in os_aio_linux_handler (global_segment=1, m1=0x7fbf0b3fdba8, m2=0x7fbf0b3fdbb0, request=0x7fbf0b3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561fada2a728 in os_aio_handler (segment=1, m1=0x7fbf0b3fdba8, m2=0x7fbf0b3fdbb0, request=0x7fbf0b3fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561fadc7d5ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000561fadb239fb in io_handler_thread (arg=0x561faecaa628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fbf0abfd700 (LWP 97220)): #0 0x00007fbf3a417644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561fada25578 in LinuxAIOHandler::collect (this=0x7fbf0abfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561fada25aa1 in LinuxAIOHandler::poll (this=0x7fbf0abfcb00, m1=0x7fbf0abfcba8, m2=0x7fbf0abfcbb0, request=0x7fbf0abfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561fada25c04 in os_aio_linux_handler (global_segment=2, m1=0x7fbf0abfcba8, m2=0x7fbf0abfcbb0, request=0x7fbf0abfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561fada2a728 in os_aio_handler (segment=2, m1=0x7fbf0abfcba8, m2=0x7fbf0abfcbb0, request=0x7fbf0abfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561fadc7d5ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000561fadb239fb in io_handler_thread (arg=0x561faecaa630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fbef9ffb700 (LWP 97273)): #0 0x00007fbf3a84de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000561fada3255d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000561fadbd8b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fbf3014b700 (LWP 97276)): #0 0x00007fbf38b3e40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000561fad2bbf44 in my_sigwait (set=0x7fbf3014acf0, sig=0x7fbf3014acb4, code=0x7fbf3014acb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000561fad2c2584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb082ceb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fbf0bbff700 (LWP 97218)): #0 0x00007fbf3a417644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561fada25578 in LinuxAIOHandler::collect (this=0x7fbf0bbfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561fada25aa1 in LinuxAIOHandler::poll (this=0x7fbf0bbfeb00, m1=0x7fbf0bbfeba8, m2=0x7fbf0bbfebb0, request=0x7fbf0bbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561fada25c04 in os_aio_linux_handler (global_segment=0, m1=0x7fbf0bbfeba8, m2=0x7fbf0bbfebb0, request=0x7fbf0bbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561fada2a728 in os_aio_handler (segment=0, m1=0x7fbf0bbfeba8, m2=0x7fbf0bbfebb0, request=0x7fbf0bbfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561fadc7d5ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000561fadb239fb in io_handler_thread (arg=0x561faecaa620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fbf30100700 (LWP 97374)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefa8d in safe_cond_wait (cond=0x561faec2f1e0 , mp=0x561faec32560 , file=0x561fadfbadc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561fad2ed4ee in inline_mysql_cond_wait (that=0x561faec2f1e0 , mutex=0x561faec32560 , src_file=0x561fadfbb930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000561fad2eed80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb0bcb100) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fbefaffd700 (LWP 97271)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada3225b in os_event::wait (this=0x561fb08697a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561fada31ce1 in os_event::wait_low (this=0x561fb08697a0, reset_sig_count=7696) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561fada32052 in os_event_wait_low (event=0x561fb08697a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561fadb1dee5 in srv_resume_thread (slot=0x561faea6c7f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000561fadb21ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fbf3006a700 (LWP 98090)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefd8f in safe_cond_timedwait (cond=0x561faec2f1a0 , mp=0x561faec31ba0 , abstime=0x7fbf30069c60, file=0x561fadfab5c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000561fad2bc845 in inline_mysql_cond_timedwait (that=0x561faec2f1a0 , mutex=0x561faec31ba0 , abstime=0x7fbf30069c60, src_file=0x561fadfad638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000561fad2c161b in cache_thread (thd=0x7fbeb8000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000561fad2c18c9 in one_thread_per_connection_end (thd=0x7fbeb8000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000561fad55e0f3 in do_handle_one_connection (connect=0x561fb0afd0e0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000561fad55ddad in handle_one_connection (arg=0x561fb0afd0e0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb06dda70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fbefb7fe700 (LWP 97270)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada3225b in os_event::wait (this=0x561fb08696a0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561fada31ce1 in os_event::wait_low (this=0x561fb08696a0, reset_sig_count=8668) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561fada32052 in os_event_wait_low (event=0x561fb08696a0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561fadb1dee5 in srv_resume_thread (slot=0x561faea6c780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000561fadb21ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fbf00ff9700 (LWP 97268)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada3225b in os_event::wait (this=0x561fb0869620) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561fada31ce1 in os_event::wait_low (this=0x561fb0869620, reset_sig_count=2105) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561fada32052 in os_event_wait_low (event=0x561fb0869620, reset_sig_count=2105) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561fadb1dee5 in srv_resume_thread (slot=0x561faea6c748 , sig_count=2105, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000561fadb21ef6 in srv_purge_coordinator_suspend (slot=0x561faea6c748 , rseg_history_len=31) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000561fadb2218c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fbf02ffd700 (LWP 97263)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada31b76 in os_event::timed_wait (this=0x561fb0869820, abstime=0x7fbf02ffcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561fada31e8d in os_event::wait_time_low (this=0x561fb0869820, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561fada3202d in os_event_wait_time_low (event=0x561fb0869820, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561fadb200ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fbf017fa700 (LWP 97267)): #0 0x00007fbf3a84de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000561fada3255d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000561fadb2144d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000561fadb2157d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fbf093fa700 (LWP 97223)): #0 0x00007fbf3a417644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561fada25578 in LinuxAIOHandler::collect (this=0x7fbf093f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561fada25aa1 in LinuxAIOHandler::poll (this=0x7fbf093f9b00, m1=0x7fbf093f9ba8, m2=0x7fbf093f9bb0, request=0x7fbf093f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561fada25c04 in os_aio_linux_handler (global_segment=5, m1=0x7fbf093f9ba8, m2=0x7fbf093f9bb0, request=0x7fbf093f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561fada2a728 in os_aio_handler (segment=5, m1=0x7fbf093f9ba8, m2=0x7fbf093f9bb0, request=0x7fbf093f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561fadc7d5ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000561fadb239fb in io_handler_thread (arg=0x561faecaa648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fbf0a3fc700 (LWP 97221)): #0 0x00007fbf3a417644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000561fada25578 in LinuxAIOHandler::collect (this=0x7fbf0a3fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000561fada25aa1 in LinuxAIOHandler::poll (this=0x7fbf0a3fbb00, m1=0x7fbf0a3fbba8, m2=0x7fbf0a3fbbb0, request=0x7fbf0a3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000561fada25c04 in os_aio_linux_handler (global_segment=3, m1=0x7fbf0a3fbba8, m2=0x7fbf0a3fbbb0, request=0x7fbf0a3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000561fada2a728 in os_aio_handler (segment=3, m1=0x7fbf0a3fbba8, m2=0x7fbf0a3fbbb0, request=0x7fbf0a3fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000561fadc7d5ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000561fadb239fb in io_handler_thread (arg=0x561faecaa638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fbf294d6700 (LWP 97217)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefd8f in safe_cond_timedwait (cond=0x561faf4bbea0 , mp=0x561faf4bbde0 , abstime=0x7fbf294d5cb0, file=0x561fae3336f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000561fadd59bc1 in inline_mysql_cond_timedwait (that=0x561faf4bbea0 , mutex=0x561faf4bbde0 , abstime=0x7fbf294d5cb0, src_file=0x561fae333738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000561fadd5a0e5 in my_service_thread_sleep (control=0x561faea70b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000561fadd4e57e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb0821840) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fbf01ffb700 (LWP 97266)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada31b76 in os_event::timed_wait (this=0x561fb0a09aa0, abstime=0x7fbf01ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561fada31e8d in os_event::wait_time_low (this=0x561fb0a09aa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561fada3202d in os_event_wait_time_low (event=0x561fb0a09aa0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561fadb80403 in ib_wqueue_timedwait (wq=0x561fb0a09970, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000561fadcc0f9a in fts_optimize_thread (arg=0x561fb0a09970) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fbefa7fc700 (LWP 97272)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada3225b in os_event::wait (this=0x561fb0869920) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561fada31ce1 in os_event::wait_low (this=0x561fb0869920, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561fada32052 in os_event_wait_low (event=0x561fb0869920, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561fadbfdfb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fbf305f0700 (LWP 97196)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefa8d in safe_cond_wait (cond=0x7fbf14015608, mp=0x7fbf14015558, file=0x561fae059c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561fad647126 in inline_mysql_cond_wait (that=0x7fbf14015608, mutex=0x7fbf14015558, src_file=0x561fae059f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000561fad649dbe in Wsrep_thd_queue::pop_front (this=0x7fbf14015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000561fad6487de in wsrep_rollback_process (rollbacker=0x7fbf14000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000561fad63931f in start_wsrep_THD (arg=0x561fb0771320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fbf08bf9700 (LWP 97224)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada3225b in os_event::wait (this=0x561fb0a7f650) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000561fada31ce1 in os_event::wait_low (this=0x561fb0a7f650, reset_sig_count=107) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000561fada32052 in os_event_wait_low (event=0x561fb0a7f650, reset_sig_count=107) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000561fadbfa9dd in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:999 #5 0x0000561fadc054ad in buf_flush_end (buf_pool=0x561fb094f120, flush_type=BUF_FLUSH_LRU) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1880 #6 0x0000561fadc0566b in buf_flush_do_batch (buf_pool=0x561fb094f120, type=BUF_FLUSH_LRU, min_n=100, lsn_limit=0, n=0x7fbf08bf8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1949 #7 0x0000561fadc06004 in buf_flush_LRU_list (buf_pool=0x561fb094f120) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2210 #8 0x0000561fadc082c7 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2730 #9 0x0000561fadc092a3 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3159 #10 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fbf027fc700 (LWP 97265)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fada31b76 in os_event::timed_wait (this=0x561fb0a73e10, abstime=0x7fbf027fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000561fada31e8d in os_event::wait_time_low (this=0x561fb0a73e10, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000561fada3202d in os_event_wait_time_low (event=0x561fb0a73e10, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000561fadc6724a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fbf316f2700 (LWP 97152)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fbf32754fd7 in gu::Lock::wait (cond=..., this=0x7fbf316f1bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fbf32755069 in galera::ServiceThd::thd_func (arg=0x561fb0721998) at galera/src/galera_service_thd.cpp:30 #3 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fbf3329f700 (LWP 97148)): #0 0x00007fbf3a84ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefd8f in safe_cond_timedwait (cond=0x561faf4ce640 , mp=0x561faf4ce580 , abstime=0x7fbf3329ed70, file=0x561fae377920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000561fadef0d49 in inline_mysql_cond_timedwait (that=0x561faf4ce640 , mutex=0x561faf4ce580 , abstime=0x7fbf3329ed70, src_file=0x561fae377968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000561fadef18ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000561fade82555 in pfs_spawn_thread (arg=0x561fb06ea730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fbf305a5700 (LWP 97197)): #0 0x00007fbf3a84a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000561fadeefa8d in safe_cond_wait (cond=0x7fbf20016858, mp=0x7fbf200167a8, file=0x561fae059c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000561fad647126 in inline_mysql_cond_wait (that=0x7fbf20016858, mutex=0x7fbf200167a8, src_file=0x561fae059f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000561fad649dbe in Wsrep_thd_queue::pop_front (this=0x7fbf20016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000561fad648b9e in wsrep_post_rollback_process (post_rollbacker=0x7fbf20001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000561fad63931f in start_wsrep_THD (arg=0x561fb075b810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fbf3a846dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fbf38c04ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fbf3ac57880 (LWP 97101)): #0 0x00007fbf3a84b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000561fadeea45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000561fad7285a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fbf38bfa20d in poll () from /lib64/libc.so.6 #5 0x0000561fad2c8a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000561fad2c7d25 in mysqld_main (argc=161, argv=0x561fb05a2d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000561fad2bbec5 in main (argc=23, argv=0x7ffc33d48bb8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.98615' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.GCF-1043A-innodb/mysqld.2/data/core.98615 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 98615] [New LWP 98703] [New LWP 99016] [New LWP 99049] [New LWP 99056] [New LWP 98666] [New LWP 99050] [New LWP 98633] [New LWP 99059] [New LWP 98706] [New LWP 99017] [New LWP 99072] [New LWP 98632] [New LWP 98705] [New LWP 99013] [New LWP 98704] [New LWP 99048] [New LWP 99015] [New LWP 99057] [New LWP 99047] [New LWP 99058] [New LWP 99014] [New LWP 99060] [New LWP 99067] [New LWP 99068] [New LWP 99019] [New LWP 99069] [New LWP 99066] [New LWP 99070] [New LWP 99071] [New LWP 99046] [New LWP 99018] [New LWP 99012] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f6a2217b9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f6a2217b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005558f2b3f45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005558f237d5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6a2052a20d in poll () from /lib64/libc.so.6 #5 0x00005558f1f1da8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005558f1f1cd25 in mysqld_main (argc=162, argv=0x5558f466cd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005558f1f10ec5 in main (argc=23, argv=0x7ffc99af2d18) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f6a08941700 (LWP 99012)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2b44d8f in safe_cond_timedwait (cond=0x5558f4110ea0 , mp=0x5558f4110de0 , abstime=0x7f6a08940cb0, file=0x5558f2f886f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005558f29aebc1 in inline_mysql_cond_timedwait (that=0x5558f4110ea0 , mutex=0x5558f4110de0 , abstime=0x7f6a08940cb0, src_file=0x5558f2f88738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005558f29af0e5 in my_service_thread_sleep (control=0x5558f36c5b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005558f29a357e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005558f2ad7555 in pfs_spawn_thread (arg=0x5558f48f06f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f69dffff700 (LWP 99018)): #0 0x00007f6a21d47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005558f267a578 in LinuxAIOHandler::collect (this=0x7f69dfffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005558f267aaa1 in LinuxAIOHandler::poll (this=0x7f69dfffeb00, m1=0x7f69dfffeba8, m2=0x7f69dfffebb0, request=0x7f69dfffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005558f267ac04 in os_aio_linux_handler (global_segment=5, m1=0x7f69dfffeba8, m2=0x7f69dfffebb0, request=0x7f69dfffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005558f267f728 in os_aio_handler (segment=5, m1=0x7f69dfffeba8, m2=0x7f69dfffebb0, request=0x7f69dfffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005558f28d25ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005558f27789fb in io_handler_thread (arg=0x5558f38ff648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f69de7fc700 (LWP 99046)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2686b76 in os_event::timed_wait (this=0x5558f4b10680, abstime=0x7f69de7fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005558f2686e8d in os_event::wait_time_low (this=0x5558f4b10680, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005558f268702d in os_event_wait_time_low (event=0x5558f4b10680, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005558f26539e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f6a080f5700 (LWP 99071)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2b44a8d in safe_cond_wait (cond=0x5558f38841e0 , mp=0x5558f3887560 , file=0x5558f2c0fdc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005558f1f424ee in inline_mysql_cond_wait (that=0x5558f38841e0 , mutex=0x5558f3887560 , src_file=0x5558f2c10930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005558f1f43d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005558f2ad7555 in pfs_spawn_thread (arg=0x5558f4c9b040) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f6a08140700 (LWP 99070)): #0 0x00007f6a2046e40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005558f1f10f44 in my_sigwait (set=0x7f6a0813fcf0, sig=0x7f6a0813fcb4, code=0x7f6a0813fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005558f1f17584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005558f2ad7555 in pfs_spawn_thread (arg=0x5558f48fbd60) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f69cd7fa700 (LWP 99066)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f268725b in os_event::wait (this=0x5558f4938690) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005558f2686ce1 in os_event::wait_low (this=0x5558f4938690, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005558f2687052 in os_event_wait_low (event=0x5558f4938690, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005558f2852fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f69b37fe700 (LWP 99069)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2b44a8d in safe_cond_wait (cond=0x7f69b37fdc70, mp=0x7f69b37fdcb0, file=0x5558f2e12d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005558f259e525 in inline_mysql_cond_wait (that=0x7f69b37fdc70, mutex=0x7f69b37fdcb0, src_file=0x5558f2e146d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005558f25a3161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005558f2ad7555 in pfs_spawn_thread (arg=0x5558f4b7ae40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f69df7fe700 (LWP 99019)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2686b76 in os_event::timed_wait (this=0x5558f4938710, abstime=0x7f69df7fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005558f2686e8d in os_event::wait_time_low (this=0x5558f4938710, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005558f268702d in os_event_wait_time_low (event=0x5558f4938710, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005558f285ce25 in pc_sleep_if_needed (next_loop_time=1549624064534, sig_count=2, cur_time=1549624063534) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x00005558f285dd5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f69b3fff700 (LWP 99068)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f268725b in os_event::wait (this=0x5558f4938790) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005558f2686ce1 in os_event::wait_low (this=0x5558f4938790, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005558f2687052 in os_event_wait_low (event=0x5558f4938790, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005558f283bc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f69ccff9700 (LWP 99067)): #0 0x00007f6a2217de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005558f268755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005558f282db44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f69cdffb700 (LWP 99060)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f268725b in os_event::wait (this=0x5558f4938510) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005558f2686ce1 in os_event::wait_low (this=0x5558f4938510, reset_sig_count=7136) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005558f2687052 in os_event_wait_low (event=0x5558f4938510, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005558f2772ee5 in srv_resume_thread (slot=0x5558f36c17f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005558f2776ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f69e67ad700 (LWP 99014)): #0 0x00007f6a21d47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005558f267a578 in LinuxAIOHandler::collect (this=0x7f69e67acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005558f267aaa1 in LinuxAIOHandler::poll (this=0x7f69e67acb00, m1=0x7f69e67acba8, m2=0x7f69e67acbb0, request=0x7f69e67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005558f267ac04 in os_aio_linux_handler (global_segment=1, m1=0x7f69e67acba8, m2=0x7f69e67acbb0, request=0x7f69e67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005558f267f728 in os_aio_handler (segment=1, m1=0x7f69e67acba8, m2=0x7f69e67acbb0, request=0x7f69e67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005558f28d25ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005558f27789fb in io_handler_thread (arg=0x5558f38ff628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f69ceffd700 (LWP 99058)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f268725b in os_event::wait (this=0x5558f4938490) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005558f2686ce1 in os_event::wait_low (this=0x5558f4938490, reset_sig_count=7599) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005558f2687052 in os_event_wait_low (event=0x5558f4938490, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005558f2772ee5 in srv_resume_thread (slot=0x5558f36c17b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005558f2776ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f69deffd700 (LWP 99047)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2686b76 in os_event::timed_wait (this=0x5558f4938590, abstime=0x7f69deffcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005558f2686e8d in os_event::wait_time_low (this=0x5558f4938590, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005558f268702d in os_event_wait_time_low (event=0x5558f4938590, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005558f27750ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f69cf7fe700 (LWP 99057)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f268725b in os_event::wait (this=0x5558f4938390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005558f2686ce1 in os_event::wait_low (this=0x5558f4938390, reset_sig_count=2073) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005558f2687052 in os_event_wait_low (event=0x5558f4938390, reset_sig_count=2073) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005558f2772ee5 in srv_resume_thread (slot=0x5558f36c1748 , sig_count=2073, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005558f2776ef6 in srv_purge_coordinator_suspend (slot=0x5558f36c1748 , rseg_history_len=38) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005558f277718c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f69e5fac700 (LWP 99015)): #0 0x00007f6a21d47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005558f267a578 in LinuxAIOHandler::collect (this=0x7f69e5fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005558f267aaa1 in LinuxAIOHandler::poll (this=0x7f69e5fabb00, m1=0x7f69e5fabba8, m2=0x7f69e5fabbb0, request=0x7f69e5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005558f267ac04 in os_aio_linux_handler (global_segment=2, m1=0x7f69e5fabba8, m2=0x7f69e5fabbb0, request=0x7f69e5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005558f267f728 in os_aio_handler (segment=2, m1=0x7f69e5fabba8, m2=0x7f69e5fabbb0, request=0x7f69e5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005558f28d25ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005558f27789fb in io_handler_thread (arg=0x5558f38ff630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f69ddffb700 (LWP 99048)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2686b76 in os_event::timed_wait (this=0x5558f4938610, abstime=0x7f69ddffad20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005558f2686e8d in os_event::wait_time_low (this=0x5558f4938610, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005558f268702d in os_event_wait_time_low (event=0x5558f4938610, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005558f2774bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f6a18120700 (LWP 98704)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2b44a8d in safe_cond_wait (cond=0x7f6a00017b68, mp=0x7f6a00017ab8, file=0x5558f2caec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005558f229c126 in inline_mysql_cond_wait (that=0x7f6a00017b68, mutex=0x7f6a00017ab8, src_file=0x5558f2caef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005558f229edbe in Wsrep_thd_queue::pop_front (this=0x7f6a00017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005558f229d7de in wsrep_rollback_process (rollbacker=0x7f6a00001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005558f228e31f in start_wsrep_THD (arg=0x5558f4848c40) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f69e6fae700 (LWP 99013)): #0 0x00007f6a21d47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005558f267a578 in LinuxAIOHandler::collect (this=0x7f69e6fadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005558f267aaa1 in LinuxAIOHandler::poll (this=0x7f69e6fadb00, m1=0x7f69e6fadba8, m2=0x7f69e6fadbb0, request=0x7f69e6fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005558f267ac04 in os_aio_linux_handler (global_segment=0, m1=0x7f69e6fadba8, m2=0x7f69e6fadbb0, request=0x7f69e6fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005558f267f728 in os_aio_handler (segment=0, m1=0x7f69e6fadba8, m2=0x7f69e6fadbb0, request=0x7f69e6fadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005558f28d25ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005558f27789fb in io_handler_thread (arg=0x5558f38ff620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f6a180d5700 (LWP 98705)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2b44a8d in safe_cond_wait (cond=0x7f69f8015608, mp=0x7f69f8015558, file=0x5558f2caec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005558f229c126 in inline_mysql_cond_wait (that=0x7f69f8015608, mutex=0x7f69f8015558, src_file=0x5558f2caef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005558f229edbe in Wsrep_thd_queue::pop_front (this=0x7f69f8015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005558f229db9e in wsrep_post_rollback_process (post_rollbacker=0x7f69f8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005558f228e31f in start_wsrep_THD (arg=0x5558f483fdb0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f6a1abcf700 (LWP 98632)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2b44d8f in safe_cond_timedwait (cond=0x5558f4123640 , mp=0x5558f4123580 , abstime=0x7f6a1abced70, file=0x5558f2fcc920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005558f2b45d49 in inline_mysql_cond_timedwait (that=0x5558f4123640 , mutex=0x5558f4123580 , abstime=0x7f6a1abced70, src_file=0x5558f2fcc968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005558f2b468ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005558f2ad7555 in pfs_spawn_thread (arg=0x5558f47b47d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f6a080aa700 (LWP 99072)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2b44d8f in safe_cond_timedwait (cond=0x5558f38841a0 , mp=0x5558f3886ba0 , abstime=0x7f6a080a9c60, file=0x5558f2c005c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005558f1f11845 in inline_mysql_cond_timedwait (that=0x5558f38841a0 , mutex=0x5558f3886ba0 , abstime=0x7f6a080a9c60, src_file=0x5558f2c02638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005558f1f1661b in cache_thread (thd=0x7f69f4000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005558f1f168c9 in one_thread_per_connection_end (thd=0x7f69f4000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005558f21b30f3 in do_handle_one_connection (connect=0x5558f4c99d20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005558f21b2dad in handle_one_connection (arg=0x5558f4c99d20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005558f2ad7555 in pfs_spawn_thread (arg=0x5558f4cbf7f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f69e4faa700 (LWP 99017)): #0 0x00007f6a21d47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005558f267a578 in LinuxAIOHandler::collect (this=0x7f69e4fa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005558f267aaa1 in LinuxAIOHandler::poll (this=0x7f69e4fa9b00, m1=0x7f69e4fa9ba8, m2=0x7f69e4fa9bb0, request=0x7f69e4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005558f267ac04 in os_aio_linux_handler (global_segment=4, m1=0x7f69e4fa9ba8, m2=0x7f69e4fa9bb0, request=0x7f69e4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005558f267f728 in os_aio_handler (segment=4, m1=0x7f69e4fa9ba8, m2=0x7f69e4fa9bb0, request=0x7f69e4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005558f28d25ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005558f27789fb in io_handler_thread (arg=0x5558f38ff640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f6a1808a700 (LWP 98706)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6a19f79132 in fifo_lock_get (q=q@entry=0x5558f481cf10) at galerautils/src/gu_fifo.c:262 #2 0x00007f6a19f797a1 in gu_fifo_get_head (q=0x5558f481cf10, err=0x7f6a1808906c) at galerautils/src/gu_fifo.c:330 #3 0x00007f6a1a05b0cc in gcs_recv (conn=0x5558f47f6890, action=0x7f6a180890b0) at gcs/src/gcs.cpp:2024 #4 0x00007f6a1a08816c in galera::GcsActionSource::process (this=0x5558f47f2ae0, recv_ctx=0x7f6a180894d0, exit_loop=@0x7f6a1808929e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f6a1a0a7033 in galera::ReplicatorSMM::async_recv (this=0x5558f47eb1f0, recv_ctx=0x7f6a180894d0) at galera/src/replicator_smm.cpp:383 #6 0x00005558f2bf936c in wsrep::wsrep_provider_v26::run_applier (this=0x5558f4760e50, applier_ctx=0x7f6a180894d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005558f229c6d4 in wsrep_replication_process (thd=0x7f69f0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005558f228e31f in start_wsrep_THD (arg=0x5558f48488c0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f69ce7fc700 (LWP 99059)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f268725b in os_event::wait (this=0x5558f4938410) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005558f2686ce1 in os_event::wait_low (this=0x5558f4938410, reset_sig_count=8274) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005558f2687052 in os_event_wait_low (event=0x5558f4938410, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005558f2772ee5 in srv_resume_thread (slot=0x5558f36c1780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005558f2776ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f6a19a23700 (LWP 98633)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6a1a084fd7 in gu::Lock::wait (cond=..., this=0x7f6a19a22bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6a1a085069 in galera::ServiceThd::thd_func (arg=0x5558f47eb908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f69dcff9700 (LWP 99050)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2686b76 in os_event::timed_wait (this=0x5558f4ae0930, abstime=0x7f69dcff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005558f2686e8d in os_event::wait_time_low (this=0x5558f4ae0930, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005558f268702d in os_event_wait_time_low (event=0x5558f4ae0930, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005558f27d5403 in ib_wqueue_timedwait (wq=0x5558f4ae0800, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005558f2915f9a in fts_optimize_thread (arg=0x5558f4ae0800) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f6a18921700 (LWP 98666)): #0 0x00007f6a20535483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6a1a0266f8 in asio::detail::epoll_reactor::run (this=0x5558f4828050, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f6a1a02adf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5558f4827ef0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f6a1a02b0a2 in asio::detail::task_io_service::run (this=0x5558f4827ef0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f6a1a042afd in asio::io_service::run (this=0x5558f4826d40) at asio/asio/impl/io_service.ipp:58 #5 0x00007f6a1a043251 in gcomm::AsioProtonet::event_loop (this=0x5558f4826ca0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f6a1a0612ed in GCommConn::run (this=0x5558f4827630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f6a1a064399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f69cffff700 (LWP 99056)): #0 0x00007f6a2217de3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005558f268755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005558f277644d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005558f277657d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f69dd7fa700 (LWP worker[3] Trying to dump core for [mysqltest - pid: 120628, winpid: 120628] worker[3] Trying to dump core for [mysqld.1 - pid: 119753, winpid: 119753] worker[3] Trying to dump core for [mysqld.2 - pid: 120066, winpid: 120066] 99049)): #0 0x00007f6a2217ad12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005558f2686b76 in os_event::timed_wait (this=0x5558f4b42740, abstime=0x7f69dd7f9d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005558f2686e8d in os_event::wait_time_low (this=0x5558f4b42740, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005558f268702d in os_event_wait_time_low (event=0x5558f4b42740, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005558f28bc24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f69e57ab700 (LWP 99016)): #0 0x00007f6a21d47644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005558f267a578 in LinuxAIOHandler::collect (this=0x7f69e57aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005558f267aaa1 in LinuxAIOHandler::poll (this=0x7f69e57aab00, m1=0x7f69e57aaba8, m2=0x7f69e57aabb0, request=0x7f69e57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005558f267ac04 in os_aio_linux_handler (global_segment=3, m1=0x7f69e57aaba8, m2=0x7f69e57aabb0, request=0x7f69e57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005558f267f728 in os_aio_handler (segment=3, m1=0x7f69e57aaba8, m2=0x7f69e57aabb0, request=0x7f69e57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005558f28d25ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005558f27789fb in io_handler_thread (arg=0x5558f38ff638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f6a0ac7c700 (LWP 98703)): #0 0x00007f6a2217a965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6a1a05f867 in gu::Lock::wait (cond=..., this=0x7f6a0ac7b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6a1a0623ab in RecvBuf::front (this=this@entry=0x5558f4827818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f6a1a05ff46 in gcomm_recv (backend=, msg=0x5558f47f6b70, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f6a1a053eec in core_msg_recv (backend=backend@entry=0x5558f47f6cb8, recv_msg=recv_msg@entry=0x5558f47f6b70, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f6a1a057164 in gcs_core_recv (conn=0x5558f47f6b10, recv_act=recv_act@entry=0x7f6a0ac7bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f6a1a05e3d8 in gcs_recv_thread (arg=0x5558f47f6890) at gcs/src/gcs.cpp:1398 #7 0x00007f6a22176dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6a20534ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f6a22585880 (LWP 98615)): #0 0x00007f6a2217b9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005558f2b3f45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005558f237d5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6a2052a20d in poll () from /lib64/libc.so.6 #5 0x00005558f1f1da8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005558f1f1cd25 in mysqld_main (argc=162, argv=0x5558f466cd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005558f1f10ec5 in main (argc=23, argv=0x7ffc99af2d18) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.GCF-1043A-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043A-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1043A galera_sr.GCF-1018 galera_sr.mysql-wsrep-features#9 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1060 'innodb' w2 [ pass ] 626893 galera_sr.GCF-561 'innodb' w2 [ pass ] 12296 galera_sr.GCF-571 'innodb' w2 [ pass ] 12241 galera_sr.galera_sr_many_fragments 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 13:17:35 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr_many_fragments.log == connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.119755' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.1/data/core.119755 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 119755] [New LWP 119844] [New LWP 119845] [New LWP 119846] [New LWP 119849] [New LWP 119857] [New LWP 120010] [New LWP 120708] [New LWP 119802] [New LWP 119800] [New LWP 119864] [New LWP 119863] [New LWP 119862] [New LWP 119856] [New LWP 120008] [New LWP 119865] [New LWP 119861] [New LWP 119860] [New LWP 119855] [New LWP 119854] [New LWP 119852] [New LWP 119859] [New LWP 119843] [New LWP 119851] [New LWP 119858] [New LWP 119764] [New LWP 119848] [New LWP 119853] [New LWP 119761] [New LWP 119801] [New LWP 119803] [New LWP 119842] [New LWP 119799] [New LWP 119847] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f29b4e969d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f29b4e969d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005590061bf45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005590059fd5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f29b324520d in poll () from /lib64/libc.so.6 #5 0x000055900559da8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055900559cd25 in mysqld_main (argc=161, argv=0x5590085d7d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559005590ec5 in main (argc=23, argv=0x7ffc971492d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f2983fff700 (LWP 119847)): #0 0x00007f29b4e98d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000559005cfbc27 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000559005cfbf7d in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000559005f444aa in pfs_os_file_flush_func (file=..., src_file=0x5590065beb50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559005f47225 in fil_flush_low (space=0x559008aa8400, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000559005f52aff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000559005f52d86 in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x0000559005ecf10a in buf_dblwr_update (bpage=0x7f298a41ddf0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x0000559005ed6af7 in buf_flush_write_complete (bpage=0x7f298a41ddf0, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x0000559005ec57b1 in buf_page_io_complete (bpage=0x7f298a41ddf0, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x0000559005f52859 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x0000559005df89fb in io_handler_thread (arg=0x559006f7f640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f29a517a700 (LWP 119799)): #0 0x00007f29b3250483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f29acd416f8 in asio::detail::epoll_reactor::run (this=0x559008797040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f29acd45df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559008796ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f29acd460a2 in asio::detail::task_io_service::run (this=0x559008796ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f29acd5dafd in asio::io_service::run (this=0x559008795fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f29acd5e251 in gcomm::AsioProtonet::event_loop (this=0x559008795f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f29acd7c2ed in GCommConn::run (this=0x5590087966c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f29acd7f399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f298b85a700 (LWP 119842)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590061c4d8f in safe_cond_timedwait (cond=0x559007790ea0 , mp=0x559007790de0 , abstime=0x7f298b859cb0, file=0x5590066086f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055900602ebc1 in inline_mysql_cond_timedwait (that=0x559007790ea0 , mutex=0x559007790de0 , abstime=0x7f298b859cb0, src_file=0x559006608738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055900602f0e5 in my_service_thread_sleep (control=0x559006d45b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055900602357e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559006157555 in pfs_spawn_thread (arg=0x559008856880) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f29ac6a8700 (LWP 119803)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f29acc94132 in fifo_lock_get (q=q@entry=0x559008784bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f29acc947a1 in gu_fifo_get_head (q=0x559008784bd0, err=0x7f29ac6a706c) at galerautils/src/gu_fifo.c:330 #3 0x00007f29acd760cc in gcs_recv (conn=0x55900875e550, action=0x7f29ac6a70b0) at gcs/src/gcs.cpp:2024 #4 0x00007f29acda316c in galera::GcsActionSource::process (this=0x55900875df30, recv_ctx=0x7f29ac6a74d0, exit_loop=@0x7f29ac6a729e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f29acdc2033 in galera::ReplicatorSMM::async_recv (this=0x559008756280, recv_ctx=0x7f29ac6a74d0) at galera/src/replicator_smm.cpp:383 #6 0x000055900627936c in wsrep::wsrep_provider_v26::run_applier (this=0x5590086dd360, applier_ctx=0x7f29ac6a74d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055900591c6d4 in wsrep_replication_process (thd=0x7f2994001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055900590e31f in start_wsrep_THD (arg=0x5590087af2a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f29ac73e700 (LWP 119801)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590061c4a8d in safe_cond_wait (cond=0x7f299c015608, mp=0x7f299c015558, file=0x55900632ec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055900591c126 in inline_mysql_cond_wait (that=0x7f299c015608, mutex=0x7f299c015558, src_file=0x55900632ef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055900591edbe in Wsrep_thd_queue::pop_front (this=0x7f299c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055900591d7de in wsrep_rollback_process (rollbacker=0x7f299c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055900590e31f in start_wsrep_THD (arg=0x5590087afe90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f29ad8ea700 (LWP 119761)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590061c4d8f in safe_cond_timedwait (cond=0x5590077a3640 , mp=0x5590077a3580 , abstime=0x7f29ad8e9d70, file=0x55900664c920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005590061c5d49 in inline_mysql_cond_timedwait (that=0x5590077a3640 , mutex=0x5590077a3580 , abstime=0x7f29ad8e9d70, src_file=0x55900664c968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005590061c68ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559006157555 in pfs_spawn_thread (arg=0x55900871f730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f29827fc700 (LWP 119853)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d06b76 in os_event::timed_wait (this=0x55900889e870, abstime=0x7f29827fbd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559005d06e8d in os_event::wait_time_low (this=0x55900889e870, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559005d0702d in os_event_wait_time_low (event=0x55900889e870, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559005df4bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f29837fe700 (LWP 119848)): #0 0x00007f29b4a62644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559005cfa578 in LinuxAIOHandler::collect (this=0x7f29837fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559005cfaaa1 in LinuxAIOHandler::poll (this=0x7f29837fdb00, m1=0x7f29837fdba8, m2=0x7f29837fdbb0, request=0x7f29837fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559005cfac04 in os_aio_linux_handler (global_segment=5, m1=0x7f29837fdba8, m2=0x7f29837fdbb0, request=0x7f29837fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559005cff728 in os_aio_handler (segment=5, m1=0x7f29837fdba8, m2=0x7f29837fdbb0, request=0x7f29837fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559005f525ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559005df89fb in io_handler_thread (arg=0x559006f7f648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f29a75fe700 (LWP 119764)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f29acd9ffd7 in gu::Lock::wait (cond=..., this=0x7f29a75fdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f29acda0069 in galera::ServiceThd::thd_func (arg=0x559008756998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f297a7fc700 (LWP 119858)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d0725b in os_event::wait (this=0x55900889e770) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559005d06ce1 in os_event::wait_low (this=0x55900889e770, reset_sig_count=7248) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559005d07052 in os_event_wait_low (event=0x55900889e770, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559005df2ee5 in srv_resume_thread (slot=0x559006d417f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559005df6ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f2981ffb700 (LWP 119851)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d06b76 in os_event::timed_wait (this=0x559008a76d20, abstime=0x7f2981ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559005d06e8d in os_event::wait_time_low (this=0x559008a76d20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559005d0702d in os_event_wait_time_low (event=0x559008a76d20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559005cd39e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f298a208700 (LWP 119843)): #0 0x00007f29b4a62644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559005cfa578 in LinuxAIOHandler::collect (this=0x7f298a207b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559005cfaaa1 in LinuxAIOHandler::poll (this=0x7f298a207b00, m1=0x7f298a207ba8, m2=0x7f298a207bb0, request=0x7f298a207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559005cfac04 in os_aio_linux_handler (global_segment=0, m1=0x7f298a207ba8, m2=0x7f298a207bb0, request=0x7f298a207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559005cff728 in os_aio_handler (segment=0, m1=0x7f298a207ba8, m2=0x7f298a207bb0, request=0x7f298a207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559005f525ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559005df89fb in io_handler_thread (arg=0x559006f7f620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f2979ffb700 (LWP 119859)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d0725b in os_event::wait (this=0x55900889e670) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559005d06ce1 in os_event::wait_low (this=0x55900889e670, reset_sig_count=8778) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559005d07052 in os_event_wait_low (event=0x55900889e670, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559005df2ee5 in srv_resume_thread (slot=0x559006d41780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559005df6ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f29817fa700 (LWP 119852)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d06b76 in os_event::timed_wait (this=0x55900889e7f0, abstime=0x7f29817f9ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559005d06e8d in os_event::wait_time_low (this=0x55900889e7f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559005d0702d in os_event_wait_time_low (event=0x55900889e7f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559005df50ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f2980ff9700 (LWP 119854)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d06b76 in os_event::timed_wait (this=0x559008aa8de0, abstime=0x7f2980ff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559005d06e8d in os_event::wait_time_low (this=0x559008aa8de0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559005d0702d in os_event_wait_time_low (event=0x559008aa8de0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559005f3c24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f297bfff700 (LWP 119855)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d06b76 in os_event::timed_wait (this=0x559008a3ea70, abstime=0x7f297bffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559005d06e8d in os_event::wait_time_low (this=0x559008a3ea70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559005d0702d in os_event_wait_time_low (event=0x559008a3ea70, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559005e55403 in ib_wqueue_timedwait (wq=0x559008a3e940, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559005f95f9a in fts_optimize_thread (arg=0x559008a3e940) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f29797fa700 (LWP 119860)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d0725b in os_event::wait (this=0x55900889e6f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559005d06ce1 in os_event::wait_low (this=0x55900889e6f0, reset_sig_count=7660) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559005d07052 in os_event_wait_low (event=0x55900889e6f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559005df2ee5 in srv_resume_thread (slot=0x559006d417b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559005df6ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f2978ff9700 (LWP 119861)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d0725b in os_event::wait (this=0x55900889e8f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559005d06ce1 in os_event::wait_low (this=0x55900889e8f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559005d07052 in os_event_wait_low (event=0x55900889e8f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559005ed2fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f29ac099700 (LWP 119865)): #0 0x00007f29b318940a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000559005590f44 in my_sigwait (set=0x7f29ac098cf0, sig=0x7f29ac098cb4, code=0x7f29ac098cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000559005597584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000559006157555 in pfs_spawn_thread (arg=0x559008861ef0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f29ac04e700 (LWP 120008)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590061c4a8d in safe_cond_wait (cond=0x559006f041e0 , mp=0x559006f07560 , file=0x55900628fdc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005590055c24ee in inline_mysql_cond_wait (that=0x559006f041e0 , mutex=0x559006f07560 , src_file=0x559006290930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005590055c3d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559006157555 in pfs_spawn_thread (arg=0x559008c000d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f297b7fe700 (LWP 119856)): #0 0x00007f29b4e98e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559005d0755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559005df644d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000559005df657d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f295ffff700 (LWP 119862)): #0 0x00007f29b4e98e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559005d0755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559005eadb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f295f7fe700 (LWP 119863)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d0725b in os_event::wait (this=0x55900889e9f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559005d06ce1 in os_event::wait_low (this=0x55900889e9f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559005d07052 in os_event_wait_low (event=0x55900889e9f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559005ebbc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f295effd700 (LWP 119864)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590061c4a8d in safe_cond_wait (cond=0x7f295effcc70, mp=0x7f295effccb0, file=0x559006492d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559005c1e525 in inline_mysql_cond_wait (that=0x7f295effcc70, mutex=0x7f295effccb0, src_file=0x5590064946d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559005c23161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000559006157555 in pfs_spawn_thread (arg=0x559008adf930) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f29a4979700 (LWP 119800)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f29acd7a867 in gu::Lock::wait (cond=..., this=0x7f29a4978850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f29acd7d3ab in RecvBuf::front (this=this@entry=0x5590087968a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f29acd7af46 in gcomm_recv (backend=, msg=0x55900875e830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f29acd6eeec in core_msg_recv (backend=backend@entry=0x55900875e978, recv_msg=recv_msg@entry=0x55900875e830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f29acd72164 in gcs_core_recv (conn=0x55900875e7d0, recv_act=recv_act@entry=0x7f29a4978d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f29acd793d8 in gcs_recv_thread (arg=0x55900875e550) at gcs/src/gcs.cpp:1398 #7 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f29ac6f3700 (LWP 119802)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590061c4a8d in safe_cond_wait (cond=0x7f298c015608, mp=0x7f298c015558, file=0x55900632ec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055900591c126 in inline_mysql_cond_wait (that=0x7f298c015608, mutex=0x7f298c015558, src_file=0x55900632ef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055900591edbe in Wsrep_thd_queue::pop_front (this=0x7f298c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055900591db9e in wsrep_post_rollback_process (post_rollbacker=0x7f298c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055900590e31f in start_wsrep_THD (arg=0x5590087a6320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f29a412d700 (LWP 120708)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d0725b in os_event::wait (this=0x5590089847c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559005d06ce1 in os_event::wait_low (this=0x5590089847c0, reset_sig_count=879) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559005d07052 in os_event_wait_low (event=0x5590089847c0, reset_sig_count=879) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559005e00cee in sync_array_wait_event (arr=0x559008875a50, cell=@0x7f29a4127330: 0x559008875c80) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000559005c52195 in TTASEventMutex::enter (this=0x5590089840f0, max_spins=60, max_delay=4, filename=0x55900658b498 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc", line=3599) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x0000559005c503f0 in PolicyMutex >::enter (this=0x5590089840f0, n_spins=30, n_delay=4, name=0x55900658b498 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc", line=3599) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x0000559005ebdc0f in buf_page_make_young (bpage=0x7f298a43db60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3599 #8 0x0000559005ebdd4a in buf_page_make_young_if_needed (bpage=0x7f298a43db60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3626 #9 0x0000559005ec1246 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x559006563a08 "/home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0undo.cc", line=1343, mtr=0x7f29a4127840, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4789 #10 0x0000559005e4fed6 in trx_undo_reuse_cached (trx=0x7f298ac5c390, rseg=0x559008ac7630, pundo=0x7f298ac5d3b0, mtr=0x7f29a4127840) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0undo.cc:1341 #11 0x0000559005e50616 in trx_undo_assign_low (trx=0x7f298ac5c390, rseg=0x559008ac7630, undo=0x7f298ac5d3b0, err=0x7f29a41277c8, mtr=0x7f29a4127840) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0undo.cc:1471 #12 0x0000559005e31ad7 in trx_undo_report_row_operation (thr=0x7f2934159bc8, index=0x559008c39fb8, clust_entry=0x7f293415b158, update=0x0, cmpl_info=0, rec=0x0, offsets=0x0, roll_ptr=0x7f29a4127dd0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0rec.cc:2115 #13 0x0000559005e85fd5 in btr_cur_ins_lock_and_undo (flags=2, cursor=0x7f29a4128280, entry=0x7f293415b158, thr=0x7f2934159bc8, mtr=0x7f29a41286a0, inherit=0x7f29a4127e49) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3272 #14 0x0000559005e869d0 in btr_cur_optimistic_insert (flags=2, cursor=0x7f29a4128280, offsets=0x7f29a4128218, heap=0x7f29a4128210, entry=0x7f293415b158, rec=0x7f29a4128220, big_rec=0x7f29a4128208, n_ext=0, thr=0x7f2934159bc8, mtr=0x7f29a41286a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3483 #15 0x0000559005d70a9c in row_ins_clust_index_entry_low (flags=2, mode=2, index=0x559008c39fb8, n_uniq=3, entry=0x7f293415b158, n_ext=0, thr=0x7f2934159bc8, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:2756 #16 0x0000559005d7246e in row_ins_clust_index_entry (index=0x559008c39fb8, entry=0x7f293415b158, thr=0x7f2934159bc8, n_ext=0, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3280 #17 0x0000559005d7299c in row_ins_index_entry (index=0x559008c39fb8, entry=0x7f293415b158, thr=0x7f2934159bc8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3405 #18 0x0000559005d730db in row_ins_index_entry_step (node=0x7f29341598d0, thr=0x7f2934159bc8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3574 #19 0x0000559005d7350c in row_ins (node=0x7f29341598d0, thr=0x7f2934159bc8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3717 #20 0x0000559005d73d0b in row_ins_step (thr=0x7f2934159bc8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3962 #21 0x0000559005d93878 in row_insert_for_mysql (mysql_rec=0x7f2934158250 "db87973b-2b90-11e9-8d26-8b17a8e555a1F", prebuilt=0x7f29341592a8, ins_mode=ROW_INS_NORMAL) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:1460 #22 0x0000559005c313f6 in ha_innobase::write_row (this=0x7f2934158698, record=0x7f2934158250 "db87973b-2b90-11e9-8d26-8b17a8e555a1F") at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:8050 #23 0x0000559005a10869 in handler::ha_write_row (this=0x7f2934158698, buf=0x7f2934158250 "db87973b-2b90-11e9-8d26-8b17a8e555a1F") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6463 #24 0x0000559005921305 in Wsrep_schema_impl::insert (table=0x7f2934157600) at /home/stepan/galera/git/10.4/server/sql/wsrep_schema.cc:369 #25 0x0000559005923911 in Wsrep_schema::append_fragment (this=0x559008b9c4b0, thd=0x7f293414c5a0, server_id=..., transaction_id=..., seqno=..., flags=0, data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_schema.cc:915 #26 0x0000559005c06cba in Wsrep_storage_service::append_fragment (this=0x7f293405cec0, server_id=..., transaction_id=..., flags=0, data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:114 #27 0x000055900626f36b in wsrep::transaction::certify_fragment (this=0x7f29340069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1125 #28 0x000055900626ed2d in wsrep::transaction::streaming_step (this=0x7f29340069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #29 0x000055900626c0e8 in wsrep::transaction::after_row (this=0x7f29340069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #30 0x0000559005a15446 in wsrep::client_state::after_row (this=0x7f2934006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #31 0x00005590059fe6c3 in wsrep_after_row (thd=0x7f2934000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #32 0x0000559005a105c8 in wsrep_after_row (thd=0x7f2934000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #33 0x0000559005a109b9 in handler::ha_write_row (this=0x7f293407ca68, buf=0x7f293407d218 "\376)\020") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #34 0x0000559005686443 in write_record (thd=0x7f2934000b00, table=0x7f29340779a0, info=0x7f2934017040) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #35 0x000055900568b742 in select_insert::send_data (this=0x7f2934016ff8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #36 0x0000559005743e07 in end_send (join=0x7f29340170a8, join_tab=0x7f29340887c8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #37 0x0000559005880fa7 in JOIN_CACHE::generate_full_extensions (this=0x7f2934088dd8, rec_ptr=0x7f293410bb60 "\024\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #38 0x0000559005880cc2 in JOIN_CACHE::join_matching_records (this=0x7f2934088dd8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #39 0x000055900588068e in JOIN_CACHE::join_records (this=0x7f2934088dd8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #40 0x0000559005880828 in JOIN_CACHE::join_records (this=0x7f2934088ca8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #41 0x0000559005880828 in JOIN_CACHE::join_records (this=0x7f2934088b78, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #42 0x0000559005740abb in sub_select_cache (join=0x7f29340170a8, join_tab=0x7f2934087cb8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #43 0x0000559005740cc8 in sub_select (join=0x7f29340170a8, join_tab=0x7f2934087908, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #44 0x0000559005740493 in do_select (join=0x7f29340170a8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #45 0x0000559005718fb1 in JOIN::exec_inner (this=0x7f29340170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #46 0x00005590057183fa in JOIN::exec (this=0x7f29340170a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #47 0x00005590057196ec in mysql_select (thd=0x7f2934000b00, tables=0x7f2934014dd0, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7f2934016ff8, unit=0x7f29340049c0, select_lex=0x7f2934014700) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #48 0x000055900570afed in handle_select (thd=0x7f2934000b00, lex=0x7f29340048f8, result=0x7f2934016ff8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #49 0x00005590056cd3fc in mysql_execute_command (thd=0x7f2934000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #50 0x00005590056d8935 in mysql_parse (thd=0x7f2934000b00, rawbuf=0x7f2934013e58 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f29a412c0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #51 0x00005590056d7f68 in wsrep_mysql_parse (thd=0x7f2934000b00, rawbuf=0x7f2934013e58 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7f29a412c0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #52 0x00005590056c3e25 in dispatch_command (command=COM_QUERY, thd=0x7f2934000b00, packet=0x7f293400a431 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #53 0x00005590056c280d in do_command (thd=0x7f2934000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #54 0x0000559005833069 in do_handle_one_connection (connect=0x559008b320b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #55 0x0000559005832dad in handle_one_connection (arg=0x559008b320b0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #56 0x0000559006157555 in pfs_spawn_thread (arg=0x559008712a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #57 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #58 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f29a4178700 (LWP 120010)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005590061c4d8f in safe_cond_timedwait (cond=0x559006f041a0 , mp=0x559006f06ba0 , abstime=0x7f29a4177c60, file=0x5590062805c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559005591845 in inline_mysql_cond_timedwait (that=0x559006f041a0 , mutex=0x559006f06ba0 , abstime=0x7f29a4177c60, src_file=0x559006282638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055900559661b in cache_thread (thd=0x7f2940000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005590055968c9 in one_thread_per_connection_end (thd=0x7f2940000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005590058330f3 in do_handle_one_connection (connect=0x559008c29bf0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000559005832dad in handle_one_connection (arg=0x559008c29bf0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000559006157555 in pfs_spawn_thread (arg=0x559008c2f350) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f297affd700 (LWP 119857)): #0 0x00007f29b4e95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d0725b in os_event::wait (this=0x55900889e5f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559005d06ce1 in os_event::wait_low (this=0x55900889e5f0, reset_sig_count=2003) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559005d07052 in os_event_wait_low (event=0x55900889e5f0, reset_sig_count=2003) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559005df2ee5 in srv_resume_thread (slot=0x559006d41748 , sig_count=2003, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000559005df6ef6 in srv_purge_coordinator_suspend (slot=0x559006d41748 , rseg_history_len=18) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000559005df718c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f2982ffd700 (LWP 119849)): #0 0x00007f29b4e95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559005d06b76 in os_event::timed_wait (this=0x55900889e970, abstime=0x7f2982ffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559005d06e8d in os_event::wait_time_low (this=0x55900889e970, time_in_usec=352000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559005d0702d in os_event_wait_time_low (event=0x55900889e970, time_in_usec=352000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559005edce25 in pc_sleep_if_needed (next_loop_time=1549624594064, sig_count=2, cur_time=1549624593712) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000559005eddd5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f2988a05700 (LWP 119846)): #0 0x00007f29b4a62644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559005cfa578 in LinuxAIOHandler::collect (this=0x7f2988a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559005cfaaa1 in LinuxAIOHandler::poll (this=0x7f2988a04b00, m1=0x7f2988a04ba8, m2=0x7f2988a04bb0, request=0x7f2988a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559005cfac04 in os_aio_linux_handler (global_segment=3, m1=0x7f2988a04ba8, m2=0x7f2988a04bb0, request=0x7f2988a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559005cff728 in os_aio_handler (segment=3, m1=0x7f2988a04ba8, m2=0x7f2988a04bb0, request=0x7f2988a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559005f525ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559005df89fb in io_handler_thread (arg=0x559006f7f638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f2989206700 (LWP 119845)): #0 0x00007f29b4a62644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559005cfa578 in LinuxAIOHandler::collect (this=0x7f2989205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559005cfaaa1 in LinuxAIOHandler::poll (this=0x7f2989205b00, m1=0x7f2989205ba8, m2=0x7f2989205bb0, request=0x7f2989205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559005cfac04 in os_aio_linux_handler (global_segment=2, m1=0x7f2989205ba8, m2=0x7f2989205bb0, request=0x7f2989205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559005cff728 in os_aio_handler (segment=2, m1=0x7f2989205ba8, m2=0x7f2989205bb0, request=0x7f2989205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559005f525ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559005df89fb in io_handler_thread (arg=0x559006f7f630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f2989a07700 (LWP 119844)): #0 0x00007f29b4a62644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559005cfa578 in LinuxAIOHandler::collect (this=0x7f2989a06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559005cfaaa1 in LinuxAIOHandler::poll (this=0x7f2989a06b00, m1=0x7f2989a06ba8, m2=0x7f2989a06bb0, request=0x7f2989a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559005cfac04 in os_aio_linux_handler (global_segment=1, m1=0x7f2989a06ba8, m2=0x7f2989a06bb0, request=0x7f2989a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559005cff728 in os_aio_handler (segment=1, m1=0x7f2989a06ba8, m2=0x7f2989a06bb0, request=0x7f2989a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000559005f525ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000559005df89fb in io_handler_thread (arg=0x559006f7f628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f29b4e91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f29b324fead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f29b52a2880 (LWP 119755)): #0 0x00007f29b4e969d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005590061bf45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005590059fd5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f29b324520d in poll () from /lib64/libc.so.6 #5 0x000055900559da8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055900559cd25 in mysqld_main (argc=161, argv=0x5590085d7d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559005590ec5 in main (argc=23, argv=0x7ffc971492d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.120068' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.2/data/core.120068 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 120068] [New LWP 120171] [New LWP 120070] [New LWP 120516] [New LWP 120141] [New LWP 120518] [New LWP 120515] [New LWP 120578] [New LWP 120519] [New LWP 120579] [New LWP 120580] [New LWP 120581] [New LWP 120582] [New LWP 120602] [New LWP 120588] [New LWP 120603] [New LWP 120609] [New LWP 120613] [New LWP 120610] [New LWP 120604] [New LWP 120592] [New LWP 120591] [New LWP 120601] [New LWP 120590] [New LWP 120589] [New LWP 120520] [New LWP 120517] [New LWP 120124] [New LWP 120170] [New LWP 120522] [New LWP 120521] [New LWP 120172] [New LWP 120173] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f18453e19d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f18453e19d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005594e290445e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005594e21425a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f184379020d in poll () from /lib64/libc.so.6 #5 0x00005594e1ce2a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005594e1ce1d25 in mysqld_main (argc=162, argv=0x5594e4447d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005594e1cd5ec5 in main (argc=23, argv=0x7ffc6a6cba68) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f183c3f2700 (LWP 120173)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f183d1df132 in fifo_lock_get (q=q@entry=0x5594e45f4ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007f183d1df7a1 in gu_fifo_get_head (q=0x5594e45f4ac0, err=0x7f183c3f106c) at galerautils/src/gu_fifo.c:330 #3 0x00007f183d2c10cc in gcs_recv (conn=0x5594e45ce440, action=0x7f183c3f10b0) at gcs/src/gcs.cpp:2024 #4 0x00007f183d2ee16c in galera::GcsActionSource::process (this=0x5594e45fee60, recv_ctx=0x7f183c3f14d0, exit_loop=@0x7f183c3f129e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f183d30d033 in galera::ReplicatorSMM::async_recv (this=0x5594e45c61f0, recv_ctx=0x7f183c3f14d0) at galera/src/replicator_smm.cpp:383 #6 0x00005594e29be36c in wsrep::wsrep_provider_v26::run_applier (this=0x5594e453be50, applier_ctx=0x7f183c3f14d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x00005594e20616d4 in wsrep_replication_process (thd=0x7f1818001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005594e205331f in start_wsrep_THD (arg=0x5594e45e7010) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f183c43d700 (LWP 120172)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e2909a8d in safe_cond_wait (cond=0x7f1820015608, mp=0x7f1820015558, file=0x5594e2a73c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005594e2061126 in inline_mysql_cond_wait (that=0x7f1820015608, mutex=0x7f1820015558, src_file=0x5594e2a73f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005594e2063dbe in Wsrep_thd_queue::pop_front (this=0x7f1820015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005594e2062b9e in wsrep_post_rollback_process (post_rollbacker=0x7f1820000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005594e205331f in start_wsrep_THD (arg=0x5594e45e6d70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f181f7fe700 (LWP 120521)): #0 0x00007f1844fad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005594e243f578 in LinuxAIOHandler::collect (this=0x7f181f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005594e243faa1 in LinuxAIOHandler::poll (this=0x7f181f7fdb00, m1=0x7f181f7fdba8, m2=0x7f181f7fdbb0, request=0x7f181f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005594e243fc04 in os_aio_linux_handler (global_segment=5, m1=0x7f181f7fdba8, m2=0x7f181f7fdbb0, request=0x7f181f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005594e2444728 in os_aio_handler (segment=5, m1=0x7f181f7fdba8, m2=0x7f181f7fdbb0, request=0x7f181f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005594e26975ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005594e253d9fb in io_handler_thread (arg=0x5594e36c4648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f181effd700 (LWP 120522)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244bb76 in os_event::timed_wait (this=0x5594e47175d0, abstime=0x7f181effcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005594e244be8d in os_event::wait_time_low (this=0x5594e47175d0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005594e244c02d in os_event_wait_time_low (event=0x5594e47175d0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005594e2621e25 in pc_sleep_if_needed (next_loop_time=1549624601342, sig_count=2, cur_time=1549624600342) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x00005594e2622d5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f1827fff700 (LWP 120170)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f183d2c5867 in gu::Lock::wait (cond=..., this=0x7f1827ffe850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f183d2c83ab in RecvBuf::front (this=this@entry=0x5594e4606818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f183d2c5f46 in gcomm_recv (backend=, msg=0x5594e45ce720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f183d2b9eec in core_msg_recv (backend=backend@entry=0x5594e45ce868, recv_msg=recv_msg@entry=0x5594e45ce720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f183d2bd164 in gcs_core_recv (conn=0x5594e45ce6c0, recv_act=recv_act@entry=0x7f1827ffed70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f183d2c43d8 in gcs_recv_thread (arg=0x5594e45ce440) at gcs/src/gcs.cpp:1398 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f183cc89700 (LWP 120124)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f183d2eafd7 in gu::Lock::wait (cond=..., this=0x7f183cc88bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f183d2eb069 in galera::ServiceThd::thd_func (arg=0x5594e45c6908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f18259ab700 (LWP 120517)): #0 0x00007f1844fad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005594e243f578 in LinuxAIOHandler::collect (this=0x7f18259aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005594e243faa1 in LinuxAIOHandler::poll (this=0x7f18259aab00, m1=0x7f18259aaba8, m2=0x7f18259aabb0, request=0x7f18259aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005594e243fc04 in os_aio_linux_handler (global_segment=1, m1=0x7f18259aaba8, m2=0x7f18259aabb0, request=0x7f18259aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005594e2444728 in os_aio_handler (segment=1, m1=0x7f18259aaba8, m2=0x7f18259aabb0, request=0x7f18259aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005594e26975ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005594e253d9fb in io_handler_thread (arg=0x5594e36c4628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f181ffff700 (LWP 120520)): #0 0x00007f1844fad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005594e243f578 in LinuxAIOHandler::collect (this=0x7f181fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005594e243faa1 in LinuxAIOHandler::poll (this=0x7f181fffeb00, m1=0x7f181fffeba8, m2=0x7f181fffebb0, request=0x7f181fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005594e243fc04 in os_aio_linux_handler (global_segment=4, m1=0x7f181fffeba8, m2=0x7f181fffebb0, request=0x7f181fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005594e2444728 in os_aio_handler (segment=4, m1=0x7f181fffeba8, m2=0x7f181fffebb0, request=0x7f181fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005594e26975ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005594e253d9fb in io_handler_thread (arg=0x5594e36c4640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f1816ffd700 (LWP 120589)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244c25b in os_event::wait (this=0x5594e4717250) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005594e244bce1 in os_event::wait_low (this=0x5594e4717250, reset_sig_count=2065) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005594e244c052 in os_event_wait_low (event=0x5594e4717250, reset_sig_count=2065) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005594e2537ee5 in srv_resume_thread (slot=0x5594e3486748 , sig_count=2065, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005594e253bef6 in srv_purge_coordinator_suspend (slot=0x5594e3486748 , rseg_history_len=7) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x00005594e253c18c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f18167fc700 (LWP 120590)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244c25b in os_event::wait (this=0x5594e4717350) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005594e244bce1 in os_event::wait_low (this=0x5594e4717350, reset_sig_count=7784) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005594e244c052 in os_event_wait_low (event=0x5594e4717350, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005594e2537ee5 in srv_resume_thread (slot=0x5594e34867b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005594e253bae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f1814ff9700 (LWP 120601)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244c25b in os_event::wait (this=0x5594e4717550) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005594e244bce1 in os_event::wait_low (this=0x5594e4717550, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005594e244c052 in os_event_wait_low (event=0x5594e4717550, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005594e2617fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f1815ffb700 (LWP 120591)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244c25b in os_event::wait (this=0x5594e47172d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005594e244bce1 in os_event::wait_low (this=0x5594e47172d0, reset_sig_count=8638) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005594e244c052 in os_event_wait_low (event=0x5594e47172d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005594e2537ee5 in srv_resume_thread (slot=0x5594e3486780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005594e253bae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f18157fa700 (LWP 120592)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244c25b in os_event::wait (this=0x5594e47173d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005594e244bce1 in os_event::wait_low (this=0x5594e47173d0, reset_sig_count=7225) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005594e244c052 in os_event_wait_low (event=0x5594e47173d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005594e2537ee5 in srv_resume_thread (slot=0x5594e34867f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x00005594e253bae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f17daffd700 (LWP 120604)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e2909a8d in safe_cond_wait (cond=0x7f17daffcc70, mp=0x7f17daffccb0, file=0x5594e2bd7d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005594e2363525 in inline_mysql_cond_wait (that=0x7f17daffcc70, mutex=0x7f17daffccb0, src_file=0x5594e2bd96d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005594e2368161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005594e289c555 in pfs_spawn_thread (arg=0x5594e48d2610) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f182c152700 (LWP 120610)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e2909a8d in safe_cond_wait (cond=0x5594e36491e0 , mp=0x5594e364c560 , file=0x5594e29d4dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005594e1d074ee in inline_mysql_cond_wait (that=0x5594e36491e0 , mutex=0x5594e364c560 , src_file=0x5594e29d5930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005594e1d08d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005594e289c555 in pfs_spawn_thread (arg=0x5594e4a79d70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f182c107700 (LWP 120613)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e2909d8f in safe_cond_timedwait (cond=0x5594e36491a0 , mp=0x5594e364bba0 , abstime=0x7f182c106c60, file=0x5594e29c55c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005594e1cd6845 in inline_mysql_cond_timedwait (that=0x5594e36491a0 , mutex=0x5594e364bba0 , abstime=0x7f182c106c60, src_file=0x5594e29c7638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005594e1cdb61b in cache_thread (thd=0x7f180c000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005594e1cdb8c9 in one_thread_per_connection_end (thd=0x7f180c000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005594e1f780f3 in do_handle_one_connection (connect=0x5594e4a17410) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x00005594e1f77dad in handle_one_connection (arg=0x5594e4a17410) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x00005594e289c555 in pfs_spawn_thread (arg=0x5594e4a455a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f183c064700 (LWP 120609)): #0 0x00007f18436d440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005594e1cd5f44 in my_sigwait (set=0x7f183c063cf0, sig=0x7f183c063cb4, code=0x7f183c063cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005594e1cdc584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005594e289c555 in pfs_spawn_thread (arg=0x5594e46dab50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f17db7fe700 (LWP 120603)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244c25b in os_event::wait (this=0x5594e4717650) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005594e244bce1 in os_event::wait_low (this=0x5594e4717650, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005594e244c052 in os_event_wait_low (event=0x5594e4717650, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005594e2600c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f18177fe700 (LWP 120588)): #0 0x00007f18453e3e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005594e244c55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005594e253b44d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x00005594e253b57d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f17dbfff700 (LWP 120602)): #0 0x00007f18453e3e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005594e244c55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005594e25f2b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f1817fff700 (LWP 120582)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244bb76 in os_event::timed_wait (this=0x5594e48bfd00, abstime=0x7f1817ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005594e244be8d in os_event::wait_time_low (this=0x5594e48bfd00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005594e244c02d in os_event_wait_time_low (event=0x5594e48bfd00, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005594e259a403 in ib_wqueue_timedwait (wq=0x5594e48bfbd0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005594e26daf9a in fts_optimize_thread (arg=0x5594e48bfbd0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f181cdf4700 (LWP 120581)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244bb76 in os_event::timed_wait (this=0x5594e4921a40, abstime=0x7f181cdf3d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005594e244be8d in os_event::wait_time_low (this=0x5594e4921a40, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005594e244c02d in os_event_wait_time_low (event=0x5594e4921a40, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005594e268124a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f181d5f5700 (LWP 120580)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244bb76 in os_event::timed_wait (this=0x5594e47174d0, abstime=0x7f181d5f4d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005594e244be8d in os_event::wait_time_low (this=0x5594e47174d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005594e244c02d in os_event_wait_time_low (event=0x5594e47174d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005594e2539bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f181e5f7700 (LWP 120579)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244bb76 in os_event::timed_wait (this=0x5594e4717450, abstime=0x7f181e5f6ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005594e244be8d in os_event::wait_time_low (this=0x5594e4717450, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005594e244c02d in os_event_wait_time_low (event=0x5594e4717450, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005594e253a0ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f18249a9700 (LWP 120519)): #0 0x00007f1844fad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005594e243f578 in LinuxAIOHandler::collect (this=0x7f18249a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005594e243faa1 in LinuxAIOHandler::poll (this=0x7f18249a8b00, m1=0x7f18249a8ba8, m2=0x7f18249a8bb0, request=0x7f18249a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005594e243fc04 in os_aio_linux_handler (global_segment=3, m1=0x7f18249a8ba8, m2=0x7f18249a8bb0, request=0x7f18249a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005594e2444728 in os_aio_handler (segment=3, m1=0x7f18249a8ba8, m2=0x7f18249a8bb0, request=0x7f18249a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005594e26975ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005594e253d9fb in io_handler_thread (arg=0x5594e36c4638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f181ddf6700 (LWP 120578)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e244bb76 in os_event::timed_wait (this=0x5594e48ef980, abstime=0x7f181ddf5d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005594e244be8d in os_event::wait_time_low (this=0x5594e48ef980, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005594e244c02d in os_event_wait_time_low (event=0x5594e48ef980, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005594e24189e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f18277fe700 (LWP 120515)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e2909d8f in safe_cond_timedwait (cond=0x5594e3ed5ea0 , mp=0x5594e3ed5de0 , abstime=0x7f18277fdcb0, file=0x5594e2d4d6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005594e2773bc1 in inline_mysql_cond_timedwait (that=0x5594e3ed5ea0 , mutex=0x5594e3ed5de0 , abstime=0x7f18277fdcb0, src_file=0x5594e2d4d738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005594e27740e5 in my_service_thread_sleep (control=0x5594e348ab00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005594e276857e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005594e289c555 in pfs_spawn_thread (arg=0x5594e46cf4a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f18251aa700 (LWP 120518)): #0 0x00007f1844fad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005594e243f578 in LinuxAIOHandler::collect (this=0x7f18251a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005594e243faa1 in LinuxAIOHandler::poll (this=0x7f18251a9b00, m1=0x7f18251a9ba8, m2=0x7f18251a9bb0, request=0x7f18251a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005594e243fc04 in os_aio_linux_handler (global_segment=2, m1=0x7f18251a9ba8, m2=0x7f18251a9bb0, request=0x7f18251a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005594e2444728 in os_aio_handler (segment=2, m1=0x7f18251a9ba8, m2=0x7f18251a9bb0, request=0x7f18251a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005594e26975ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005594e253d9fb in io_handler_thread (arg=0x5594e36c4630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f182e37b700 (LWP 120141)): #0 0x00007f184379b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f183d28c6f8 in asio::detail::epoll_reactor::run (this=0x5594e4607000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f183d290df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5594e4606ea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f183d2910a2 in asio::detail::task_io_service::run (this=0x5594e4606ea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f183d2a8afd in asio::io_service::run (this=0x5594e4605bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f183d2a9251 in gcomm::AsioProtonet::event_loop (this=0x5594e4605b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f183d2c72ed in GCommConn::run (this=0x5594e4606630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f183d2ca399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f18261ac700 (LWP 120516)): #0 0x00007f1844fad644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005594e243f578 in LinuxAIOHandler::collect (this=0x7f18261abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000worker[1] Trying to dump core for [mysqltest - pid: 123715, winpid: 123715] worker[1] Trying to dump core for [mysqld.1 - pid: 122852, winpid: 122852] worker[1] Trying to dump core for [mysqld.2 - pid: 123183, winpid: 123183] 5594e243faa1 in LinuxAIOHandler::poll (this=0x7f18261abb00, m1=0x7f18261abba8, m2=0x7f18261abbb0, request=0x7f18261abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005594e243fc04 in os_aio_linux_handler (global_segment=0, m1=0x7f18261abba8, m2=0x7f18261abbb0, request=0x7f18261abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005594e2444728 in os_aio_handler (segment=0, m1=0x7f18261abba8, m2=0x7f18261abbb0, request=0x7f18261abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x00005594e26975ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x00005594e253d9fb in io_handler_thread (arg=0x5594e36c4620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f183de35700 (LWP 120070)): #0 0x00007f18453e0d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e2909d8f in safe_cond_timedwait (cond=0x5594e3ee8640 , mp=0x5594e3ee8580 , abstime=0x7f183de34d70, file=0x5594e2d91920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005594e290ad49 in inline_mysql_cond_timedwait (that=0x5594e3ee8640 , mutex=0x5594e3ee8580 , abstime=0x7f183de34d70, src_file=0x5594e2d91968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005594e290b8ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005594e289c555 in pfs_spawn_thread (arg=0x5594e458f7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f183c488700 (LWP 120171)): #0 0x00007f18453e0965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005594e2909a8d in safe_cond_wait (cond=0x7f1808015608, mp=0x7f1808015558, file=0x5594e2a73c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005594e2061126 in inline_mysql_cond_wait (that=0x7f1808015608, mutex=0x7f1808015558, src_file=0x5594e2a73f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005594e2063dbe in Wsrep_thd_queue::pop_front (this=0x7f1808015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005594e20627de in wsrep_rollback_process (rollbacker=0x7f1808000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005594e205331f in start_wsrep_THD (arg=0x5594e460cc50) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f18453dcdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f184379aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f18457ed880 (LWP 120068)): #0 0x00007f18453e19d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005594e290445e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005594e21425a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f184379020d in poll () from /lib64/libc.so.6 #5 0x00005594e1ce2a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005594e1ce1d25 in mysqld_main (argc=162, argv=0x5594e4447d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005594e1cd5ec5 in main (argc=23, argv=0x7ffc6a6cba68) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_many_fragments-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_many_fragments Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-572 'innodb' w3 [ pass ] 14255 galera_sr.GCF-580 'innodb' w3 [ pass ] 6352 galera_sr.galera_sr_small_gcache 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 13:18:53 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr_small_gcache.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE t1 (f1 INTEGER) ENGINE=InnoDB; INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.122853' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.1/data/core.122853 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 122853] [New LWP 122880] [New LWP 122935] [New LWP 122901] [New LWP 122895] [New LWP 122949] [New LWP 122902] [New LWP 122900] [New LWP 122950] [New LWP 122940] [New LWP 122903] [New LWP 122959] [New LWP 122946] [New LWP 122937] [New LWP 122960] [New LWP 122961] [New LWP 122948] [New LWP 122947] [New LWP 122962] [New LWP 122936] [New LWP 123781] [New LWP 122963] [New LWP 122938] [New LWP 122939] [New LWP 122964] [New LWP 122942] [New LWP 122994] [New LWP 123133] [New LWP 123134] [New LWP 122896] [New LWP 122965] [New LWP 122958] [New LWP 122957] [New LWP 122941] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fa63267e9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fa63267e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055c35d0b445e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055c35c8f25a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa630a2d20d in poll () from /lib64/libc.so.6 #5 0x000055c35c492a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055c35c491d25 in mysqld_main (argc=161, argv=0x55c35fa10d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055c35c485ec5 in main (argc=23, argv=0x7ffffcccce58) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fa6017fa700 (LWP 122941)): #0 0x00007fa632680d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055c35cbf0c27 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055c35cbf0f7d in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055c35ce394aa in pfs_os_file_flush_func (file=..., src_file=0x55c35d4b3b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055c35ce3c225 in fil_flush_low (space=0x55c35fee9400, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055c35ce47aff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055c35ce47d86 in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x000055c35cdc410a in buf_dblwr_update (bpage=0x7fa620a47e10, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x000055c35cdcbaf7 in buf_flush_write_complete (bpage=0x7fa620a47e10, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000055c35cdba7b1 in buf_page_io_complete (bpage=0x7fa620a47e10, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x000055c35ce47859 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x000055c35cced9fb in io_handler_thread (arg=0x55c35de74648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fa5f97fa700 (LWP 122957)): #0 0x00007fa632680d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055c35cbf0c27 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055c35cbf0f7d in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055c35ce394aa in pfs_os_file_flush_func (file=..., src_file=0x55c35d4b3b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055c35ce3c225 in fil_flush_low (space=0x55c35fee97a0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055c35ce47aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055c35cbcc321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055c35cbccd83 in log_write_up_to (lsn=3472147, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055c35cbcceb3 in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x000055c35ccea522 in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1982 #10 0x000055c35cceaae5 in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2149 #11 0x000055c35cceb643 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2356 #12 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fa5f8ff9700 (LWP 122958)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fcdf5f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fcdf5f0, reset_sig_count=2413) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fcdf5f0, reset_sig_count=2413) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35cce7ee5 in srv_resume_thread (slot=0x55c35dc36748 , sig_count=2413, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055c35ccebef6 in srv_purge_coordinator_suspend (slot=0x55c35dc36748 , rseg_history_len=7) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055c35ccec18c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fa5e8ff9700 (LWP 122965)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35d0b9a8d in safe_cond_wait (cond=0x7fa5e8ff8c70, mp=0x7fa5e8ff8cb0, file=0x55c35d387d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055c35cb13525 in inline_mysql_cond_wait (that=0x7fa5e8ff8c70, mutex=0x7fa5e8ff8cb0, src_file=0x55c35d3896d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055c35cb18161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055c35d04c555 in pfs_spawn_thread (arg=0x55c35ff20a30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fa62367e700 (LWP 122896)): #0 0x00007fa630a38483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa62a5296f8 in asio::detail::epoll_reactor::run (this=0x55c35fbd7f00, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fa62a52ddf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55c35fbd6c10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fa62a52e0a2 in asio::detail::task_io_service::run (this=0x55c35fbd6c10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fa62a545afd in asio::io_service::run (this=0x55c35fbd7be0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fa62a546251 in gcomm::AsioProtonet::event_loop (this=0x55c35fbd7b40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fa62a5642ed in GCommConn::run (this=0x55c35fbd7530) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fa62a567399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fa62076e700 (LWP 123134)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35d0b9d8f in safe_cond_timedwait (cond=0x55c35ddf91a0 , mp=0x55c35ddfbba0 , abstime=0x7fa62076dc60, file=0x55c35d1755c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055c35c486845 in inline_mysql_cond_timedwait (that=0x55c35ddf91a0 , mutex=0x55c35ddfbba0 , abstime=0x7fa62076dc60, src_file=0x55c35d177638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055c35c48b61b in cache_thread (thd=0x7fa5b4000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055c35c48b8c9 in one_thread_per_connection_end (thd=0x7fa5b4000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055c35c7280f3 in do_handle_one_connection (connect=0x55c35ffde650) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055c35c727dad in handle_one_connection (arg=0x55c35ffde650) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055c35d04c555 in pfs_spawn_thread (arg=0x55c3600733c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fa6207b9700 (LWP 123133)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35d0b9a8d in safe_cond_wait (cond=0x55c35ddf91e0 , mp=0x55c35ddfc560 , file=0x55c35d184dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055c35c4b74ee in inline_mysql_cond_wait (that=0x55c35ddf91e0 , mutex=0x55c35ddfc560 , src_file=0x55c35d185930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055c35c4b8d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055c35d04c555 in pfs_spawn_thread (arg=0x55c35fca3190) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fa620804700 (LWP 122994)): #0 0x00007fa63097140a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055c35c485f44 in my_sigwait (set=0x7fa620803cf0, sig=0x7fa620803cb4, code=0x7fa620803cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055c35c48c584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055c35d04c555 in pfs_spawn_thread (arg=0x55c35fca1ad0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fa600ff9700 (LWP 122942)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fdc57c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fdc57c0, reset_sig_count=960) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fdc57c0, reset_sig_count=960) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35ccf5cee in sync_array_wait_event (arr=0x55c35fcb68e0, cell=@0x7fa600ff8a20: 0x55c35fcb6a90) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055c35cb47195 in TTASEventMutex::enter (this=0x55c35fdc50f0, max_spins=60, max_delay=4, filename=0x55c35d48a608 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1833) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x000055c35cb453f0 in PolicyMutex >::enter (this=0x55c35fdc50f0, n_spins=30, n_delay=4, name=0x55c35d48a608 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1833) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x000055c35cdcf28b in buf_flush_start (buf_pool=0x55c35fdc50f0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1833 #8 0x000055c35cdcf62b in buf_flush_do_batch (buf_pool=0x55c35fdc50f0, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7fa600ff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1943 #9 0x000055c35cdcf92e in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7fa600ff8bd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2053 #10 0x000055c35cdd3679 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3205 #11 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fa5e97fa700 (LWP 122964)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fcdf9f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fcdf9f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fcdf9f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35cdb0c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fa6027fc700 (LWP 122939)): #0 0x00007fa63224a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055c35cbef578 in LinuxAIOHandler::collect (this=0x7fa6027fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055c35cbefaa1 in LinuxAIOHandler::poll (this=0x7fa6027fbb00, m1=0x7fa6027fbba8, m2=0x7fa6027fbbb0, request=0x7fa6027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055c35cbefc04 in os_aio_linux_handler (global_segment=3, m1=0x7fa6027fbba8, m2=0x7fa6027fbbb0, request=0x7fa6027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055c35cbf4728 in os_aio_handler (segment=3, m1=0x7fa6027fbba8, m2=0x7fa6027fbbb0, request=0x7fa6027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055c35ce475ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055c35cced9fb in io_handler_thread (arg=0x55c35de74638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fa602ffd700 (LWP 122938)): #0 0x00007fa63224a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055c35cbef578 in LinuxAIOHandler::collect (this=0x7fa602ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055c35cbefaa1 in LinuxAIOHandler::poll (this=0x7fa602ffcb00, m1=0x7fa602ffcba8, m2=0x7fa602ffcbb0, request=0x7fa602ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055c35cbefc04 in os_aio_linux_handler (global_segment=2, m1=0x7fa602ffcba8, m2=0x7fa602ffcbb0, request=0x7fa602ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055c35cbf4728 in os_aio_handler (segment=2, m1=0x7fa602ffcba8, m2=0x7fa602ffcbb0, request=0x7fa602ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055c35ce475ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055c35cced9fb in io_handler_thread (arg=0x55c35de74630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fa5e9ffb700 (LWP 122963)): #0 0x00007fa632680e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055c35cbfc55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055c35cda2b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fa620723700 (LWP 123781)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fdc57c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fdc57c0, reset_sig_count=960) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fdc57c0, reset_sig_count=960) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35ccf5cee in sync_array_wait_event (arr=0x55c35fcb68e0, cell=@0x7fa62071e5b0: 0x55c35fcb69d0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055c35cb47195 in TTASEventMutex::enter (this=0x55c35fdc50f0, max_spins=60, max_delay=4, filename=0x55c35d480498 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc", line=3599) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x000055c35cb453f0 in PolicyMutex >::enter (this=0x55c35fdc50f0, n_spins=30, n_delay=4, name=0x55c35d480498 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc", line=3599) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x000055c35cdb2c0f in buf_page_make_young (bpage=0x7fa620a47ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3599 #8 0x000055c35cdb2d4a in buf_page_make_young_if_needed (bpage=0x7fa620a47ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3626 #9 0x000055c35cdb6246 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=2, guess=0x0, mode=10, file=0x55c35d4469c0 "/home/stepan/galera/git/10.4/server/storage/innobase/include/trx0rseg.ic", line=44, mtr=0x7fa62071ef50, err=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4789 #10 0x000055c35cd117e6 in trx_rsegf_get (space=0x55c35fee9400, page_no=246, mtr=0x7fa62071ef50) at /home/stepan/galera/git/10.4/server/storage/innobase/include/trx0rseg.ic:43 #11 0x000055c35cd1252d in trx_purge_add_undo_to_history (trx=0x7fa621258390, undo=@0x7fa6212593b0: 0x7fa5b01629b0, mtr=0x7fa62071ef50) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:219 #12 0x000055c35cd37b8b in trx_write_serialisation_history (trx=0x7fa621258390, mtr=0x7fa62071ef50) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1092 #13 0x000055c35cd3966b in trx_commit_low (trx=0x7fa621258390, mtr=0x7fa62071ef50) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1503 #14 0x000055c35cd397bd in trx_commit (trx=0x7fa621258390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1570 #15 0x000055c35cd39bc7 in trx_commit_for_mysql (trx=0x7fa621258390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1706 #16 0x000055c35cb1dcdf in innobase_commit_low (trx=0x7fa621258390) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4382 #17 0x000055c35cb1e09f in innobase_commit_ordered_2 (trx=0x7fa621258390, thd=0x7fa5b016b330) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4504 #18 0x000055c35cb1e595 in innobase_commit (hton=0x55c35fcb39a0, thd=0x7fa5b016b330, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4620 #19 0x000055c35c8f80d7 in commit_one_phase_2 (thd=0x7fa5b016b330, all=true, trans=0x7fa5b016e880, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #20 0x000055c35c8f7fd9 in ha_commit_one_phase (thd=0x7fa5b016b330, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #21 0x000055c35c8f7355 in ha_commit_trans (thd=0x7fa5b016b330, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #22 0x000055c35c73ce8e in trans_commit (thd=0x7fa5b016b330) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #23 0x000055c35cafc22a in Wsrep_storage_service::commit (this=0x7fa5b0173be0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #24 0x000055c35d1645c9 in wsrep::transaction::certify_fragment (this=0x7fa5b00069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #25 0x000055c35d163d2d in wsrep::transaction::streaming_step (this=0x7fa5b00069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #26 0x000055c35d1610e8 in wsrep::transaction::after_row (this=0x7fa5b00069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #27 0x000055c35c90a446 in wsrep::client_state::after_row (this=0x7fa5b0006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #28 0x000055c35c8f36c3 in wsrep_after_row (thd=0x7fa5b0000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #29 0x000055c35c9055c8 in wsrep_after_row (thd=0x7fa5b0000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #30 0x000055c35c9059b9 in handler::ha_write_row (this=0x7fa5b007c9f8, buf=0x7fa5b007acd0 "\375\001") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #31 0x000055c35c57b443 in write_record (thd=0x7fa5b0000b00, table=0x7fa5b007bdc0, info=0x7fa5b0016d38) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #32 0x000055c35c580742 in select_insert::send_data (this=0x7fa5b0016cf0, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #33 0x000055c35c638e07 in end_send (join=0x7fa5b0016da0, join_tab=0x7fa5b00874b8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #34 0x000055c35c775fa7 in JOIN_CACHE::generate_full_extensions (this=0x7fa5b0019c70, rec_ptr=0x7fa5b010aef8 "|\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #35 0x000055c35c775cc2 in JOIN_CACHE::join_matching_records (this=0x7fa5b0019c70, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #36 0x000055c35c77568e in JOIN_CACHE::join_records (this=0x7fa5b0019c70, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #37 0x000055c35c775828 in JOIN_CACHE::join_records (this=0x7fa5b0019b40, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #38 0x000055c35c775828 in JOIN_CACHE::join_records (this=0x7fa5b0019a10, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #39 0x000055c35c635abb in sub_select_cache (join=0x7fa5b0016da0, join_tab=0x7fa5b00869a8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #40 0x000055c35c635cc8 in sub_select (join=0x7fa5b0016da0, join_tab=0x7fa5b00865f8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #41 0x000055c35c635493 in do_select (join=0x7fa5b0016da0, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #42 0x000055c35c60dfb1 in JOIN::exec_inner (this=0x7fa5b0016da0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #43 0x000055c35c60d3fa in JOIN::exec (this=0x7fa5b0016da0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #44 0x000055c35c60e6ec in mysql_select (thd=0x7fa5b0000b00, tables=0x7fa5b0014ac8, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7fa5b0016cf0, unit=0x7fa5b00049c0, select_lex=0x7fa5b00145a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #45 0x000055c35c5fffed in handle_select (thd=0x7fa5b0000b00, lex=0x7fa5b00048f8, result=0x7fa5b0016cf0, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #46 0x000055c35c5c23fc in mysql_execute_command (thd=0x7fa5b0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #47 0x000055c35c5cd935 in mysql_parse (thd=0x7fa5b0000b00, rawbuf=0x7fa5b0013e58 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7fa6207220d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #48 0x000055c35c5ccf68 in wsrep_mysql_parse (thd=0x7fa5b0000b00, rawbuf=0x7fa5b0013e58 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7fa6207220d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #49 0x000055c35c5b8e25 in dispatch_command (command=COM_QUERY, thd=0x7fa5b0000b00, packet=0x7fa5b000a431 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=71, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #50 0x000055c35c5b780d in do_command (thd=0x7fa5b0000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #51 0x000055c35c728069 in do_handle_one_connection (connect=0x55c360020b40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #52 0x000055c35c727dad in handle_one_connection (arg=0x55c360020b40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #53 0x000055c35d04c555 in pfs_spawn_thread (arg=0x55c35fb4ba70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #54 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #55 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fa603fff700 (LWP 122936)): #0 0x00007fa63224a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055c35cbef578 in LinuxAIOHandler::collect (this=0x7fa603ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055c35cbefaa1 in LinuxAIOHandler::poll (this=0x7fa603ffeb00, m1=0x7fa603ffeba8, m2=0x7fa603ffebb0, request=0x7fa603ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055c35cbefc04 in os_aio_linux_handler (global_segment=0, m1=0x7fa603ffeba8, m2=0x7fa603ffebb0, request=0x7fa603ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055c35cbf4728 in os_aio_handler (segment=0, m1=0x7fa603ffeba8, m2=0x7fa603ffebb0, request=0x7fa603ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055c35ce475ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055c35cced9fb in io_handler_thread (arg=0x55c35de74620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fa5ea7fc700 (LWP 122962)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fcdf8f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fcdf8f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fcdf8f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35cdc7fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fa5faffd700 (LWP 122947)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfbb76 in os_event::timed_wait (this=0x55c35fcdf7f0, abstime=0x7fa5faffcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055c35cbfbe8d in os_event::wait_time_low (this=0x55c35fcdf7f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055c35cbfc02d in os_event_wait_time_low (event=0x55c35fcdf7f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055c35ccea0ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fa5fa7fc700 (LWP 122948)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfbb76 in os_event::timed_wait (this=0x55c35fcdf870, abstime=0x7fa5fa7fbd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055c35cbfbe8d in os_event::wait_time_low (this=0x55c35fcdf870, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055c35cbfc02d in os_event_wait_time_low (event=0x55c35fcdf870, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055c35cce9bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fa5eaffd700 (LWP 122961)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fcdf770) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fcdf770, reset_sig_count=8716) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fcdf770, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35cce7ee5 in srv_resume_thread (slot=0x55c35dc367f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055c35ccebae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fa5eb7fe700 (LWP 122960)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fcdf6f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fcdf6f0, reset_sig_count=9335) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fcdf6f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35cce7ee5 in srv_resume_thread (slot=0x55c35dc367b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055c35ccebae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fa6037fe700 (LWP 122937)): #0 0x00007fa63224a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055c35cbef578 in LinuxAIOHandler::collect (this=0x7fa6037fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055c35cbefaa1 in LinuxAIOHandler::poll (this=0x7fa6037fdb00, m1=0x7fa6037fdba8, m2=0x7fa6037fdbb0, request=0x7fa6037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055c35cbefc04 in os_aio_linux_handler (global_segment=1, m1=0x7fa6037fdba8, m2=0x7fa6037fdbb0, request=0x7fa6037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055c35cbf4728 in os_aio_handler (segment=1, m1=0x7fa6037fdba8, m2=0x7fa6037fdbb0, request=0x7fa6037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055c35ce475ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055c35cced9fb in io_handler_thread (arg=0x55c35de74628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fa5fb7fe700 (LWP 122946)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfbb76 in os_event::timed_wait (this=0x55c35feb7d20, abstime=0x7fa5fb7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055c35cbfbe8d in os_event::wait_time_low (this=0x55c35feb7d20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055c35cbfc02d in os_event_wait_time_low (event=0x55c35feb7d20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055c35cbc89e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fa5ebfff700 (LWP 122959)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfc25b in os_event::wait (this=0x55c35fcdf670) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055c35cbfbce1 in os_event::wait_low (this=0x55c35fcdf670, reset_sig_count=10767) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055c35cbfc052 in os_event_wait_low (event=0x55c35fcdf670, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055c35cce7ee5 in srv_resume_thread (slot=0x55c35dc36780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055c35ccebae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fa62838d700 (LWP 122903)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa62a47c132 in fifo_lock_get (q=q@entry=0x55c35fbbda90) at galerautils/src/gu_fifo.c:262 #2 0x00007fa62a47c7a1 in gu_fifo_get_head (q=0x55c35fbbda90, err=0x7fa62838c06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fa62a55e0cc in gcs_recv (conn=0x55c35fb974f0, action=0x7fa62838c0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fa62a58b16c in galera::GcsActionSource::process (this=0x55c35fb97140, recv_ctx=0x7fa62838c4d0, exit_loop=@0x7fa62838c29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fa62a5aa033 in galera::ReplicatorSMM::async_recv (this=0x55c35fb8f170, recv_ctx=0x7fa62838c4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055c35d16e36c in wsrep::wsrep_provider_v26::run_applier (this=0x55c35fb16360, applier_ctx=0x7fa62838c4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055c35c8116d4 in wsrep_replication_process (thd=0x7fa610000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055c35c80331f in start_wsrep_THD (arg=0x55c35fbf0cc0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fa601ffb700 (LWP 122940)): #0 0x00007fa63224a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055c35cbef578 in LinuxAIOHandler::collect (this=0x7fa601ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055c35cbefaa1 in LinuxAIOHandler::poll (this=0x7fa601ffab00, m1=0x7fa601ffaba8, m2=0x7fa601ffabb0, request=0x7fa601ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055c35cbefc04 in os_aio_linux_handler (global_segment=4, m1=0x7fa601ffaba8, m2=0x7fa601ffabb0, request=0x7fa601ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055c35cbf4728 in os_aio_handler (segment=4, m1=0x7fa601ffaba8, m2=0x7fa601ffabb0, request=0x7fa601ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055c35ce475ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055c35cced9fb in io_handler_thread (arg=0x55c35de74640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fa5fbfff700 (LWP 122950)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfbb76 in os_event::timed_wait (this=0x55c35fe7fcf0, abstime=0x7fa5fbffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055c35cbfbe8d in os_event::wait_time_low (this=0x55c35fe7fcf0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055c35cbfc02d in os_event_wait_time_low (event=0x55c35fe7fcf0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055c35cd4a403 in ib_wqueue_timedwait (wq=0x55c35fe7fbc0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055c35ce8af9a in fts_optimize_thread (arg=0x55c35fe7fbc0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fa622e7d700 (LWP 122900)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa62a562867 in gu::Lock::wait (cond=..., this=0x7fa622e7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fa62a5653ab in RecvBuf::front (this=this@entry=0x55c35fbd7718, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fa62a562f46 in gcomm_recv (backend=, msg=0x55c35fb977d0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fa62a556eec in core_msg_recv (backend=backend@entry=0x55c35fb97918, recv_msg=recv_msg@entry=0x55c35fb977d0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fa62a55a164 in gcs_core_recv (conn=0x55c35fb97770, recv_act=recv_act@entry=0x7fa622e7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fa62a5613d8 in gcs_recv_thread (arg=0x55c35fb974f0) at gcs/src/gcs.cpp:1398 #7 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fa6283d8700 (LWP 122902)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35d0b9a8d in safe_cond_wait (cond=0x7fa60c015608, mp=0x7fa60c015558, file=0x55c35d223c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055c35c811126 in inline_mysql_cond_wait (that=0x7fa60c015608, mutex=0x7fa60c015558, src_file=0x55c35d223f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055c35c813dbe in Wsrep_thd_queue::pop_front (this=0x7fa60c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055c35c812b9e in wsrep_post_rollback_process (post_rollbacker=0x7fa60c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055c35c80331f in start_wsrep_THD (arg=0x55c35fbf0ce0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fa5f9ffb700 (LWP 122949)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35cbfbb76 in os_event::timed_wait (this=0x55c35fee9de0, abstime=0x7fa5f9ffad60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055c35cbfbe8d in os_event::wait_time_low (this=0x55c35fee9de0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055c35cbfc02d in os_event_wait_time_low (event=0x55c35fee9de0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055c35ce3124a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fa629f26700 (LWP 122895)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa62a587fd7 in gu::Lock::wait (cond=..., this=0x7fa629f25bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fa62a588069 in galera::ServiceThd::thd_func (arg=0x55c35fb8f888) at galera/src/galera_service_thd.cpp:30 #3 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fa628423700 (LWP 122901)): #0 0x00007fa63267d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35d0b9a8d in safe_cond_wait (cond=0x7fa618017b68, mp=0x7fa618017ab8, file=0x55c35d223c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055c35c811126 in inline_mysql_cond_wait (that=0x7fa618017b68, mutex=0x7fa618017ab8, src_file=0x55c35d223f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055c35c813dbe in Wsrep_thd_queue::pop_front (this=0x7fa618017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055c35c8127de in wsrep_rollback_process (rollbacker=0x7fa618001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055c35c80331f in start_wsrep_THD (arg=0x55c35fbe72b0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fa621e56700 (LWP 122935)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35d0b9d8f in safe_cond_timedwait (cond=0x55c35e685ea0 , mp=0x55c35e685de0 , abstime=0x7fa621e55cb0, file=0x55c35d4fd6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055c35cf23bc1 in inline_mysql_cond_timedwait (that=0x55c35e685ea0 , mutex=0x55c35e685de0 , abstime=0x7fa621e55cb0, src_file=0x55c35d4fd738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055c35cf240e5 in my_service_thread_sleep (control=0x55c35dc3ab00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055c35cf1857e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055c35d04c555 in pfs_spawn_thread (arg=0x55c35fc97670) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fa62b0d2700 (LWP 122880)): #0 0x00007fa63267dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055c35d0b9d8f in safe_cond_timedwait (cond=0x55c35e698640 , mp=0x55c35e698580 , abstime=0x7fa62b0d1d70, file=0x55c35d541920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055c35d0bad49 in inline_mysql_cond_timedwait (that=0x55c35e698640 , mutex=0x55c35e698580 , abstime=0x7fa62b0d1d70, src_file=0x55c35d541968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055c35d0bb8ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055c35d04c555 in pfs_spawn_thread (arg=0x55c35fb58730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa632679dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa630a37ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fa632a8a880 (LWP 122853)): #0 0x00007fa63267e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055c35d0b445e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055c35c8f25a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa630a2d20d in poll () from /lib64/libc.so.6 #5 0x000055c35c492a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055c35c491d25 in mysqld_main (argc=161, argv=0x55c35fa10d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055c35c485ec5 in main (argc=23, argv=0x7ffffcccce58) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.123185' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.2/data/core.123185 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 123185] [New LWP 123187] [New LWP 123210] [New LWP 123659] [New LWP 123233] [New LWP 123664] [New LWP 123606] [New LWP 123610] [New LWP 123648] [New LWP 123649] [New LWP 123650] [New LWP 123652] [New LWP 123654] [New LWP 123657] [New LWP 123660] [New LWP 123608] [New LWP 123270] [New LWP 123604] [New LWP 123661] [New LWP 123609] [New LWP 123611] [New LWP 123646] [New LWP 123653] [New LWP 123647] [New LWP 123651] [New LWP 123655] [New LWP 123656] [New LWP 123658] [New LWP 123605] [New LWP 123607] [New LWP 123268] [New LWP 123269] [New LWP 123271] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f8fd99919d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f8fd99919d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563f3d62145e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563f3ce5f5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f8fd7d4020d in poll () from /lib64/libc.so.6 #5 0x0000563f3c9ffa8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000563f3c9fed25 in mysqld_main (argc=162, argv=0x563f3f1b7d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000563f3c9f2ec5 in main (argc=23, argv=0x7ffed29496b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f8fd00a1700 (LWP 123271)): #0 0x00007f8fd9993d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000563f3d15dc27 in os_file_fsync_posix (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000563f3d15df7d in os_file_flush_func (file=21) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000563f3d3a64aa in pfs_os_file_flush_func (file=..., src_file=0x563f3da20b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000563f3d3a9225 in fil_flush_low (space=0x563f3f6914e0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000563f3d3b4aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000563f3d139321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000563f3d139d83 in log_write_up_to (lsn=2041156, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000563f3d2a4e99 in trx_flush_log_if_needed_low (lsn=2041156) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000563f3d2a4ee2 in trx_flush_log_if_needed (lsn=2041156, trx=0x7f8fa73ff140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000563f3d2a6d0b in trx_commit_complete_for_mysql (trx=0x7f8fa73ff140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000563f3d08b5b2 in innobase_commit (hton=0x563f3f45b9c0, thd=0x7f8fb0000b00, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x0000563f3ce650d7 in commit_one_phase_2 (thd=0x7f8fb0000b00, all=true, trans=0x7f8fb0004050, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000563f3ce64fd9 in ha_commit_one_phase (thd=0x7f8fb0000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000563f3ce64355 in ha_commit_trans (thd=0x7f8fb0000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000563f3cca9e8e in trans_commit (thd=0x7f8fb0000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000563f3cd5d3d6 in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7f8fd00a04d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:264 #17 0x0000563f3d6c3b47 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:67 #18 0x0000563f3d6c45a7 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:241 #19 0x0000563f3d6c78fd in wsrep::server_state::on_apply (this=0x563f3f300740, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:944 #20 0x0000563f3d6dcfc7 in wsrep::high_priority_service::apply (this=0x7f8fd00a04d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x0000563f3d6da6ae in (anonymous namespace)::apply_cb (ctx=0x7f8fd00a04d0, wsh=0x7f8fd009f3e0, flags=0, buf=0x7f8fd009f420, meta=0x7f8fd009f6b0, exit_loop=0x7f8fd009f5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007f8fd18843b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7f8fb0084770, recv_ctx=recv_ctx@entry=0x7f8fd00a04d0, apply_cb=0x563f3d6da48b <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f8fd009f5ef: false) at galera/src/trx_handle.cpp:418 #23 0x00007f8fd18b6d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x563f3f336170, recv_ctx=recv_ctx@entry=0x7f8fd00a04d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007f8fd18bca40 in galera::ReplicatorSMM::process_trx (this=0x563f3f336170, recv_ctx=0x7f8fd00a04d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #25 0x00007f8fd189dc6b in galera::GcsActionSource::process_writeset (this=this@entry=0x563f3f33e140, recv_ctx=0x7f8fd00a04d0, act=..., exit_loop=@0x7f8fd00a029e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007f8fd189de45 in galera::GcsActionSource::dispatch (this=this@entry=0x563f3f33e140, recv_ctx=recv_ctx@entry=0x7f8fd00a04d0, act=..., exit_loop=@0x7f8fd00a029e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007f8fd189e21d in galera::GcsActionSource::process (this=0x563f3f33e140, recv_ctx=0x7f8fd00a04d0, exit_loop=@0x7f8fd00a029e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007f8fd18bd033 in galera::ReplicatorSMM::async_recv (this=0x563f3f336170, recv_ctx=0x7f8fd00a04d0) at galera/src/replicator_smm.cpp:383 #29 0x0000563f3d6db36c in wsrep::wsrep_provider_v26::run_applier (this=0x563f3f2bd360, applier_ctx=0x7f8fd00a04d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x0000563f3cd7e6d4 in wsrep_replication_process (thd=0x7f8fb0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x0000563f3cd7031f in start_wsrep_THD (arg=0x563f3f3976d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #32 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f8fd0137700 (LWP 123269)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d626a8d in safe_cond_wait (cond=0x7f8fc0016858, mp=0x7f8fc00167a8, file=0x563f3d790c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563f3cd7e126 in inline_mysql_cond_wait (that=0x7f8fc0016858, mutex=0x7f8fc00167a8, src_file=0x563f3d790f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563f3cd80dbe in Wsrep_thd_queue::pop_front (this=0x7f8fc0016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563f3cd7f7de in wsrep_rollback_process (rollbacker=0x7f8fc0001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000563f3cd7031f in start_wsrep_THD (arg=0x563f3f3989b0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f8fca47c700 (LWP 123268)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f8fd1875867 in gu::Lock::wait (cond=..., this=0x7f8fca47b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f8fd18783ab in RecvBuf::front (this=this@entry=0x563f3f376718, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f8fd1875f46 in gcomm_recv (backend=, msg=0x563f3f33e7d0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f8fd1869eec in core_msg_recv (backend=backend@entry=0x563f3f33e918, recv_msg=recv_msg@entry=0x563f3f33e7d0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f8fd186d164 in gcs_core_recv (conn=0x563f3f33e770, recv_act=recv_act@entry=0x7f8fca47bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f8fd18743d8 in gcs_recv_thread (arg=0x563f3f33e4f0) at gcs/src/gcs.cpp:1398 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f8fa57ab700 (LWP 123607)): #0 0x00007f8fd955d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563f3d15c578 in LinuxAIOHandler::collect (this=0x7f8fa57aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563f3d15caa1 in LinuxAIOHandler::poll (this=0x7f8fa57aab00, m1=0x7f8fa57aaba8, m2=0x7f8fa57aabb0, request=0x7f8fa57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563f3d15cc04 in os_aio_linux_handler (global_segment=2, m1=0x7f8fa57aaba8, m2=0x7f8fa57aabb0, request=0x7f8fa57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563f3d161728 in os_aio_handler (segment=2, m1=0x7f8fa57aaba8, m2=0x7f8fa57aabb0, request=0x7f8fa57aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563f3d3b45ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563f3d25a9fb in io_handler_thread (arg=0x563f3e3e1630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f8fa67ad700 (LWP 123605)): #0 0x00007f8fd955d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563f3d15c578 in LinuxAIOHandler::collect (this=0x7f8fa67acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563f3d15caa1 in LinuxAIOHandler::poll (this=0x7f8fa67acb00, m1=0x7f8fa67acba8, m2=0x7f8fa67acbb0, request=0x7f8fa67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563f3d15cc04 in os_aio_linux_handler (global_segment=0, m1=0x7f8fa67acba8, m2=0x7f8fa67acbb0, request=0x7f8fa67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563f3d161728 in os_aio_handler (segment=0, m1=0x7f8fa67acba8, m2=0x7f8fa67acbb0, request=0x7f8fa67acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563f3d3b45ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563f3d25a9fb in io_handler_thread (arg=0x563f3e3e1620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f8f7f7fe700 (LWP 123658)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d16925b in os_event::wait (this=0x563f3f487730) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563f3d168ce1 in os_event::wait_low (this=0x563f3f487730, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563f3d169052 in os_event_wait_low (event=0x563f3f487730, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563f3d31dc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f8f8cff9700 (LWP 123656)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d16925b in os_event::wait (this=0x563f3f487630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563f3d168ce1 in os_event::wait_low (this=0x563f3f487630, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563f3d169052 in os_event_wait_low (event=0x563f3f487630, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563f3d334fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f8f8d7fa700 (LWP 123655)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d16925b in os_event::wait (this=0x563f3f4874b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563f3d168ce1 in os_event::wait_low (this=0x563f3f4874b0, reset_sig_count=8527) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563f3d169052 in os_event_wait_low (event=0x563f3f4874b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563f3d254ee5 in srv_resume_thread (slot=0x563f3e1a37f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563f3d258ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f8f8f7fe700 (LWP 123651)): #0 0x00007f8fd9993e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563f3d16955d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563f3d25844d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000563f3d25857d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f8f9e7fc700 (LWP 123647)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d168b76 in os_event::timed_wait (this=0x563f3f487530, abstime=0x7f8f9e7fbba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563f3d168e8d in os_event::wait_time_low (this=0x563f3f487530, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563f3d16902d in os_event_wait_time_low (event=0x563f3f487530, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563f3d2570ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f8f8e7fc700 (LWP 123653)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d16925b in os_event::wait (this=0x563f3f487430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563f3d168ce1 in os_event::wait_low (this=0x563f3f487430, reset_sig_count=9348) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563f3d169052 in os_event_wait_low (event=0x563f3f487430, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563f3d254ee5 in srv_resume_thread (slot=0x563f3e1a37b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563f3d258ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f8f9dffb700 (LWP 123646)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d168b76 in os_event::timed_wait (this=0x563f3f65fa60, abstime=0x7f8f9dffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563f3d168e8d in os_event::wait_time_low (this=0x563f3f65fa60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563f3d16902d in os_event_wait_time_low (event=0x563f3f65fa60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563f3d1359e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f8f9effd700 (LWP 123611)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d168b76 in os_event::timed_wait (this=0x563f3f4876b0, abstime=0x7f8f9effcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563f3d168e8d in os_event::wait_time_low (this=0x563f3f4876b0, time_in_usec=999000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563f3d16902d in os_event_wait_time_low (event=0x563f3f4876b0, time_in_usec=999000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563f3d33ee25 in pc_sleep_if_needed (next_loop_time=1549624679246, sig_count=2, cur_time=1549624678247) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000563f3d33fd5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f8f9ffff700 (LWP 123609)): #0 0x00007f8fd955d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563f3d15c578 in LinuxAIOHandler::collect (this=0x7f8f9fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563f3d15caa1 in LinuxAIOHandler::poll (this=0x7f8f9fffeb00, m1=0x7f8f9fffeba8, m2=0x7f8f9fffebb0, request=0x7f8f9fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563f3d15cc04 in os_aio_linux_handler (global_segment=4, m1=0x7f8f9fffeba8, m2=0x7f8f9fffebb0, request=0x7f8f9fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563f3d161728 in os_aio_handler (segment=4, m1=0x7f8f9fffeba8, m2=0x7f8f9fffebb0, request=0x7f8f9fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563f3d3b45ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563f3d25a9fb in io_handler_thread (arg=0x563f3e3e1640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f8fc80f6700 (LWP 123661)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d626a8d in safe_cond_wait (cond=0x563f3e3661e0 , mp=0x563f3e369560 , file=0x563f3d6f1dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563f3ca244ee in inline_mysql_cond_wait (that=0x563f3e3661e0 , mutex=0x563f3e369560 , src_file=0x563f3d6f2930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000563f3ca25d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000563f3d5b9555 in pfs_spawn_thread (arg=0x563f3f44b1f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f8fa7fff700 (LWP 123604)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d626d8f in safe_cond_timedwait (cond=0x563f3ebf2ea0 , mp=0x563f3ebf2de0 , abstime=0x7f8fa7ffecb0, file=0x563f3da6a6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563f3d490bc1 in inline_mysql_cond_timedwait (that=0x563f3ebf2ea0 , mutex=0x563f3ebf2de0 , abstime=0x7f8fa7ffecb0, src_file=0x563f3da6a738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563f3d4910e5 in my_service_thread_sleep (control=0x563f3e1a7b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000563f3d48557e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000563f3d5b9555 in pfs_spawn_thread (arg=0x563f3f43f690) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f8fd00ec700 (LWP 123270)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d626a8d in safe_cond_wait (cond=0x7f8fb8015608, mp=0x7f8fb8015558, file=0x563f3d790c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563f3cd7e126 in inline_mysql_cond_wait (that=0x7f8fb8015608, mutex=0x7f8fb8015558, src_file=0x563f3d790f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563f3cd80dbe in Wsrep_thd_queue::pop_front (this=0x7f8fb8015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563f3cd7fb9e in wsrep_post_rollback_process (post_rollbacker=0x7f8fb8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000563f3cd7031f in start_wsrep_THD (arg=0x563f3f397320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f8fa4faa700 (LWP 123608)): #0 0x00007f8fd955d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563f3d15c578 in LinuxAIOHandler::collect (this=0x7f8fa4fa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563f3d15caa1 in LinuxAIOHandler::poll (this=0x7f8fa4fa9b00, m1=0x7f8fa4fa9ba8, m2=0x7f8fa4fa9bb0, request=0x7f8fa4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563f3d15cc04 in os_aio_linux_handler (global_segment=3, m1=0x7f8fa4fa9ba8, m2=0x7f8fa4fa9bb0, request=0x7f8fa4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563f3d161728 in os_aio_handler (segment=3, m1=0x7f8fa4fa9ba8, m2=0x7f8fa4fa9bb0, request=0x7f8fa4fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563f3d3b45ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563f3d25a9fb in io_handler_thread (arg=0x563f3e3e1638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f8fc8141700 (LWP 123660)): #0 0x00007f8fd7c8440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000563f3c9f2f44 in my_sigwait (set=0x7f8fc8140cf0, sig=0x7f8fc8140cb4, code=0x7f8fc8140cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000563f3c9f9584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000563f3d5b9555 in pfs_spawn_thread (arg=0x563f3f44ad00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f8f7ffff700 (LWP 123657)): #0 0x00007f8fd9993e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563f3d16955d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563f3d30fb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f8f8dffb700 (LWP 123654)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d16925b in os_event::wait (this=0x563f3f4873b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563f3d168ce1 in os_event::wait_low (this=0x563f3f4873b0, reset_sig_count=10532) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563f3d169052 in os_event_wait_low (event=0x563f3f4873b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563f3d254ee5 in srv_resume_thread (slot=0x563f3e1a3780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563f3d258ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f8f8effd700 (LWP 123652)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d16925b in os_event::wait (this=0x563f3f487330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563f3d168ce1 in os_event::wait_low (this=0x563f3f487330, reset_sig_count=2464) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563f3d169052 in os_event_wait_low (event=0x563f3f487330, reset_sig_count=2464) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563f3d254ee5 in srv_resume_thread (slot=0x563f3e1a3748 , sig_count=2464, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563f3d258ef6 in srv_purge_coordinator_suspend (slot=0x563f3e1a3748 , rseg_history_len=31) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000563f3d25918c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f8f8ffff700 (LWP 123650)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d168b76 in os_event::timed_wait (this=0x563f3f62f4d0, abstime=0x7f8f8fffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563f3d168e8d in os_event::wait_time_low (this=0x563f3f62f4d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563f3d16902d in os_event_wait_time_low (event=0x563f3f62f4d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563f3d2b7403 in ib_wqueue_timedwait (wq=0x563f3f62f3a0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000563f3d3f7f9a in fts_optimize_thread (arg=0x563f3f62f3a0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f8f9cff9700 (LWP 123649)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d168b76 in os_event::timed_wait (this=0x563f3f691b20, abstime=0x7f8f9cff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563f3d168e8d in os_event::wait_time_low (this=0x563f3f691b20, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563f3d16902d in os_event_wait_time_low (event=0x563f3f691b20, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563f3d39e24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f8f9d7fa700 (LWP 123648)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d168b76 in os_event::timed_wait (this=0x563f3f4875b0, abstime=0x7f8f9d7f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563f3d168e8d in os_event::wait_time_low (this=0x563f3f4875b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563f3d16902d in os_event_wait_time_low (event=0x563f3f4875b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563f3d256bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f8f9f7fe700 (LWP 123610)): #0 0x00007f8fd955d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563f3d15c578 in LinuxAIOHandler::collect (this=0x7f8f9f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563f3d15caa1 in LinuxAIOHandler::poll (this=0x7f8f9f7fdb00, m1=0x7f8f9f7fdba8, m2=0x7f8f9f7fdbb0, request=0x7f8f9f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563f3d15cc04 in os_aio_linux_handler (global_segment=5, m1=0x7f8f9f7fdba8, m2=0x7f8f9f7fdbb0, request=0x7f8f9f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563f3d161728 in os_aio_handler (segment=5, m1=0x7f8f9f7fdba8, m2=0x7f8f9f7fdbb0, request=0x7f8f9f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563f3d3b45ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563f3d25a9fb in io_handler_thread (arg=0x563f3e3e1648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f8fa5fac700 (LWP 123606)): #0 0x00007f8fd955d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563f3d15c578 in LinuxAIOHandler::collect (this=0x7f8fa5fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563f3d15caa1 in LinuxAIOHandler::poll (this=0x7f8fa5fabb00, m1=0x7f8fa5fabba8, m2=0x7f8fa5fabbb0, request=0x7f8fa5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563f3d15cc04 in os_aio_linux_handler (global_segment=1, m1=0x7f8fa5fabba8, m2=0x7f8fa5fabbb0, request=0x7f8fa5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563f3d161728 in os_aio_handler (segment=1, m1=0x7f8fa5fabba8, m2=0x7f8fa5fabbb0, request=0x7f8fa5fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563f3d3b45ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563f3d25a9fb in io_handler_thread (arg=0x563f3e3e1628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f8fc80ab700 (LWP 123664)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d626d8f in safe_cond_timedwait (cond=0x563f3e3661a0 , mp=0x563f3e368ba0 , abstime=0x7f8fc80aac60, file=0x563f3d6e25c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563f3c9f3845 in inline_mysql_cond_timedwait (that=0x563f3e3661a0 , mutex=0x563f3e368ba0 , abstime=0x7f8fc80aac60, src_file=0x563f3d6e4638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563f3c9f861b in cache_thread (thd=0x7f8fb4000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000563f3c9f88c9 in one_thread_per_connection_end (thd=0x7f8fb4000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000563f3cc950f3 in do_handle_one_connection (connect=0x563f3f7874c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000563f3cc94dad in handle_one_connection (arg=0x563f3f7874c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000563f3d5b9555 in pfs_spawn_thread (arg=0x563f3f785630) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f8fcac7d700 (LWP 123233)): #0 0x00007f8fd7d4b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8fd183c6f8 in asio::detail::epoll_reactor::run (this=0x563f3f376f50, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f8fd1840df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x563f3f376df0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f8fd18410a2 in asio::detail::task_io_service::run (this=0x563f3f376df0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f8fd1858afd in asio::io_service::run (this=0x563f3f375b50) at asio/asio/impl/io_service.ipp:58 #5 0x00007f8fd1859251 in gcomm::AsioProtonet::event_loop (this=0x563f3f375ab0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f8fd18772ed in GCommConn::run (this=0x563f3f376530) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f8fd187a399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f8f7effd700 (LWP 123659)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d626a8d in safe_cond_wait (cond=0x7f8f7effcc70, mp=0x7f8f7effccb0, file=0x563f3d8f4d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563f3d080525 in inline_mysql_cond_wait (that=0x7f8f7effcc70, mutex=0x7f8f7effccb0, src_file=0x563f3d8f66d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563f3d085161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000563f3d5b9555 in pfs_spawn_thread (arg=0x563f3f642310) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f8fd1239700 (LWP 123210)): #0 0x00007f8fd9990965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f8fd189afd7 in gu::Lock::wait (cond=..., this=0x7f8fd1238bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f8fd189b069 in galera::ServiceThd::thd_func (arg=0x563f3f336888) at galera/src/galera_service_thd.cpp:30 #3 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f8fd23e5700 (LWP 123187)): #0 0x00007f8fd9990d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563f3d626d8f in safe_cond_timedwait (cond=0x563f3ec05640 , mp=0x563f3ec05580 , abstime=0x7f8fd23e4d70, file=0x563f3daae920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563f3d627d49 in inline_mysql_cond_timedwait (that=0x563f3ec05640 , mutex=0x563f3ec05580 , abstime=0x7f8fd23e4d70, src_file=0x563f3daae968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563f3d6288ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000563f3d5b9555 in pfs_spawn_thread (arg=0x563f3f2ff7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8fd998cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8fd7d4aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f8fd9d9d880 (LWP 123185)): #0 0x00007f8fd99919d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563f3d62145e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563f3ce5f5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f8fd7d4020d in poll () from /lib64/libc.so.6 #5 0x0000563f3c9ffa8a in handle_connections_sworker[4] Trying to dump core for [mysqltest - pid: 137420, winpid: 137420] worker[4] Trying to dump core for [mysqld.1 - pid: 136580, winpid: 136580] worker[4] Trying to dump core for [mysqld.2 - pid: 136894, winpid: 136894] ockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000563f3c9fed25 in mysqld_main (argc=162, argv=0x563f3f1b7d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000563f3c9f2ec5 in main (argc=23, argv=0x7ffed29496b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.galera_sr_small_gcache-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_small_gcache-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_small_gcache Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1043B 'innodb' w4 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 13:24:56 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/GCF-1043B.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 INSERT INTO t1 VALUES (1,'y'),(2,'x') UPDATE t1 SET f2 = 'y' WHERE f1 = 1 OR f1 = 2; connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.' (111) - found 'core.136582' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.GCF-1043B-innodb/mysqld.1/data/core.136582 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 136582] [New LWP 136627] [New LWP 136668] [New LWP 136675] [New LWP 136683] [New LWP 136622] [New LWP 137501] [New LWP 136605] [New LWP 136667] [New LWP 136624] [New LWP 136670] [New LWP 136625] [New LWP 136682] [New LWP 136626] [New LWP 136685] [New LWP 136669] [New LWP 136688] [New LWP 136676] [New LWP 136832] [New LWP 136835] [New LWP 136680] [New LWP 136684] [New LWP 136623] [New LWP 136686] [New LWP 136673] [New LWP 136687] [New LWP 136672] [New LWP 136711] [New LWP 137500] [New LWP 136679] [New LWP 136678] [New LWP 136681] [New LWP 136666] [New LWP 136677] [New LWP 136671] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fc9ea2249d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fc9ea2249d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000555f70f2f45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000555f7076d5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fc9e85d320d in poll () from /lib64/libc.so.6 #5 0x0000555f7030da8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000555f7030cd25 in mysqld_main (argc=161, argv=0x555f7403ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000555f70300ec5 in main (argc=23, argv=0x7ffede0ef778) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7fc9b95ab700 (LWP 136671)): #0 0x00007fc9ea226d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000555f70a6bc27 in os_file_fsync_posix (file=81) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000555f70a6bf7d in os_file_flush_func (file=81) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000555f70cb44aa in pfs_os_file_flush_func (file=..., src_file=0x555f7132eb50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000555f70cb7225 in fil_flush_low (space=0x7fc96c0565b0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x0000555f70cc2aff in fil_flush (space_id=8) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x0000555f70cc2d86 in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x0000555f70c3f10a in buf_dblwr_update (bpage=0x7fc9bb7e1830, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x0000555f70c46af7 in buf_flush_write_complete (bpage=0x7fc9bb7e1830, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x0000555f70c357b1 in buf_page_io_complete (bpage=0x7fc9bb7e1830, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x0000555f70cc2859 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x0000555f70b689fb in io_handler_thread (arg=0x555f71cef640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fc9b6ba1700 (LWP 136677)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a76b76 in os_event::timed_wait (this=0x555f74305860, abstime=0x7fc9b6ba0d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000555f70a76e8d in os_event::wait_time_low (this=0x555f74305860, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000555f70a7702d in os_event_wait_time_low (event=0x555f74305860, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555f70b64bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fc9d8ff8700 (LWP 136666)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34d8f in safe_cond_timedwait (cond=0x555f72500ea0 , mp=0x555f72500de0 , abstime=0x7fc9d8ff7cb0, file=0x555f713786f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000555f70d9ebc1 in inline_mysql_cond_timedwait (that=0x555f72500ea0 , mutex=0x555f72500de0 , abstime=0x7fc9d8ff7cb0, src_file=0x555f71378738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000555f70d9f0e5 in my_service_thread_sleep (control=0x555f71ab5b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000555f70d9357e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f742bd7e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fc9b4b9d700 (LWP 136681)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a7725b in os_event::wait (this=0x555f743055e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000555f70a76ce1 in os_event::wait_low (this=0x555f743055e0, reset_sig_count=2965) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555f70a77052 in os_event_wait_low (event=0x555f743055e0, reset_sig_count=2965) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000555f70b62ee5 in srv_resume_thread (slot=0x555f71ab1748 , sig_count=2965, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000555f70b66ef6 in srv_purge_coordinator_suspend (slot=0x555f71ab1748 , rseg_history_len=34) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000555f70b6718c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fc9b63a0700 (LWP 136678)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a76b76 in os_event::timed_wait (this=0x555f7450fdd0, abstime=0x7fc9b639fd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000555f70a76e8d in os_event::wait_time_low (this=0x555f7450fdd0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000555f70a7702d in os_event_wait_time_low (event=0x555f7450fdd0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555f70cac24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fc9b5b9f700 (LWP 136679)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a76b76 in os_event::timed_wait (this=0x555f744a5a60, abstime=0x7fc9b5b9eb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000555f70a76e8d in os_event::wait_time_low (this=0x555f744a5a60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000555f70a7702d in os_event_wait_time_low (event=0x555f744a5a60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555f70bc5403 in ib_wqueue_timedwait (wq=0x555f744a5930, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000555f70d05f9a in fts_optimize_thread (arg=0x555f744a5930) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fc9d8115700 (LWP 137500)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34d8f in safe_cond_timedwait (cond=0x555f71c741a0 , mp=0x555f71c76ba0 , abstime=0x7fc9d8114c60, file=0x555f70ff05c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000555f70301845 in inline_mysql_cond_timedwait (that=0x555f71c741a0 , mutex=0x555f71c76ba0 , abstime=0x7fc9d8114c60, src_file=0x555f70ff2638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000555f7030661b in cache_thread (thd=0x7fc96c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000555f703068c9 in one_thread_per_connection_end (thd=0x7fc96c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000555f705a30f3 in do_handle_one_connection (connect=0x555f745990a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000555f705a2dad in handle_one_connection (arg=0x555f745990a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f74179a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fc9d81f6700 (LWP 136711)): #0 0x00007fc9e851740a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000555f70300f44 in my_sigwait (set=0x7fc9d81f5cf0, sig=0x7fc9d81f5cb4, code=0x7fc9d81f5cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000555f70307584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f742c8e50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fc9b8daa700 (LWP 136672)): #0 0x00007fc9e9df0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000555f70a6a578 in LinuxAIOHandler::collect (this=0x7fc9b8da9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000555f70a6aaa1 in LinuxAIOHandler::poll (this=0x7fc9b8da9b00, m1=0x7fc9b8da9ba8, m2=0x7fc9b8da9bb0, request=0x7fc9b8da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000555f70a6ac04 in os_aio_linux_handler (global_segment=5, m1=0x7fc9b8da9ba8, m2=0x7fc9b8da9bb0, request=0x7fc9b8da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000555f70a6f728 in os_aio_handler (segment=5, m1=0x7fc9b8da9ba8, m2=0x7fc9b8da9bb0, request=0x7fc9b8da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000555f70cc25ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000555f70b689fb in io_handler_thread (arg=0x555f71cef648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fc9a57fa700 (LWP 136687)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a7725b in os_event::wait (this=0x555f743059e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000555f70a76ce1 in os_event::wait_low (this=0x555f743059e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555f70a77052 in os_event_wait_low (event=0x555f743059e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000555f70c2bc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fc9b85a9700 (LWP 136673)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a76b76 in os_event::timed_wait (this=0x555f74305960, abstime=0x7fc9b85a8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000555f70a76e8d in os_event::wait_time_low (this=0x555f74305960, time_in_usec=514000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000555f70a7702d in os_event_wait_time_low (event=0x555f74305960, time_in_usec=514000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555f70c4ce25 in pc_sleep_if_needed (next_loop_time=1549625036311, sig_count=2, cur_time=1549625035797) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000555f70c4dd5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fc9a5ffb700 (LWP 136686)): #0 0x00007fc9ea226e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000555f70a7755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000555f70c1db44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fc9e08ca700 (LWP 136623)): #0 0x00007fc9e85de483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fc9e20cf6f8 in asio::detail::epoll_reactor::run (this=0x555f741fe040, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fc9e20d3df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x555f741fdee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fc9e20d40a2 in asio::detail::task_io_service::run (this=0x555f741fdee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fc9e20ebafd in asio::io_service::run (this=0x555f741fcfe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fc9e20ec251 in gcomm::AsioProtonet::event_loop (this=0x555f741fcf40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fc9e210a2ed in GCommConn::run (this=0x555f741fd6c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fc9e210d399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fc9a6ffd700 (LWP 136684)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a7725b in os_event::wait (this=0x555f74305760) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000555f70a76ce1 in os_event::wait_low (this=0x555f74305760, reset_sig_count=11014) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555f70a77052 in os_event_wait_low (event=0x555f74305760, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000555f70b62ee5 in srv_resume_thread (slot=0x555f71ab17f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000555f70b66ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fc9b539e700 (LWP 136680)): #0 0x00007fc9ea226e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000555f70a7755d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000555f70b6644d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000555f70b6657d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fc9d8160700 (LWP 136835)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34d8f in safe_cond_timedwait (cond=0x555f71c741a0 , mp=0x555f71c76ba0 , abstime=0x7fc9d815fc60, file=0x555f70ff05c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000555f70301845 in inline_mysql_cond_timedwait (that=0x555f71c741a0 , mutex=0x555f71c76ba0 , abstime=0x7fc9d815fc60, src_file=0x555f70ff2638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000555f7030661b in cache_thread (thd=0x7fc978000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000555f703068c9 in one_thread_per_connection_end (thd=0x7fc978000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000555f705a30f3 in do_handle_one_connection (connect=0x555f74690be0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000555f705a2dad in handle_one_connection (arg=0x555f74690be0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f74696340) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fc9d81ab700 (LWP 136832)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34a8d in safe_cond_wait (cond=0x555f71c741e0 , mp=0x555f71c77560 , file=0x555f70fffdc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000555f703324ee in inline_mysql_cond_wait (that=0x555f71c741e0 , mutex=0x555f71c77560 , src_file=0x555f71000930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000555f70333d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f746670c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fc9b7ba3700 (LWP 136676)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a76b76 in os_event::timed_wait (this=0x555f743057e0, abstime=0x7fc9b7ba2ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000555f70a76e8d in os_event::wait_time_low (this=0x555f743057e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000555f70a7702d in os_event_wait_time_low (event=0x555f743057e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555f70b650ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fc9a4ff9700 (LWP 136688)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34a8d in safe_cond_wait (cond=0x7fc9a4ff8c70, mp=0x7fc9a4ff8cb0, file=0x555f71202d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000555f7098e525 in inline_mysql_cond_wait (that=0x7fc9a4ff8c70, mutex=0x7fc9a4ff8cb0, src_file=0x555f712046d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000555f70993161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f74546920) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fc9ba5ad700 (LWP 136669)): #0 0x00007fc9e9df0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000555f70a6a578 in LinuxAIOHandler::collect (this=0x7fc9ba5acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000555f70a6aaa1 in LinuxAIOHandler::poll (this=0x7fc9ba5acb00, m1=0x7fc9ba5acba8, m2=0x7fc9ba5acbb0, request=0x7fc9ba5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000555f70a6ac04 in os_aio_linux_handler (global_segment=2, m1=0x7fc9ba5acba8, m2=0x7fc9ba5acbb0, request=0x7fc9ba5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000555f70a6f728 in os_aio_handler (segment=2, m1=0x7fc9ba5acba8, m2=0x7fc9ba5acbb0, request=0x7fc9ba5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000555f70cc25ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000555f70b689fb in io_handler_thread (arg=0x555f71cef630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fc9a67fc700 (LWP 136685)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a7725b in os_event::wait (this=0x555f743058e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000555f70a76ce1 in os_event::wait_low (this=0x555f743058e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555f70a77052 in os_event_wait_low (event=0x555f743058e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000555f70c42fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fc9e007e700 (LWP 136626)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34a8d in safe_cond_wait (cond=0x7fc9cc017b68, mp=0x7fc9cc017ab8, file=0x555f7109ec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000555f7068c126 in inline_mysql_cond_wait (that=0x7fc9cc017b68, mutex=0x7fc9cc017ab8, src_file=0x555f7109ef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000555f7068edbe in Wsrep_thd_queue::pop_front (this=0x7fc9cc017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000555f7068db9e in wsrep_post_rollback_process (post_rollbacker=0x7fc9cc001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000555f7067e31f in start_wsrep_THD (arg=0x555f74215c00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fc9a7fff700 (LWP 136682)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a7725b in os_event::wait (this=0x555f743056e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000555f70a76ce1 in os_event::wait_low (this=0x555f743056e0, reset_sig_count=11574) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555f70a77052 in os_event_wait_low (event=0x555f743056e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000555f70b62ee5 in srv_resume_thread (slot=0x555f71ab17b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000555f70b66ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc9e00c9700 (LWP 136625)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34a8d in safe_cond_wait (cond=0x7fc9d0015608, mp=0x7fc9d0015558, file=0x555f7109ec60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000555f7068c126 in inline_mysql_cond_wait (that=0x7fc9d0015608, mutex=0x7fc9d0015558, src_file=0x555f7109ef80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000555f7068edbe in Wsrep_thd_queue::pop_front (this=0x7fc9d0015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000555f7068d7de in wsrep_rollback_process (rollbacker=0x7fc9d0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000555f7067e31f in start_wsrep_THD (arg=0x555f7420d320) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc9b9dac700 (LWP 136670)): #0 0x00007fc9e9df0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000555f70a6a578 in LinuxAIOHandler::collect (this=0x7fc9b9dabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000555f70a6aaa1 in LinuxAIOHandler::poll (this=0x7fc9b9dabb00, m1=0x7fc9b9dabba8, m2=0x7fc9b9dabbb0, request=0x7fc9b9dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000555f70a6ac04 in os_aio_linux_handler (global_segment=3, m1=0x7fc9b9dabba8, m2=0x7fc9b9dabbb0, request=0x7fc9b9dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000555f70a6f728 in os_aio_handler (segment=3, m1=0x7fc9b9dabba8, m2=0x7fc9b9dabbb0, request=0x7fc9b9dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000555f70cc25ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000555f70b689fb in io_handler_thread (arg=0x555f71cef638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fc9da37c700 (LWP 136624)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc9e2108867 in gu::Lock::wait (cond=..., this=0x7fc9da37b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fc9e210b3ab in RecvBuf::front (this=this@entry=0x555f741fd8a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fc9e2108f46 in gcomm_recv (backend=, msg=0x555f741c5830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fc9e20fceec in core_msg_recv (backend=backend@entry=0x555f741c5978, recv_msg=recv_msg@entry=0x555f741c5830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fc9e2100164 in gcs_core_recv (conn=0x555f741c57d0, recv_act=recv_act@entry=0x7fc9da37bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fc9e21073d8 in gcs_recv_thread (arg=0x555f741c5550) at gcs/src/gcs.cpp:1398 #7 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc9bb5af700 (LWP 136667)): #0 0x00007fc9e9df0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000555f70a6a578 in LinuxAIOHandler::collect (this=0x7fc9bb5aeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000555f70a6aaa1 in LinuxAIOHandler::poll (this=0x7fc9bb5aeb00, m1=0x7fc9bb5aeba8, m2=0x7fc9bb5aebb0, request=0x7fc9bb5aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000555f70a6ac04 in os_aio_linux_handler (global_segment=0, m1=0x7fc9bb5aeba8, m2=0x7fc9bb5aebb0, request=0x7fc9bb5aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000555f70a6f728 in os_aio_handler (segment=0, m1=0x7fc9bb5aeba8, m2=0x7fc9bb5aebb0, request=0x7fc9bb5aebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000555f70cc25ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000555f70b689fb in io_handler_thread (arg=0x555f71cef620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc9e2c78700 (LWP 136605)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70f34d8f in safe_cond_timedwait (cond=0x555f72513640 , mp=0x555f72513580 , abstime=0x7fc9e2c77d70, file=0x555f713bc920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000555f70f35d49 in inline_mysql_cond_timedwait (that=0x555f72513640 , mutex=0x555f72513580 , abstime=0x7fc9e2c77d70, src_file=0x555f713bc968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000555f70f368ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f74186730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fc9d80ca700 (LWP 137501)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a7725b in os_event::wait (this=0x555f7442c710) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000555f70a76ce1 in os_event::wait_low (this=0x555f7442c710, reset_sig_count=493) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555f70a77052 in os_event_wait_low (event=0x555f7442c710, reset_sig_count=493) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000555f70b70cee in sync_array_wait_event (arr=0x555f742dc9b0, cell=@0x7fc9d80c5160: 0x555f742dcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x0000555f70b7456d in rw_lock_sx_lock_func (lock=0x7fc9bb7e18f8, pass=0, file_name=0x555f7127b5e8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x0000555f70c1f8fb in pfs_rw_lock_sx_lock_func (lock=0x7fc9bb7e18f8, pass=0, file_name=0x555f7127b5e8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x0000555f70c31398 in buf_page_get_gen (page_id=..., zip_size=0, rw_latch=4, guess=0x0, mode=10, file=0x555f7127b5e8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, mtr=0x7fc9d80c6a80, err=0x7fc9d80c54d0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:4817 #8 0x0000555f70bf0d3c in btr_cur_search_to_nth_level_func (index=0x7fc96c05c278, level=0, tuple=0x7fc964081108, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x7fc9d80c6660, ahi_latch=0x0, file=0x555f7127b5e8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, mtr=0x7fc9d80c6a80, autoinc=2) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:1548 #9 0x0000555f70ad9512 in btr_pcur_open_low (index=0x7fc96c05c278, level=0, tuple=0x7fc964081108, mode=PAGE_CUR_LE, latch_mode=2, cursor=0x7fc9d80c6660, file=0x555f7127b5e8 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc", line=2639, autoinc=2, mtr=0x7fc9d80c6a80) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:464 #10 0x0000555f70ae03bc in row_ins_clust_index_entry_low (flags=0, mode=2, index=0x7fc96c05c278, n_uniq=1, entry=0x7fc964081108, n_ext=0, thr=0x7fc964079928, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:2639 #11 0x0000555f70ae246e in row_ins_clust_index_entry (index=0x7fc96c05c278, entry=0x7fc964081108, thr=0x7fc964079928, n_ext=0, dup_chk_only=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3280 #12 0x0000555f70ae299c in row_ins_index_entry (index=0x7fc96c05c278, entry=0x7fc964081108, thr=0x7fc964079928) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3405 #13 0x0000555f70ae30db in row_ins_index_entry_step (node=0x7fc9c40330c0, thr=0x7fc964079928) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3574 #14 0x0000555f70ae350c in row_ins (node=0x7fc9c40330c0, thr=0x7fc964079928) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3717 #15 0x0000555f70ae3d0b in row_ins_step (thr=0x7fc964079928) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0ins.cc:3962 #16 0x0000555f70b03878 in row_insert_for_mysql (mysql_rec=0x7fc9c403f3c0 "\375\002", prebuilt=0x7fc9c4032548, ins_mode=ROW_INS_NORMAL) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0mysql.cc:1460 #17 0x0000555f709a13f6 in ha_innobase::write_row (this=0x7fc9c403f818, record=0x7fc9c403f3c0 "\375\002") at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:8050 #18 0x0000555f70780817 in handler::ha_write_row (this=0x7fc9c403f818, buf=0x7fc9c403f3c0 "\375\002") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6463 #19 0x0000555f703f6443 in write_record (thd=0x7fc964000b00, table=0x7fc9c403e780, info=0x7fc9d80c7f50) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #20 0x0000555f703f3805 in mysql_insert (thd=0x7fc964000b00, table_list=0x7fc96401fa00, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:1067 #21 0x0000555f7043cb4a in mysql_execute_command (thd=0x7fc964000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4754 #22 0x0000555f70448935 in mysql_parse (thd=0x7fc964000b00, rawbuf=0x7fc96401f908 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7fc9d80c90d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #23 0x0000555f70447f68 in wsrep_mysql_parse (thd=0x7fc964000b00, rawbuf=0x7fc96401f908 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7fc9d80c90d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #24 0x0000555f70433e25 in dispatch_command (command=COM_QUERY, thd=0x7fc964000b00, packet=0x7fc9640177d1 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", packet_length=37, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #25 0x0000555f7043280d in do_command (thd=0x7fc964000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #26 0x0000555f705a3069 in do_handle_one_connection (connect=0x555f745e2a30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #27 0x0000555f705a2dad in handle_one_connection (arg=0x555f745e2a30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #28 0x0000555f70ec7555 in pfs_spawn_thread (arg=0x555f745fbf10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #29 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #30 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc9e10cb700 (LWP 136622)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc9e212dfd7 in gu::Lock::wait (cond=..., this=0x7fc9e10cabf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fc9e212e069 in galera::ServiceThd::thd_func (arg=0x555f741bd998) at galera/src/galera_service_thd.cpp:30 #3 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc9a77fe700 (LWP 136683)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a7725b in os_event::wait (this=0x555f74305660) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000555f70a76ce1 in os_event::wait_low (this=0x555f74305660, reset_sig_count=12786) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555f70a77052 in os_event_wait_low (event=0x555f74305660, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000555f70b62ee5 in srv_resume_thread (slot=0x555f71ab1780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000555f70b66ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc9b73a2700 (LWP 136675)): #0 0x00007fc9ea223d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555f70a76b76 in os_event::timed_wait (this=0x555f744ddd10, abstime=0x7fc9b73a1d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000555f70a76e8d in os_event::wait_time_low (this=0x555f744ddd10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000555f70a7702d in os_event_wait_time_low (event=0x555f744ddd10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555f70a439e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc9badae700 (LWP 136668)): #0 0x00007fc9e9df0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000555f70a6a578 in LinuxAIOHandler::collect (this=0x7fc9badadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000555f70a6aaa1 in LinuxAIOHandler::poll (this=0x7fc9badadb00, m1=0x7fc9badadba8, m2=0x7fc9badadbb0, request=0x7fc9badadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000555f70a6ac04 in os_aio_linux_handler (global_segment=1, m1=0x7fc9badadba8, m2=0x7fc9badadbb0, request=0x7fc9badadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000555f70a6f728 in os_aio_handler (segment=1, m1=0x7fc9badadba8, m2=0x7fc9badadbb0, request=0x7fc9badadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000555f70cc25ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000555f70b689fb in io_handler_thread (arg=0x555f71cef628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fc9d9b7b700 (LWP 136627)): #0 0x00007fc9ea223965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc9e2022132 in fifo_lock_get (q=q@entry=0x555f741ebbd0) at galerautils/src/gu_fifo.c:262 #2 0x00007fc9e20227a1 in gu_fifo_get_head (q=0x555f741ebbd0, err=0x7fc9d9b7a06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fc9e21040cc in gcs_recv (conn=0x555f741c5550, action=0x7fc9d9b7a0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fc9e213116c in galera::GcsActionSource::process (this=0x555f741c4f30, recv_ctx=0x7fc9d9b7a4d0, exit_loop=@0x7fc9d9b7a29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fc9e2150033 in galera::ReplicatorSMM::async_recv (this=0x555f741bd280, recv_ctx=0x7fc9d9b7a4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000555f70fe936c in wsrep::wsrep_provider_v26::run_applier (this=0x555f74144360, applier_ctx=0x7fc9d9b7a4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000555f7068c6d4 in wsrep_replication_process (thd=0x7fc9c4000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000555f7067e31f in start_wsrep_THD (arg=0x555f742162a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fc9ea21fdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fc9e85ddead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc9ea630880 (LWP 136582)): #0 0x00007fc9ea2249d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000555f70f2f45e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000555f7076d5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fc9e85d320d in poll () from /lib64/libc.so.6 #5 0x0000555f7030da8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000555f7030cd25 in mysqld_main (argc=161, argv=0x555f7403ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000555f70300ec5 in main (argc=23, argv=0x7ffede0ef778) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.136896' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.GCF-1043B-innodb/mysqld.2/data/core.136896 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 136896] [New LWP 136899] [New LWP 136898] [New LWP 136976] [New LWP 136979] [New LWP 136978] [New LWP 136977] [New LWP 137346] [New LWP 137343] [New LWP 137350] [New LWP 137397] [New LWP 137395] [New LWP 137348] [New LWP 137398] [New LWP 137396] [New LWP 137349] [New LWP 137402] [New LWP 137401] [New LWP 137399] [New LWP 137403] [New LWP 137404] [New LWP 137405] [New LWP 137407] [New LWP 137413] [New LWP 137408] [New LWP 137344] [New LWP 137347] [New LWP 137400] [New LWP 136938] [New LWP 137345] [New LWP 137406] [New LWP 137410] [New LWP 137409] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fae00d9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fae00d9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055eb4e53045e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055eb4dd6e5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fadff14d20d in poll () from /lib64/libc.so.6 #5 0x000055eb4d90ea8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055eb4d90dd25 in mysqld_main (argc=162, argv=0x55eb4fef4d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055eb4d901ec5 in main (argc=23, argv=0x7ffdd3888e68) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fadf81a1700 (LWP 137409)): #0 0x00007fadff09140a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055eb4d901f44 in my_sigwait (set=0x7fadf81a0cf0, sig=0x7fadf81a0cb4, code=0x7fadf81a0cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055eb4d908584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055eb4e4c8555 in pfs_spawn_thread (arg=0x55eb50181c10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fadf8156700 (LWP 137410)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e535a8d in safe_cond_wait (cond=0x55eb4f2751e0 , mp=0x55eb4f278560 , file=0x55eb4e600dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055eb4d9334ee in inline_mysql_cond_wait (that=0x55eb4f2751e0 , mutex=0x55eb4f278560 , src_file=0x55eb4e601930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055eb4d934d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055eb4e4c8555 in pfs_spawn_thread (arg=0x55eb50182100) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fad977fe700 (LWP 137406)): #0 0x00007fae00da0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055eb4e07855d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055eb4e21eb44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fadc9206700 (LWP 137345)): #0 0x00007fae0096a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055eb4e06b578 in LinuxAIOHandler::collect (this=0x7fadc9205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055eb4e06baa1 in LinuxAIOHandler::poll (this=0x7fadc9205b00, m1=0x7fadc9205ba8, m2=0x7fadc9205bb0, request=0x7fadc9205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055eb4e06bc04 in os_aio_linux_handler (global_segment=1, m1=0x7fadc9205ba8, m2=0x7fadc9205bb0, request=0x7fadc9205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055eb4e070728 in os_aio_handler (segment=1, m1=0x7fadc9205ba8, m2=0x7fadc9205bb0, request=0x7fadc9205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055eb4e2c35ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055eb4e1699fb in io_handler_thread (arg=0x55eb4f2f0628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fade9b7a700 (LWP 136938)): #0 0x00007fadff158483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fadf8c496f8 in asio::detail::epoll_reactor::run (this=0x55eb500ae000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fadf8c4ddf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55eb500adea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fadf8c4e0a2 in asio::detail::task_io_service::run (this=0x55eb500adea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fadf8c65afd in asio::io_service::run (this=0x55eb500acbc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007fadf8c66251 in gcomm::AsioProtonet::event_loop (this=0x55eb500acb20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fadf8c842ed in GCommConn::run (this=0x55eb500ad630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fadf8c87399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fadaeffd700 (LWP 137400)): #0 0x00007fae00da0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055eb4e07855d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055eb4e16744d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055eb4e16757d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fadc3fff700 (LWP 137347)): #0 0x00007fae0096a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055eb4e06b578 in LinuxAIOHandler::collect (this=0x7fadc3ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055eb4e06baa1 in LinuxAIOHandler::poll (this=0x7fadc3ffeb00, m1=0x7fadc3ffeba8, m2=0x7fadc3ffebb0, request=0x7fadc3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055eb4e06bc04 in os_aio_linux_handler (global_segment=3, m1=0x7fadc3ffeba8, m2=0x7fadc3ffebb0, request=0x7fadc3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055eb4e070728 in os_aio_handler (segment=3, m1=0x7fadc3ffeba8, m2=0x7fadc3ffebb0, request=0x7fadc3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055eb4e2c35ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055eb4e1699fb in io_handler_thread (arg=0x55eb4f2f0638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fadc9a07700 (LWP 137344)): #0 0x00007fae0096a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055eb4e06b578 in LinuxAIOHandler::collect (this=0x7fadc9a06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055eb4e06baa1 in LinuxAIOHandler::poll (this=0x7fadc9a06b00, m1=0x7fadc9a06ba8, m2=0x7fadc9a06bb0, request=0x7fadc9a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055eb4e06bc04 in os_aio_linux_handler (global_segment=0, m1=0x7fadc9a06ba8, m2=0x7fadc9a06bb0, request=0x7fadc9a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055eb4e070728 in os_aio_handler (segment=0, m1=0x7fadc9a06ba8, m2=0x7fadc9a06bb0, request=0x7fadc9a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055eb4e2c35ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055eb4e1699fb in io_handler_thread (arg=0x55eb4f2f0620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fad967fc700 (LWP 137408)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e535a8d in safe_cond_wait (cond=0x7fad967fbc70, mp=0x7fad967fbcb0, file=0x55eb4e803d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055eb4df8f525 in inline_mysql_cond_wait (that=0x7fad967fbc70, mutex=0x7fad967fbcb0, src_file=0x55eb4e8056d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055eb4df94161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055eb4e4c8555 in pfs_spawn_thread (arg=0x55eb50379680) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fadf810b700 (LWP 137413)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e535d8f in safe_cond_timedwait (cond=0x55eb4f2751a0 , mp=0x55eb4f277ba0 , abstime=0x7fadf810ac60, file=0x55eb4e5f15c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055eb4d902845 in inline_mysql_cond_timedwait (that=0x55eb4f2751a0 , mutex=0x55eb4f277ba0 , abstime=0x7fadf810ac60, src_file=0x55eb4e5f3638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055eb4d90761b in cache_thread (thd=0x7fadcc000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055eb4d9078c9 in one_thread_per_connection_end (thd=0x7fadcc000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055eb4dba40f3 in do_handle_one_connection (connect=0x55eb504be2a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055eb4dba3dad in handle_one_connection (arg=0x55eb504be2a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055eb4e4c8555 in pfs_spawn_thread (arg=0x55eb504bc410) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fad96ffd700 (LWP 137407)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e07825b in os_event::wait (this=0x55eb501be640) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055eb4e077ce1 in os_event::wait_low (this=0x55eb501be640, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055eb4e078052 in os_event_wait_low (event=0x55eb501be640, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055eb4e22cc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fad97fff700 (LWP 137405)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e07825b in os_event::wait (this=0x55eb501be540) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055eb4e077ce1 in os_event::wait_low (this=0x55eb501be540, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055eb4e078052 in os_event_wait_low (event=0x55eb501be540, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055eb4e243fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fadacff9700 (LWP 137404)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e07825b in os_event::wait (this=0x55eb501be340) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055eb4e077ce1 in os_event::wait_low (this=0x55eb501be340, reset_sig_count=11131) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055eb4e078052 in os_event_wait_low (event=0x55eb501be340, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055eb4e163ee5 in srv_resume_thread (slot=0x55eb4f0b27b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055eb4e167ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fadad7fa700 (LWP 137403)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e07825b in os_event::wait (this=0x55eb501be3c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055eb4e077ce1 in os_event::wait_low (this=0x55eb501be3c0, reset_sig_count=10500) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055eb4e078052 in os_event_wait_low (event=0x55eb501be3c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055eb4e163ee5 in srv_resume_thread (slot=0x55eb4f0b27f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055eb4e167ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fadaf7fe700 (LWP 137399)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e077b76 in os_event::timed_wait (this=0x55eb503669c0, abstime=0x7fadaf7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055eb4e077e8d in os_event::wait_time_low (this=0x55eb503669c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055eb4e07802d in os_event_wait_time_low (event=0x55eb503669c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055eb4e1c6403 in ib_wqueue_timedwait (wq=0x55eb503cb580, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055eb4e306f9a in fts_optimize_thread (arg=0x55eb503cb580) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fadae7fc700 (LWP 137401)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e07825b in os_event::wait (this=0x55eb501be240) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055eb4e077ce1 in os_event::wait_low (this=0x55eb501be240, reset_sig_count=3092) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055eb4e078052 in os_event_wait_low (event=0x55eb501be240, reset_sig_count=3092) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055eb4e163ee5 in srv_resume_thread (slot=0x55eb4f0b2748 , sig_count=3092, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055eb4e167ef6 in srv_purge_coordinator_suspend (slot=0x55eb4f0b2748 , rseg_history_len=27) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055eb4e16818c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fadadffb700 (LWP 137402)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e07825b in os_event::wait (this=0x55eb501be2c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055eb4e077ce1 in os_event::wait_low (this=0x55eb501be2c0, reset_sig_count=12577) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055eb4e078052 in os_event_wait_low (event=0x55eb501be2c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055eb4e163ee5 in srv_resume_thread (slot=0x55eb4f0b2780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055eb4e167ae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fadc2ffd700 (LWP 137349)): #0 0x00007fae0096a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055eb4e06b578 in LinuxAIOHandler::collect (this=0x7fadc2ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055eb4e06baa1 in LinuxAIOHandler::poll (this=0x7fadc2ffcb00, m1=0x7fadc2ffcba8, m2=0x7fadc2ffcbb0, request=0x7fadc2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055eb4e06bc04 in os_aio_linux_handler (global_segment=5, m1=0x7fadc2ffcba8, m2=0x7fadc2ffcbb0, request=0x7fadc2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055eb4e070728 in os_aio_handler (segment=5, m1=0x7fadc2ffcba8, m2=0x7fadc2ffcbb0, request=0x7fadc2ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055eb4e2c35ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055eb4e1699fb in io_handler_thread (arg=0x55eb4f2f0648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fadc1ffb700 (LWP 137396)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e077b76 in os_event::timed_wait (this=0x55eb501be440, abstime=0x7fadc1ffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055eb4e077e8d in os_event::wait_time_low (this=0x55eb501be440, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055eb4e07802d in os_event_wait_time_low (event=0x55eb501be440, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055eb4e1660ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fadaffff700 (LWP 137398)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e077b76 in os_event::timed_wait (this=0x55eb503c8a30, abstime=0x7fadafffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055eb4e077e8d in os_event::wait_time_low (this=0x55eb503c8a30, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055eb4e07802d in os_event_wait_time_low (event=0x55eb503c8a30, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055eb4e2ad24a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fadc37fe700 (LWP 137348)): #0 0x00007fae0096a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055eb4e06b578 in LinuxAIOHandler::collect (this=0x7fadc37fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055eb4e06baa1 in LinuxAIOHandler::poll (this=0x7fadc37fdb00, m1=0x7fadc37fdba8, m2=0x7fadc37fdbb0, request=0x7fadc37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055eb4e06bc04 in os_aio_linux_handler (global_segment=4, m1=0x7fadc37fdba8, m2=0x7fadc37fdbb0, request=0x7fadc37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055eb4e070728 in os_aio_handler (segment=4, m1=0x7fadc37fdba8, m2=0x7fadc37fdbb0, request=0x7fadc37fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055eb4e2c35ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055eb4e1699fb in io_handler_thread (arg=0x55eb4f2f0640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fadc17fa700 (LWP 137395)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e077b76 in os_event::timed_wait (this=0x55eb50396970, abstime=0x7fadc17f9d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055eb4e077e8d in os_event::wait_time_low (this=0x55eb50396970, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055eb4e07802d in os_event_wait_time_low (event=0x55eb50396970, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055eb4e0449e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fadc0ff9700 (LWP 137397)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e077b76 in os_event::timed_wait (this=0x55eb501be4c0, abstime=0x7fadc0ff8d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055eb4e077e8d in os_event::wait_time_low (this=0x55eb501be4c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055eb4e07802d in os_event_wait_time_low (event=0x55eb501be4c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055eb4e165bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fadc27fc700 (LWP 137350)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e077b76 in os_event::timed_wait (this=0x55eb501be5c0, abstime=0x7fadc27fbaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055eb4e077e8d in os_event::wait_time_low (this=0x55eb501be5c0, time_in_usec=746000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055eb4e07802d in os_event_wait_time_low (event=0x55eb501be5c0, time_in_usec=746000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055eb4e24de25 in pc_sleep_if_needed (next_loop_time=1549625042666, sig_count=2, cur_time=1549625041920) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055eb4e24ed5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fadcb059700 (LWP 137343)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e535d8f in safe_cond_timedwait (cond=0x55eb4fb01ea0 , mp=0x55eb4fb01de0 , abstime=0x7fadcb058cb0, file=0x55eb4e9796f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055eb4e39fbc1 in inline_mysql_cond_timedwait (that=0x55eb4fb01ea0 , mutex=0x55eb4fb01de0 , abstime=0x7fadcb058cb0, src_file=0x55eb4e979738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055eb4e3a00e5 in my_service_thread_sleep (control=0x55eb4f0b6b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055eb4e39457e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055eb4e4c8555 in pfs_spawn_thread (arg=0x55eb501765a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fadc8a05700 (LWP 137346)): #0 0x00007fae0096a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055eb4e06b578 in LinuxAIOHandler::collect (this=0x7fadc8a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055eb4e06baa1 in LinuxAIOHandler::poll (this=0x7fadc8a04b00, m1=0x7fadc8a04ba8, m2=0x7fadc8a04bb0, request=0x7fadc8a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055eb4e06bc04 in os_aio_linux_handler (global_segment=2, m1=0x7fadc8a04ba8, m2=0x7fadc8a04bb0, request=0x7fadc8a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055eb4e070728 in os_aio_handler (segment=2, m1=0x7fadc8a04ba8, m2=0x7fadc8a04bb0, request=0x7fadc8a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055eb4e2c35ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055eb4e1699fb in io_handler_thread (arg=0x55eb4f2f0630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fadf8646700 (LWP 136977)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e535a8d in safe_cond_wait (cond=0x7fadd8015608, mp=0x7fadd8015558, file=0x55eb4e69fc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055eb4dc8d126 in inline_mysql_cond_wait (that=0x7fadd8015608, mutex=0x7fadd8015558, src_file=0x55eb4e69ff80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055eb4dc8fdbe in Wsrep_thd_queue::pop_front (this=0x7fadd8015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055eb4dc8e7de in wsrep_rollback_process (rollbacker=0x7fadd8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055eb4dc7f31f in start_wsrep_THD (arg=0x55eb500cf2c0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fadf85fb700 (LWP 136978)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e535a8d in safe_cond_wait (cond=0x7faddc017b68, mp=0x7faddc017ab8, file=0x55eb4e69fc60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055eb4dc8d126 in inline_mysql_cond_wait (that=0x7faddc017b68, mutex=0x7faddc017ab8, src_file=0x55eb4e69ff80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055eb4dc8fdbe in Wsrep_thd_queue::pop_front (this=0x7faddc017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055eb4dc8eb9e in wsrep_post_rollback_process (post_rollbacker=0x7faddc001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055eb4dc7f31f in start_wsrep_THD (arg=0x55eb500cf550) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fadf85b0700 (LWP 136979)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fadf8b9c132 in fifo_lock_get (q=q@entry=0x55eb500a1ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007fadf8b9c7a1 in gu_fifo_get_head (q=0x55eb500a1ac0, err=0x7fadf85af06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fadf8c7e0cc in gcs_recv (conn=0x55eb5007b440, action=0x7fadf85af0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fadf8cab16c in galera::GcsActionSource::process (this=0x55eb500a5e60, recv_ctx=0x7fadf85af4d0, exit_loop=@0x7fadf85af29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fadf8cca033 in galera::ReplicatorSMM::async_recv (this=0x55eb500731f0, recv_ctx=0x7fadf85af4d0) at galerworker[1] Trying to dump core for [mysqltest - pid: 160610, winpid: 160610] worker[1] Trying to dump core for [mysqld.1 - pid: 159814, winpid: 159814] worker[1] Trying to dump core for [mysqld.2 - pid: 160064, winpid: 160064] a/src/replicator_smm.cpp:383 #6 0x000055eb4e5ea36c in wsrep::wsrep_provider_v26::run_applier (this=0x55eb4ffe8e50, applier_ctx=0x7fadf85af4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055eb4dc8d6d4 in wsrep_replication_process (thd=0x7fade0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055eb4dc7f31f in start_wsrep_THD (arg=0x55eb50093d70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fade9379700 (LWP 136976)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fadf8c82867 in gu::Lock::wait (cond=..., this=0x7fade9378850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fadf8c853ab in RecvBuf::front (this=this@entry=0x55eb500ad818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fadf8c82f46 in gcomm_recv (backend=, msg=0x55eb5007b720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fadf8c76eec in core_msg_recv (backend=backend@entry=0x55eb5007b868, recv_msg=recv_msg@entry=0x55eb5007b720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fadf8c7a164 in gcs_core_recv (conn=0x55eb5007b6c0, recv_act=recv_act@entry=0x7fade9378d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fadf8c813d8 in gcs_recv_thread (arg=0x55eb5007b440) at gcs/src/gcs.cpp:1398 #7 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fadf97f2700 (LWP 136898)): #0 0x00007fae00d9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055eb4e535d8f in safe_cond_timedwait (cond=0x55eb4fb14640 , mp=0x55eb4fb14580 , abstime=0x7fadf97f1d70, file=0x55eb4e9bd920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055eb4e536d49 in inline_mysql_cond_timedwait (that=0x55eb4fb14640 , mutex=0x55eb4fb14580 , abstime=0x7fadf97f1d70, src_file=0x55eb4e9bd968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055eb4e5378ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055eb4e4c8555 in pfs_spawn_thread (arg=0x55eb5003c7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fadebffe700 (LWP 136899)): #0 0x00007fae00d9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fadf8ca7fd7 in gu::Lock::wait (cond=..., this=0x7fadebffdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fadf8ca8069 in galera::ServiceThd::thd_func (arg=0x55eb50073908) at galera/src/galera_service_thd.cpp:30 #3 0x00007fae00d99dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fadff157ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fae011aa880 (LWP 136896)): #0 0x00007fae00d9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055eb4e53045e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055eb4dd6e5a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fadff14d20d in poll () from /lib64/libc.so.6 #5 0x000055eb4d90ea8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055eb4d90dd25 in mysqld_main (argc=162, argv=0x55eb4fef4d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055eb4d901ec5 in main (argc=23, argv=0x7ffdd3888e68) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.GCF-1043B-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043B-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1043B Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-597 'innodb' w4 [ pass ] 5336 galera_sr.GCF-620 'innodb' w4 [ pass ] 4994 galera_sr.mysql-wsrep-features#8 'innodb' w1 [ fail ] timeout after 900 seconds Test ended at 2019-02-08 13:35:09 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/mysql-wsrep-features#8.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); connection node_1; CREATE TABLE t1 (f1 INT PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(100), FULLTEXT (f2)) ENGINE=InnoDB; connection node_2; SELECT COUNT(*) = 13 FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE name LIKE 'test/%'; COUNT(*) = 13 1 connection node_1; INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/1/mysqld.' (111) - found 'core.159815' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.1/data/core.159815 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 159815] [New LWP 159862] [New LWP 159819] [New LWP 159863] [New LWP 159838] [New LWP 159865] [New LWP 159866] [New LWP 159868] [New LWP 159877] [New LWP 159871] [New LWP 159878] [New LWP 159873] [New LWP 159896] [New LWP 159879] [New LWP 160008] [New LWP 159885] [New LWP 159882] [New LWP 159893] [New LWP 159895] [New LWP 160009] [New LWP 159922] [New LWP 159884] [New LWP 159883] [New LWP 159894] [New LWP 159881] [New LWP 159876] [New LWP 159875] [New LWP 159870] [New LWP 159869] [New LWP 159867] [New LWP 159864] [New LWP 159861] [New LWP 159872] [New LWP 160692] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f6ea19239d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f6ea19239d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055cdab1d345e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055cdaaa115a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6e9fcd220d in poll () from /lib64/libc.so.6 #5 0x000055cdaa5b1a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055cdaa5b0d25 in mysqld_main (argc=161, argv=0x55cdad4a5d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055cdaa5a4ec5 in main (argc=23, argv=0x7ffe45f94108) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f6e903d4700 (LWP 160692)): #0 0x00007f6ea1925d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055cdaad0fc27 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055cdaad0ff7d in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055cdaaf584aa in pfs_os_file_flush_func (file=..., src_file=0x55cdab5d2b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055cdaaf5b225 in fil_flush_low (space=0x55cdad97e7f0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055cdaaf66aff in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055cdaaceb321 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055cdaacebd83 in log_write_up_to (lsn=6424872, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055cdaae56e99 in trx_flush_log_if_needed_low (lsn=6424872) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000055cdaae56ee2 in trx_flush_log_if_needed (lsn=6424872, trx=0x7f6e7b401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000055cdaae58d0b in trx_commit_complete_for_mysql (trx=0x7f6e7b401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000055cdaac3d5b2 in innobase_commit (hton=0x55cdad748a50, thd=0x7f6e20172960, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4633 #12 0x000055cdaaa170d7 in commit_one_phase_2 (thd=0x7f6e20172960, all=true, trans=0x7f6e20175eb0, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000055cdaaa16fd9 in ha_commit_one_phase (thd=0x7f6e20172960, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000055cdaaa16355 in ha_commit_trans (thd=0x7f6e20172960, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000055cdaa85be8e in trans_commit (thd=0x7f6e20172960) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000055cdaac1b22a in Wsrep_storage_service::commit (this=0x7f6e201b7e50, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x000055cdab2835c9 in wsrep::transaction::certify_fragment (this=0x7f6e200069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1157 #18 0x000055cdab282d2d in wsrep::transaction::streaming_step (this=0x7f6e200069d0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1031 #19 0x000055cdab2800e8 in wsrep::transaction::after_row (this=0x7f6e200069d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x000055cdaaa29446 in wsrep::client_state::after_row (this=0x7f6e20006960) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x000055cdaaa126c3 in wsrep_after_row (thd=0x7f6e20000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x000055cdaaa245c8 in wsrep_after_row (thd=0x7f6e20000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6442 #23 0x000055cdaaa249b9 in handler::ha_write_row (this=0x7f6e2009eea8, buf=0x7f6e2009ea50 "\376\017\"") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6473 #24 0x000055cdaa69a443 in write_record (thd=0x7f6e20000b00, table=0x7f6e2009de10, info=0x7f6e20016ea0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x000055cdaa69f742 in select_insert::send_data (this=0x7f6e20016e58, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x000055cdaa757e07 in end_send (join=0x7f6e20016f08, join_tab=0x7f6e200aa7f8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x000055cdaa894fa7 in JOIN_CACHE::generate_full_extensions (this=0x7f6e200aaba8, rec_ptr=0x7f6e2012e01c "\360") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x000055cdaa894cc2 in JOIN_CACHE::join_matching_records (this=0x7f6e200aaba8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x000055cdaa89468e in JOIN_CACHE::join_records (this=0x7f6e200aaba8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x000055cdaa894828 in JOIN_CACHE::join_records (this=0x7f6e20019ca8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x000055cdaa894828 in JOIN_CACHE::join_records (this=0x7f6e20019b78, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x000055cdaa754abb in sub_select_cache (join=0x7f6e20016f08, join_tab=0x7f6e200a9ce8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x000055cdaa754cc8 in sub_select (join=0x7f6e20016f08, join_tab=0x7f6e200a9938, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x000055cdaa754493 in do_select (join=0x7f6e20016f08, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x000055cdaa72cfb1 in JOIN::exec_inner (this=0x7f6e20016f08) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x000055cdaa72c3fa in JOIN::exec (this=0x7f6e20016f08) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x000055cdaa72d6ec in mysql_select (thd=0x7f6e20000b00, tables=0x7f6e20014c30, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f6e20016e58, unit=0x7f6e200049c0, select_lex=0x7f6e200146f0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x000055cdaa71efed in handle_select (thd=0x7f6e20000b00, lex=0x7f6e200048f8, result=0x7f6e20016e58, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x000055cdaa6e13fc in mysql_execute_command (thd=0x7f6e20000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4859 #40 0x000055cdaa6ec935 in mysql_parse (thd=0x7f6e20000b00, rawbuf=0x7f6e20013e58 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7f6e903d30d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #41 0x000055cdaa6ebf68 in wsrep_mysql_parse (thd=0x7f6e20000b00, rawbuf=0x7f6e20013e58 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7f6e903d30d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #42 0x000055cdaa6d7e25 in dispatch_command (command=COM_QUERY, thd=0x7f6e20000b00, packet=0x7f6e2000a431 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=86, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #43 0x000055cdaa6d680d in do_command (thd=0x7f6e20000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x000055cdaa847069 in do_handle_one_connection (connect=0x55cdada08100) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #45 0x000055cdaa846dad in handle_one_connection (arg=0x55cdada08100) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #46 0x000055cdab16b555 in pfs_spawn_thread (arg=0x55cdad5e0a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f6e6ffff700 (LWP 159872)): #0 0x00007f6ea1925d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055cdaad0fc27 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055cdaad0ff7d in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055cdaaf584aa in pfs_os_file_flush_func (file=..., src_file=0x55cdab5d2b50 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=827) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055cdaaf5b225 in fil_flush_low (space=0x55cdad97e450, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:827 #5 0x000055cdaaf66aff in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4449 #6 0x000055cdaaf66d86 in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4522 #7 0x000055cdaaee310a in buf_dblwr_update (bpage=0x7f6e7abe8a70, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:756 #8 0x000055cdaaeeaaf7 in buf_flush_write_complete (bpage=0x7f6e7abe8a70, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000055cdaaed97b1 in buf_page_io_complete (bpage=0x7f6e7abe8a70, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6138 #10 0x000055cdaaf66859 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4409 #11 0x000055cdaae0c9fb in io_handler_thread (arg=0x55cdabf93648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f6e91b7b700 (LWP 159861)): #0 0x00007f6e9fcdd483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6e997ce6f8 in asio::detail::epoll_reactor::run (this=0x55cdad66cfc0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f6e997d2df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55cdad66cee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f6e997d30a2 in asio::detail::task_io_service::run (this=0x55cdad66cee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f6e997eaafd in asio::io_service::run (this=0x55cdad66bfe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f6e997eb251 in gcomm::AsioProtonet::event_loop (this=0x55cdad66bf40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f6e998092ed in GCommConn::run (this=0x55cdad66c6c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f6e9980c399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f6e9877f700 (LWP 159864)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdab1d8a8d in safe_cond_wait (cond=0x7f6e7c015608, mp=0x7f6e7c015558, file=0x55cdab342c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cdaa930126 in inline_mysql_cond_wait (that=0x7f6e7c015608, mutex=0x7f6e7c015558, src_file=0x55cdab342f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055cdaa932dbe in Wsrep_thd_queue::pop_front (this=0x7f6e7c015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055cdaa931b9e in wsrep_post_rollback_process (post_rollbacker=0x7f6e7c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055cdaa92231f in start_wsrep_THD (arg=0x55cdad666810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f6e7abad700 (LWP 159867)): #0 0x00007f6ea14ef644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cdaad0e578 in LinuxAIOHandler::collect (this=0x7f6e7abacb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cdaad0eaa1 in LinuxAIOHandler::poll (this=0x7f6e7abacb00, m1=0x7f6e7abacba8, m2=0x7f6e7abacbb0, request=0x7f6e7abacbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cdaad0ec04 in os_aio_linux_handler (global_segment=0, m1=0x7f6e7abacba8, m2=0x7f6e7abacbb0, request=0x7f6e7abacbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cdaad13728 in os_aio_handler (segment=0, m1=0x7f6e7abacba8, m2=0x7f6e7abacbb0, request=0x7f6e7abacbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cdaaf665ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cdaae0c9fb in io_handler_thread (arg=0x55cdabf93620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f6e79bab700 (LWP 159869)): #0 0x00007f6ea14ef644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cdaad0e578 in LinuxAIOHandler::collect (this=0x7f6e79baab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cdaad0eaa1 in LinuxAIOHandler::poll (this=0x7f6e79baab00, m1=0x7f6e79baaba8, m2=0x7f6e79baabb0, request=0x7f6e79baabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cdaad0ec04 in os_aio_linux_handler (global_segment=2, m1=0x7f6e79baaba8, m2=0x7f6e79baabb0, request=0x7f6e79baabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cdaad13728 in os_aio_handler (segment=2, m1=0x7f6e79baaba8, m2=0x7f6e79baabb0, request=0x7f6e79baabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cdaaf665ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cdaae0c9fb in io_handler_thread (arg=0x55cdabf93630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f6e793aa700 (LWP 159870)): #0 0x00007f6ea14ef644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cdaad0e578 in LinuxAIOHandler::collect (this=0x7f6e793a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cdaad0eaa1 in LinuxAIOHandler::poll (this=0x7f6e793a9b00, m1=0x7f6e793a9ba8, m2=0x7f6e793a9bb0, request=0x7f6e793a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cdaad0ec04 in os_aio_linux_handler (global_segment=3, m1=0x7f6e793a9ba8, m2=0x7f6e793a9bb0, request=0x7f6e793a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cdaad13728 in os_aio_handler (segment=3, m1=0x7f6e793a9ba8, m2=0x7f6e793a9bb0, request=0x7f6e793a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cdaaf665ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cdaae0c9fb in io_handler_thread (arg=0x55cdabf93638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f6e6e7fc700 (LWP 159875)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1ab76 in os_event::timed_wait (this=0x55cdad94cd70, abstime=0x7f6e6e7fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cdaad1ae8d in os_event::wait_time_low (this=0x55cdad94cd70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cdaad1b02d in os_event_wait_time_low (event=0x55cdad94cd70, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cdaace79e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f6e6dffb700 (LWP 159876)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1ab76 in os_event::timed_wait (this=0x55cdad774840, abstime=0x7f6e6dffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cdaad1ae8d in os_event::wait_time_low (this=0x55cdad774840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cdaad1b02d in os_event_wait_time_low (event=0x55cdad774840, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cdaae090ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f6e5ffff700 (LWP 159881)): #0 0x00007f6ea1925e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055cdaad1b55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055cdaae0a44d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055cdaae0a57d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f6e5cff9700 (LWP 159894)): #0 0x00007f6ea1925e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055cdaad1b55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055cdaaec1b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f6e5effd700 (LWP 159883)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1b25b in os_event::wait (this=0x55cdad774740) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cdaad1ace1 in os_event::wait_low (this=0x55cdad774740, reset_sig_count=17125) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cdaad1b052 in os_event_wait_low (event=0x55cdad774740, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cdaae06ee5 in srv_resume_thread (slot=0x55cdabd557b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055cdaae0aae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f6e5e7fc700 (LWP 159884)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1b25b in os_event::wait (this=0x55cdad7747c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cdaad1ace1 in os_event::wait_low (this=0x55cdad7747c0, reset_sig_count=14822) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cdaad1b052 in os_event_wait_low (event=0x55cdad7747c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cdaae06ee5 in srv_resume_thread (slot=0x55cdabd557f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055cdaae0aae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f6e98125700 (LWP 159922)): #0 0x00007f6e9fc1640a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055cdaa5a4f44 in my_sigwait (set=0x7f6e98124cf0, sig=0x7f6e98124cb4, code=0x7f6e98124cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055cdaa5ab584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055cdab16b555 in pfs_spawn_thread (arg=0x55cdad737d90) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f6e9808f700 (LWP 160009)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdab1d8d8f in safe_cond_timedwait (cond=0x55cdabf181a0 , mp=0x55cdabf1aba0 , abstime=0x7f6e9808ec60, file=0x55cdab2945c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055cdaa5a5845 in inline_mysql_cond_timedwait (that=0x55cdabf181a0 , mutex=0x55cdabf1aba0 , abstime=0x7f6e9808ec60, src_file=0x55cdab296638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055cdaa5aa61b in cache_thread (thd=0x7f6e24000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055cdaa5aa8c9 in one_thread_per_connection_end (thd=0x7f6e24000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055cdaa8470f3 in do_handle_one_connection (connect=0x55cdadaffc40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055cdaa846dad in handle_one_connection (arg=0x55cdadaffc40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055cdab16b555 in pfs_spawn_thread (arg=0x55cdadb053a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f6e47fff700 (LWP 159895)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1b25b in os_event::wait (this=0x55cdad774a40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cdaad1ace1 in os_event::wait_low (this=0x55cdad774a40, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cdaad1b052 in os_event_wait_low (event=0x55cdad774a40, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cdaaecfc2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f6e5d7fa700 (LWP 159893)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1b25b in os_event::wait (this=0x55cdad774940) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cdaad1ace1 in os_event::wait_low (this=0x55cdad774940, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cdaad1b052 in os_event_wait_low (event=0x55cdad774940, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cdaaee6fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f6e5f7fe700 (LWP 159882)): #0 0x00007f6e9fcc1d47 in sched_yield () from /lib64/libc.so.6 #1 0x000055cdaad1b4ef in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x000055cdaae34ccb in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x000055cdaae34ed3 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x000055cdaae0adbc in srv_do_purge (n_total_purged=0x7f6e5f7fdd98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2577 #5 0x000055cdaae0b1e9 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2700 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f6e5dffb700 (LWP 159885)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1b25b in os_event::wait (this=0x55cdad85a810) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055cdaad1ace1 in os_event::wait_low (this=0x55cdad85a810, reset_sig_count=1179) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055cdaad1b052 in os_event_wait_low (event=0x55cdad85a810, reset_sig_count=1179) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055cdaae14cee in sync_array_wait_event (arr=0x55cdad74b990, cell=@0x7f6e5dff8e70: 0x55cdad74bac0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000055cdaac66195 in TTASEventMutex::enter (this=0x55cdad85a140, max_spins=60, max_delay=4, filename=0x55cdab5ad388 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x000055cdaac643f0 in PolicyMutex >::enter (this=0x55cdad85a140, n_spins=30, n_delay=4, name=0x55cdab5ad388 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x000055cdaaef95d3 in buf_LRU_get_free_block (buf_pool=0x55cdad85a140) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1092 #8 0x000055cdaaed6e1b in buf_page_init_for_read (err=0x7f6e5dff9884, mode=132, page_id=..., zip_size=0, unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5332 #9 0x000055cdaaf001c6 in buf_read_page_low (err=0x7f6e5dff9884, sync=false, type=192, mode=132, page_id=..., zip_size=0, unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:151 #10 0x000055cdaaf00ef0 in buf_read_page_background (page_id=..., zip_size=0, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:444 #11 0x000055cdaae9a10a in btr_cur_prefetch_siblings (block=0x7f6e7abb0970) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3304 #12 0x000055cdaaea2905 in btr_cur_optimistic_delete_func (cursor=0x55cdad9961f8, flags=0, mtr=0x7f6e5dffa1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5806 #13 0x000055cdaadc80e6 in row_purge_remove_clust_if_poss_low (node=0x55cdad996160, mode=2) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:168 #14 0x000055cdaadc82d5 in row_purge_remove_clust_if_poss (node=0x55cdad996160) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:219 #15 0x000055cdaadc9b28 in row_purge_del_mark (node=0x55cdad996160) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #16 0x000055cdaadcb0b0 in row_purge_record_func (node=0x55cdad996160, undo_rec=0x55cdad996638 "", thr=0x55cdad9960a8, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #17 0x000055cdaadcb3ed in row_purge (node=0x55cdad996160, undo_rec=0x55cdad996638 "", thr=0x55cdad9960a8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #18 0x000055cdaadcb798 in row_purge_step (thr=0x55cdad9960a8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #19 0x000055cdaad4dbf6 in que_thr_step (thr=0x55cdad9960a8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #20 0x000055cdaad4de26 in que_run_threads_low (thr=0x55cdad9960a8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #21 0x000055cdaad4e016 in que_run_threads (thr=0x55cdad9960a8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #22 0x000055cdaae0a947 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2437 #23 0x000055cdaae0aaee in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #24 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f6e980da700 (LWP 160008)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdab1d8a8d in safe_cond_wait (cond=0x55cdabf181e0 , mp=0x55cdabf1b560 , file=0x55cdab2a3dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cdaa5d64ee in inline_mysql_cond_wait (that=0x55cdabf181e0 , mutex=0x55cdabf1b560 , src_file=0x55cdab2a4930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055cdaa5d7d80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055cdab16b555 in pfs_spawn_thread (arg=0x55cdadad6120) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f6e6cff9700 (LWP 159879)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1ab76 in os_event::timed_wait (this=0x55cdad914ac0, abstime=0x7f6e6cff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cdaad1ae8d in os_event::wait_time_low (this=0x55cdad914ac0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cdaad1b02d in os_event_wait_time_low (event=0x55cdad914ac0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cdaae69403 in ib_wqueue_timedwait (wq=0x55cdad914990, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055cdaafa9f9a in fts_optimize_thread (arg=0x55cdad914990) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f6e477fe700 (LWP 159896)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdab1d8a8d in safe_cond_wait (cond=0x7f6e477fdc70, mp=0x7f6e477fdcb0, file=0x55cdab4a6d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cdaac32525 in inline_mysql_cond_wait (that=0x7f6e477fdc70, mutex=0x7f6e477fdcb0, src_file=0x55cdab4a86d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055cdaac37161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055cdab16b555 in pfs_spawn_thread (arg=0x55cdad9b5980) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f6e6f7fe700 (LWP 159873)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1ab76 in os_event::timed_wait (this=0x55cdad7749c0, abstime=0x7f6e6f7fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cdaad1ae8d in os_event::wait_time_low (this=0x55cdad7749c0, time_in_usec=545000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cdaad1b02d in os_event_wait_time_low (event=0x55cdad7749c0, time_in_usec=545000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cdaaef0e25 in pc_sleep_if_needed (next_loop_time=1549625649158, sig_count=2, cur_time=1549625648613) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055cdaaef1d5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f6e6d7fa700 (LWP 159878)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1ab76 in os_event::timed_wait (this=0x55cdad97ee30, abstime=0x7f6e6d7f9d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cdaad1ae8d in os_event::wait_time_low (this=0x55cdad97ee30, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cdaad1b02d in os_event_wait_time_low (event=0x55cdad97ee30, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cdaaf5024a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f6e78ba9700 (LWP 159871)): #0 0x00007f6ea14ef644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cdaad0e578 in LinuxAIOHandler::collect (this=0x7f6e78ba8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cdaad0eaa1 in LinuxAIOHandler::poll (this=0x7f6e78ba8b00, m1=0x7f6e78ba8ba8, m2=0x7f6e78ba8bb0, request=0x7f6e78ba8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cdaad0ec04 in os_aio_linux_handler (global_segment=4, m1=0x7f6e78ba8ba8, m2=0x7f6e78ba8bb0, request=0x7f6e78ba8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cdaad13728 in os_aio_handler (segment=4, m1=0x7f6e78ba8ba8, m2=0x7f6e78ba8bb0, request=0x7f6e78ba8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cdaaf665ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cdaae0c9fb in io_handler_thread (arg=0x55cdabf93640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f6e6effd700 (LWP 159877)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdaad1ab76 in os_event::timed_wait (this=0x55cdad7748c0, abstime=0x7f6e6effcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055cdaad1ae8d in os_event::wait_time_low (this=0x55cdad7748c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055cdaad1b02d in os_event_wait_time_low (event=0x55cdad7748c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055cdaae08bcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f6e7a3ac700 (LWP 159868)): #0 0x00007f6ea14ef644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055cdaad0e578 in LinuxAIOHandler::collect (this=0x7f6e7a3abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055cdaad0eaa1 in LinuxAIOHandler::poll (this=0x7f6e7a3abb00, m1=0x7f6e7a3abba8, m2=0x7f6e7a3abbb0, request=0x7f6e7a3abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055cdaad0ec04 in os_aio_linux_handler (global_segment=1, m1=0x7f6e7a3abba8, m2=0x7f6e7a3abbb0, request=0x7f6e7a3abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055cdaad13728 in os_aio_handler (segment=1, m1=0x7f6e7a3abba8, m2=0x7f6e7a3abbb0, request=0x7f6e7a3abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055cdaaf665ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055cdaae0c9fb in io_handler_thread (arg=0x55cdabf93628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f6e7bfff700 (LWP 159866)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdab1d8d8f in safe_cond_timedwait (cond=0x55cdac7a4ea0 , mp=0x55cdac7a4de0 , abstime=0x7f6e7bffecb0, file=0x55cdab61c6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055cdab042bc1 in inline_mysql_cond_timedwait (that=0x55cdac7a4ea0 , mutex=0x55cdac7a4de0 , abstime=0x7f6e7bffecb0, src_file=0x55cdab61c738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055cdab0430e5 in my_service_thread_sleep (control=0x55cdabd59b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055cdab03757e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055cdab16b555 in pfs_spawn_thread (arg=0x55cdad72c720) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f6e98734700 (LWP 159865)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6e99721132 in fifo_lock_get (q=q@entry=0x55cdad652bd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f6e997217a1 in gu_fifo_get_head (q=0x55cdad652bd0, err=0x7f6e9873306c) at galerautils/src/gu_fifo.c:330 #3 0x00007f6e998030cc in gcs_recv (conn=0x55cdad62c550, action=0x7f6e987330b0) at gcs/src/gcs.cpp:2024 #4 0x00007f6e9983016c in galera::GcsActionSource::process (this=0x55cdad62bf30, recv_ctx=0x7f6e987334d0, exit_loop=@0x7f6e9873329e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f6e9984f033 in galera::ReplicatorSMM::async_recv (this=0x55cdad624280, recv_ctx=0x7f6e987334d0) at galera/src/replicator_smm.cpp:383 #6 0x000055cdab28d36c in wsrep::wsrep_provider_v26::run_applier (this=0x55cdad5ab360, applier_ctx=0x7f6e987334d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055cdaa9306d4 in wsrep_replication_process (thd=0x7f6e80000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055cdaa92231f in start_wsrep_THD (arg=0x55cdad686550) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f6e93fff700 (LWP 159838)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6e9982cfd7 in gu::Lock::wait (cond=..., this=0x7f6e93ffebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6e9982d069 in galera::ServiceThd::thd_func (arg=0x55cdad624998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f6e987ca700 (LWP 159863)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdab1d8a8d in safe_cond_wait (cond=0x7f6e88017b68, mp=0x7f6e88017ab8, file=0x55cdab342c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055cdaa930126 in inline_mysql_cond_wait (that=0x7f6e88017b68, mutex=0x7f6e88017ab8, src_file=0x55cdab342f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055cdaa932dbe in Wsrep_thd_queue::pop_front (this=0x7f6e88017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055cdaa9317de in wsrep_rollback_process (rollbacker=0x7f6e88001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055cdaa92231f in start_wsrep_THD (arg=0x55cdad67c340) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f6e9a377700 (LWP 159819)): #0 0x00007f6ea1922d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055cdab1d8d8f in safe_cond_timedwait (cond=0x55cdac7b7640 , mp=0x55cdac7b7580 , abstime=0x7f6e9a376d70, file=0x55cdab660920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055cdab1d9d49 in inline_mysql_cond_timedwait (that=0x55cdac7b7640 , mutex=0x55cdac7b7580 , abstime=0x7f6e9a376d70, src_file=0x55cdab660968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055cdab1da8ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055cdab16b555 in pfs_spawn_thread (arg=0x55cdad5ed730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f6e9137a700 (LWP 159862)): #0 0x00007f6ea1922965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6e99807867 in gu::Lock::wait (cond=..., this=0x7f6e91379850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f6e9980a3ab in RecvBuf::front (this=this@entry=0x55cdad66c8a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f6e99807f46 in gcomm_recv (backend=, msg=0x55cdad62c830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f6e997fbeec in core_msg_recv (backend=backend@entry=0x55cdad62c978, recv_msg=recv_msg@entry=0x55cdad62c830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f6e997ff164 in gcs_core_recv (conn=0x55cdad62c7d0, recv_act=recv_act@entry=0x7f6e91379d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f6e998063d8 in gcs_recv_thread (arg=0x55cdad62c550) at gcs/src/gcs.cpp:1398 #7 0x00007f6ea191edd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6e9fcdcead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f6ea1d2f880 (LWP 159815)): #0 0x00007f6ea19239d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055cdab1d345e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055cdaaa115a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f6e9fcd220d in poll () from /lib64/libc.so.6 #5 0x000055cdaa5b1a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055cdaa5b0d25 in mysqld_main (argc=161, argv=0x55cdad4a5d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055cdaa5a4ec5 in main (argc=23, argv=0x7ffe45f94108) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.160066' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.2/data/core.160066 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 160066] [New LWP 160151] [New LWP 160069] [New LWP 160535] [New LWP 160110] [New LWP 160152] [New LWP 160534] [New LWP 160536] [New LWP 160537] [New LWP 160538] [New LWP 160556] [New LWP 160568] [New LWP 160579] [New LWP 160580] [New LWP 160600] [New LWP 160566] [New LWP 160567] [New LWP 160578] [New LWP 160599] [New LWP 160564] [New LWP 160558] [New LWP 160603] [New LWP 160577] [New LWP 160565] [New LWP 160559] [New LWP 160557] [New LWP 160532] [New LWP 160555] [New LWP 160531] [New LWP 160533] [New LWP 160154] [New LWP 160068] [New LWP 160153] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f893d1869d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f893d1869d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563c75b4645e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563c753845a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f893b53520d in poll () from /lib64/libc.so.6 #5 0x0000563c74f24a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000563c74f23d25 in mysqld_main (argc=162, argv=0x563c78eabd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000563c74f17ec5 in main (argc=23, argv=0x7fff3efdb328) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f89341e2700 (LWP 160153)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c75b4ba8d in safe_cond_wait (cond=0x7f8910016918, mp=0x7f8910016868, file=0x563c75cb5c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563c752a3126 in inline_mysql_cond_wait (that=0x7f8910016918, mutex=0x7f8910016868, src_file=0x563c75cb5f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563c752a5dbe in Wsrep_thd_queue::pop_front (this=0x7f8910016810) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563c752a4b9e in wsrep_post_rollback_process (post_rollbacker=0x7f8910000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000563c7529531f in start_wsrep_THD (arg=0x563c7908ba30) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f8935bda700 (LWP 160068)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c75b4bd8f in safe_cond_timedwait (cond=0x563c7712a640 , mp=0x563c7712a580 , abstime=0x7f8935bd9d70, file=0x563c75fd3920 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563c75b4cd49 in inline_mysql_cond_timedwait (that=0x563c7712a640 , mutex=0x563c7712a580 , abstime=0x7f8935bd9d70, src_file=0x563c75fd3968 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563c75b4d8ac in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000563c75ade555 in pfs_spawn_thread (arg=0x563c78ff37d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f8934197700 (LWP 160154)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f8934f84132 in fifo_lock_get (q=q@entry=0x563c79058ac0) at galerautils/src/gu_fifo.c:262 #2 0x00007f8934f847a1 in gu_fifo_get_head (q=0x563c79058ac0, err=0x7f893419606c) at galerautils/src/gu_fifo.c:330 #3 0x00007f89350660cc in gcs_recv (conn=0x563c79032440, action=0x7f89341960b0) at gcs/src/gcs.cpp:2024 #4 0x00007f893509316c in galera::GcsActionSource::process (this=0x563c79062e60, recv_ctx=0x7f89341964d0, exit_loop=@0x7f893419629e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f89350b2033 in galera::ReplicatorSMM::async_recv (this=0x563c7902a1f0, recv_ctx=0x7f89341964d0) at galera/src/replicator_smm.cpp:383 #6 0x0000563c75c0036c in wsrep::wsrep_provider_v26::run_applier (this=0x563c78f9fe50, applier_ctx=0x7f89341964d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x0000563c752a36d4 in wsrep_replication_process (thd=0x7f891c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000563c7529531f in start_wsrep_THD (arg=0x563c7908c980) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f8905807700 (LWP 160533)): #0 0x00007f893cd52644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563c75681578 in LinuxAIOHandler::collect (this=0x7f8905806b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563c75681aa1 in LinuxAIOHandler::poll (this=0x7f8905806b00, m1=0x7f8905806ba8, m2=0x7f8905806bb0, request=0x7f8905806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563c75681c04 in os_aio_linux_handler (global_segment=1, m1=0x7f8905806ba8, m2=0x7f8905806bb0, request=0x7f8905806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563c75686728 in os_aio_handler (segment=1, m1=0x7f8905806ba8, m2=0x7f8905806bb0, request=0x7f8905806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563c758d95ad in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563c7577f9fb in io_handler_thread (arg=0x563c76906628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f890785a700 (LWP 160531)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c75b4bd8f in safe_cond_timedwait (cond=0x563c77117ea0 , mp=0x563c77117de0 , abstime=0x7f8907859cb0, file=0x563c75f8f6f0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563c759b5bc1 in inline_mysql_cond_timedwait (that=0x563c77117ea0 , mutex=0x563c77117de0 , abstime=0x7f8907859cb0, src_file=0x563c75f8f738 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563c759b60e5 in my_service_thread_sleep (control=0x563c766ccb00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000563c759aa57e in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000563c75ade555 in pfs_spawn_thread (arg=0x563c79133660) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f88f9df6700 (LWP 160555)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568db76 in os_event::timed_wait (this=0x563c79353a10, abstime=0x7f88f9df5d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563c7568de8d in os_event::wait_time_low (this=0x563c79353a10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563c7568e02d in os_event_wait_time_low (event=0x563c79353a10, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563c7565a9e0 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f8906008700 (LWP 160532)): #0 0x00007f893cd52644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563c75681578 in LinuxAIOHandler::collect (this=0x7f8906007b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563c75681aa1 in LinuxAIOHandler::poll (this=0x7f8906007b00, m1=0x7f8906007ba8, m2=0x7f8906007bb0, request=0x7f8906007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563c75681c04 in os_aio_linux_handler (global_segment=0, m1=0x7f8906007ba8, m2=0x7f8906007bb0, request=0x7f8906007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563c75686728 in os_aio_handler (segment=0, m1=0x7f8906007ba8, m2=0x7f8906007bb0, request=0x7f8906007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563c758d95ad in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563c7577f9fb in io_handler_thread (arg=0x563c76906620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f88f8df4700 (LWP 160557)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568db76 in os_event::timed_wait (this=0x563c7917b560, abstime=0x7f88f8df3d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563c7568de8d in os_event::wait_time_low (this=0x563c7917b560, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563c7568e02d in os_event_wait_time_low (event=0x563c7917b560, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563c7577bbcc in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f88eb7fe700 (LWP 160559)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568db76 in os_event::timed_wait (this=0x563c79323730, abstime=0x7f88eb7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563c7568de8d in os_event::wait_time_low (this=0x563c79323730, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563c7568e02d in os_event_wait_time_low (event=0x563c79323730, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563c757dc403 in ib_wqueue_timedwait (wq=0x563c79323600, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000563c7591cf9a in fts_optimize_thread (arg=0x563c79323600) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f88eaffd700 (LWP 160565)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568e25b in os_event::wait (this=0x563c7917b2e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563c7568dce1 in os_event::wait_low (this=0x563c7917b2e0, reset_sig_count=4359) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563c7568e052 in os_event_wait_low (event=0x563c7917b2e0, reset_sig_count=4359) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563c75779ee5 in srv_resume_thread (slot=0x563c766c8748 , sig_count=4359, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563c7577def6 in srv_purge_coordinator_suspend (slot=0x563c766c8748 , rseg_history_len=28) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000563c7577e18c in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f88e8ff9700 (LWP 160577)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568e25b in os_event::wait (this=0x563c7917b5e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563c7568dce1 in os_event::wait_low (this=0x563c7917b5e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563c7568e052 in os_event_wait_low (event=0x563c7917b5e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563c75859fb9 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f8924069700 (LWP 160603)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c75b4bd8f in safe_cond_timedwait (cond=0x563c7688b1a0 , mp=0x563c7688dba0 , abstime=0x7f8924068c60, file=0x563c75c075c8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000563c74f18845 in inline_mysql_cond_timedwait (that=0x563c7688b1a0 , mutex=0x563c7688dba0 , abstime=0x7f8924068c60, src_file=0x563c75c09638 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000563c74f1d61b in cache_thread (thd=0x7f8908000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000563c74f1d8c9 in one_thread_per_connection_end (thd=0x7f8908000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000563c751ba0f3 in do_handle_one_connection (connect=0x563c7947b430) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x0000563c751b9dad in handle_one_connection (arg=0x563c7947b430) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x0000563c75ade555 in pfs_spawn_thread (arg=0x563c794795a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f88ebfff700 (LWP 160558)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568db76 in os_event::timed_wait (this=0x563c79385ad0, abstime=0x7f88ebffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563c7568de8d in os_event::wait_time_low (this=0x563c79385ad0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563c7568e02d in os_event_wait_time_low (event=0x563c79385ad0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563c758c324a in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f88fa5f7700 (LWP 160564)): #0 0x00007f893d188e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563c7568e55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563c7577d44d in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000563c7577d57d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f893404b700 (LWP 160599)): #0 0x00007f893b47940a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000563c74f17f44 in my_sigwait (set=0x7f893404acf0, sig=0x7f893404acb4, code=0x7f893404acb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000563c74f1e584 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000563c75ade555 in pfs_spawn_thread (arg=0x563c7913ecd0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f88cbfff700 (LWP 160578)): #0 0x00007f893d188e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000563c7568e55d in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000563c75834b44 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f88e9ffb700 (LWP 160567)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568e25b in os_event::wait (this=0x563c7917b460) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563c7568dce1 in os_event::wait_low (this=0x563c7917b460, reset_sig_count=15026) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563c7568e052 in os_event_wait_low (event=0x563c7917b460, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563c75779ee5 in srv_resume_thread (slot=0x563c766c87f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563c7577dae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f88ea7fc700 (LWP 160566)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568e25b in os_event::wait (this=0x563c7917b360) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563c7568dce1 in os_event::wait_low (this=0x563c7917b360, reset_sig_count=20725) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563c7568e052 in os_event_wait_low (event=0x563c7917b360, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563c75779ee5 in srv_resume_thread (slot=0x563c766c8780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563c7577dae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f89240b4700 (LWP 160600)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c75b4ba8d in safe_cond_wait (cond=0x563c7688b1e0 , mp=0x563c7688e560 , file=0x563c75c16dc8 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563c74f494ee in inline_mysql_cond_wait (that=0x563c7688b1e0 , mutex=0x563c7688e560 , src_file=0x563c75c17930 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000563c74f4ad80 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000563c75ade555 in pfs_spawn_thread (arg=0x563c7913f1c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f88caffd700 (LWP 160580)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c75b4ba8d in safe_cond_wait (cond=0x7f88caffcc70, mp=0x7f88caffccb0, file=0x563c75e19d40 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563c755a5525 in inline_mysql_cond_wait (that=0x7f88caffcc70, mutex=0x7f88caffccb0, src_file=0x563c75e1b6d8 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563c755aa161 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000563c75ade555 in pfs_spawn_thread (arg=0x563c79336250) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f88cb7fe700 (LWP 160579)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568e25b in os_event::wait (this=0x563c7917b6e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563c7568dce1 in os_event::wait_low (this=0x563c7917b6e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563c7568e052 in os_event_wait_low (event=0x563c7917b6e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563c75842c2e in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f88e97fa700 (LWP 160568)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568e25b in os_event::wait (this=0x563c7917b3e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000563c7568dce1 in os_event::wait_low (this=0x563c7917b3e0, reset_sig_count=16680) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000563c7568e052 in os_event_wait_low (event=0x563c7917b3e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000563c75779ee5 in srv_resume_thread (slot=0x563c766c87b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000563c7577dae9 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f88f95f5700 (LWP 160556)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568db76 in os_event::timed_wait (this=0x563c7917b4e0, abstime=0x7f88f95f4ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563c7568de8d in os_event::wait_time_low (this=0x563c7917b4e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563c7568e02d in os_event_wait_time_low (event=0x563c7917b4e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563c7577c0ee in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f88faffd700 (LWP 160538)): #0 0x00007f893d185d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c7568db76 in os_event::timed_wait (this=0x563c7917b660, abstime=0x7f88faffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000563c7568de8d in os_event::wait_time_low (this=0x563c7917b660, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000563c7568e02d in os_event_wait_time_low (event=0x563c7917b660, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000563c75863e25 in pc_sleep_if_needed (next_loop_time=1549625656315, sig_count=2, cur_time=1549625655315) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000563c75864d5e in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f88fb7fe700 (LWP 160537)): #0 0x00007f893cd52644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563c75681578 in LinuxAIOHandler::collect (this=0x7f88fb7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563c75681aa1 in LinuxAIOHandler::poll (this=0x7f88fb7fdb00, m1=0x7f88fb7fdba8, m2=0x7f88fb7fdbb0, request=0x7f88fb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563c75681c04 in os_aio_linux_handler (global_segment=5, m1=0x7f88fb7fdba8, m2=0x7f88fb7fdbb0, request=0x7f88fb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563c75686728 in os_aio_handler (segment=5, m1=0x7f88fb7fdba8, m2=0x7f88fb7fdbb0, request=0x7f88fb7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563c758d95ad in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563c7577f9fb in io_handler_thread (arg=0x563c76906648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f88fbfff700 (LWP 160536)): #0 0x00007f893cd52644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563c75681578 in LinuxAIOHandler::collect (this=0x7f88fbffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563c75681aa1 in LinuxAIOHandler::poll (this=0x7f88fbffeb00, m1=0x7f88fbffeba8, m2=0x7f88fbffebb0, request=0x7f88fbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563c75681c04 in os_aio_linux_handler (global_segment=4, m1=0x7f88fbffeba8, m2=0x7f88fbffebb0, request=0x7f88fbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563c75686728 in os_aio_handler (segment=4, m1=0x7f88fbffeba8, m2=0x7f88fbffebb0, request=0x7f88fbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563c758d95ad in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563c7577f9fb in io_handler_thread (arg=0x563c76906640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f8905006700 (LWP 160534)): #0 0x00007f893cd52644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563c75681578 in LinuxAIOHandler::collect (this=0x7f8905005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563c75681aa1 in LinuxAIOHandler::poll (this=0x7f8905005b00, m1=0x7f8905005ba8, m2=0x7f8905005bb0, request=0x7f8905005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563c75681c04 in os_aio_linux_handler (global_segment=2, m1=0x7f8905005ba8, m2=0x7f8905005bb0, request=0x7f8905005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563c75686728 in os_aio_handler (segment=2, m1=0x7f8905005ba8, m2=0x7f8905005bb0, request=0x7f8905005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563c758d95ad in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563c7577f9fb in io_handler_thread (arg=0x563c76906630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f893422d700 (LWP 160152)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000563c75b4ba8d in safe_cond_wait (cond=0x7f8914015608, mp=0x7f8914015558, file=0x563c75cb5c60 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000563c752a3126 in inline_mysql_cond_wait (that=0x7f8914015608, mutex=0x7f8914015558, src_file=0x563c75cb5f80 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000563c752a5dbe in Wsrep_thd_queue::pop_front (this=0x7f8914015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000563c752a47de in wsrep_rollback_process (rollbacker=0x7f8914000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000563c7529531f in start_wsrep_THD (arg=0x563c7908cb90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f892637b700 (LWP 160110)): #0 0x00007f893b540483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f89350316f8 in asio::detail::epoll_reactor::run (this=0x563c7906b000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f8935035df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x563c7906aea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f89350360a2 in asio::detail::task_io_service::run (this=0x563c7906aea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f893504dafd in asio::io_service::run (this=0x563c79069bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f893504e251 in gcomm::AsioProtonet::event_loop (this=0x563c79069b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f893506c2ed in GCommConn::run (this=0x563c7906a630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f893506f399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f8904805700 (LWP 160535)): #0 0x00007f893cd52644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000563c75681578 in LinuxAIOHandler::collect (this=0x7f8904804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000563c75681aa1 in LinuxAIOHandler::poll (this=0x7f8904804b00, m1=0x7f8904804ba8, m2=0x7f8904804bb0, request=0x7f8904804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000563c75681c04 in os_aio_linux_handler (global_segment=3, m1=0x7f8904804ba8, m2=0x7f8904804bb0, request=0x7f8904804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000563c75686728 in os_aio_handler (segment=3, m1=0x7f8904804ba8, m2=0x7f8904804bb0, request=0x7f8904804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000563c758d95ad in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000563c7577f9fb in io_handler_thread (arg=0x563c76906638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f8934a2e700 (LWP 160069)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f893508ffd7 in gu::Lock::wait (cond=..., this=0x7f8934a2dbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f8935090069 in galera::ServiceThd::thd_func (arg=0x563c7902a908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f8925b7a700 (LWP 160151)): #0 0x00007f893d185965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f893506a867 in gu::Lock::wait (cond=..., this=0x7f8925b79850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f893506d3ab in RecvBuf::front (this=this@entry=0x563c7906a818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f893506af46 in gcomm_recv (backend=, msg=0x563c79032720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f893505eeec in core_msg_recv (backend=backend@entry=0x563c79032868, recv_msg=recv_msg@entry=0x563c79032720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f8935062164 in gcs_core_recv (conn=0x563c790326c0, recv_act=recv_act@entry=0x7f8925b79d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f89350693d8 in gcs_recv_thread (arg=0x563c79032440) at gcs/src/gcs.cpp:1398 #7 0x00007f893d181dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f893b53fead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f893d592880 (LWP 160066)): #0 0x00007f893d1869d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000563c75b4645e in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000563c753845a1 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f893b53520d in poll () from /lib64/libc.so.6 #5 0x0000563c74f24a8a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000563c74f23d25 in mysqld_main (argc=162, argv=0x563c78eabd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000563c74f17ec5 in main (argc=23, argv=0x7fff3efdb328) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/1/log/galera_sr.mysql-wsrep-features#8-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#8 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1008 'innodb' w1 [ pass ] 7689 galera_sr.GCF-585 'innodb' w1 [ pass ] 17420 -------------------------------------------------------------------------- The servers were restarted 27 times Spent 1631.643 of 4429 seconds executing testcases Check of testcase failed for: galera_sr.mysql-wsrep-features#148 galera_sr.galera_sr_kill_all_norecovery Completed: Failed 17/83 tests, 79.52% were successful. Failing test(s): galera_sr.galera_sr_large_fragment galera_sr.GCF-1018B galera_sr.galera-features#56 galera_sr.galera_sr_insert_select galera_sr.GCF-437 galera_sr.galera_sr_cc_master galera_sr.galera_sr_rollback_statement galera_sr.mysql-wsrep-features#165 galera_sr.galera_sr_table_contents galera_sr.galera_sr_load_data galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_mysqldump_sst galera_sr.GCF-1043A galera_sr.galera_sr_many_fragments galera_sr.galera_sr_small_gcache galera_sr.GCF-1043B galera_sr.mysql-wsrep-features#8 The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html Errors/warnings were found in logfiles during server shutdown after running the following sequence(s) of tests: galera_sr.galera_sr_large_fragment galera_sr.GCF-1018B galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.GCF-623 galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera-features#56 galera_sr.GCF-845 galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_conflict_with_rollback_master galera_sr.galera_sr_ddl_master galera_sr.galera_sr_insert_select galera_sr.galera_sr_cc_master galera_sr.galera_sr_bf_abort galera_sr.galera_sr_blob galera_sr.galera_sr_kill_query galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 galera_sr.galera_sr_load_data galera_sr.galera_sr_kill_connection galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery galera_sr.GCF-1043A galera_sr.GCF-1018 galera_sr.mysql-wsrep-features#9 galera_sr.galera_sr_many_fragments galera_sr.galera_sr_small_gcache galera_sr.GCF-1043B galera_sr.mysql-wsrep-features#8 mysql-test-run: *** ERROR: there were failing test cases