Checking supported features... MariaDB Version 10.4.3-MariaDB-debug - SSL connections supported - binaries are debug compiled Using suites: galera_sr Collecting tests... Installing system database... worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 worker[1] Trying to dump core for [mysqltest - pid: 19180, winpid: 19180] worker[1] Trying to dump core for [mysqld.1 - pid: 18934, winpid: 18934] worker[1] Trying to dump core for [mysqld.2 - pid: 18971, winpid: 18971] ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- galera_sr.GCF-574 [ disabled ] CTAS is not supported together with SR galera_sr.galera_sr_sbr [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.mysql-wsrep-features#29 [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.galera_sr_large_fragment 'innodb' [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-04 17:58:44 line 2019-02-04 17:52:12 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:52:12 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:52:43 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:52:43 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:53:14 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:53:14 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:53:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:53:45 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:54:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:54:16 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:54:16 0 [Warning] InnoDB: A long semaphore wait: ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.1.err ok line 2019-02-04 17:52:50 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:53:21 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:53:52 0 [Warning] InnoDB: A long semaphore wait: 2019-02-04 17:54:23 0 [Warning] InnoDB: A long semaphore wait: ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.2.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_large_fragment-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_large_fragment-innodb/' galera_sr.mysql-wsrep-features#32 'innodb' [ pass ] 6162 galera_sr.galera_sr_gtid 'innodb' [ pass ] 4049 galera_sr.galera_sr_log_bin 'innodb' [ pass ] 8105 galera_sr.mysql-wsrep-features#136 'innodb' [ pass ] 3910 galera_sr.galera_sr_v1_row_events 'innodb' [ pass ] 3800 galera_sr.GCF-1018B 'innodb' [ pass ] 485914 galera_sr.GCF-437 'innodb' [ fail ] Test ended at 2019-02-04 18:10:55 CURRENT_TEST: galera_sr.GCF-437 mysqltest: At line 13: query 'INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5' succeeded - should have failed with errno 1534... The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=MyISAM; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 2 * 1024 * 1024; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-437-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-437-innodb/' galera_sr.galera_sr_kill_all_nobootstrap 'innodb' [ pass ] 24327 galera_sr.galera_sr_kill_all_pcrecovery 'innodb' [ pass ] 25663 galera_sr.galera_sr_load_data_splitting 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 18:28:02 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr_load_data_splitting.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 512; SET GLOBAL wsrep_load_data_splitting = TRUE; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; connection node_2; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) - found 'core.18935' (0/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.1/data/core.18935 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 18935] [New LWP 18936] [New LWP 18941] [New LWP 18937] [New LWP 18938] [New LWP 18939] [New LWP 18940] [New LWP 18944] [New LWP 18942] [New LWP 18947] [New LWP 18943] [New LWP 18946] [New LWP 18953] [New LWP 18949] [New LWP 18956] [New LWP 18957] [New LWP 18954] [New LWP 18963] [New LWP 18967] [New LWP 18960] [New LWP 18959] [New LWP 18964] [New LWP 18962] [New LWP 18966] [New LWP 18968] [New LWP 18965] [New LWP 18969] [New LWP 18955] [New LWP 18961] [New LWP 18948] [New LWP 18958] [New LWP 18945] [New LWP 18950] [New LWP 19182] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007ff3bb7709d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007ff3bb7709d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005616c448b37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005616c3cc7d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff3b9b1f20d in poll () from /lib64/libc.so.6 #5 0x00005616c386996a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005616c3868c05 in mysqld_main (argc=161, argv=0x5616c65a9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005616c385cda5 in main (argc=23, argv=0x7ffd732f6be8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7ff3a8ded700 (LWP 19182)): #0 0x00007ff3bb772d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005616c3fc6255 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005616c3fc65ab in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005616c420f751 in pfs_os_file_flush_func (file=..., src_file=0x5616c488b8b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005616c4212593 in fil_flush_low (space=0x5616c6a82100, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x00005616c421e113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x00005616c3fa1c59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005616c3fa26bb in log_write_up_to (lsn=11788409, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005616c410d551 in trx_flush_log_if_needed_low (lsn=11788409) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005616c410d59a in trx_flush_log_if_needed (lsn=11788409, trx=0x7ff3a9922390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005616c410f3c3 in trx_commit_complete_for_mysql (trx=0x7ff3a9922390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005616c3ef3a6f in innobase_commit (hton=0x5616c684c910, thd=0x7ff33c0c90c0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #12 0x00005616c3ccd89b in commit_one_phase_2 (thd=0x7ff33c0c90c0, all=true, trans=0x7ff33c0cc608, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x00005616c3ccd79d in ha_commit_one_phase (thd=0x7ff33c0c90c0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x00005616c3cccb19 in ha_commit_trans (thd=0x7ff33c0c90c0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x00005616c3b12a76 in trans_commit (thd=0x7ff33c0c90c0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005616c3ed1988 in Wsrep_storage_service::commit (this=0x7ff33c079270, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x00005616c453a85f in wsrep::transaction::certify_fragment (this=0x7ff33c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #18 0x00005616c4539fc3 in wsrep::transaction::streaming_step (this=0x7ff33c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #19 0x00005616c453737e in wsrep::transaction::after_row (this=0x7ff33c0069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x00005616c3cdfba4 in wsrep::client_state::after_row (this=0x7ff33c006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x00005616c3cc8e87 in wsrep_after_row (thd=0x7ff33c000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x00005616c3cdad26 in wsrep_after_row (thd=0x7ff33c000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #23 0x00005616c3cdb117 in handler::ha_write_row (this=0x7ff33c026c58, buf=0x7ff33c024ed0 "\377v6") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #24 0x00005616c395118d in write_record (thd=0x7ff33c000b00, table=0x7ff33c026020, info=0x7ff3a8deaaf0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x00005616c398414f in read_sep_field (thd=0x7ff33c000b00, info=..., table_list=0x7ff33c014090, fields_vars=..., set_fields=..., set_values=..., read_info=..., enclosed=..., skip_lines=0, ignore_check_option_errors=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:1152 #26 0x00005616c398270a in mysql_load (thd=0x7ff33c000b00, ex=0x7ff33c014008, table_list=0x7ff33c014090, fields_vars=..., set_fields=..., set_values=..., handle_duplicates=DUP_ERROR, ignore=false, read_file_from_client=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:660 #27 0x00005616c399949d in mysql_execute_command (thd=0x7ff33c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5145 #28 0x00005616c39a3675 in mysql_parse (thd=0x7ff33c000b00, rawbuf=0x7ff33c013e68 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=111, parser_state=0x7ff3a8dec0b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #29 0x00005616c39a2ca8 in wsrep_mysql_parse (thd=0x7ff33c000b00, rawbuf=0x7ff33c013e68 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=111, parser_state=0x7ff3a8dec0b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #30 0x00005616c398eb65 in dispatch_command (command=COM_QUERY, thd=0x7ff33c000b00, packet=0x7ff33c00a451 "", packet_length=112, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #31 0x00005616c398d599 in do_command (thd=0x7ff33c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #32 0x00005616c3afdc50 in do_handle_one_connection (connect=0x5616c6790d00) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #33 0x00005616c3afd994 in handle_one_connection (arg=0x5616c6790d00) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #34 0x00005616c4423471 in pfs_spawn_thread (arg=0x5616c66e49a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #35 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #36 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7ff3897fa700 (LWP 18950)): #0 0x00007ff3bb772d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005616c3fc6255 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005616c3fc65ab in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005616c420f751 in pfs_os_file_flush_func (file=..., src_file=0x5616c488b8b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005616c4212593 in fil_flush_low (space=0x5616c6a81d60, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x00005616c421e113 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x00005616c419ab29 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1073 #7 0x00005616c41a52df in buf_flush_end (buf_pool=0x5616c695dfb0, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1879 #8 0x00005616c41a549d in buf_flush_do_batch (buf_pool=0x5616c695dfb0, type=BUF_FLUSH_LIST, min_n=7, lsn_limit=18446744073709551615, n=0x7ff3897f9b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1948 #9 0x00005616c41a8187 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2749 #10 0x00005616c41a90d5 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3158 #11 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff38bfff700 (LWP 18945)): #0 0x00007ff3bb33c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005616c3fc4ba6 in LinuxAIOHandler::collect (this=0x7ff38bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005616c3fc50cf in LinuxAIOHandler::poll (this=0x7ff38bffeb00, m1=0x7ff38bffeba8, m2=0x7ff38bffebb0, request=0x7ff38bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005616c3fc5232 in os_aio_linux_handler (global_segment=1, m1=0x7ff38bffeba8, m2=0x7ff38bffebb0, request=0x7ff38bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005616c3fc9d56 in os_aio_handler (segment=1, m1=0x7ff38bffeba8, m2=0x7ff38bffebb0, request=0x7ff38bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005616c421dbc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005616c40c312f in io_handler_thread (arg=0x5616c524c428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff3797fa700 (LWP 18958)): #0 0x00007ff3bb772e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005616c3fd1b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005616c40c0b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005616c40c0c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff38a7fc700 (LWP 18948)): #0 0x00007ff3bb33c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005616c3fc4ba6 in LinuxAIOHandler::collect (this=0x7ff38a7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005616c3fc50cf in LinuxAIOHandler::poll (this=0x7ff38a7fbb00, m1=0x7ff38a7fbba8, m2=0x7ff38a7fbbb0, request=0x7ff38a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005616c3fc5232 in os_aio_linux_handler (global_segment=4, m1=0x7ff38a7fbba8, m2=0x7ff38a7fbbb0, request=0x7ff38a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005616c3fc9d56 in os_aio_handler (segment=4, m1=0x7ff38a7fbba8, m2=0x7ff38a7fbbb0, request=0x7ff38a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005616c421dbc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005616c40c312f in io_handler_thread (arg=0x5616c524c440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff3677fe700 (LWP 18961)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd1889 in os_event::wait (this=0x5616c697fe40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005616c3fd130f in os_event::wait_low (this=0x5616c697fe40, reset_sig_count=561) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005616c3fd1680 in os_event_wait_low (event=0x5616c697fe40, reset_sig_count=561) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005616c40cb42e in sync_array_wait_event (arr=0x5616c684f7b0, cell=@0x7ff3677fbb48: 0x5616c684f920) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005616c40ce9fa in rw_lock_x_lock_func (lock=0x7ff3a90e8d18, pass=0, file_name=0x5616c483fdf8 "/home/stepan/galera/git/10.4/server/storage/innobase/include/mtr0mtr.ic", line=147) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:735 #6 0x00005616c4140516 in pfs_rw_lock_x_lock_func (lock=0x7ff3a90e8d18, pass=0, file_name=0x5616c483fdf8 "/home/stepan/galera/git/10.4/server/storage/innobase/include/mtr0mtr.ic", line=147) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:558 #7 0x00005616c4160b41 in mtr_t::x_latch_at_savepoint (this=0x7ff3677fd1a0, savepoint=16, block=0x7ff3a90e8c48) at /home/stepan/galera/git/10.4/server/storage/innobase/include/mtr0mtr.ic:147 #8 0x00005616c414d738 in btr_cur_search_to_nth_level_func (index=0x7ff358013f48, level=0, tuple=0x7ff35c00c648, mode=PAGE_CUR_LE, latch_mode=33, cursor=0x5616c6a9c368, ahi_latch=0x0, file=0x5616c47f25e0 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7ff3677fd1a0, autoinc=0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:2191 #9 0x00005616c4163d22 in btr_pcur_open_with_no_init_func (index=0x7ff358013f48, tuple=0x7ff35c00c648, mode=PAGE_CUR_LE, latch_mode=65569, cursor=0x5616c6a9c368, ahi_latch=0x0, file=0x5616c47f25e0 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7ff3677fd1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/btr0pcur.ic:527 #10 0x00005616c416502b in btr_pcur_restore_position_func (latch_mode=65569, cursor=0x5616c6a9c368, file=0x5616c47f25e0 "/home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc", line=100, mtr=0x7ff3677fd1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0pcur.cc:391 #11 0x00005616c407e3cb in row_purge_reposition_pcur (mode=65569, node=0x5616c6a9c2d0, mtr=0x7ff3677fd1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:100 #12 0x00005616c407e598 in row_purge_remove_clust_if_poss_low (node=0x5616c6a9c2d0, mode=65569) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:140 #13 0x00005616c407e9ee in row_purge_remove_clust_if_poss (node=0x5616c6a9c2d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:226 #14 0x00005616c408021b in row_purge_del_mark (node=0x5616c6a9c2d0) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #15 0x00005616c40817a5 in row_purge_record_func (node=0x5616c6a9c2d0, undo_rec=0x5616c6a9c7a8 "", thr=0x5616c6a9c218, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #16 0x00005616c4081ae2 in row_purge (node=0x5616c6a9c2d0, undo_rec=0x5616c6a9c7a8 "", thr=0x5616c6a9c218) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #17 0x00005616c4081e8d in row_purge_step (thr=0x5616c6a9c218) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #18 0x00005616c40041a8 in que_thr_step (thr=0x5616c6a9c218) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #19 0x00005616c40043d8 in que_run_threads_low (thr=0x5616c6a9c218) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #20 0x00005616c40045c8 in que_run_threads (thr=0x5616c6a9c218) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #21 0x00005616c40c1061 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2439 #22 0x00005616c40c1208 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2487 #23 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #24 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff37affd700 (LWP 18955)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd11a4 in os_event::timed_wait (this=0x5616c6878730, abstime=0x7ff37affcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005616c3fd14bb in os_event::wait_time_low (this=0x5616c6878730, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005616c3fd165b in os_event_wait_time_low (event=0x5616c6878730, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005616c40bf2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff3a8e38700 (LWP 18969)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c4490cab in safe_cond_timedwait (cond=0x5616c51d0fa0 , mp=0x5616c51d39a0 , abstime=0x7ff3a8e37c60, file=0x5616c454b348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005616c385d725 in inline_mysql_cond_timedwait (that=0x5616c51d0fa0 , mutex=0x5616c51d39a0 , abstime=0x7ff3a8e37c60, src_file=0x5616c454d3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005616c38624fb in cache_thread (thd=0x7ff344000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005616c38627a9 in one_thread_per_connection_end (thd=0x7ff344000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005616c3afdcda in do_handle_one_connection (connect=0x5616c6beafb0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x00005616c3afd994 in handle_one_connection (arg=0x5616c6beafb0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x00005616c4423471 in pfs_spawn_thread (arg=0x5616c6c01ff0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff3657fa700 (LWP 18965)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd1889 in os_event::wait (this=0x5616c68788b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005616c3fd130f in os_event::wait_low (this=0x5616c68788b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005616c3fd1680 in os_event_wait_low (event=0x5616c68788b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005616c41868b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff3a8e83700 (LWP 18968)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c44909a9 in safe_cond_wait (cond=0x5616c51d0fe0 , mp=0x5616c51d4360 , file=0x5616c455ab48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005616c388e3ce in inline_mysql_cond_wait (that=0x5616c51d0fe0 , mutex=0x5616c51d4360 , src_file=0x5616c455b6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005616c388fc60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005616c4423471 in pfs_spawn_thread (arg=0x5616c683c240) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff364ff9700 (LWP 18966)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c44909a9 in safe_cond_wait (cond=0x7ff364ff8c70, mp=0x7ff364ff8cb0, file=0x5616c475d860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005616c3ee8a13 in inline_mysql_cond_wait (that=0x7ff364ff8c70, mutex=0x7ff364ff8cb0, src_file=0x5616c475f3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005616c3eed632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x00005616c4423471 in pfs_spawn_thread (arg=0x5616c6a1e130) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff366ffd700 (LWP 18962)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd1889 in os_event::wait (this=0x5616c6878630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005616c3fd130f in os_event::wait_low (this=0x5616c6878630, reset_sig_count=14955) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005616c3fd1680 in os_event_wait_low (event=0x5616c6878630, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005616c40bd5ff in srv_resume_thread (slot=0x5616c500e5f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005616c40c1203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff365ffb700 (LWP 18964)): #0 0x00007ff3bb772e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005616c3fd1b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005616c4178755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff378ff9700 (LWP 18959)): #0 0x00007ff3b9b0ed47 in sched_yield () from /lib64/libc.so.6 #1 0x00005616c3fd1b1d in os_thread_yield () at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:206 #2 0x00005616c40eb3d9 in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1274 #3 0x00005616c40eb5e1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #4 0x00005616c40c14d6 in srv_do_purge (n_total_purged=0x7ff378ff8d98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2579 #5 0x00005616c40c1903 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2702 #6 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff367fff700 (LWP 18960)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd1889 in os_event::wait (this=0x5616c6878530) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005616c3fd130f in os_event::wait_low (this=0x5616c6878530, reset_sig_count=22441) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005616c3fd1680 in os_event_wait_low (event=0x5616c6878530, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005616c40bd5ff in srv_resume_thread (slot=0x5616c500e580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005616c40c1203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff3a8ece700 (LWP 18967)): #0 0x00007ff3b9a6340a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005616c385ce24 in my_sigwait (set=0x7ff3a8ecdcf0, sig=0x7ff3a8ecdcb4, code=0x7ff3a8ecdcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005616c3863464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005616c4423471 in pfs_spawn_thread (arg=0x5616c683bd50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff3667fc700 (LWP 18963)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd1889 in os_event::wait (this=0x5616c68787b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005616c3fd130f in os_event::wait_low (this=0x5616c68787b0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005616c3fd1680 in os_event_wait_low (event=0x5616c68787b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005616c419ddde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff37b7fe700 (LWP 18954)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd11a4 in os_event::timed_wait (this=0x5616c68786b0, abstime=0x7ff37b7fdba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005616c3fd14bb in os_event::wait_time_low (this=0x5616c68786b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005616c3fd165b in os_event_wait_time_low (event=0x5616c68786b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005616c40bf808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff379ffb700 (LWP 18957)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd11a4 in os_event::timed_wait (this=0x5616c6ac9890, abstime=0x7ff379ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005616c3fd14bb in os_event::wait_time_low (this=0x5616c6ac9890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005616c3fd165b in os_event_wait_time_low (event=0x5616c6ac9890, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005616c411faad in ib_wqueue_timedwait (wq=0x5616c6a336e0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005616c4261f11 in fts_optimize_thread (arg=0x5616c6a336e0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff37a7fc700 (LWP 18956)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd11a4 in os_event::timed_wait (this=0x5616c6a827e0, abstime=0x7ff37a7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005616c3fd14bb in os_event::wait_time_low (this=0x5616c6a827e0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005616c3fd165b in os_event_wait_time_low (event=0x5616c6a827e0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005616c420752c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff389ffb700 (LWP 18949)): #0 0x00007ff3bb33c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005616c3fc4ba6 in LinuxAIOHandler::collect (this=0x7ff389ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005616c3fc50cf in LinuxAIOHandler::poll (this=0x7ff389ffab00, m1=0x7ff389ffaba8, m2=0x7ff389ffabb0, request=0x7ff389ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005616c3fc5232 in os_aio_linux_handler (global_segment=5, m1=0x7ff389ffaba8, m2=0x7ff389ffabb0, request=0x7ff389ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005616c3fc9d56 in os_aio_handler (segment=5, m1=0x7ff389ffaba8, m2=0x7ff389ffabb0, request=0x7ff389ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005616c421dbc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005616c40c312f in io_handler_thread (arg=0x5616c524c448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff37bfff700 (LWP 18953)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c3fd11a4 in os_event::timed_wait (this=0x5616c6a507c0, abstime=0x7ff37bffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005616c3fd14bb in os_event::wait_time_low (this=0x5616c6a507c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005616c3fd165b in os_event_wait_time_low (event=0x5616c6a507c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005616c3f9e314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff38b7fe700 (LWP 18946)): #0 0x00007ff3bb33c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005616c3fc4ba6 in LinuxAIOHandler::collect (this=0x7ff38b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005616c3fc50cf in LinuxAIOHandler::poll (this=0x7ff38b7fdb00, m1=0x7ff38b7fdba8, m2=0x7ff38b7fdbb0, request=0x7ff38b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005616c3fc5232 in os_aio_linux_handler (global_segment=2, m1=0x7ff38b7fdba8, m2=0x7ff38b7fdbb0, request=0x7ff38b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005616c3fc9d56 in os_aio_handler (segment=2, m1=0x7ff38b7fdba8, m2=0x7ff38b7fdbb0, request=0x7ff38b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005616c421dbc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005616c40c312f in io_handler_thread (arg=0x5616c524c430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff3aa520700 (LWP 18943)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c4490cab in safe_cond_timedwait (cond=0x5616c5a5dca0 , mp=0x5616c5a5dbe0 , abstime=0x7ff3aa51fcb0, file=0x5616c48d6130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005616c42faadd in inline_mysql_cond_timedwait (that=0x5616c5a5dca0 , mutex=0x5616c5a5dbe0 , abstime=0x7ff3aa51fcb0, src_file=0x5616c48d6178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005616c42fb001 in my_service_thread_sleep (control=0x5616c5012900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005616c42ef49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005616c4423471 in pfs_spawn_thread (arg=0x5616c68307a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff38affd700 (LWP 18947)): #0 0x00007ff3bb33c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005616c3fc4ba6 in LinuxAIOHandler::collect (this=0x7ff38affcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005616c3fc50cf in LinuxAIOHandler::poll (this=0x7ff38affcb00, m1=0x7ff38affcba8, m2=0x7ff38affcbb0, request=0x7ff38affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005616c3fc5232 in os_aio_linux_handler (global_segment=3, m1=0x7ff38affcba8, m2=0x7ff38affcbb0, request=0x7ff38affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005616c3fc9d56 in os_aio_handler (segment=3, m1=0x7ff38affcba8, m2=0x7ff38affcbb0, request=0x7ff38affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005616c421dbc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005616c40c312f in io_handler_thread (arg=0x5616c524c438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff3b00fd700 (LWP 18942)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff3b356e132 in fifo_lock_get (q=q@entry=0x5616c6756b20) at galerautils/src/gu_fifo.c:262 #2 0x00007ff3b356e7a1 in gu_fifo_get_head (q=0x5616c6756b20, err=0x7ff3b00fc06c) at galerautils/src/gu_fifo.c:330 #3 0x00007ff3b36500cc in gcs_recv (conn=0x5616c67304a0, action=0x7ff3b00fc0b0) at gcs/src/gcs.cpp:2024 #4 0x00007ff3b367d16c in galera::GcsActionSource::process (this=0x5616c672fe80, recv_ctx=0x7ff3b00fc4d0, exit_loop=@0x7ff3b00fc29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007ff3b369c033 in galera::ReplicatorSMM::async_recv (this=0x5616c67281b0, recv_ctx=0x7ff3b00fc4d0) at galera/src/replicator_smm.cpp:383 #6 0x00005616c454420e in wsrep::wsrep_provider_v26::run_applier (this=0x5616c66af280, applier_ctx=0x7ff3b00fc4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005616c3be72b4 in wsrep_replication_process (thd=0x7ff398000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005616c3bd8f01 in start_wsrep_THD (arg=0x5616c678a580) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff3a8cce700 (LWP 18944)): #0 0x00007ff3bb33c644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005616c3fc4ba6 in LinuxAIOHandler::collect (this=0x7ff3a8ccdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005616c3fc50cf in LinuxAIOHandler::poll (this=0x7ff3a8ccdb00, m1=0x7ff3a8ccdba8, m2=0x7ff3a8ccdbb0, request=0x7ff3a8ccdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005616c3fc5232 in os_aio_linux_handler (global_segment=0, m1=0x7ff3a8ccdba8, m2=0x7ff3a8ccdbb0, request=0x7ff3a8ccdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005616c3fc9d56 in os_aio_handler (segment=0, m1=0x7ff3a8ccdba8, m2=0x7ff3a8ccdbb0, request=0x7ff3a8ccdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005616c421dbc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005616c40c312f in io_handler_thread (arg=0x5616c524c420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff3b0193700 (LWP 18940)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c44909a9 in safe_cond_wait (cond=0x7ff3a0017b88, mp=0x7ff3a0017ad8, file=0x5616c45f9a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005616c3be6d06 in inline_mysql_cond_wait (that=0x7ff3a0017b88, mutex=0x7ff3a0017ad8, src_file=0x5616c45f9d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005616c3be999e in Wsrep_thd_queue::pop_front (this=0x7ff3a0017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005616c3be83be in wsrep_rollback_process (rollbacker=0x7ff3a0001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005616c3bd8f01 in start_wsrep_THD (arg=0x5616c67803f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff3ab7fe700 (LWP 18939)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff3b3654867 in gu::Lock::wait (cond=..., this=0x7ff3ab7fd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff3b36573ab in RecvBuf::front (this=this@entry=0x5616c67707f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007ff3b3654f46 in gcomm_recv (backend=, msg=0x5616c6730780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007ff3b3648eec in core_msg_recv (backend=backend@entry=0x5616c67308c8, recv_msg=recv_msg@entry=0x5616c6730780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007ff3b364c164 in gcs_core_recv (conn=0x5616c6730720, recv_act=recv_act@entry=0x7ff3ab7fdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007ff3b36533d8 in gcs_recv_thread (arg=0x5616c67304a0) at gcs/src/gcs.cpp:1398 #7 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff3abfff700 (LWP 18938)): #0 0x00007ff3b9b2a483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff3b361b6f8 in asio::detail::epoll_reactor::run (this=0x5616c6770f10, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007ff3b361fdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5616c6770e30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007ff3b36200a2 in asio::detail::task_io_service::run (this=0x5616c6770e30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007ff3b3637afd in asio::io_service::run (this=0x5616c676ff30) at asio/asio/impl/io_service.ipp:58 #5 0x00007ff3b3638251 in gcomm::AsioProtonet::event_loop (this=0x5616c676fe90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007ff3b36562ed in GCommConn::run (this=0x5616c6770610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007ff3b3659399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff3b2617700 (LWP 18937)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff3b3679fd7 in gu::Lock::wait (cond=..., this=0x7ff3b2616bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007ff3b367a069 in galera::ServiceThd::thd_func (arg=0x5616c67288c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff3b0148700 (LWP 18941)): #0 0x00007ff3bb76f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c44909a9 in safe_cond_wait (cond=0x7ff394015628, mp=0x7ff394015578, file=0x5616c45f9a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005616c3be6d06 in inline_mysql_cond_wait (that=0x7ff394015628, mutex=0x7ff394015578, src_file=0x5616c45f9d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005616c3be999e in Wsrep_thd_queue::pop_front (this=0x7ff394015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005616c3be877e in wsrep_post_rollback_process (post_rollbacker=0x7ff394000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005616c3bd8f01 in start_wsrep_THD (arg=0x5616c676a760) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff3b41c4700 (LWP 18936)): #0 0x00007ff3bb76fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005616c4490cab in safe_cond_timedwait (cond=0x5616c5a70440 , mp=0x5616c5a70380 , abstime=0x7ff3b41c3d70, file=0x5616c491a360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005616c4491c65 in inline_mysql_cond_timedwait (that=0x5616c5a70440 , mutex=0x5616c5a70380 , abstime=0x7ff3b41c3d70, src_file=0x5616c491a3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005616c44927c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005616c4423471 in pfs_spawn_thread (arg=0x5616c66f16f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007ff3bb76bdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff3b9b29ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff3bbb7c880 (LWP 18935)): #0 0x00007ff3bb7709d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005616c448b37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005616c3cc7d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007ff3b9b1f20d in poll () from /lib64/libc.so.6 #5 0x00005616c386996a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005616c3868c05 in mysqld_main (argc=161, argv=0x5616c65a9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005616c385cda5 in main (argc=23, argv=0x7ffd732f6be8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.1/data/core.18935 - found 'core.18972' (1/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.2/data/core.18972 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 18972] [New LWP 19149] [New LWP 19151] [New LWP 18976] [New LWP 19152] [New LWP 18978] [New LWP 19153] [New LWP 18977] [New LWP 19138] [New LWP 19162] [New LWP 19158] [New LWP 19155] [New LWP 18979] [New LWP 18974] [New LWP 19141] [New LWP 19139] [New LWP 19142] [New LWP 19140] [New LWP 19144] [New LWP 19148] [New LWP 19143] [New LWP 19150] [New LWP 19147] [New LWP 19156] [New LWP 19154] [New LWP 19157] [New LWP 19159] [New LWP 19163] [New LWP 19160] [New LWP 18975] [New LWP 19161] [New LWP 19137] [New LWP 18973] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fa9e5c519d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fa9e5c519d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005649717b037a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000564970fecd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa9e400020d in poll () from /lib64/libc.so.6 #5 0x0000564970b8e96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000564970b8dc05 in mysqld_main (argc=162, argv=0x56497460fd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000564970b81da5 in main (argc=23, argv=0x7ffd5fd5a928) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fa9de6a5700 (LWP 18973)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649717b5cab in safe_cond_timedwait (cond=0x564972d95440 , mp=0x564972d95380 , abstime=0x7fa9de6a4d70, file=0x564971c3f360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005649717b6c65 in inline_mysql_cond_timedwait (that=0x564972d95440 , mutex=0x564972d95380 , abstime=0x7fa9de6a4d70, src_file=0x564971c3f3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005649717b77c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000564971748471 in pfs_spawn_thread (arg=0x564974757700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fa9bffff700 (LWP 19137)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649717b5cab in safe_cond_timedwait (cond=0x564972d82ca0 , mp=0x564972d82be0 , abstime=0x7fa9bfffecb0, file=0x564971bfb130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056497161fadd in inline_mysql_cond_timedwait (that=0x564972d82ca0 , mutex=0x564972d82be0 , abstime=0x7fa9bfffecb0, src_file=0x564971bfb178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564971620001 in my_service_thread_sleep (control=0x564972337900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056497161449a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000564971748471 in pfs_spawn_thread (arg=0x56497489f6c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fa9dc139700 (LWP 19161)): #0 0x00007fa9e3f4440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000564970b81e24 in my_sigwait (set=0x7fa9dc138cf0, sig=0x7fa9dc138cb4, code=0x7fa9dc138cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000564970b88464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000564971748471 in pfs_spawn_thread (arg=0x5649748a9c90) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fa9cec7c700 (LWP 18975)): #0 0x00007fa9e400b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa9ddafc6f8 in asio::detail::epoll_reactor::run (this=0x5649747d6f50, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fa9ddb00df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5649747d6390, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fa9ddb010a2 in asio::detail::task_io_service::run (this=0x5649747d6390, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fa9ddb18afd in asio::io_service::run (this=0x5649747d3f30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fa9ddb19251 in gcomm::AsioProtonet::event_loop (this=0x5649747d3e90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fa9ddb372ed in GCommConn::run (this=0x5649747d5c80) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fa9ddb3a399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fa9777fe700 (LWP 19160)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649717b59a9 in safe_cond_wait (cond=0x7fa9777fdc70, mp=0x7fa9777fdcb0, file=0x564971a82860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056497120da13 in inline_mysql_cond_wait (that=0x7fa9777fdc70, mutex=0x7fa9777fdcb0, src_file=0x564971a843a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564971212632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x0000564971748471 in pfs_spawn_thread (arg=0x564974b29e70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fa9dc0a3700 (LWP 19163)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649717b5cab in safe_cond_timedwait (cond=0x5649724f5fa0 , mp=0x5649724f89a0 , abstime=0x7fa9dc0a2c60, file=0x564971870348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000564970b82725 in inline_mysql_cond_timedwait (that=0x5649724f5fa0 , mutex=0x5649724f89a0 , abstime=0x7fa9dc0a2c60, src_file=0x5649718723b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000564970b874fb in cache_thread (thd=0x7fa960000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000564970b877a9 in one_thread_per_connection_end (thd=0x7fa960000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000564970e22cda in do_handle_one_connection (connect=0x564974c57be0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000564970e22994 in handle_one_connection (arg=0x564974c57be0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x0000564971748471 in pfs_spawn_thread (arg=0x564974c6ec70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fa977fff700 (LWP 19159)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f6889 in os_event::wait (this=0x5649748e74c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649712f630f in os_event::wait_low (this=0x5649748e74c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649712f6680 in os_event_wait_low (event=0x5649748e74c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649714ab8b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fa9917fa700 (LWP 19157)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f6889 in os_event::wait (this=0x5649748e73c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649712f630f in os_event::wait_low (this=0x5649748e73c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649712f6680 in os_event_wait_low (event=0x5649748e73c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649714c2dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fa992ffd700 (LWP 19154)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f6889 in os_event::wait (this=0x5649748e7240) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649712f630f in os_event::wait_low (this=0x5649748e7240, reset_sig_count=15317) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649712f6680 in os_event_wait_low (event=0x5649748e7240, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649713e25ff in srv_resume_thread (slot=0x5649723335f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005649713e6203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fa991ffb700 (LWP 19156)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f6889 in os_event::wait (this=0x5649748e71c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649712f630f in os_event::wait_low (this=0x5649748e71c0, reset_sig_count=18399) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649712f6680 in os_event_wait_low (event=0x5649748e71c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649713e25ff in srv_resume_thread (slot=0x5649723335b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005649713e6203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fa9a65f7700 (LWP 19147)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f61a4 in os_event::timed_wait (this=0x564974abf3d0, abstime=0x7fa9a65f6d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649712f64bb in os_event::wait_time_low (this=0x564974abf3d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649712f665b in os_event_wait_time_low (event=0x564974abf3d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649712c3314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fa9a55f5700 (LWP 19150)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f61a4 in os_event::timed_wait (this=0x564974af13f0, abstime=0x7fa9a55f4d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649712f64bb in os_event::wait_time_low (this=0x564974af13f0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649712f665b in os_event_wait_time_low (event=0x564974af13f0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056497152c52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fa9a7fff700 (LWP 19143)): #0 0x00007fa9e581d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649712e9ba6 in LinuxAIOHandler::collect (this=0x7fa9a7ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649712ea0cf in LinuxAIOHandler::poll (this=0x7fa9a7ffeb00, m1=0x7fa9a7ffeba8, m2=0x7fa9a7ffebb0, request=0x7fa9a7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649712ea232 in os_aio_linux_handler (global_segment=5, m1=0x7fa9a7ffeba8, m2=0x7fa9a7ffebb0, request=0x7fa9a7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649712eed56 in os_aio_handler (segment=5, m1=0x7fa9a7ffeba8, m2=0x7fa9a7ffebb0, request=0x7fa9a7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000564971542bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005649713e812f in io_handler_thread (arg=0x564972571448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fa9a6df8700 (LWP 19148)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f61a4 in os_event::timed_wait (this=0x5649748e72c0, abstime=0x7fa9a6df7ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649712f64bb in os_event::wait_time_low (this=0x5649748e72c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649712f665b in os_event_wait_time_low (event=0x5649748e72c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649713e4808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fa9a77fe700 (LWP 19144)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f61a4 in os_event::timed_wait (this=0x5649748e7440, abstime=0x7fa9a77fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649712f64bb in os_event::wait_time_low (this=0x5649748e7440, time_in_usec=605000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649712f665b in os_event_wait_time_low (event=0x5649748e7440, time_in_usec=605000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649714ccc57 in pc_sleep_if_needed (next_loop_time=1549294027902, sig_count=2, cur_time=1549294027297) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x00005649714cdb90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fa9bd9ab700 (LWP 19140)): #0 0x00007fa9e581d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649712e9ba6 in LinuxAIOHandler::collect (this=0x7fa9bd9aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649712ea0cf in LinuxAIOHandler::poll (this=0x7fa9bd9aab00, m1=0x7fa9bd9aaba8, m2=0x7fa9bd9aabb0, request=0x7fa9bd9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649712ea232 in os_aio_linux_handler (global_segment=2, m1=0x7fa9bd9aaba8, m2=0x7fa9bd9aabb0, request=0x7fa9bd9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649712eed56 in os_aio_handler (segment=2, m1=0x7fa9bd9aaba8, m2=0x7fa9bd9aabb0, request=0x7fa9bd9aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000564971542bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005649713e812f in io_handler_thread (arg=0x564972571430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fa9bc9a9700 (LWP 19142)): #0 0x00007fa9e581d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649712e9ba6 in LinuxAIOHandler::collect (this=0x7fa9bc9a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649712ea0cf in LinuxAIOHandler::poll (this=0x7fa9bc9a8b00, m1=0x7fa9bc9a8ba8, m2=0x7fa9bc9a8bb0, request=0x7fa9bc9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649712ea232 in os_aio_linux_handler (global_segment=4, m1=0x7fa9bc9a8ba8, m2=0x7fa9bc9a8bb0, request=0x7fa9bc9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649712eed56 in os_aio_handler (segment=4, m1=0x7fa9bc9a8ba8, m2=0x7fa9bc9a8bb0, request=0x7fa9bc9a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000564971542bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005649713e812f in io_handler_thread (arg=0x564972571440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fa9be1ac700 (LWP 19139)): #0 0x00007fa9e581d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649712e9ba6 in LinuxAIOHandler::collect (this=0x7fa9be1abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649712ea0cf in LinuxAIOHandler::poll (this=0x7fa9be1abb00, m1=0x7fa9be1abba8, m2=0x7fa9be1abbb0, request=0x7fa9be1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649712ea232 in os_aio_linux_handler (global_segment=1, m1=0x7fa9be1abba8, m2=0x7fa9be1abbb0, request=0x7fa9be1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649712eed56 in os_aio_handler (segment=1, m1=0x7fa9be1abba8, m2=0x7fa9be1abbb0, request=0x7fa9be1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000564971542bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005649713e812f in io_handler_thread (arg=0x564972571428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fa9bd1aa700 (LWP 19141)): #0 0x00007fa9e581d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649712e9ba6 in LinuxAIOHandler::collect (this=0x7fa9bd1a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649712ea0cf in LinuxAIOHandler::poll (this=0x7fa9bd1a9b00, m1=0x7fa9bd1a9ba8, m2=0x7fa9bd1a9bb0, request=0x7fa9bd1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649712ea232 in os_aio_linux_handler (global_segment=3, m1=0x7fa9bd1a9ba8, m2=0x7fa9bd1a9bb0, request=0x7fa9bd1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649712eed56 in os_aio_handler (segment=3, m1=0x7fa9bd1a9ba8, m2=0x7fa9bd1a9bb0, request=0x7fa9bd1a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000564971542bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005649713e812f in io_handler_thread (arg=0x564972571438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fa9dd4f9700 (LWP 18974)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa9ddb5afd7 in gu::Lock::wait (cond=..., this=0x7fa9dd4f8bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fa9ddb5b069 in galera::ServiceThd::thd_func (arg=0x56497478e878) at galera/src/galera_service_thd.cpp:30 #3 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fa9dc361700 (LWP 18979)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa9dda4f132 in fifo_lock_get (q=q@entry=0x5649747c00c0) at galerautils/src/gu_fifo.c:262 #2 0x00007fa9dda4f7a1 in gu_fifo_get_head (q=0x5649747c00c0, err=0x7fa9dc36006c) at galerautils/src/gu_fifo.c:330 #3 0x00007fa9ddb310cc in gcs_recv (conn=0x564974799b20, action=0x7fa9dc3600b0) at gcs/src/gcs.cpp:2024 #4 0x00007fa9ddb5e16c in galera::GcsActionSource::process (this=0x564974799800, recv_ctx=0x7fa9dc3604d0, exit_loop=@0x7fa9dc36029e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fa9ddb7d033 in galera::ReplicatorSMM::async_recv (this=0x56497478e160, recv_ctx=0x7fa9dc3604d0) at galera/src/replicator_smm.cpp:383 #6 0x000056497186920e in wsrep::wsrep_provider_v26::run_applier (this=0x564974703d80, applier_ctx=0x7fa9dc3604d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x0000564970f0c2b4 in wsrep_replication_process (thd=0x7fa9b8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000564970efdf01 in start_wsrep_THD (arg=0x5649747eedc0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fa9927fc700 (LWP 19155)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f6889 in os_event::wait (this=0x5649748e7140) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649712f630f in os_event::wait_low (this=0x5649748e7140, reset_sig_count=22660) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649712f6680 in os_event_wait_low (event=0x5649748e7140, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649713e25ff in srv_resume_thread (slot=0x564972333580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005649713e6203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fa990ff9700 (LWP 19158)): #0 0x00007fa9e5c53e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005649712f6b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056497149d755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fa9dc0ee700 (LWP 19162)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649717b59a9 in safe_cond_wait (cond=0x5649724f5fe0 , mp=0x5649724f9360 , file=0x56497187fb48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564970bb33ce in inline_mysql_cond_wait (that=0x5649724f5fe0 , mutex=0x5649724f9360 , src_file=0x5649718806b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000564970bb4c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000564971748471 in pfs_spawn_thread (arg=0x564974bea1c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fa9be9ad700 (LWP 19138)): #0 0x00007fa9e581d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005649712e9ba6 in LinuxAIOHandler::collect (this=0x7fa9be9acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005649712ea0cf in LinuxAIOHandler::poll (this=0x7fa9be9acb00, m1=0x7fa9be9acba8, m2=0x7fa9be9acbb0, request=0x7fa9be9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005649712ea232 in os_aio_linux_handler (global_segment=0, m1=0x7fa9be9acba8, m2=0x7fa9be9acbb0, request=0x7fa9be9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005649712eed56 in os_aio_handler (segment=0, m1=0x7fa9be9acba8, m2=0x7fa9be9acbb0, request=0x7fa9be9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000564971542bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005649713e812f in io_handler_thread (arg=0x564972571420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fa9dc3f7700 (LWP 18977)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649717b59a9 in safe_cond_wait (cond=0x7fa9c4017b88, mp=0x7fa9c4017ad8, file=0x56497191ea00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564970f0bd06 in inline_mysql_cond_wait (that=0x7fa9c4017b88, mutex=0x7fa9c4017ad8, src_file=0x56497191ed20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564970f0e99e in Wsrep_thd_queue::pop_front (this=0x7fa9c4017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000564970f0d3be in wsrep_rollback_process (rollbacker=0x7fa9c4001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000564970efdf01 in start_wsrep_THD (arg=0x5649747f8160) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fa9937fe700 (LWP 19153)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f6889 in os_event::wait (this=0x5649748e70c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005649712f630f in os_event::wait_low (this=0x5649748e70c0, reset_sig_count=4620) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005649712f6680 in os_event_wait_low (event=0x5649748e70c0, reset_sig_count=4620) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005649713e25ff in srv_resume_thread (slot=0x564972333548 , sig_count=4620, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005649713e6610 in srv_purge_coordinator_suspend (slot=0x564972333548 , rseg_history_len=35) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x00005649713e68a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fa9dc3ac700 (LWP 18978)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649717b59a9 in safe_cond_wait (cond=0x7fa9b0015628, mp=0x7fa9b0015578, file=0x56497191ea00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000564970f0bd06 in inline_mysql_cond_wait (that=0x7fa9b0015628, mutex=0x7fa9b0015578, src_file=0x56497191ed20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000564970f0e99e in Wsrep_thd_queue::pop_front (this=0x7fa9b0015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000564970f0d77e in wsrep_post_rollback_process (post_rollbacker=0x7fa9b0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000564970efdf01 in start_wsrep_THD (arg=0x5649747f74c0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fa993fff700 (LWP 19152)): #0 0x00007fa9e5c53e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005649712f6b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005649713e5b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005649713e5c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fa9ce47b700 (LWP 18976)): #0 0x00007fa9e5c50965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa9ddb35867 in gu::Lock::wait (cond=..., this=0x7fa9ce47a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fa9ddb383ab in RecvBuf::front (this=this@entry=0x5649747d5e68, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fa9ddb35f46 in gcomm_recv (backend=, msg=0x564974799e00, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fa9ddb29eec in core_msg_recv (backend=backend@entry=0x564974799f48, recv_msg=recv_msg@entry=0x564974799e00, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fa9ddb2d164 in gcs_core_recv (conn=0x564974799da0, recv_act=recv_act@entry=0x7fa9ce47ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fa9ddb343d8 in gcs_recv_thread (arg=0x564974799b20) at gcs/src/gcs.cpp:1398 #7 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fa9a4df4700 (LWP 19151)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f61a4 in os_event::timed_wait (this=0x564974a8f650, abstime=0x7fa9a4df3b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649712f64bb in os_event::wait_time_low (this=0x564974a8f650, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649712f665b in os_event_wait_time_low (event=0x564974a8f650, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000564971444aad in ib_wqueue_timedwait (wq=0x564974a8f520, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000564971586f11 in fts_optimize_thread (arg=0x564974a8f520) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fa9a5df6700 (LWP 19149)): #0 0x00007fa9e5c50d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005649712f61a4 in os_event::timed_wait (this=0x5649748e7340, abstime=0x7fa9a5df5d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005649712f64bb in os_event::wait_time_low (this=0x5649748e7340, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005649712f665b in os_event_wait_time_low (event=0x5649748e7340, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005649713e42e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fa9e5c4cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa9e400aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fa9e605d880 (LWP 18972)): #0 0x00007fa9e5c519d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005649717b037a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000564970fecd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa9e400020d in poll () from /lib64/libc.so.6 #5 0x0000564970b8e96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000564970b8dc05 in mysqld_main (argc=162, argv=0x56497460fd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000564970b81da5 in main (argc=23, argv=0x7ffd5fd5a928) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 worker[1] Trying to dump core for [mysqltest - pid: 19648, winpid: 19648] worker[1] Trying to dump core for [mysqld.1 - pid: 19393, winpid: 19393] worker[1] Trying to dump core for [mysqld.2 - pid: 19440, winpid: 19440] Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/mysqld.2/data/core.18972 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_many_fragments 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 18:44:34 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr_many_fragments.log == connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) - found 'core.19394' (2/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.1/data/core.19394 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 19394] [New LWP 19408] [New LWP 19400] [New LWP 19409] [New LWP 19404] [New LWP 19421] [New LWP 19395] [New LWP 19416] [New LWP 19398] [New LWP 19422] [New LWP 19399] [New LWP 19423] [New LWP 19429] [New LWP 19397] [New LWP 19413] [New LWP 19396] [New LWP 19424] [New LWP 19414] [New LWP 19403] [New LWP 19415] [New LWP 19401] [New LWP 19417] [New LWP 19425] [New LWP 19402] [New LWP 19428] [New LWP 19420] [New LWP 19405] [New LWP 19419] [New LWP 19406] [New LWP 19407] [New LWP 19412] [New LWP 19411] [New LWP 19418] [New LWP 19650] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fe62d8d79d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fe62d8d79d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055ceefc7837a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055ceef4b4d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fe62bc8620d in poll () from /lib64/libc.so.6 #5 0x000055ceef05696a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055ceef055c05 in mysqld_main (argc=161, argv=0x55cef1696d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055ceef049da5 in main (argc=23, argv=0x7fff1af05b68) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fe61c389700 (LWP 19650)): #0 0x00007fe62d8d9d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055ceef7b3255 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055ceef7b35ab in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055ceef9fc751 in pfs_os_file_flush_func (file=..., src_file=0x55cef00788b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055ceef9ff593 in fil_flush_low (space=0x55cef1b6f030, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x000055ceefa0b113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x000055ceef78ec59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055ceef78f6bb in log_write_up_to (lsn=12883795, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055ceef8fa551 in trx_flush_log_if_needed_low (lsn=12883795) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000055ceef8fa59a in trx_flush_log_if_needed (lsn=12883795, trx=0x7fe607401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000055ceef8fc3c3 in trx_commit_complete_for_mysql (trx=0x7fe607401390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000055ceef6e0a6f in innobase_commit (hton=0x55cef19397b0, thd=0x7fe5b41a2cb0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #12 0x000055ceef4ba89b in commit_one_phase_2 (thd=0x7fe5b41a2cb0, all=true, trans=0x7fe5b41a61f8, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000055ceef4ba79d in ha_commit_one_phase (thd=0x7fe5b41a2cb0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000055ceef4b9b19 in ha_commit_trans (thd=0x7fe5b41a2cb0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000055ceef2ffa76 in trans_commit (thd=0x7fe5b41a2cb0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000055ceef6be988 in Wsrep_storage_service::commit (this=0x7fe5b418e050, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x000055ceefd2785f in wsrep::transaction::certify_fragment (this=0x7fe5b40069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #18 0x000055ceefd26fc3 in wsrep::transaction::streaming_step (this=0x7fe5b40069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #19 0x000055ceefd2437e in wsrep::transaction::after_row (this=0x7fe5b40069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x000055ceef4ccba4 in wsrep::client_state::after_row (this=0x7fe5b4006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x000055ceef4b5e87 in wsrep_after_row (thd=0x7fe5b4000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x000055ceef4c7d26 in wsrep_after_row (thd=0x7fe5b4000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #23 0x000055ceef4c8117 in handler::ha_write_row (this=0x7fe5b407d788, buf=0x7fe5b40774a8 "\376\r!") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #24 0x000055ceef13e18d in write_record (thd=0x7fe5b4000b00, table=0x7fe5b407cb50, info=0x7fe5b4017140) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x000055ceef14348c in select_insert::send_data (this=0x7fe5b40170f8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x000055ceef1fbb47 in end_send (join=0x7fe5b40171a8, join_tab=0x7fe5b4088bd8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x000055ceef338b89 in JOIN_CACHE::generate_full_extensions (this=0x7fe5b40891e8, rec_ptr=0x7fe5b410c1f8 "|") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x000055ceef3388a4 in JOIN_CACHE::join_matching_records (this=0x7fe5b40891e8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x000055ceef338270 in JOIN_CACHE::join_records (this=0x7fe5b40891e8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x000055ceef33840a in JOIN_CACHE::join_records (this=0x7fe5b40890b8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x000055ceef33840a in JOIN_CACHE::join_records (this=0x7fe5b4088f88, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x000055ceef1f87fb in sub_select_cache (join=0x7fe5b40171a8, join_tab=0x7fe5b40880c8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x000055ceef1f8a08 in sub_select (join=0x7fe5b40171a8, join_tab=0x7fe5b4087d18, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x000055ceef1f81d3 in do_select (join=0x7fe5b40171a8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x000055ceef1d0cf1 in JOIN::exec_inner (this=0x7fe5b40171a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x000055ceef1d013a in JOIN::exec (this=0x7fe5b40171a8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x000055ceef1d142c in mysql_select (thd=0x7fe5b4000b00, tables=0x7fe5b4014e08, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7fe5b40170f8, unit=0x7fe5b40049b8, select_lex=0x7fe5b4014738) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x000055ceef1c2d2d in handle_select (thd=0x7fe5b4000b00, lex=0x7fe5b40048f0, result=0x7fe5b40170f8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x000055ceef18513c in mysql_execute_command (thd=0x7fe5b4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4856 #40 0x000055ceef190675 in mysql_parse (thd=0x7fe5b4000b00, rawbuf=0x7fe5b4013e68 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fe61c3880b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #41 0x000055ceef18fca8 in wsrep_mysql_parse (thd=0x7fe5b4000b00, rawbuf=0x7fe5b4013e68 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fe61c3880b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #42 0x000055ceef17bb65 in dispatch_command (command=COM_QUERY, thd=0x7fe5b4000b00, packet=0x7fe5b400a451 "INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #43 0x000055ceef17a599 in do_command (thd=0x7fe5b4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x000055ceef2eac50 in do_handle_one_connection (connect=0x55cef1bac340) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #45 0x000055ceef2ea994 in handle_one_connection (arg=0x55cef1bac340) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #46 0x000055ceefc10471 in pfs_spawn_thread (arg=0x55cef1c671a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fe5eaffd700 (LWP 19418)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be889 in os_event::wait (this=0x55cef1965460) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055ceef7be30f in os_event::wait_low (this=0x55cef1965460, reset_sig_count=20323) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055ceef7be680 in os_event_wait_low (event=0x55cef1965460, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055ceef8aa5ff in srv_resume_thread (slot=0x55cef07fb580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055ceef8ae203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fe5fa7fc700 (LWP 19411)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be1a4 in os_event::timed_wait (this=0x55cef1b3d6f0, abstime=0x7fe5fa7fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055ceef7be4bb in os_event::wait_time_low (this=0x55cef1b3d6f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055ceef7be65b in os_event_wait_time_low (event=0x55cef1b3d6f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055ceef78b314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fe5f9ffb700 (LWP 19412)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be1a4 in os_event::timed_wait (this=0x55cef19655e0, abstime=0x7fe5f9ffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055ceef7be4bb in os_event::wait_time_low (this=0x55cef19655e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055ceef7be65b in os_event_wait_time_low (event=0x55cef19655e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055ceef8ac808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fe604ba9700 (LWP 19407)): #0 0x00007fe62d4a3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055ceef7b1ba6 in LinuxAIOHandler::collect (this=0x7fe604ba8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055ceef7b20cf in LinuxAIOHandler::poll (this=0x7fe604ba8b00, m1=0x7fe604ba8ba8, m2=0x7fe604ba8bb0, request=0x7fe604ba8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055ceef7b2232 in os_aio_linux_handler (global_segment=4, m1=0x7fe604ba8ba8, m2=0x7fe604ba8bb0, request=0x7fe604ba8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055ceef7b6d56 in os_aio_handler (segment=4, m1=0x7fe604ba8ba8, m2=0x7fe604ba8bb0, request=0x7fe604ba8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055ceefa0abc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055ceef8b012f in io_handler_thread (arg=0x55cef0a39440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fe6053aa700 (LWP 19406)): #0 0x00007fe62d4a3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055ceef7b1ba6 in LinuxAIOHandler::collect (this=0x7fe6053a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055ceef7b20cf in LinuxAIOHandler::poll (this=0x7fe6053a9b00, m1=0x7fe6053a9ba8, m2=0x7fe6053a9bb0, request=0x7fe6053a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055ceef7b2232 in os_aio_linux_handler (global_segment=3, m1=0x7fe6053a9ba8, m2=0x7fe6053a9bb0, request=0x7fe6053a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055ceef7b6d56 in os_aio_handler (segment=3, m1=0x7fe6053a9ba8, m2=0x7fe6053a9bb0, request=0x7fe6053a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055ceefa0abc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055ceef8b012f in io_handler_thread (arg=0x55cef0a39438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fe5ea7fc700 (LWP 19419)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be889 in os_event::wait (this=0x55cef1965560) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055ceef7be30f in os_event::wait_low (this=0x55cef1965560, reset_sig_count=13607) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055ceef7be680 in os_event_wait_low (event=0x55cef1965560, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055ceef8aa5ff in srv_resume_thread (slot=0x55cef07fb5f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055ceef8ae203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fe605bab700 (LWP 19405)): #0 0x00007fe62d4a3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055ceef7b1ba6 in LinuxAIOHandler::collect (this=0x7fe605baab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055ceef7b20cf in LinuxAIOHandler::poll (this=0x7fe605baab00, m1=0x7fe605baaba8, m2=0x7fe605baabb0, request=0x7fe605baabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055ceef7b2232 in os_aio_linux_handler (global_segment=2, m1=0x7fe605baaba8, m2=0x7fe605baabb0, request=0x7fe605baabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055ceef7b6d56 in os_aio_handler (segment=2, m1=0x7fe605baaba8, m2=0x7fe605baabb0, request=0x7fe605baabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055ceefa0abc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055ceef8b012f in io_handler_thread (arg=0x55cef0a39430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fe5e9ffb700 (LWP 19420)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be889 in os_event::wait (this=0x55cef19654e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055ceef7be30f in os_event::wait_low (this=0x55cef19654e0, reset_sig_count=17321) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055ceef7be680 in os_event_wait_low (event=0x55cef19654e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055ceef8aa5ff in srv_resume_thread (slot=0x55cef07fb5b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055ceef8ae203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fe62408e700 (LWP 19428)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceefc7d9a9 in safe_cond_wait (cond=0x55cef09bdfe0 , mp=0x55cef09c1360 , file=0x55ceefd47b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055ceef07b3ce in inline_mysql_cond_wait (that=0x55cef09bdfe0 , mutex=0x55cef09c1360 , src_file=0x55ceefd486b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055ceef07cc60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055ceefc10471 in pfs_spawn_thread (arg=0x55cef1c66d00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fe607fff700 (LWP 19402)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceefc7dcab in safe_cond_timedwait (cond=0x55cef124aca0 , mp=0x55cef124abe0 , abstime=0x7fe607ffecb0, file=0x55cef00c3130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055ceefae7add in inline_mysql_cond_timedwait (that=0x55cef124aca0 , mutex=0x55cef124abe0 , abstime=0x7fe607ffecb0, src_file=0x55cef00c3178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055ceefae8001 in my_service_thread_sleep (control=0x55cef07ff900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055ceefadc49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055ceefc10471 in pfs_spawn_thread (arg=0x55cef191d640) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fe6240d9700 (LWP 19425)): #0 0x00007fe62bbca40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055ceef049e24 in my_sigwait (set=0x7fe6240d8cf0, sig=0x7fe6240d8cb4, code=0x7fe6240d8cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055ceef050464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055ceefc10471 in pfs_spawn_thread (arg=0x55cef1928f10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fe5eb7fe700 (LWP 19417)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be889 in os_event::wait (this=0x55cef19653e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055ceef7be30f in os_event::wait_low (this=0x55cef19653e0, reset_sig_count=4166) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055ceef7be680 in os_event_wait_low (event=0x55cef19653e0, reset_sig_count=4166) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055ceef8aa5ff in srv_resume_thread (slot=0x55cef07fb548 , sig_count=4166, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055ceef8ae610 in srv_purge_coordinator_suspend (slot=0x55cef07fb548 , rseg_history_len=42) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x000055ceef8ae8a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fe6246e8700 (LWP 19401)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe6256d5132 in fifo_lock_get (q=q@entry=0x55cef1843b20) at galerautils/src/gu_fifo.c:262 #2 0x00007fe6256d57a1 in gu_fifo_get_head (q=0x55cef1843b20, err=0x7fe6246e706c) at galerautils/src/gu_fifo.c:330 #3 0x00007fe6257b70cc in gcs_recv (conn=0x55cef181d4a0, action=0x7fe6246e70b0) at gcs/src/gcs.cpp:2024 #4 0x00007fe6257e416c in galera::GcsActionSource::process (this=0x55cef181ce80, recv_ctx=0x7fe6246e74d0, exit_loop=@0x7fe6246e729e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fe625803033 in galera::ReplicatorSMM::async_recv (this=0x55cef18151b0, recv_ctx=0x7fe6246e74d0) at galera/src/replicator_smm.cpp:383 #6 0x000055ceefd3120e in wsrep::wsrep_provider_v26::run_applier (this=0x55cef179c280, applier_ctx=0x7fe6246e74d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x000055ceef3d42b4 in wsrep_replication_process (thd=0x7fe60c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055ceef3c5f01 in start_wsrep_THD (arg=0x55cef18774a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fe5f8ff9700 (LWP 19415)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be1a4 in os_event::timed_wait (this=0x55cef1b051f0, abstime=0x7fe5f8ff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055ceef7be4bb in os_event::wait_time_low (this=0x55cef1b051f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055ceef7be65b in os_event_wait_time_low (event=0x55cef1b051f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055ceef90caad in ib_wqueue_timedwait (wq=0x55cef1b05140, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055ceefa4ef11 in fts_optimize_thread (arg=0x55cef1b05140) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fe606bad700 (LWP 19403)): #0 0x00007fe62d4a3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055ceef7b1ba6 in LinuxAIOHandler::collect (this=0x7fe606bacb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055ceef7b20cf in LinuxAIOHandler::poll (this=0x7fe606bacb00, m1=0x7fe606bacba8, m2=0x7fe606bacbb0, request=0x7fe606bacbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055ceef7b2232 in os_aio_linux_handler (global_segment=0, m1=0x7fe606bacba8, m2=0x7fe606bacbb0, request=0x7fe606bacbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055ceef7b6d56 in os_aio_handler (segment=0, m1=0x7fe606bacba8, m2=0x7fe606bacbb0, request=0x7fe606bacbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055ceefa0abc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055ceef8b012f in io_handler_thread (arg=0x55cef0a39420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fe5f97fa700 (LWP 19414)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be1a4 in os_event::timed_wait (this=0x55cef1b6f710, abstime=0x7fe5f97f9d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055ceef7be4bb in os_event::wait_time_low (this=0x55cef1b6f710, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055ceef7be65b in os_event_wait_time_low (event=0x55cef1b6f710, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055ceef9f452c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fe5d37fe700 (LWP 19424)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceefc7d9a9 in safe_cond_wait (cond=0x7fe5d37fdc70, mp=0x7fe5d37fdcb0, file=0x55ceeff4a860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055ceef6d5a13 in inline_mysql_cond_wait (that=0x7fe5d37fdc70, mutex=0x7fe5d37fdcb0, src_file=0x55ceeff4c3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055ceef6da632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000055ceefc10471 in pfs_spawn_thread (arg=0x55cef1ba6290) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fe61ffff700 (LWP 19396)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe6257e0fd7 in gu::Lock::wait (cond=..., this=0x7fe61fffebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fe6257e1069 in galera::ServiceThd::thd_func (arg=0x55cef18158c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fe5faffd700 (LWP 19413)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be1a4 in os_event::timed_wait (this=0x55cef1965660, abstime=0x7fe5faffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055ceef7be4bb in os_event::wait_time_low (this=0x55cef1965660, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055ceef7be65b in os_event_wait_time_low (event=0x55cef1965660, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055ceef8ac2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fe61db7b700 (LWP 19397)): #0 0x00007fe62bc91483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe6257826f8 in asio::detail::epoll_reactor::run (this=0x55cef185df10, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fe625786df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55cef185de30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fe6257870a2 in asio::detail::task_io_service::run (this=0x55cef185de30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fe62579eafd in asio::io_service::run (this=0x55cef185cf30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fe62579f251 in gcomm::AsioProtonet::event_loop (this=0x55cef185ce90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fe6257bd2ed in GCommConn::run (this=0x55cef185d610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fe6257c0399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fe61c3d4700 (LWP 19429)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceefc7dcab in safe_cond_timedwait (cond=0x55cef09bdfa0 , mp=0x55cef09c09a0 , abstime=0x7fe61c3d3c60, file=0x55ceefd38348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055ceef04a725 in inline_mysql_cond_timedwait (that=0x55cef09bdfa0 , mutex=0x55cef09c09a0 , abstime=0x7fe61c3d3c60, src_file=0x55ceefd3a3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055ceef04f4fb in cache_thread (thd=0x7fe5bc000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055ceef04f7a9 in one_thread_per_connection_end (thd=0x7fe5bc000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055ceef2eacda in do_handle_one_connection (connect=0x55cef1c64db0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x000055ceef2ea994 in handle_one_connection (arg=0x55cef1c64db0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000055ceefc10471 in pfs_spawn_thread (arg=0x55cef1cf3cb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fe5d3fff700 (LWP 19423)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be889 in os_event::wait (this=0x55cef19657e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055ceef7be30f in os_event::wait_low (this=0x55cef19657e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055ceef7be680 in os_event_wait_low (event=0x55cef19657e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055ceef9738b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fe62477e700 (LWP 19399)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceefc7d9a9 in safe_cond_wait (cond=0x7fe614017b88, mp=0x7fe614017ad8, file=0x55ceefde6a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055ceef3d3d06 in inline_mysql_cond_wait (that=0x7fe614017b88, mutex=0x7fe614017ad8, src_file=0x55ceefde6d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055ceef3d699e in Wsrep_thd_queue::pop_front (this=0x7fe614017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055ceef3d53be in wsrep_rollback_process (rollbacker=0x7fe614001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055ceef3c5f01 in start_wsrep_THD (arg=0x55cef186d290) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fe5e8ff9700 (LWP 19422)): #0 0x00007fe62d8d9e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055ceef7beb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055ceef965755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fe61d37a700 (LWP 19398)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe6257bb867 in gu::Lock::wait (cond=..., this=0x7fe61d379850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fe6257be3ab in RecvBuf::front (this=this@entry=0x55cef185d7f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fe6257bbf46 in gcomm_recv (backend=, msg=0x55cef181d780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fe6257afeec in core_msg_recv (backend=backend@entry=0x55cef181d8c8, recv_msg=recv_msg@entry=0x55cef181d780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fe6257b3164 in gcs_core_recv (conn=0x55cef181d720, recv_act=recv_act@entry=0x7fe61d379d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fe6257ba3d8 in gcs_recv_thread (arg=0x55cef181d4a0) at gcs/src/gcs.cpp:1398 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fe5ebfff700 (LWP 19416)): #0 0x00007fe62d8d9e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055ceef7beb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055ceef8adb67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x000055ceef8adc97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fe62632b700 (LWP 19395)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceefc7dcab in safe_cond_timedwait (cond=0x55cef125d440 , mp=0x55cef125d380 , abstime=0x7fe62632ad70, file=0x55cef0107360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055ceefc7ec65 in inline_mysql_cond_timedwait (that=0x55cef125d440 , mutex=0x55cef125d380 , abstime=0x7fe62632ad70, src_file=0x55cef01073a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055ceefc7f7c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055ceefc10471 in pfs_spawn_thread (arg=0x55cef17de6f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fe5e97fa700 (LWP 19421)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be889 in os_event::wait (this=0x55cef19656e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055ceef7be30f in os_event::wait_low (this=0x55cef19656e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055ceef7be680 in os_event_wait_low (event=0x55cef19656e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055ceef98adde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fe6063ac700 (LWP 19404)): #0 0x00007fe62d4a3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055ceef7b1ba6 in LinuxAIOHandler::collect (this=0x7fe6063abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055ceef7b20cf in LinuxAIOHandler::poll (this=0x7fe6063abb00, m1=0x7fe6063abba8, m2=0x7fe6063abbb0, request=0x7fe6063abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055ceef7b2232 in os_aio_linux_handler (global_segment=1, m1=0x7fe6063abba8, m2=0x7fe6063abbb0, request=0x7fe6063abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055ceef7b6d56 in os_aio_handler (segment=1, m1=0x7fe6063abba8, m2=0x7fe6063abbb0, request=0x7fe6063abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055ceefa0abc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055ceef8b012f in io_handler_thread (arg=0x55cef0a39428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fe5fb7fe700 (LWP 19409)): #0 0x00007fe62d8d6d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceef7be1a4 in os_event::timed_wait (this=0x55cef1965760, abstime=0x7fe5fb7fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055ceef7be4bb in os_event::wait_time_low (this=0x55cef1965760, time_in_usec=591000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055ceef7be65b in os_event_wait_time_low (event=0x55cef1965760, time_in_usec=591000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055ceef994c57 in pc_sleep_if_needed (next_loop_time=1549295015355, sig_count=2, cur_time=1549295014764) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x000055ceef995b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fe624733700 (LWP 19400)): #0 0x00007fe62d8d6965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055ceefc7d9a9 in safe_cond_wait (cond=0x7fe608015628, mp=0x7fe608015578, file=0x55ceefde6a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055ceef3d3d06 in inline_mysql_cond_wait (that=0x7fe608015628, mutex=0x7fe608015578, src_file=0x55ceefde6d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055ceef3d699e in Wsrep_thd_queue::pop_front (this=0x7fe608015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055ceef3d577e in wsrep_post_rollback_process (post_rollbacker=0x7fe608000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055ceef3c5f01 in start_wsrep_THD (arg=0x55cef1857760) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fe5fbfff700 (LWP 19408)): #0 0x00007fe62d4a3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055ceef7b1ba6 in LinuxAIOHandler::collect (this=0x7fe5fbffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055ceef7b20cf in LinuxAIOHandler::poll (this=0x7fe5fbffeb00, m1=0x7fe5fbffeba8, m2=0x7fe5fbffebb0, request=0x7fe5fbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055ceef7b2232 in os_aio_linux_handler (global_segment=5, m1=0x7fe5fbffeba8, m2=0x7fe5fbffebb0, request=0x7fe5fbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055ceef7b6d56 in os_aio_handler (segment=5, m1=0x7fe5fbffeba8, m2=0x7fe5fbffebb0, request=0x7fe5fbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055ceefa0abc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055ceef8b012f in io_handler_thread (arg=0x55cef0a39448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe62d8d2dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe62bc90ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fe62dce3880 (LWP 19394)): #0 0x00007fe62d8d79d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055ceefc7837a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055ceef4b4d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fe62bc8620d in poll () from /lib64/libc.so.6 #5 0x000055ceef05696a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055ceef055c05 in mysqld_main (argc=161, argv=0x55cef1696d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055ceef049da5 in main (argc=23, argv=0x7fff1af05b68) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.1/data/core.19394 - found 'core.19442' (3/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.2/data/core.19442 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 19442] [New LWP 19444] [New LWP 19445] [New LWP 19449] [New LWP 19447] [New LWP 19613] [New LWP 19450] [New LWP 19448] [New LWP 19625] [New LWP 19614] [New LWP 19626] [New LWP 19627] [New LWP 19617] [New LWP 19632] [New LWP 19631] [New LWP 19618] [New LWP 19635] [New LWP 19638] [New LWP 19624] [New LWP 19637] [New LWP 19630] [New LWP 19636] [New LWP 19641] [New LWP 19619] [New LWP 19633] [New LWP 19615] [New LWP 19629] [New LWP 19634] [New LWP 19623] [New LWP 19616] [New LWP 19620] [New LWP 19446] [New LWP 19628] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f05d6f759d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f05d6f759d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005589010a337a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005589008dfd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f05d532420d in poll () from /lib64/libc.so.6 #5 0x000055890048196a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000558900480c05 in mysqld_main (argc=162, argv=0x5589041efd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000558900474da5 in main (argc=23, argv=0x7fffd7f4e1f8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f05917fa700 (LWP 19628)): #0 0x00007f05d6f77d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000558900bde255 in os_file_fsync_posix (file=22) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000558900bde5ab in os_file_flush_func (file=22) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000558900e27751 in pfs_os_file_flush_func (file=..., src_file=0x5589014a38b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000558900e2a593 in fil_flush_low (space=0x5589046c90f0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x0000558900e36113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x0000558900bb9c59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000558900bba6bb in log_write_up_to (lsn=7913524, flush_to_disk=true, rotate_key=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000558900bbba52 in log_checkpoint (sync=true, write_always=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1494 #9 0x0000558900cd8986 in srv_master_do_idle_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2258 #10 0x0000558900cd8d64 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2360 #11 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f05bfffe700 (LWP 19446)): #0 0x00007f05d532f483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f05cee206f8 in asio::detail::epoll_reactor::run (this=0x5589043aef70, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f05cee24df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5589043aee10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f05cee250a2 in asio::detail::task_io_service::run (this=0x5589043aee10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f05cee3cafd in asio::io_service::run (this=0x5589043adb30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f05cee3d251 in gcomm::AsioProtonet::event_loop (this=0x5589043ada90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f05cee5b2ed in GCommConn::run (this=0x5589043ae5a0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f05cee5e399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f05acdf9700 (LWP 19620)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be91a4 in os_event::timed_wait (this=0x5589044bf3e0, abstime=0x7f05acdf8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558900be94bb in os_event::wait_time_low (this=0x5589044bf3e0, time_in_usec=777000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558900be965b in os_event_wait_time_low (event=0x5589044bf3e0, time_in_usec=777000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558900dbfc57 in pc_sleep_if_needed (next_loop_time=1549295020866, sig_count=2, cur_time=1549295020089) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x0000558900dc0b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f05aedfd700 (LWP 19616)): #0 0x00007f05d6b41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558900bdcba6 in LinuxAIOHandler::collect (this=0x7f05aedfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558900bdd0cf in LinuxAIOHandler::poll (this=0x7f05aedfcb00, m1=0x7f05aedfcba8, m2=0x7f05aedfcbb0, request=0x7f05aedfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558900bdd232 in os_aio_linux_handler (global_segment=2, m1=0x7f05aedfcba8, m2=0x7f05aedfcbb0, request=0x7f05aedfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558900be1d56 in os_aio_handler (segment=2, m1=0x7f05aedfcba8, m2=0x7f05aedfcbb0, request=0x7f05aedfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558900e35bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558900cdb12f in io_handler_thread (arg=0x558901e64430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f05937fe700 (LWP 19623)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be91a4 in os_event::timed_wait (this=0x5589046977b0, abstime=0x7f05937fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558900be94bb in os_event::wait_time_low (this=0x5589046977b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558900be965b in os_event_wait_time_low (event=0x5589046977b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558900bb6314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f057dffb700 (LWP 19634)): #0 0x00007f05d6f77e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000558900be9b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000558900d90755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f0590ff9700 (LWP 19629)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be9889 in os_event::wait (this=0x5589044bf060) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558900be930f in os_event::wait_low (this=0x5589044bf060, reset_sig_count=4221) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558900be9680 in os_event_wait_low (event=0x5589044bf060, reset_sig_count=4221) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558900cd55ff in srv_resume_thread (slot=0x558901c26548 , sig_count=4221, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558900cd9610 in srv_purge_coordinator_suspend (slot=0x558901c26548 , rseg_history_len=21) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x0000558900cd98a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f05af5fe700 (LWP 19615)): #0 0x00007f05d6b41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558900bdcba6 in LinuxAIOHandler::collect (this=0x7f05af5fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558900bdd0cf in LinuxAIOHandler::poll (this=0x7f05af5fdb00, m1=0x7f05af5fdba8, m2=0x7f05af5fdbb0, request=0x7f05af5fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558900bdd232 in os_aio_linux_handler (global_segment=1, m1=0x7f05af5fdba8, m2=0x7f05af5fdbb0, request=0x7f05af5fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558900be1d56 in os_aio_handler (segment=1, m1=0x7f05af5fdba8, m2=0x7f05af5fdbb0, request=0x7f05af5fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558900e35bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558900cdb12f in io_handler_thread (arg=0x558901e64428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f057e7fc700 (LWP 19633)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be9889 in os_event::wait (this=0x5589044bf360) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558900be930f in os_event::wait_low (this=0x5589044bf360, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558900be9680 in os_event_wait_low (event=0x5589044bf360, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558900db5dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f05ad5fa700 (LWP 19619)): #0 0x00007f05d6b41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558900bdcba6 in LinuxAIOHandler::collect (this=0x7f05ad5f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558900bdd0cf in LinuxAIOHandler::poll (this=0x7f05ad5f9b00, m1=0x7f05ad5f9ba8, m2=0x7f05ad5f9bb0, request=0x7f05ad5f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558900bdd232 in os_aio_linux_handler (global_segment=5, m1=0x7f05ad5f9ba8, m2=0x7f05ad5f9bb0, request=0x7f05ad5f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558900be1d56 in os_aio_handler (segment=5, m1=0x7f05ad5f9ba8, m2=0x7f05ad5f9bb0, request=0x7f05ad5f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558900e35bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558900cdb12f in io_handler_thread (arg=0x558901e64448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f05affff700 (LWP 19641)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005589010a8cab in safe_cond_timedwait (cond=0x558901de8fa0 , mp=0x558901deb9a0 , abstime=0x7f05afffec60, file=0x558901163348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558900475725 in inline_mysql_cond_timedwait (that=0x558901de8fa0 , mutex=0x558901deb9a0 , abstime=0x7f05afffec60, src_file=0x5589011653b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055890047a4fb in cache_thread (thd=0x7f059c000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055890047a7a9 in one_thread_per_connection_end (thd=0x7f059c000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000558900715cda in do_handle_one_connection (connect=0x55890482f960) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000558900715994 in handle_one_connection (arg=0x55890482f960) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000055890103b471 in pfs_spawn_thread (arg=0x558904846c10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f057cff9700 (LWP 19636)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005589010a89a9 in safe_cond_wait (cond=0x7f057cff8c70, mp=0x7f057cff8cb0, file=0x558901375860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558900b00a13 in inline_mysql_cond_wait (that=0x7f057cff8c70, mutex=0x7f057cff8cb0, src_file=0x5589013773a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000558900b05632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000055890103b471 in pfs_spawn_thread (arg=0x55890467a3c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f057ffff700 (LWP 19630)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be9889 in os_event::wait (this=0x5589044bf0e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558900be930f in os_event::wait_low (this=0x5589044bf0e0, reset_sig_count=20073) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558900be9680 in os_event_wait_low (event=0x5589044bf0e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558900cd55ff in srv_resume_thread (slot=0x558901c26580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558900cd9203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f05cc0db700 (LWP 19637)): #0 0x00007f05d526840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000558900474e24 in my_sigwait (set=0x7f05cc0dacf0, sig=0x7f05cc0dacb4, code=0x7f05cc0dacb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055890047b464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055890103b471 in pfs_spawn_thread (arg=0x558904482df0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f0593fff700 (LWP 19624)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be91a4 in os_event::timed_wait (this=0x5589044bf260, abstime=0x7f0593ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558900be94bb in os_event::wait_time_low (this=0x5589044bf260, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558900be965b in os_event_wait_time_low (event=0x5589044bf260, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558900cd7808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f05cc090700 (LWP 19638)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005589010a89a9 in safe_cond_wait (cond=0x558901de8fe0 , mp=0x558901dec360 , file=0x558901172b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005589004a63ce in inline_mysql_cond_wait (that=0x558901de8fe0 , mutex=0x558901dec360 , src_file=0x5589011736b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005589004a7c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055890103b471 in pfs_spawn_thread (arg=0x5589044832e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f057d7fa700 (LWP 19635)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be9889 in os_event::wait (this=0x5589044bf460) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558900be930f in os_event::wait_low (this=0x5589044bf460, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558900be9680 in os_event_wait_low (event=0x5589044bf460, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558900d9e8b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f05addfb700 (LWP 19618)): #0 0x00007f05d6b41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558900bdcba6 in LinuxAIOHandler::collect (this=0x7f05addfab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558900bdd0cf in LinuxAIOHandler::poll (this=0x7f05addfab00, m1=0x7f05addfaba8, m2=0x7f05addfabb0, request=0x7f05addfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558900bdd232 in os_aio_linux_handler (global_segment=4, m1=0x7f05addfaba8, m2=0x7f05addfabb0, request=0x7f05addfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558900be1d56 in os_aio_handler (segment=4, m1=0x7f05addfaba8, m2=0x7f05addfabb0, request=0x7f05addfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558900e35bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558900cdb12f in io_handler_thread (arg=0x558901e64440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f057f7fe700 (LWP 19631)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be9889 in os_event::wait (this=0x5589044bf1e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558900be930f in os_event::wait_low (this=0x5589044bf1e0, reset_sig_count=14260) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558900be9680 in os_event_wait_low (event=0x5589044bf1e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558900cd55ff in srv_resume_thread (slot=0x558901c265f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558900cd9203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f057effd700 (LWP 19632)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be9889 in os_event::wait (this=0x5589044bf160) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558900be930f in os_event::wait_low (this=0x5589044bf160, reset_sig_count=16617) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558900be9680 in os_event_wait_low (event=0x5589044bf160, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558900cd55ff in srv_resume_thread (slot=0x558901c265b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558900cd9203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f05ae5fc700 (LWP 19617)): #0 0x00007f05d6b41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558900bdcba6 in LinuxAIOHandler::collect (this=0x7f05ae5fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558900bdd0cf in LinuxAIOHandler::poll (this=0x7f05ae5fbb00, m1=0x7f05ae5fbba8, m2=0x7f05ae5fbbb0, request=0x7f05ae5fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558900bdd232 in os_aio_linux_handler (global_segment=3, m1=0x7f05ae5fbba8, m2=0x7f05ae5fbbb0, request=0x7f05ae5fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558900be1d56 in os_aio_handler (segment=3, m1=0x7f05ae5fbba8, m2=0x7f05ae5fbbb0, request=0x7f05ae5fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558900e35bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558900cdb12f in io_handler_thread (arg=0x558901e64438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f0591ffb700 (LWP 19627)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be91a4 in os_event::timed_wait (this=0x558904667a10, abstime=0x7f0591ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558900be94bb in os_event::wait_time_low (this=0x558904667a10, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558900be965b in os_event_wait_time_low (event=0x558904667a10, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558900d37aad in ib_wqueue_timedwait (wq=0x5589046678e0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000558900e79f11 in fts_optimize_thread (arg=0x5589046678e0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f05927fc700 (LWP 19626)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be91a4 in os_event::timed_wait (this=0x5589046c97d0, abstime=0x7f05927fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558900be94bb in os_event::wait_time_low (this=0x5589046c97d0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558900be965b in os_event_wait_time_low (event=0x5589046c97d0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558900e1f52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f05afdff700 (LWP 19614)): #0 0x00007f05d6b41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558900bdcba6 in LinuxAIOHandler::collect (this=0x7f05afdfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558900bdd0cf in LinuxAIOHandler::poll (this=0x7f05afdfeb00, m1=0x7f05afdfeba8, m2=0x7f05afdfebb0, request=0x7f05afdfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558900bdd232 in os_aio_linux_handler (global_segment=0, m1=0x7f05afdfeba8, m2=0x7f05afdfebb0, request=0x7f05afdfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558900be1d56 in os_aio_handler (segment=0, m1=0x7f05afdfeba8, m2=0x7f05afdfebb0, request=0x7f05afdfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558900e35bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558900cdb12f in io_handler_thread (arg=0x558901e64420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f0592ffd700 (LWP 19625)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558900be91a4 in os_event::timed_wait (this=0x5589044bf2e0, abstime=0x7f0592ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558900be94bb in os_event::wait_time_low (this=0x5589044bf2e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558900be965b in os_event_wait_time_low (event=0x5589044bf2e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558900cd72e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f05cc399700 (LWP 19448)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005589010a89a9 in safe_cond_wait (cond=0x7f05b4016878, mp=0x7f05b40167c8, file=0x558901211a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005589007fed06 in inline_mysql_cond_wait (that=0x7f05b4016878, mutex=0x7f05b40167c8, src_file=0x558901211d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055890080199e in Wsrep_thd_queue::pop_front (this=0x7f05b4016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005589008003be in wsrep_rollback_process (rollbacker=0x7f05b4001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005589007f0f01 in start_wsrep_THD (arg=0x5589043d01c0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f05cc303700 (LWP 19450)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f05ced73132 in fifo_lock_get (q=q@entry=0x55890439ca30) at galerautils/src/gu_fifo.c:262 #2 0x00007f05ced737a1 in gu_fifo_get_head (q=0x55890439ca30, err=0x7f05cc30206c) at galerautils/src/gu_fifo.c:330 #3 0x00007f05cee550cc in gcs_recv (conn=0x5589043763b0, action=0x7f05cc3020b0) at gcs/src/gcs.cpp:2024 #4 0x00007f05cee8216c in galera::GcsActionSource::process (this=0x5589043a6dd0, recv_ctx=0x7f05cc3024d0, exit_loop=@0x7f05cc30229e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f05ceea1033 in galera::ReplicatorSMM::async_recv (this=0x55890436e160, recv_ctx=0x7f05cc3024d0) at galera/src/replicator_smm.cpp:383 #6 0x000055890115c20e in wsrep::wsrep_provider_v26::run_applier (this=0x5589042e3d80, applier_ctx=0x7f05cc3024d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005589007ff2b4 in wsrep_replication_process (thd=0x7f05a4000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005589007f0f01 in start_wsrep_THD (arg=0x5589043d08f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f05bd66e700 (LWP 19613)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005589010a8cab in safe_cond_timedwait (cond=0x558902675ca0 , mp=0x558902675be0 , abstime=0x7f05bd66dcb0, file=0x5589014ee130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558900f12add in inline_mysql_cond_timedwait (that=0x558902675ca0 , mutex=0x558902675be0 , abstime=0x7f05bd66dcb0, src_file=0x5589014ee178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000558900f13001 in my_service_thread_sleep (control=0x558901c2a900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000558900f0749a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055890103b471 in pfs_spawn_thread (arg=0x5589044775a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f05bf7fd700 (LWP 19447)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f05cee59867 in gu::Lock::wait (cond=..., this=0x7f05bf7fc850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f05cee5c3ab in RecvBuf::front (this=this@entry=0x5589043ae788, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f05cee59f46 in gcomm_recv (backend=, msg=0x558904376690, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f05cee4deec in core_msg_recv (backend=backend@entry=0x5589043767d8, recv_msg=recv_msg@entry=0x558904376690, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f05cee51164 in gcs_core_recv (conn=0x558904376630, recv_act=recv_act@entry=0x7f05bf7fcd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f05cee583d8 in gcs_recv_thread (arg=0x5589043763b0) at gcs/src/gcs.cpp:1398 #7 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f05cc34e700 (LWP 19449)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005589010a89a9 in safe_cond_wait (cond=0x7f05a8016938, mp=0x7f05a8016888, file=0x558901211a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005589007fed06 in inline_mysql_cond_wait (that=0x7f05a8016938, mutex=0x7f05a8016888, src_file=0x558901211d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055890080199e in Wsrep_thd_queue::pop_front (this=0x7f05a8016830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055890080077e in wsrep_post_rollback_process (post_rollbacker=0x7f05a8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005589007f0f01 in start_wsrep_THD (arg=0x5589043cf670) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f05ce81d700 (LWP 19445)): #0 0x00007f05d6f74965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f05cee7efd7 in gu::Lock::wait (cond=..., this=0x7f05ce81cbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f05cee7f069 in galera::ServiceThd::thd_func (arg=0x55890436e878) at galera/src/galera_service_thd.cpp:30 #3 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f05cf9c9700 (LWP 19444)): #0 0x00007f05d6f74d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005589010a8cab in safe_cond_timedwait (cond=0x558902688440 , mp=0x558902688380 , abstime=0x7f05cf9c8d70, file=0x558901532360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005589010a9c65 in inline_mysql_cond_timedwait (that=0x558902688440 , mutex=0x558902688380 , abstime=0x7f05cf9c8d70, src_file=0x5589015323a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005589010aa7c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055890103b471 in pfs_spawn_thread (arg=0x558904337700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f05d6f70dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f05d532eead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f05d7381880 (LWP 19442)): #0 0x00007f05d6f759d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005589010a337a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005589008dfd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f05d532420d in poll () from /lib64/libc.so.6 #5 0x000055890048196a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000558900480c05 in mysqld_main (argc=162, argv=0x5589041efd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000558900474da5 in main (argc=23, argv=0x7fffd7f4e1f8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 worker[1] Trying to dump core for [mysqltest - pid: 20120, winpid: 20120] worker[1] Trying to dump core for [mysqld.1 - pid: 19860, winpid: 19860] worker[1] Trying to dump core for [mysqld.2 - pid: 19906, winpid: 19906] Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/mysqld.2/data/core.19442 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_many_fragments-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_many_fragments Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.mysql-wsrep-features#8 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 19:00:44 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysql-wsrep-features#8.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); connection node_1; CREATE TABLE t1 (f1 INT PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(100), FULLTEXT (f2)) ENGINE=InnoDB; connection node_2; SELECT COUNT(*) = 13 FROM INFORMATION_SCHEMA.INNODB_SYS_TABLES WHERE name LIKE 'test/%'; COUNT(*) = 13 1 connection node_1; INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) - found 'core.19861' (4/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.1/data/core.19861 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 19861] [New LWP 19870] [New LWP 19866] [New LWP 19869] [New LWP 19875] [New LWP 19865] [New LWP 19873] [New LWP 19878] [New LWP 19881] [New LWP 19876] [New LWP 19882] [New LWP 19886] [New LWP 19885] [New LWP 19892] [New LWP 19891] [New LWP 19898] [New LWP 19894] [New LWP 19895] [New LWP 19884] [New LWP 19893] [New LWP 19899] [New LWP 19890] [New LWP 19879] [New LWP 19877] [New LWP 19868] [New LWP 19883] [New LWP 19867] [New LWP 19889] [New LWP 19887] [New LWP 19888] [New LWP 19874] [New LWP 19872] [New LWP 19871] [New LWP 20133] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fb8f59eb9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fb8f59eb9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559ac7aca37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559ac7306d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb8f3d9a20d in poll () from /lib64/libc.so.6 #5 0x0000559ac6ea896a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559ac6ea7c05 in mysqld_main (argc=161, argv=0x559aca6b9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559ac6e9bda5 in main (argc=23, argv=0x7ffdd9f56c48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fb8e40ca700 (LWP 20133)): #0 0x00007fb8f59edd1d in fsync () from /lib64/libpthread.so.0 #1 0x0000559ac7605255 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000559ac76055ab in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000559ac784e751 in pfs_os_file_flush_func (file=..., src_file=0x559ac7eca8b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559ac7851593 in fil_flush_low (space=0x559acab8a010, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x0000559ac785d113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x0000559ac75e0c59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000559ac75e16bb in log_write_up_to (lsn=7203972, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000559ac774c551 in trx_flush_log_if_needed_low (lsn=7203972) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000559ac774c59a in trx_flush_log_if_needed (lsn=7203972, trx=0x7fb8e43af390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000559ac774e3c3 in trx_commit_complete_for_mysql (trx=0x7fb8e43af390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000559ac7532a6f in innobase_commit (hton=0x559aca9548f0, thd=0x7fb87c1711e0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #12 0x0000559ac730c89b in commit_one_phase_2 (thd=0x7fb87c1711e0, all=true, trans=0x7fb87c174728, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000559ac730c79d in ha_commit_one_phase (thd=0x7fb87c1711e0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000559ac730bb19 in ha_commit_trans (thd=0x7fb87c1711e0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000559ac7151a76 in trans_commit (thd=0x7fb87c1711e0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000559ac7510988 in Wsrep_storage_service::commit (this=0x7fb87c1d5580, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x0000559ac7b7985f in wsrep::transaction::certify_fragment (this=0x7fb87c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #18 0x0000559ac7b78fc3 in wsrep::transaction::streaming_step (this=0x7fb87c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #19 0x0000559ac7b7637e in wsrep::transaction::after_row (this=0x7fb87c0069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x0000559ac731eba4 in wsrep::client_state::after_row (this=0x7fb87c006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x0000559ac7307e87 in wsrep_after_row (thd=0x7fb87c000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x0000559ac7319d26 in wsrep_after_row (thd=0x7fb87c000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #23 0x0000559ac731a117 in handler::ha_write_row (this=0x7fb87c09d828, buf=0x7fb87c096220 "\376\221&") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #24 0x0000559ac6f9018d in write_record (thd=0x7fb87c000b00, table=0x7fb87c09cbf0, info=0x7fb87c016fa0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x0000559ac6f9548c in select_insert::send_data (this=0x7fb87c016f58, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x0000559ac704db47 in end_send (join=0x7fb87c017008, join_tab=0x7fb87c0a9178, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x0000559ac718ab89 in JOIN_CACHE::generate_full_extensions (this=0x7fb87c0a9658, rec_ptr=0x7fb87c12d2a0 "\224") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x0000559ac718a8a4 in JOIN_CACHE::join_matching_records (this=0x7fb87c0a9658, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x0000559ac718a270 in JOIN_CACHE::join_records (this=0x7fb87c0a9658, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x0000559ac718a40a in JOIN_CACHE::join_records (this=0x7fb87c0a9528, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x0000559ac718a40a in JOIN_CACHE::join_records (this=0x7fb87c019c78, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x0000559ac704a7fb in sub_select_cache (join=0x7fb87c017008, join_tab=0x7fb87c0a8668, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x0000559ac704aa08 in sub_select (join=0x7fb87c017008, join_tab=0x7fb87c0a82b8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x0000559ac704a1d3 in do_select (join=0x7fb87c017008, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x0000559ac7022cf1 in JOIN::exec_inner (this=0x7fb87c017008) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x0000559ac702213a in JOIN::exec (this=0x7fb87c017008) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x0000559ac702342c in mysql_select (thd=0x7fb87c000b00, tables=0x7fb87c014c68, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7fb87c016f58, unit=0x7fb87c0049b8, select_lex=0x7fb87c014728) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x0000559ac7014d2d in handle_select (thd=0x7fb87c000b00, lex=0x7fb87c0048f0, result=0x7fb87c016f58, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x0000559ac6fd713c in mysql_execute_command (thd=0x7fb87c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4856 #40 0x0000559ac6fe2675 in mysql_parse (thd=0x7fb87c000b00, rawbuf=0x7fb87c013e68 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7fb8e40c90b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #41 0x0000559ac6fe1ca8 in wsrep_mysql_parse (thd=0x7fb87c000b00, rawbuf=0x7fb87c013e68 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=86, parser_state=0x7fb8e40c90b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #42 0x0000559ac6fcdb65 in dispatch_command (command=COM_QUERY, thd=0x7fb87c000b00, packet=0x7fb87c00a451 "INSERT INTO t1 (f2) SELECT 'foobarbaz' FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=86, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #43 0x0000559ac6fcc599 in do_command (thd=0x7fb87c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x0000559ac713cc50 in do_handle_one_connection (connect=0x559acac7e200) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #45 0x0000559ac713c994 in handle_one_connection (arg=0x559acac7e200) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #46 0x0000559ac7a62471 in pfs_spawn_thread (arg=0x559acac820d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fb8e532f700 (LWP 19871)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb8ed7e9132 in fifo_lock_get (q=q@entry=0x559aca866b20) at galerautils/src/gu_fifo.c:262 #2 0x00007fb8ed7e97a1 in gu_fifo_get_head (q=0x559aca866b20, err=0x7fb8e532e06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fb8ed8cb0cc in gcs_recv (conn=0x559aca8404a0, action=0x7fb8e532e0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fb8ed8f816c in galera::GcsActionSource::process (this=0x559aca83fe80, recv_ctx=0x7fb8e532e4d0, exit_loop=@0x7fb8e532e29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fb8ed917033 in galera::ReplicatorSMM::async_recv (this=0x559aca8381b0, recv_ctx=0x7fb8e532e4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000559ac7b8320e in wsrep::wsrep_provider_v26::run_applier (this=0x559aca7bf280, applier_ctx=0x7fb8e532e4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x0000559ac72262b4 in wsrep_replication_process (thd=0x7fb8cc000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000559ac7217f01 in start_wsrep_THD (arg=0x559aca890b50) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fb8cbfff700 (LWP 19872)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7acfcab in safe_cond_timedwait (cond=0x559ac909cca0 , mp=0x559ac909cbe0 , abstime=0x7fb8cbffecb0, file=0x559ac7f15130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559ac7939add in inline_mysql_cond_timedwait (that=0x559ac909cca0 , mutex=0x559ac909cbe0 , abstime=0x7fb8cbffecb0, src_file=0x559ac7f15178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559ac793a001 in my_service_thread_sleep (control=0x559ac8651900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559ac792e49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559ac7a62471 in pfs_spawn_thread (arg=0x559aca938780) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fb8ca5ad700 (LWP 19874)): #0 0x00007fb8f55b7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559ac7603ba6 in LinuxAIOHandler::collect (this=0x7fb8ca5acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559ac76040cf in LinuxAIOHandler::poll (this=0x7fb8ca5acb00, m1=0x7fb8ca5acba8, m2=0x7fb8ca5acbb0, request=0x7fb8ca5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559ac7604232 in os_aio_linux_handler (global_segment=1, m1=0x7fb8ca5acba8, m2=0x7fb8ca5acbb0, request=0x7fb8ca5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559ac7608d56 in os_aio_handler (segment=1, m1=0x7fb8ca5acba8, m2=0x7fb8ca5acbb0, request=0x7fb8ca5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559ac785cbc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559ac770212f in io_handler_thread (arg=0x559ac888b428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fb8b2ffd700 (LWP 19888)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7610889 in os_event::wait (this=0x559aca980440) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559ac761030f in os_event::wait_low (this=0x559aca980440, reset_sig_count=23989) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559ac7610680 in os_event_wait_low (event=0x559aca980440, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559ac76fc5ff in srv_resume_thread (slot=0x559ac864d580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559ac7700203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fb8b37fe700 (LWP 19887)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7610889 in os_event::wait (this=0x559aca9803c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559ac761030f in os_event::wait_low (this=0x559aca9803c0, reset_sig_count=4899) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559ac7610680 in os_event_wait_low (event=0x559aca9803c0, reset_sig_count=4899) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559ac76fc5ff in srv_resume_thread (slot=0x559ac864d548 , sig_count=4899, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559ac7700610 in srv_purge_coordinator_suspend (slot=0x559ac864d548 , rseg_history_len=23) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x0000559ac77008a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fb8b27fc700 (LWP 19889)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7610889 in os_event::wait (this=0x559aca9804c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559ac761030f in os_event::wait_low (this=0x559aca9804c0, reset_sig_count=20258) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559ac7610680 in os_event_wait_low (event=0x559aca9804c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559ac76fc5ff in srv_resume_thread (slot=0x559ac864d5b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559ac7700203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fb8e637c700 (LWP 19867)): #0 0x00007fb8f3da5483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb8ed8966f8 in asio::detail::epoll_reactor::run (this=0x559aca878f90, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fb8ed89adf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559aca878e30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fb8ed89b0a2 in asio::detail::task_io_service::run (this=0x559aca878e30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fb8ed8b2afd in asio::io_service::run (this=0x559aca877f30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fb8ed8b3251 in gcomm::AsioProtonet::event_loop (this=0x559aca877e90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fb8ed8d12ed in GCommConn::run (this=0x559aca878610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fb8ed8d4399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fb8c2ffd700 (LWP 19883)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac76101a4 in os_event::timed_wait (this=0x559aca980640, abstime=0x7fb8c2ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559ac76104bb in os_event::wait_time_low (this=0x559aca980640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559ac761065b in os_event_wait_time_low (event=0x559aca980640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559ac76fe2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fb8e5b7b700 (LWP 19868)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb8ed8cf867 in gu::Lock::wait (cond=..., this=0x7fb8e5b7a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb8ed8d23ab in RecvBuf::front (this=this@entry=0x559aca8787f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fb8ed8cff46 in gcomm_recv (backend=, msg=0x559aca840780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fb8ed8c3eec in core_msg_recv (backend=backend@entry=0x559aca8408c8, recv_msg=recv_msg@entry=0x559aca840780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fb8ed8c7164 in gcs_core_recv (conn=0x559aca840720, recv_act=recv_act@entry=0x7fb8e5b7ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fb8ed8ce3d8 in gcs_recv_thread (arg=0x559aca8404a0) at gcs/src/gcs.cpp:1398 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fb8c8daa700 (LWP 19877)): #0 0x00007fb8f55b7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559ac7603ba6 in LinuxAIOHandler::collect (this=0x7fb8c8da9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559ac76040cf in LinuxAIOHandler::poll (this=0x7fb8c8da9b00, m1=0x7fb8c8da9ba8, m2=0x7fb8c8da9bb0, request=0x7fb8c8da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559ac7604232 in os_aio_linux_handler (global_segment=4, m1=0x7fb8c8da9ba8, m2=0x7fb8c8da9bb0, request=0x7fb8c8da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559ac7608d56 in os_aio_handler (segment=4, m1=0x7fb8c8da9ba8, m2=0x7fb8c8da9bb0, request=0x7fb8c8da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559ac785cbc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559ac770212f in io_handler_thread (arg=0x559ac888b440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fb8c37fe700 (LWP 19879)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac76101a4 in os_event::timed_wait (this=0x559aca980740, abstime=0x7fb8c37fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559ac76104bb in os_event::wait_time_low (this=0x559aca980740, time_in_usec=536000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559ac761065b in os_event_wait_time_low (event=0x559aca980740, time_in_usec=536000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559ac77e6c57 in pc_sleep_if_needed (next_loop_time=1549295985541, sig_count=2, cur_time=1549295985005) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x0000559ac77e7b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fb8b1ffb700 (LWP 19890)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7610889 in os_event::wait (this=0x559aca980540) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559ac761030f in os_event::wait_low (this=0x559aca980540, reset_sig_count=15905) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559ac7610680 in os_event_wait_low (event=0x559aca980540, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559ac76fc5ff in srv_resume_thread (slot=0x559ac864d5f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559ac7700203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fb8e4115700 (LWP 19899)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7acfcab in safe_cond_timedwait (cond=0x559ac880ffa0 , mp=0x559ac88129a0 , abstime=0x7fb8e4114c60, file=0x559ac7b8a348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559ac6e9c725 in inline_mysql_cond_timedwait (that=0x559ac880ffa0 , mutex=0x559ac88129a0 , abstime=0x7fb8e4114c60, src_file=0x559ac7b8c3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559ac6ea14fb in cache_thread (thd=0x7fb884000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000559ac6ea17a9 in one_thread_per_connection_end (thd=0x7fb884000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000559ac713ccda in do_handle_one_connection (connect=0x559acacf4480) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000559ac713c994 in handle_one_connection (arg=0x559acacf4480) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x0000559ac7a62471 in pfs_spawn_thread (arg=0x559acad117c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fb89bfff700 (LWP 19893)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7610889 in os_event::wait (this=0x559aca9807c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559ac761030f in os_event::wait_low (this=0x559aca9807c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559ac7610680 in os_event_wait_low (event=0x559aca9807c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559ac77c58b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fb8c17fa700 (LWP 19884)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac76101a4 in os_event::timed_wait (this=0x559acab8a6f0, abstime=0x7fb8c17f9d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559ac76104bb in os_event::wait_time_low (this=0x559acab8a6f0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559ac761065b in os_event_wait_time_low (event=0x559acab8a6f0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559ac784652c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fb8e41ab700 (LWP 19895)): #0 0x00007fb8f3cde40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000559ac6e9be24 in my_sigwait (set=0x7fb8e41aacf0, sig=0x7fb8e41aacb4, code=0x7fb8e41aacb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000559ac6ea2464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000559ac7a62471 in pfs_spawn_thread (arg=0x559aca944060) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fb89b7fe700 (LWP 19894)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7acf9a9 in safe_cond_wait (cond=0x7fb89b7fdc70, mp=0x7fb89b7fdcb0, file=0x559ac7d9c860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559ac7527a13 in inline_mysql_cond_wait (that=0x7fb89b7fdc70, mutex=0x7fb89b7fdcb0, src_file=0x559ac7d9e3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559ac752c632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x0000559ac7a62471 in pfs_spawn_thread (arg=0x559acabc1270) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fb8e4160700 (LWP 19898)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7acf9a9 in safe_cond_wait (cond=0x559ac880ffe0 , mp=0x559ac8813360 , file=0x559ac7b99b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559ac6ecd3ce in inline_mysql_cond_wait (that=0x559ac880ffe0 , mutex=0x559ac8813360 , src_file=0x559ac7b9a6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000559ac6ecec60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559ac7a62471 in pfs_spawn_thread (arg=0x559acac81c30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fb8b17fa700 (LWP 19891)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7610889 in os_event::wait (this=0x559aca9806c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559ac761030f in os_event::wait_low (this=0x559aca9806c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559ac7610680 in os_event_wait_low (event=0x559aca9806c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559ac77dcdde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fb8b0ff9700 (LWP 19892)): #0 0x00007fb8f59ede3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559ac7610b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559ac77b7755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fb8c0ff9700 (LWP 19885)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac76101a4 in os_event::timed_wait (this=0x559acab201d0, abstime=0x7fb8c0ff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559ac76104bb in os_event::wait_time_low (this=0x559acab201d0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559ac761065b in os_event_wait_time_low (event=0x559acab201d0, time_in_usec=5000000, reset_sig_count=3) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559ac775eaad in ib_wqueue_timedwait (wq=0x559acab20120, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559ac78a0f11 in fts_optimize_thread (arg=0x559acab20120) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fb8b3fff700 (LWP 19886)): #0 0x00007fb8f59ede3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559ac7610b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559ac76ffb67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x0000559ac76ffc97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fb8c1ffb700 (LWP 19882)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac76101a4 in os_event::timed_wait (this=0x559aca9805c0, abstime=0x7fb8c1ffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559ac76104bb in os_event::wait_time_low (this=0x559aca9805c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559ac761065b in os_event_wait_time_low (event=0x559aca9805c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559ac76fe808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fb8c95ab700 (LWP 19876)): #0 0x00007fb8f55b7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559ac7603ba6 in LinuxAIOHandler::collect (this=0x7fb8c95aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559ac76040cf in LinuxAIOHandler::poll (this=0x7fb8c95aab00, m1=0x7fb8c95aaba8, m2=0x7fb8c95aabb0, request=0x7fb8c95aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559ac7604232 in os_aio_linux_handler (global_segment=3, m1=0x7fb8c95aaba8, m2=0x7fb8c95aabb0, request=0x7fb8c95aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559ac7608d56 in os_aio_handler (segment=3, m1=0x7fb8c95aaba8, m2=0x7fb8c95aabb0, request=0x7fb8c95aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559ac785cbc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559ac770212f in io_handler_thread (arg=0x559ac888b438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fb8c27fc700 (LWP 19881)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac76101a4 in os_event::timed_wait (this=0x559acab586d0, abstime=0x7fb8c27fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559ac76104bb in os_event::wait_time_low (this=0x559acab586d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559ac761065b in os_event_wait_time_low (event=0x559acab586d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559ac75dd314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fb8c3fff700 (LWP 19878)): #0 0x00007fb8f55b7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559ac7603ba6 in LinuxAIOHandler::collect (this=0x7fb8c3ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559ac76040cf in LinuxAIOHandler::poll (this=0x7fb8c3ffeb00, m1=0x7fb8c3ffeba8, m2=0x7fb8c3ffebb0, request=0x7fb8c3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559ac7604232 in os_aio_linux_handler (global_segment=5, m1=0x7fb8c3ffeba8, m2=0x7fb8c3ffebb0, request=0x7fb8c3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559ac7608d56 in os_aio_handler (segment=5, m1=0x7fb8c3ffeba8, m2=0x7fb8c3ffebb0, request=0x7fb8c3ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559ac785cbc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559ac770212f in io_handler_thread (arg=0x559ac888b448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb8cadae700 (LWP 19873)): #0 0x00007fb8f55b7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559ac7603ba6 in LinuxAIOHandler::collect (this=0x7fb8cadadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559ac76040cf in LinuxAIOHandler::poll (this=0x7fb8cadadb00, m1=0x7fb8cadadba8, m2=0x7fb8cadadbb0, request=0x7fb8cadadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559ac7604232 in os_aio_linux_handler (global_segment=0, m1=0x7fb8cadadba8, m2=0x7fb8cadadbb0, request=0x7fb8cadadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559ac7608d56 in os_aio_handler (segment=0, m1=0x7fb8cadadba8, m2=0x7fb8cadadbb0, request=0x7fb8cadadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559ac785cbc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559ac770212f in io_handler_thread (arg=0x559ac888b420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb8ee43f700 (LWP 19865)): #0 0x00007fb8f59ead12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7acfcab in safe_cond_timedwait (cond=0x559ac90af440 , mp=0x559ac90af380 , abstime=0x7fb8ee43ed70, file=0x559ac7f59360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559ac7ad0c65 in inline_mysql_cond_timedwait (that=0x559ac90af440 , mutex=0x559ac90af380 , abstime=0x7fb8ee43ed70, src_file=0x559ac7f593a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559ac7ad17c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559ac7a62471 in pfs_spawn_thread (arg=0x559aca8016f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb8c9dac700 (LWP 19875)): #0 0x00007fb8f55b7644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559ac7603ba6 in LinuxAIOHandler::collect (this=0x7fb8c9dabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559ac76040cf in LinuxAIOHandler::poll (this=0x7fb8c9dabb00, m1=0x7fb8c9dabba8, m2=0x7fb8c9dabbb0, request=0x7fb8c9dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559ac7604232 in os_aio_linux_handler (global_segment=2, m1=0x7fb8c9dabba8, m2=0x7fb8c9dabbb0, request=0x7fb8c9dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559ac7608d56 in os_aio_handler (segment=2, m1=0x7fb8c9dabba8, m2=0x7fb8c9dabbb0, request=0x7fb8c9dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559ac785cbc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559ac770212f in io_handler_thread (arg=0x559ac888b430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fb8ec091700 (LWP 19869)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7acf9a9 in safe_cond_wait (cond=0x7fb8dc017b88, mp=0x7fb8dc017ad8, file=0x559ac7c38a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559ac7225d06 in inline_mysql_cond_wait (that=0x7fb8dc017b88, mutex=0x7fb8dc017ad8, src_file=0x559ac7c38d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559ac722899e in Wsrep_thd_queue::pop_front (this=0x7fb8dc017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559ac72273be in wsrep_rollback_process (rollbacker=0x7fb8dc001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000559ac7217f01 in start_wsrep_THD (arg=0x559aca888270) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fb8ec892700 (LWP 19866)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb8ed8f4fd7 in gu::Lock::wait (cond=..., this=0x7fb8ec891bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fb8ed8f5069 in galera::ServiceThd::thd_func (arg=0x559aca8388c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb8e537a700 (LWP 19870)): #0 0x00007fb8f59ea965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559ac7acf9a9 in safe_cond_wait (cond=0x7fb8d4015628, mp=0x7fb8d4015578, file=0x559ac7c38a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559ac7225d06 in inline_mysql_cond_wait (that=0x7fb8d4015628, mutex=0x7fb8d4015578, src_file=0x559ac7c38d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559ac722899e in Wsrep_thd_queue::pop_front (this=0x7fb8d4015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559ac722777e in wsrep_post_rollback_process (post_rollbacker=0x7fb8d4000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000559ac7217f01 in start_wsrep_THD (arg=0x559aca872760) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fb8f59e6dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fb8f3da4ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb8f5df7880 (LWP 19861)): #0 0x00007fb8f59eb9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559ac7aca37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559ac7306d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fb8f3d9a20d in poll () from /lib64/libc.so.6 #5 0x0000559ac6ea896a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559ac6ea7c05 in mysqld_main (argc=161, argv=0x559aca6b9d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559ac6e9bda5 in main (argc=23, argv=0x7ffdd9f56c48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.1/data/core.19861 - found 'core.19908' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/mysqld.2/data/core.19908 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 19908] [New LWP 19910] [New LWP 19923] [New LWP 19921] [New LWP 19919] [New LWP 19920] [New LWP 19924] [New LWP 19922] [New LWP 20099] [New LWP 20089] [New LWP 20087] [New LWP 20106] [New LWP 20092] [New LWP 20090] [New LWP 20091] [New LWP 20093] [New LWP 20094] [New LWP 20098] [New LWP 20105] [New LWP 20100] [New LWP 20109] [New LWP 20115] [New LWP 20102] [New LWP 20110] [New LWP 20107] [New LWP 20103] [New LWP 20111] [New LWP 20104] [New LWP 20097] [New LWP 20108] [New LWP 20088] [New LWP 20112] [New LWP 20101] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f797b6919d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f797b6919d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b7b6fce37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b7b680ad65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f7979a4020d in poll () from /lib64/libc.so.6 #5 0x000055b7b63ac96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055b7b63abc05 in mysqld_main (argc=162, argv=0x55b7b89bcd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055b7b639fda5 in main (argc=23, argv=0x7fff8b31e7a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f7931ffb700 (LWP 20101)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b141a4 in os_event::timed_wait (this=0x55b7b8e30100, abstime=0x7f7931ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b7b6b144bb in os_event::wait_time_low (this=0x55b7b8e30100, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b7b6b1465b in os_event_wait_time_low (event=0x55b7b8e30100, time_in_usec=5000000, reset_sig_count=4) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b7b6c62aad in ib_wqueue_timedwait (wq=0x55b7b8e30050, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055b7b6da4f11 in fts_optimize_thread (arg=0x55b7b8e30050) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f796079c700 (LWP 20112)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6fd39a9 in safe_cond_wait (cond=0x55b7b7d13fe0 , mp=0x55b7b7d17360 , file=0x55b7b709db48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b7b63d13ce in inline_mysql_cond_wait (that=0x55b7b7d13fe0 , mutex=0x55b7b7d17360 , src_file=0x55b7b709e6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055b7b63d2c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055b7b6f66471 in pfs_spawn_thread (arg=0x55b7b8ffcdc0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f793bfff700 (LWP 20088)): #0 0x00007f797b25d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b7b6b07ba6 in LinuxAIOHandler::collect (this=0x7f793bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b7b6b080cf in LinuxAIOHandler::poll (this=0x7f793bffeb00, m1=0x7f793bffeba8, m2=0x7f793bffebb0, request=0x7f793bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b7b6b08232 in os_aio_linux_handler (global_segment=0, m1=0x7f793bffeba8, m2=0x7f793bffebb0, request=0x7f793bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b7b6b0cd56 in os_aio_handler (segment=0, m1=0x7f793bffeba8, m2=0x7f793bffebb0, request=0x7f793bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055b7b6d60bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055b7b6c0612f in io_handler_thread (arg=0x55b7b7d8f420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f7921ffb700 (LWP 20108)): #0 0x00007f797b693e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055b7b6b14b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055b7b6cbb755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f79337fe700 (LWP 20097)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b141a4 in os_event::timed_wait (this=0x55b7b8e607c0, abstime=0x7f79337fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b7b6b144bb in os_event::wait_time_low (this=0x55b7b8e607c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b7b6b1465b in os_event_wait_time_low (event=0x55b7b8e607c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b7b6ae1314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f7923fff700 (LWP 20104)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b14889 in os_event::wait (this=0x55b7b8c881f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b7b6b1430f in os_event::wait_low (this=0x55b7b8c881f0, reset_sig_count=16517) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b7b6b14680 in os_event_wait_low (event=0x55b7b8c881f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b7b6c005ff in srv_resume_thread (slot=0x55b7b7b515f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055b7b6c04203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f7970077700 (LWP 20111)): #0 0x00007f797998440a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055b7b639fe24 in my_sigwait (set=0x7f7970076cf0, sig=0x7f7970076cb4, code=0x7f7970076cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055b7b63a6464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055b7b6f66471 in pfs_spawn_thread (arg=0x55b7b8c4bf10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f7930ff9700 (LWP 20103)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b14889 in os_event::wait (this=0x55b7b8c88070) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b7b6b1430f in os_event::wait_low (this=0x55b7b8c88070, reset_sig_count=4933) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b7b6b14680 in os_event_wait_low (event=0x55b7b8c88070, reset_sig_count=4933) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b7b6c005ff in srv_resume_thread (slot=0x55b7b7b51548 , sig_count=4933, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055b7b6c04610 in srv_purge_coordinator_suspend (slot=0x55b7b7b51548 , rseg_history_len=1) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x000055b7b6c048a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f79227fc700 (LWP 20107)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b14889 in os_event::wait (this=0x55b7b8c88370) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b7b6b1430f in os_event::wait_low (this=0x55b7b8c88370, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b7b6b14680 in os_event_wait_low (event=0x55b7b8c88370, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b7b6ce0dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f7920ff9700 (LWP 20110)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6fd39a9 in safe_cond_wait (cond=0x7f7920ff8c70, mp=0x7f7920ff8cb0, file=0x55b7b72a0860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b7b6a2ba13 in inline_mysql_cond_wait (that=0x7f7920ff8c70, mutex=0x7f7920ff8cb0, src_file=0x55b7b72a23a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b7b6a30632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000055b7b6f66471 in pfs_spawn_thread (arg=0x55b7b8e42fb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f79317fa700 (LWP 20102)): #0 0x00007f797b693e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055b7b6b14b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055b7b6c03b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x000055b7b6c03c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f7960751700 (LWP 20115)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6fd3cab in safe_cond_timedwait (cond=0x55b7b7d13fa0 , mp=0x55b7b7d169a0 , abstime=0x7f7960750c60, file=0x55b7b708e348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b7b63a0725 in inline_mysql_cond_timedwait (that=0x55b7b7d13fa0 , mutex=0x55b7b7d169a0 , abstime=0x7f7960750c60, src_file=0x55b7b70903b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b7b63a54fb in cache_thread (thd=0x7f793c00a040) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055b7b63a57a9 in one_thread_per_connection_end (thd=0x7f793c00a040, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055b7b6640cda in do_handle_one_connection (connect=0x55b7b8fc5720) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x000055b7b6640994 in handle_one_connection (arg=0x55b7b8fc5720) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000055b7b6f66471 in pfs_spawn_thread (arg=0x55b7b900f870) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f79217fa700 (LWP 20109)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b14889 in os_event::wait (this=0x55b7b8c88470) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b7b6b1430f in os_event::wait_low (this=0x55b7b8c88470, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b7b6b14680 in os_event_wait_low (event=0x55b7b8c88470, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b7b6cc98b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f79327fc700 (LWP 20100)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b141a4 in os_event::timed_wait (this=0x55b7b8e927e0, abstime=0x7f79327fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b7b6b144bb in os_event::wait_time_low (this=0x55b7b8e927e0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b7b6b1465b in os_event_wait_time_low (event=0x55b7b8e927e0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b7b6d4a52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f79237fe700 (LWP 20105)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b14889 in os_event::wait (this=0x55b7b8c880f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b7b6b1430f in os_event::wait_low (this=0x55b7b8c880f0, reset_sig_count=24196) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b7b6b14680 in os_event_wait_low (event=0x55b7b8c880f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b7b6c005ff in srv_resume_thread (slot=0x55b7b7b51580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055b7b6c04203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f7933fff700 (LWP 20098)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b141a4 in os_event::timed_wait (this=0x55b7b8c88270, abstime=0x7f7933ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b7b6b144bb in os_event::wait_time_low (this=0x55b7b8c88270, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b7b6b1465b in os_event_wait_time_low (event=0x55b7b8c88270, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b7b6c02808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f7938ff9700 (LWP 20094)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b141a4 in os_event::timed_wait (this=0x55b7b8c883f0, abstime=0x7f7938ff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b7b6b144bb in os_event::wait_time_low (this=0x55b7b8c883f0, time_in_usec=682000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b7b6b1465b in os_event_wait_time_low (event=0x55b7b8c883f0, time_in_usec=682000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b7b6ceac57 in pc_sleep_if_needed (next_loop_time=1549295991042, sig_count=2, cur_time=1549295990360) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x000055b7b6cebb90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f79397fa700 (LWP 20093)): #0 0x00007f797b25d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b7b6b07ba6 in LinuxAIOHandler::collect (this=0x7f79397f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b7b6b080cf in LinuxAIOHandler::poll (this=0x7f79397f9b00, m1=0x7f79397f9ba8, m2=0x7f79397f9bb0, request=0x7f79397f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b7b6b08232 in os_aio_linux_handler (global_segment=5, m1=0x7f79397f9ba8, m2=0x7f79397f9bb0, request=0x7f79397f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b7b6b0cd56 in os_aio_handler (segment=5, m1=0x7f79397f9ba8, m2=0x7f79397f9bb0, request=0x7f79397f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055b7b6d60bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055b7b6c0612f in io_handler_thread (arg=0x55b7b7d8f448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f793a7fc700 (LWP 20091)): #0 0x00007f797b25d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b7b6b07ba6 in LinuxAIOHandler::collect (this=0x7f793a7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b7b6b080cf in LinuxAIOHandler::poll (this=0x7f793a7fbb00, m1=0x7f793a7fbba8, m2=0x7f793a7fbbb0, request=0x7f793a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b7b6b08232 in os_aio_linux_handler (global_segment=3, m1=0x7f793a7fbba8, m2=0x7f793a7fbbb0, request=0x7f793a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b7b6b0cd56 in os_aio_handler (segment=3, m1=0x7f793a7fbba8, m2=0x7f793a7fbbb0, request=0x7f793a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055b7b6d60bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055b7b6c0612f in io_handler_thread (arg=0x55b7b7d8f438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f793affd700 (LWP 20090)): #0 0x00007f797b25d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b7b6b07ba6 in LinuxAIOHandler::collect (this=0x7f793affcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b7b6b080cf in LinuxAIOHandler::poll (this=0x7f793affcb00, m1=0x7f793affcba8, m2=0x7f793affcbb0, request=0x7f793affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b7b6b08232 in os_aio_linux_handler (global_segment=2, m1=0x7f793affcba8, m2=0x7f793affcbb0, request=0x7f793affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b7b6b0cd56 in os_aio_handler (segment=2, m1=0x7f793affcba8, m2=0x7f793affcbb0, request=0x7f793affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055b7b6d60bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055b7b6c0612f in io_handler_thread (arg=0x55b7b7d8f430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f7939ffb700 (LWP 20092)): #0 0x00007f797b25d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b7b6b07ba6 in LinuxAIOHandler::collect (this=0x7f7939ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b7b6b080cf in LinuxAIOHandler::poll (this=0x7f7939ffab00, m1=0x7f7939ffaba8, m2=0x7f7939ffabb0, request=0x7f7939ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b7b6b08232 in os_aio_linux_handler (global_segment=4, m1=0x7f7939ffaba8, m2=0x7f7939ffabb0, request=0x7f7939ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b7b6b0cd56 in os_aio_handler (segment=4, m1=0x7f7939ffaba8, m2=0x7f7939ffabb0, request=0x7f7939ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055b7b6d60bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055b7b6c0612f in io_handler_thread (arg=0x55b7b7d8f440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f7922ffd700 (LWP 20106)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b14889 in os_event::wait (this=0x55b7b8c88170) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055b7b6b1430f in os_event::wait_low (this=0x55b7b8c88170, reset_sig_count=19541) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055b7b6b14680 in os_event_wait_low (event=0x55b7b8c88170, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055b7b6c005ff in srv_resume_thread (slot=0x55b7b7b515b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055b7b6c04203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f7961dee700 (LWP 20087)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6fd3cab in safe_cond_timedwait (cond=0x55b7b85a0ca0 , mp=0x55b7b85a0be0 , abstime=0x7f7961dedcb0, file=0x55b7b7419130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b7b6e3dadd in inline_mysql_cond_timedwait (that=0x55b7b85a0ca0 , mutex=0x55b7b85a0be0 , abstime=0x7f7961dedcb0, src_file=0x55b7b7419178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b7b6e3e001 in my_service_thread_sleep (control=0x55b7b7b55900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055b7b6e3249a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055b7b6f66471 in pfs_spawn_thread (arg=0x55b7b8c406c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f793b7fe700 (LWP 20089)): #0 0x00007f797b25d644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055b7b6b07ba6 in LinuxAIOHandler::collect (this=0x7f793b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055b7b6b080cf in LinuxAIOHandler::poll (this=0x7f793b7fdb00, m1=0x7f793b7fdba8, m2=0x7f793b7fdbb0, request=0x7f793b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055b7b6b08232 in os_aio_linux_handler (global_segment=1, m1=0x7f793b7fdba8, m2=0x7f793b7fdbb0, request=0x7f793b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055b7b6b0cd56 in os_aio_handler (segment=1, m1=0x7f793b7fdba8, m2=0x7f793b7fdbb0, request=0x7f793b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055b7b6d60bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055b7b6c0612f in io_handler_thread (arg=0x55b7b7d8f428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f7932ffd700 (LWP 20099)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6b141a4 in os_event::timed_wait (this=0x55b7b8c882f0, abstime=0x7f7932ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055b7b6b144bb in os_event::wait_time_low (this=0x55b7b8c882f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055b7b6b1465b in os_event_wait_time_low (event=0x55b7b8c882f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055b7b6c022e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f79702b4700 (LWP 19922)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6fd39a9 in safe_cond_wait (cond=0x7f7950015628, mp=0x7f7950015578, file=0x55b7b713ca00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b7b6729d06 in inline_mysql_cond_wait (that=0x7f7950015628, mutex=0x7f7950015578, src_file=0x55b7b713cd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b7b672c99e in Wsrep_thd_queue::pop_front (this=0x7f7950015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b7b672b3be in wsrep_rollback_process (rollbacker=0x7f7950000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055b7b671bf01 in start_wsrep_THD (arg=0x55b7b8b99b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f797021e700 (LWP 19924)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f797348f132 in fifo_lock_get (q=q@entry=0x55b7b8b69a30) at galerautils/src/gu_fifo.c:262 #2 0x00007f797348f7a1 in gu_fifo_get_head (q=0x55b7b8b69a30, err=0x7f797021d06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f79735710cc in gcs_recv (conn=0x55b7b8b433b0, action=0x7f797021d0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f797359e16c in galera::GcsActionSource::process (this=0x55b7b8b6fdd0, recv_ctx=0x7f797021d4d0, exit_loop=@0x7f797021d29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f79735bd033 in galera::ReplicatorSMM::async_recv (this=0x55b7b8b3b160, recv_ctx=0x7f797021d4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055b7b708720e in wsrep::wsrep_provider_v26::run_applier (this=0x55b7b8ab0d80, applier_ctx=0x7f797021d4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x000055b7b672a2b4 in wsrep_replication_process (thd=0x7f7958001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055b7b671bf01 in start_wsrep_THD (arg=0x55b7b8b99660) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f7970ab5700 (LWP 19920)): #0 0x00007f7979a4b483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f797353c6f8 in asio::detail::epoll_reactor::run (this=0x55b7b8b77f70, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f7973540df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55b7b8b77e10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f79735410a2 in asio::detail::task_io_service::run (this=0x55b7b8b77e10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f7973558afd in asio::io_service::run (this=0x55b7b8b76b30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f7973559251 in gcomm::AsioProtonet::event_loop (this=0x55b7b8b76a90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f79735772ed in GCommConn::run (this=0x55b7b8b775a0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f797357a399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f7972f39700 (LWP 19919)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f797359afd7 in gu::Lock::wait (cond=..., this=0x7f7972f38bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f797359b069 in galera::ServiceThd::thd_func (arg=0x55b7b8b3b878) at galera/src/galera_service_thd.cpp:30 #3 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f7963ffe700 (LWP 19921)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000worker[1] Trying to dump core for [mysqltest - pid: 20883, winpid: 20883] worker[1] Trying to dump core for [mysqld.1 - pid: 20341, winpid: 20341] worker[1] Trying to dump core for [mysqld.2 - pid: 20636, winpid: 20636] 7f7973575867 in gu::Lock::wait (cond=..., this=0x7f7963ffd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f79735783ab in RecvBuf::front (this=this@entry=0x55b7b8b77788, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f7973575f46 in gcomm_recv (backend=, msg=0x55b7b8b43690, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f7973569eec in core_msg_recv (backend=backend@entry=0x55b7b8b437d8, recv_msg=recv_msg@entry=0x55b7b8b43690, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f797356d164 in gcs_core_recv (conn=0x55b7b8b43630, recv_act=recv_act@entry=0x7f7963ffdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f79735743d8 in gcs_recv_thread (arg=0x55b7b8b433b0) at gcs/src/gcs.cpp:1398 #7 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f7970269700 (LWP 19923)): #0 0x00007f797b690965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6fd39a9 in safe_cond_wait (cond=0x7f794c015628, mp=0x7f794c015578, file=0x55b7b713ca00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055b7b6729d06 in inline_mysql_cond_wait (that=0x7f794c015628, mutex=0x7f794c015578, src_file=0x55b7b713cd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055b7b672c99e in Wsrep_thd_queue::pop_front (this=0x7f794c015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055b7b672b77e in wsrep_post_rollback_process (post_rollbacker=0x7f794c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055b7b671bf01 in start_wsrep_THD (arg=0x55b7b8b98670) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f79740e5700 (LWP 19910)): #0 0x00007f797b690d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055b7b6fd3cab in safe_cond_timedwait (cond=0x55b7b85b3440 , mp=0x55b7b85b3380 , abstime=0x7f79740e4d70, file=0x55b7b745d360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055b7b6fd4c65 in inline_mysql_cond_timedwait (that=0x55b7b85b3440 , mutex=0x55b7b85b3380 , abstime=0x7f79740e4d70, src_file=0x55b7b745d3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055b7b6fd57c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055b7b6f66471 in pfs_spawn_thread (arg=0x55b7b8b04700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f797b68cdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7979a4aead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f797ba9d880 (LWP 19908)): #0 0x00007f797b6919d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055b7b6fce37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055b7b680ad65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f7979a4020d in poll () from /lib64/libc.so.6 #5 0x000055b7b63ac96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055b7b63abc05 in mysqld_main (argc=162, argv=0x55b7b89bcd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055b7b639fda5 in main (argc=23, argv=0x7fff8b31e7a8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#8-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#8 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.mysql-wsrep-features#9 'innodb' [ pass ] 21746 galera_sr.GCF-1008 'innodb' [ pass ] 8099 galera_sr.GCF-1018 'innodb' [ pass ] 4157 galera_sr.GCF-1043A 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 19:17:28 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/GCF-1043A.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 REPLACE INTO t1 VALUES (1,'y'),(2,'x') REPLACE INTO t1 VALUES (1,'y'),(2,'y'),(3,'y') connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) - found 'core.20342' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043A-innodb/mysqld.1/data/core.20342 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 20342] [New LWP 20589] [New LWP 20359] [New LWP 20364] [New LWP 20349] [New LWP 20367] [New LWP 20362] [New LWP 20348] [New LWP 20366] [New LWP 20363] [New LWP 20380] [New LWP 20365] [New LWP 20373] [New LWP 20885] [New LWP 20369] [New LWP 20376] [New LWP 20371] [New LWP 20379] [New LWP 20355] [New LWP 20353] [New LWP 20374] [New LWP 20351] [New LWP 20358] [New LWP 20347] [New LWP 20357] [New LWP 20356] [New LWP 20346] [New LWP 20375] [New LWP 20372] [New LWP 20352] [New LWP 20370] [New LWP 20350] [New LWP 20368] [New LWP 20354] [New LWP 20360] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f1f95a279d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f1f95a279d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005569e7d3837a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005569e7574d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f1f93dd620d in poll () from /lib64/libc.so.6 #5 0x00005569e711696a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005569e7115c05 in mysqld_main (argc=161, argv=0x5569e9e1fd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005569e7109da5 in main (argc=23, argv=0x7ffe5509d3b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f1f637fe700 (LWP 20360)): #0 0x00007f1f95a29d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005569e7873255 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005569e78735ab in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005569e7abc751 in pfs_os_file_flush_func (file=..., src_file=0x5569e81388b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005569e7abf593 in fil_flush_low (space=0x5569ea2efc40, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x00005569e7acb113 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x00005569e7a47b29 in buf_dblwr_flush_buffered_writes () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:1073 #7 0x00005569e7a522df in buf_flush_end (buf_pool=0x5569ea1cbe90, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1879 #8 0x00005569e7a5249d in buf_flush_do_batch (buf_pool=0x5569ea1cbe90, type=BUF_FLUSH_LIST, min_n=10, lsn_limit=18446744073709551615, n=0x7f1f637fdb70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1948 #9 0x00005569e7a55187 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2749 #10 0x00005569e7a560d5 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3158 #11 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f1f6edae700 (LWP 20354)): #0 0x00007f1f955f3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005569e7871ba6 in LinuxAIOHandler::collect (this=0x7f1f6edadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005569e78720cf in LinuxAIOHandler::poll (this=0x7f1f6edadb00, m1=0x7f1f6edadba8, m2=0x7f1f6edadbb0, request=0x7f1f6edadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005569e7872232 in os_aio_linux_handler (global_segment=0, m1=0x7f1f6edadba8, m2=0x7f1f6edadbb0, request=0x7f1f6edadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005569e7876d56 in os_aio_handler (segment=0, m1=0x7f1f6edadba8, m2=0x7f1f6edadbb0, request=0x7f1f6edadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005569e7acabc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005569e797012f in io_handler_thread (arg=0x5569e8af9420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f1f537fe700 (LWP 20368)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e889 in os_event::wait (this=0x5569ea0e6390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005569e787e30f in os_event::wait_low (this=0x5569ea0e6390, reset_sig_count=5554) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005569e787e680 in os_event_wait_low (event=0x5569ea0e6390, reset_sig_count=5554) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005569e796a5ff in srv_resume_thread (slot=0x5569e88bb548 , sig_count=5554, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005569e796e610 in srv_purge_coordinator_suspend (slot=0x5569e88bb548 , rseg_history_len=35) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x00005569e796e8a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f1f8c0cd700 (LWP 20350)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3d9a9 in safe_cond_wait (cond=0x7f1f7c017b88, mp=0x7f1f7c017ad8, file=0x5569e7ea6a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005569e7493d06 in inline_mysql_cond_wait (that=0x7f1f7c017b88, mutex=0x7f1f7c017ad8, src_file=0x5569e7ea6d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005569e749699e in Wsrep_thd_queue::pop_front (this=0x7f1f7c017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005569e74953be in wsrep_rollback_process (rollbacker=0x7f1f7c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005569e7485f01 in start_wsrep_THD (arg=0x5569e9fee270) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f1f527fc700 (LWP 20370)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e889 in os_event::wait (this=0x5569ea0e6490) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005569e787e30f in os_event::wait_low (this=0x5569ea0e6490, reset_sig_count=21939) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005569e787e680 in os_event_wait_low (event=0x5569ea0e6490, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005569e796a5ff in srv_resume_thread (slot=0x5569e88bb5b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005569e796e203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f1f8537a700 (LWP 20352)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1f8d825132 in fifo_lock_get (q=q@entry=0x5569e9fccb20) at galerautils/src/gu_fifo.c:262 #2 0x00007f1f8d8257a1 in gu_fifo_get_head (q=0x5569e9fccb20, err=0x7f1f8537906c) at galerautils/src/gu_fifo.c:330 #3 0x00007f1f8d9070cc in gcs_recv (conn=0x5569e9fa64a0, action=0x7f1f853790b0) at gcs/src/gcs.cpp:2024 #4 0x00007f1f8d93416c in galera::GcsActionSource::process (this=0x5569e9fa5e80, recv_ctx=0x7f1f853794d0, exit_loop=@0x7f1f8537929e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f1f8d953033 in galera::ReplicatorSMM::async_recv (this=0x5569e9f9e1b0, recv_ctx=0x7f1f853794d0) at galera/src/replicator_smm.cpp:383 #6 0x00005569e7df120e in wsrep::wsrep_provider_v26::run_applier (this=0x5569e9f25280, applier_ctx=0x7f1f853794d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005569e74942b4 in wsrep_replication_process (thd=0x7f1f74000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005569e7485f01 in start_wsrep_THD (arg=0x5569e9ff6b50) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f1f517fa700 (LWP 20372)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e889 in os_event::wait (this=0x5569ea0e6690) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005569e787e30f in os_event::wait_low (this=0x5569ea0e6690, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005569e787e680 in os_event_wait_low (event=0x5569ea0e6690, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005569e7a4adde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f1f437fe700 (LWP 20375)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3d9a9 in safe_cond_wait (cond=0x7f1f437fdc70, mp=0x7f1f437fdcb0, file=0x5569e800a860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005569e7795a13 in inline_mysql_cond_wait (that=0x7f1f437fdc70, mutex=0x7f1f437fdcb0, src_file=0x5569e800c3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005569e779a632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569ea327240) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f1f8e47b700 (LWP 20346)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3dcab in safe_cond_timedwait (cond=0x5569e931d440 , mp=0x5569e931d380 , abstime=0x7f1f8e47ad70, file=0x5569e81c7360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005569e7d3ec65 in inline_mysql_cond_timedwait (that=0x5569e931d440 , mutex=0x5569e931d380 , abstime=0x7f1f8e47ad70, src_file=0x5569e81c73a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005569e7d3f7c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569e9f676f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f1f6ddac700 (LWP 20356)): #0 0x00007f1f955f3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005569e7871ba6 in LinuxAIOHandler::collect (this=0x7f1f6ddabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005569e78720cf in LinuxAIOHandler::poll (this=0x7f1f6ddabb00, m1=0x7f1f6ddabba8, m2=0x7f1f6ddabbb0, request=0x7f1f6ddabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005569e7872232 in os_aio_linux_handler (global_segment=2, m1=0x7f1f6ddabba8, m2=0x7f1f6ddabbb0, request=0x7f1f6ddabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005569e7876d56 in os_aio_handler (segment=2, m1=0x7f1f6ddabba8, m2=0x7f1f6ddabbb0, request=0x7f1f6ddabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005569e7acabc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005569e797012f in io_handler_thread (arg=0x5569e8af9430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f1f6d5ab700 (LWP 20357)): #0 0x00007f1f955f3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005569e7871ba6 in LinuxAIOHandler::collect (this=0x7f1f6d5aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005569e78720cf in LinuxAIOHandler::poll (this=0x7f1f6d5aab00, m1=0x7f1f6d5aaba8, m2=0x7f1f6d5aabb0, request=0x7f1f6d5aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005569e7872232 in os_aio_linux_handler (global_segment=3, m1=0x7f1f6d5aaba8, m2=0x7f1f6d5aabb0, request=0x7f1f6d5aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005569e7876d56 in os_aio_handler (segment=3, m1=0x7f1f6d5aaba8, m2=0x7f1f6d5aabb0, request=0x7f1f6d5aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005569e7acabc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005569e797012f in io_handler_thread (arg=0x5569e8af9438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f1f8c8ce700 (LWP 20347)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1f8d930fd7 in gu::Lock::wait (cond=..., this=0x7f1f8c8cdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f1f8d931069 in galera::ServiceThd::thd_func (arg=0x5569e9f9e8c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f1f6cdaa700 (LWP 20358)): #0 0x00007f1f955f3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005569e7871ba6 in LinuxAIOHandler::collect (this=0x7f1f6cda9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005569e78720cf in LinuxAIOHandler::poll (this=0x7f1f6cda9b00, m1=0x7f1f6cda9ba8, m2=0x7f1f6cda9bb0, request=0x7f1f6cda9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005569e7872232 in os_aio_linux_handler (global_segment=4, m1=0x7f1f6cda9ba8, m2=0x7f1f6cda9bb0, request=0x7f1f6cda9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005569e7876d56 in os_aio_handler (segment=4, m1=0x7f1f6cda9ba8, m2=0x7f1f6cda9bb0, request=0x7f1f6cda9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005569e7acabc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005569e797012f in io_handler_thread (arg=0x5569e8af9440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f1f8c082700 (LWP 20351)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3d9a9 in safe_cond_wait (cond=0x7f1f70015628, mp=0x7f1f70015578, file=0x5569e7ea6a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005569e7493d06 in inline_mysql_cond_wait (that=0x7f1f70015628, mutex=0x7f1f70015578, src_file=0x5569e7ea6d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005569e749699e in Wsrep_thd_queue::pop_front (this=0x7f1f70015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005569e749577e in wsrep_post_rollback_process (post_rollbacker=0x7f1f70000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005569e7485f01 in start_wsrep_THD (arg=0x5569e9fd8760) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f1f43fff700 (LWP 20374)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e889 in os_event::wait (this=0x5569ea0e6790) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005569e787e30f in os_event::wait_low (this=0x5569ea0e6790, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005569e787e680 in os_event_wait_low (event=0x5569ea0e6790, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005569e7a338b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f1f6ffff700 (LWP 20353)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3dcab in safe_cond_timedwait (cond=0x5569e930aca0 , mp=0x5569e930abe0 , abstime=0x7f1f6fffecb0, file=0x5569e8183130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005569e7ba7add in inline_mysql_cond_timedwait (that=0x5569e930aca0 , mutex=0x5569e930abe0 , abstime=0x7f1f6fffecb0, src_file=0x5569e8183178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005569e7ba8001 in my_service_thread_sleep (control=0x5569e88bf900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005569e7b9c49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569ea09e710) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f1f6e5ad700 (LWP 20355)): #0 0x00007f1f955f3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005569e7871ba6 in LinuxAIOHandler::collect (this=0x7f1f6e5acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005569e78720cf in LinuxAIOHandler::poll (this=0x7f1f6e5acb00, m1=0x7f1f6e5acba8, m2=0x7f1f6e5acbb0, request=0x7f1f6e5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005569e7872232 in os_aio_linux_handler (global_segment=1, m1=0x7f1f6e5acba8, m2=0x7f1f6e5acbb0, request=0x7f1f6e5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005569e7876d56 in os_aio_handler (segment=1, m1=0x7f1f6e5acba8, m2=0x7f1f6e5acbb0, request=0x7f1f6e5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005569e7acabc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005569e797012f in io_handler_thread (arg=0x5569e8af9428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f1f841ab700 (LWP 20379)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3d9a9 in safe_cond_wait (cond=0x5569e8a7dfe0 , mp=0x5569e8a81360 , file=0x5569e7e07b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005569e713b3ce in inline_mysql_cond_wait (that=0x5569e8a7dfe0 , mutex=0x5569e8a81360 , src_file=0x5569e7e086b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005569e713cc60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569ea3e7c00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f1f51ffb700 (LWP 20371)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e889 in os_event::wait (this=0x5569ea0e6510) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005569e787e30f in os_event::wait_low (this=0x5569ea0e6510, reset_sig_count=19048) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005569e787e680 in os_event_wait_low (event=0x5569ea0e6510, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005569e796a5ff in srv_resume_thread (slot=0x5569e88bb5f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005569e796e203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f1f841f6700 (LWP 20376)): #0 0x00007f1f93d1a40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005569e7109e24 in my_sigwait (set=0x7f1f841f5cf0, sig=0x7f1f841f5cb4, code=0x7f1f841f5cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005569e7110464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569ea0a9ff0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f1f52ffd700 (LWP 20369)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e889 in os_event::wait (this=0x5569ea0e6410) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005569e787e30f in os_event::wait_low (this=0x5569ea0e6410, reset_sig_count=25766) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005569e787e680 in os_event_wait_low (event=0x5569ea0e6410, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005569e796a5ff in srv_resume_thread (slot=0x5569e88bb580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005569e796e203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f1f840ca700 (LWP 20885)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3dcab in safe_cond_timedwait (cond=0x5569e8a7dfa0 , mp=0x5569e8a809a0 , abstime=0x7f1f840c9c60, file=0x5569e7df8348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005569e710a725 in inline_mysql_cond_timedwait (that=0x5569e8a7dfa0 , mutex=0x5569e8a809a0 , abstime=0x7f1f840c9c60, src_file=0x5569e7dfa3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005569e710f4fb in cache_thread (thd=0x7f1f14000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005569e710f7a9 in one_thread_per_connection_end (thd=0x7f1f14000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005569e73aacda in do_handle_one_connection (connect=0x5569ea3e4b30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x00005569e73aa994 in handle_one_connection (arg=0x5569ea3e4b30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569ea3c96d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f1f50ff9700 (LWP 20373)): #0 0x00007f1f95a29e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005569e787eb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005569e7a25755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f1f617fa700 (LWP 20365)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e1a4 in os_event::timed_wait (this=0x5569ea2f06c0, abstime=0x7f1f617f9d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005569e787e4bb in os_event::wait_time_low (this=0x5569ea2f06c0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005569e787e65b in os_event_wait_time_low (event=0x5569ea2f06c0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005569e7ab452c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f1f84160700 (LWP 20380)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3dcab in safe_cond_timedwait (cond=0x5569e8a7dfa0 , mp=0x5569e8a809a0 , abstime=0x7f1f8415fc60, file=0x5569e7df8348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005569e710a725 in inline_mysql_cond_timedwait (that=0x5569e8a7dfa0 , mutex=0x5569e8a809a0 , abstime=0x7f1f8415fc60, src_file=0x5569e7dfa3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005569e710f4fb in cache_thread (thd=0x7f1f24000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005569e710f7a9 in one_thread_per_connection_end (thd=0x7f1f24000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005569e73aacda in do_handle_one_connection (connect=0x5569ea45a450) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x00005569e73aa994 in handle_one_connection (arg=0x5569ea45a450) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569ea477790) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f1f61ffb700 (LWP 20363)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e1a4 in os_event::timed_wait (this=0x5569ea0e6590, abstime=0x7f1f61ffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005569e787e4bb in os_event::wait_time_low (this=0x5569ea0e6590, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005569e787e65b in os_event_wait_time_low (event=0x5569ea0e6590, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005569e796c808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f1f60ff9700 (LWP 20366)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e1a4 in os_event::timed_wait (this=0x5569ea2861a0, abstime=0x7f1f60ff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005569e787e4bb in os_event::wait_time_low (this=0x5569ea2861a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005569e787e65b in os_event_wait_time_low (event=0x5569ea2861a0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005569e79ccaad in ib_wqueue_timedwait (wq=0x5569ea2860f0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005569e7b0ef11 in fts_optimize_thread (arg=0x5569ea2860f0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f1f8637c700 (LWP 20348)): #0 0x00007f1f93de1483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f1f8d8d26f8 in asio::detail::epoll_reactor::run (this=0x5569e9fdef90, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f1f8d8d6df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5569e9fdee30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f1f8d8d70a2 in asio::detail::task_io_service::run (this=0x5569e9fdee30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f1f8d8eeafd in asio::io_service::run (this=0x5569e9fddf30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f1f8d8ef251 in gcomm::AsioProtonet::event_loop (this=0x5569e9fdde90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f1f8d90d2ed in GCommConn::run (this=0x5569e9fde610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f1f8d910399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f1f627fc700 (LWP 20362)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e1a4 in os_event::timed_wait (this=0x5569ea2be6a0, abstime=0x7f1f627fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005569e787e4bb in os_event::wait_time_low (this=0x5569ea2be6a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005569e787e65b in os_event_wait_time_low (event=0x5569ea2be6a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005569e784b314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f1f53fff700 (LWP 20367)): #0 0x00007f1f95a29e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005569e787eb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005569e796db67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005569e796dc97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f1f85b7b700 (LWP 20349)): #0 0x00007f1f95a26965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f1f8d90b867 in gu::Lock::wait (cond=..., this=0x7f1f85b7a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f1f8d90e3ab in RecvBuf::front (this=this@entry=0x5569e9fde7f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f1f8d90bf46 in gcomm_recv (backend=, msg=0x5569e9fa6780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f1f8d8ffeec in core_msg_recv (backend=backend@entry=0x5569e9fa68c8, recv_msg=recv_msg@entry=0x5569e9fa6780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f1f8d903164 in gcs_core_recv (conn=0x5569e9fa6720, recv_act=recv_act@entry=0x7f1f85b7ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f1f8d90a3d8 in gcs_recv_thread (arg=0x5569e9fa64a0) at gcs/src/gcs.cpp:1398 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f1f62ffd700 (LWP 20364)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e787e1a4 in os_event::timed_wait (this=0x5569ea0e6610, abstime=0x7f1f62ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005569e787e4bb in os_event::wait_time_low (this=0x5569ea0e6610, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005569e787e65b in os_event_wait_time_low (event=0x5569ea0e6610, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005569e796c2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f1f63fff700 (LWP 20359)): #0 0x00007f1f955f3644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005569e7871ba6 in LinuxAIOHandler::collect (this=0x7f1f63ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005569e78720cf in LinuxAIOHandler::poll (this=0x7f1f63ffeb00, m1=0x7f1f63ffeba8, m2=0x7f1f63ffebb0, request=0x7f1f63ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005569e7872232 in os_aio_linux_handler (global_segment=5, m1=0x7f1f63ffeba8, m2=0x7f1f63ffebb0, request=0x7f1f63ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005569e7876d56 in os_aio_handler (segment=5, m1=0x7f1f63ffeba8, m2=0x7f1f63ffebb0, request=0x7f1f63ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005569e7acabc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005569e797012f in io_handler_thread (arg=0x5569e8af9448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f1f84115700 (LWP 20589)): #0 0x00007f1f95a26d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569e7d3dcab in safe_cond_timedwait (cond=0x5569e8a7dfa0 , mp=0x5569e8a809a0 , abstime=0x7f1f84114c60, file=0x5569e7df8348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005569e710a725 in inline_mysql_cond_timedwait (that=0x5569e8a7dfa0 , mutex=0x5569e8a809a0 , abstime=0x7f1f84114c60, src_file=0x5569e7dfa3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005569e710f4fb in cache_thread (thd=0x7f1f1c000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005569e710f7a9 in one_thread_per_connection_end (thd=0x7f1f1c000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005569e73aacda in do_handle_one_connection (connect=0x5569ea3e41d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x00005569e73aa994 in handle_one_connection (arg=0x5569ea3e41d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x00005569e7cd0471 in pfs_spawn_thread (arg=0x5569ea3e80a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f1f95a22dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f1f93de0ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f1f95e33880 (LWP 20342)): #0 0x00007f1f95a279d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005569e7d3837a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005569e7574d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f1f93dd620d in poll () from /lib64/libc.so.6 #5 0x00005569e711696a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005569e7115c05 in mysqld_main (argc=161, argv=0x5569e9e1fd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005569e7109da5 in main (argc=23, argv=0x7ffe5509d3b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.20637' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043A-innodb/mysqld.2/data/core.20637 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 20637] [New LWP 20638] [New LWP 20639] [New LWP 20640] [New LWP 20641] [New LWP 20643] [New LWP 20642] [New LWP 20818] [New LWP 20644] [New LWP 20802] [New LWP 20819] [New LWP 20809] [New LWP 20803] [New LWP 20820] [New LWP 20812] [New LWP 20821] [New LWP 20804] [New LWP 20813] [New LWP 20825] [New LWP 20806] [New LWP 20816] [New LWP 20817] [New LWP 20807] [New LWP 20814] [New LWP 20823] [New LWP 20827] [New LWP 20822] [New LWP 20829] [New LWP 20826] [New LWP 20824] [New LWP 20815] [New LWP 20805] [New LWP 20808] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f50adb669d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f50adb669d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000558b9620037a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000558b95a3cd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f50abf1520d in poll () from /lib64/libc.so.6 #5 0x0000558b955de96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000558b955ddc05 in mysqld_main (argc=162, argv=0x558b97c58d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000558b955d1da5 in main (argc=23, argv=0x7ffc01ed0888) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f506bfff700 (LWP 20808)): #0 0x00007f50ad732644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558b95d39ba6 in LinuxAIOHandler::collect (this=0x7f506bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558b95d3a0cf in LinuxAIOHandler::poll (this=0x7f506bffeb00, m1=0x7f506bffeba8, m2=0x7f506bffebb0, request=0x7f506bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558b95d3a232 in os_aio_linux_handler (global_segment=5, m1=0x7f506bffeba8, m2=0x7f506bffebb0, request=0x7f506bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558b95d3ed56 in os_aio_handler (segment=5, m1=0x7f506bffeba8, m2=0x7f506bffebb0, request=0x7f506bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558b95f92bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558b95e3812f in io_handler_thread (arg=0x558b96fc1448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f50719ab700 (LWP 20805)): #0 0x00007f50ad732644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558b95d39ba6 in LinuxAIOHandler::collect (this=0x7f50719aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558b95d3a0cf in LinuxAIOHandler::poll (this=0x7f50719aab00, m1=0x7f50719aaba8, m2=0x7f50719aabb0, request=0x7f50719aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558b95d3a232 in os_aio_linux_handler (global_segment=2, m1=0x7f50719aaba8, m2=0x7f50719aabb0, request=0x7f50719aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558b95d3ed56 in os_aio_handler (segment=2, m1=0x7f50719aaba8, m2=0x7f50719aabb0, request=0x7f50719aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558b95f92bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558b95e3812f in io_handler_thread (arg=0x558b96fc1430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f50695f5700 (LWP 20815)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d461a4 in os_event::timed_wait (this=0x558b98132460, abstime=0x7f50695f4d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558b95d464bb in os_event::wait_time_low (this=0x558b98132460, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558b95d4665b in os_event_wait_time_low (event=0x558b98132460, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558b95f7c52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f503ffff700 (LWP 20824)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d46889 in os_event::wait (this=0x558b97f28530) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558b95d4630f in os_event::wait_low (this=0x558b97f28530, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558b95d46680 in os_event_wait_low (event=0x558b97f28530, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558b95efb8b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f50a404e700 (LWP 20826)): #0 0x00007f50abe5940a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000558b955d1e24 in my_sigwait (set=0x7f50a404dcf0, sig=0x7f50a404dcb4, code=0x7f50a404dcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000558b955d8464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000558b96198471 in pfs_spawn_thread (arg=0x558b97eeab50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f5094056700 (LWP 20829)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b96205cab in safe_cond_timedwait (cond=0x558b96f45fa0 , mp=0x558b96f489a0 , abstime=0x7f5094055c60, file=0x558b962c0348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558b955d2725 in inline_mysql_cond_timedwait (that=0x558b96f45fa0 , mutex=0x558b96f489a0 , abstime=0x7f5094055c60, src_file=0x558b962c23b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000558b955d74fb in cache_thread (thd=0x7f5028000ba0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000558b955d77a9 in one_thread_per_connection_end (thd=0x7f5028000ba0, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000558b95872cda in do_handle_one_connection (connect=0x558b97eeef50) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000558b95872994 in handle_one_connection (arg=0x558b97eeef50) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x0000558b96198471 in pfs_spawn_thread (arg=0x558b982aff40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f50597fa700 (LWP 20822)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d46889 in os_event::wait (this=0x558b97f28430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558b95d4630f in os_event::wait_low (this=0x558b97f28430, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558b95d46680 in os_event_wait_low (event=0x558b97f28430, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558b95f12dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f50940ec700 (LWP 20827)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b962059a9 in safe_cond_wait (cond=0x558b96f45fe0 , mp=0x558b96f49360 , file=0x558b962cfb48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558b956033ce in inline_mysql_cond_wait (that=0x558b96f45fe0 , mutex=0x558b96f49360 , src_file=0x558b962d06b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000558b95604c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000558b96198471 in pfs_spawn_thread (arg=0x558b9822ae00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f5058ff9700 (LWP 20823)): #0 0x00007f50adb68e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000558b95d46b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000558b95eed755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f5069df6700 (LWP 20814)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d461a4 in os_event::timed_wait (this=0x558b97f283b0, abstime=0x7f5069df5d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558b95d464bb in os_event::wait_time_low (this=0x558b97f283b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558b95d4665b in os_event_wait_time_low (event=0x558b97f283b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558b95e342e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f50709a9700 (LWP 20807)): #0 0x00007f50ad732644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558b95d39ba6 in LinuxAIOHandler::collect (this=0x7f50709a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558b95d3a0cf in LinuxAIOHandler::poll (this=0x7f50709a8b00, m1=0x7f50709a8ba8, m2=0x7f50709a8bb0, request=0x7f50709a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558b95d3a232 in os_aio_linux_handler (global_segment=4, m1=0x7f50709a8ba8, m2=0x7f50709a8bb0, request=0x7f50709a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558b95d3ed56 in os_aio_handler (segment=4, m1=0x7f50709a8ba8, m2=0x7f50709a8bb0, request=0x7f50709a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558b95f92bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558b95e3812f in io_handler_thread (arg=0x558b96fc1440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f505bfff700 (LWP 20817)): #0 0x00007f50adb68e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000558b95d46b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000558b95e35b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x0000558b95e35c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f5068df4700 (LWP 20816)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d461a4 in os_event::timed_wait (this=0x558b980d07b0, abstime=0x7f5068df3b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558b95d464bb in os_event::wait_time_low (this=0x558b980d07b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558b95d4665b in os_event_wait_time_low (event=0x558b980d07b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558b95e94aad in ib_wqueue_timedwait (wq=0x558b980d0680, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000558b95fd6f11 in fts_optimize_thread (arg=0x558b980d0680) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f50711aa700 (LWP 20806)): #0 0x00007f50ad732644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558b95d39ba6 in LinuxAIOHandler::collect (this=0x7f50711a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558b95d3a0cf in LinuxAIOHandler::poll (this=0x7f50711a9b00, m1=0x7f50711a9ba8, m2=0x7f50711a9bb0, request=0x7f50711a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558b95d3a232 in os_aio_linux_handler (global_segment=3, m1=0x7f50711a9ba8, m2=0x7f50711a9bb0, request=0x7f50711a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558b95d3ed56 in os_aio_handler (segment=3, m1=0x7f50711a9ba8, m2=0x7f50711a9bb0, request=0x7f50711a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558b95f92bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558b95e3812f in io_handler_thread (arg=0x558b96fc1438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f503f7fe700 (LWP 20825)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b962059a9 in safe_cond_wait (cond=0x7f503f7fdc70, mp=0x7f503f7fdcb0, file=0x558b964d2860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558b95c5da13 in inline_mysql_cond_wait (that=0x7f503f7fdc70, mutex=0x7f503f7fdcb0, src_file=0x558b964d43a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000558b95c62632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x0000558b96198471 in pfs_spawn_thread (arg=0x558b9816a9a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f506adf8700 (LWP 20813)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d461a4 in os_event::timed_wait (this=0x558b97f28330, abstime=0x7f506adf7ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558b95d464bb in os_event::wait_time_low (this=0x558b97f28330, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558b95d4665b in os_event_wait_time_low (event=0x558b97f28330, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558b95e34808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f50721ac700 (LWP 20804)): #0 0x00007f50ad732644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558b95d39ba6 in LinuxAIOHandler::collect (this=0x7f50721abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558b95d3a0cf in LinuxAIOHandler::poll (this=0x7f50721abb00, m1=0x7f50721abba8, m2=0x7f50721abbb0, request=0x7f50721abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558b95d3a232 in os_aio_linux_handler (global_segment=1, m1=0x7f50721abba8, m2=0x7f50721abbb0, request=0x7f50721abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558b95d3ed56 in os_aio_handler (segment=1, m1=0x7f50721abba8, m2=0x7f50721abbb0, request=0x7f50721abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558b95f92bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558b95e3812f in io_handler_thread (arg=0x558b96fc1428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f5059ffb700 (LWP 20821)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d46889 in os_event::wait (this=0x558b97f282b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558b95d4630f in os_event::wait_low (this=0x558b97f282b0, reset_sig_count=15658) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558b95d46680 in os_event_wait_low (event=0x558b97f282b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558b95e325ff in srv_resume_thread (slot=0x558b96d835f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558b95e36203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f506a5f7700 (LWP 20812)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d461a4 in os_event::timed_wait (this=0x558b98100440, abstime=0x7f506a5f6d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558b95d464bb in os_event::wait_time_low (this=0x558b98100440, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558b95d4665b in os_event_wait_time_low (event=0x558b98100440, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558b95d13314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f505a7fc700 (LWP 20820)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d46889 in os_event::wait (this=0x558b97f281b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558b95d4630f in os_event::wait_low (this=0x558b97f281b0, reset_sig_count=22376) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558b95d46680 in os_event_wait_low (event=0x558b97f281b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558b95e325ff in srv_resume_thread (slot=0x558b96d83580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558b95e36203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f50729ad700 (LWP 20803)): #0 0x00007f50ad732644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000558b95d39ba6 in LinuxAIOHandler::collect (this=0x7f50729acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000558b95d3a0cf in LinuxAIOHandler::poll (this=0x7f50729acb00, m1=0x7f50729acba8, m2=0x7f50729acbb0, request=0x7f50729acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000558b95d3a232 in os_aio_linux_handler (global_segment=0, m1=0x7f50729acba8, m2=0x7f50729acbb0, request=0x7f50729acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000558b95d3ed56 in os_aio_handler (segment=0, m1=0x7f50729acba8, m2=0x7f50729acbb0, request=0x7f50729acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000558b95f92bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000558b95e3812f in io_handler_thread (arg=0x558b96fc1420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f506b7fe700 (LWP 20809)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d461a4 in os_event::timed_wait (this=0x558b97f284b0, abstime=0x7f506b7fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000558b95d464bb in os_event::wait_time_low (this=0x558b97f284b0, time_in_usec=500000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000558b95d4665b in os_event_wait_time_low (event=0x558b97f284b0, time_in_usec=500000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000558b95f1cc57 in pc_sleep_if_needed (next_loop_time=1549296995033, sig_count=2, cur_time=1549296994533) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x0000558b95f1db90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f505affd700 (LWP 20819)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d46889 in os_event::wait (this=0x558b97f28230) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558b95d4630f in os_event::wait_low (this=0x558b97f28230, reset_sig_count=19025) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558b95d46680 in os_event_wait_low (event=0x558b97f28230, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558b95e325ff in srv_resume_thread (slot=0x558b96d835b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558b95e36203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f5073fff700 (LWP 20802)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b96205cab in safe_cond_timedwait (cond=0x558b977d2ca0 , mp=0x558b977d2be0 , abstime=0x7f5073ffecb0, file=0x558b9664b130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558b9606fadd in inline_mysql_cond_timedwait (that=0x558b977d2ca0 , mutex=0x558b977d2be0 , abstime=0x7f5073ffecb0, src_file=0x558b9664b178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000558b96070001 in my_service_thread_sleep (control=0x558b96d87900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000558b9606449a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000558b96198471 in pfs_spawn_thread (arg=0x558b97ee0690) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f50a4276700 (LWP 20644)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f50a5964132 in fifo_lock_get (q=q@entry=0x558b97e08e80) at galerautils/src/gu_fifo.c:262 #2 0x00007f50a59647a1 in gu_fifo_get_head (q=0x558b97e08e80, err=0x7f50a427506c) at galerautils/src/gu_fifo.c:330 #3 0x00007f50a5a460cc in gcs_recv (conn=0x558b97de2800, action=0x7f50a42750b0) at gcs/src/gcs.cpp:2024 #4 0x00007f50a5a7316c in galera::GcsActionSource::process (this=0x558b97ddea50, recv_ctx=0x7f50a42754d0, exit_loop=@0x7f50a427529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f50a5a92033 in galera::ReplicatorSMM::async_recv (this=0x558b97dd7160, recv_ctx=0x7f50a42754d0) at galera/src/replicator_smm.cpp:383 #6 0x0000558b962b920e in wsrep::wsrep_provider_v26::run_applier (this=0x558b97d4cd80, applier_ctx=0x7f50a42754d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x0000558b9595c2b4 in wsrep_replication_process (thd=0x7f507c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000558b9594df01 in start_wsrep_THD (arg=0x558b97e38810) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f505b7fe700 (LWP 20818)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b95d46889 in os_event::wait (this=0x558b97f28130) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000558b95d4630f in os_event::wait_low (this=0x558b97f28130, reset_sig_count=4927) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000558b95d46680 in os_event_wait_low (event=0x558b97f28130, reset_sig_count=4927) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000558b95e325ff in srv_resume_thread (slot=0x558b96d83548 , sig_count=4927, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000558b95e36610 in srv_purge_coordinator_suspend (slot=0x558b96d83548 , rseg_history_len=31) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x0000558b95e368a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f50a430c700 (LWP 20642)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b962059a9 in safe_cond_wait (cond=0x7f5084017b88, mp=0x7f5084017ad8, file=0x558b9636ea00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558b9595bd06 in inline_mysql_cond_wait (that=0x7f5084017b88, mutex=0x7f5084017ad8, src_file=0x558b9636ed20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000558b9595e99e in Wsrep_thd_queue::pop_front (this=0x7f5084017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000558b9595d3be in wsrep_rollback_process (rollbacker=0x7f5084001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000558b9594df01 in start_wsrep_THD (arg=0x558b97e38b90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f50a42c1700 (LWP 20643)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b962059a9 in safe_cond_wait (cond=0x7f508c015628, mp=0x7f508c015578, file=0x558b9636ea00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000558b9595bd06 in inline_mysql_cond_wait (that=0x7f508c015628, mutex=0x7f508c015578, src_file=0x558b9636ed20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000558b9595e99e in Wsrep_thd_queue::pop_front (this=0x7f508c015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000558b9595d77e in wsrep_post_rollback_process (post_rollbacker=0x7f508c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000558b9594df01 in start_wsrep_THD (arg=0x558b97e2fd00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f509647b700 (LWP 20641)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f50a5a4a867 in gu::Lock::wait (cond=..., this=0x7f509647a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f50a5a4d3ab in RecvBuf::front (this=this@entry=0x558b97e16cd8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f50a5a4af46 in gcomm_recv (backend=, msg=0x558b97de2ae0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f50a5a3eeec in core_msg_recv (backend=backend@entry=0x558b97de2c28, recv_msworker[1] Trying to dump core for [mysqltest - pid: 21334, winpid: 21334] worker[1] Trying to dump core for [mysqld.1 - pid: 21074, winpid: 21074] worker[1] Trying to dump core for [mysqld.2 - pid: 21119, winpid: 21119] g=recv_msg@entry=0x558b97de2ae0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f50a5a42164 in gcs_core_recv (conn=0x558b97de2a80, recv_act=recv_act@entry=0x7f509647ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f50a5a493d8 in gcs_recv_thread (arg=0x558b97de2800) at gcs/src/gcs.cpp:1398 #7 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f5096c7c700 (LWP 20640)): #0 0x00007f50abf20483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f50a5a116f8 in asio::detail::epoll_reactor::run (this=0x558b97e17f40, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f50a5a15df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x558b97e171a0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f50a5a160a2 in asio::detail::task_io_service::run (this=0x558b97e171a0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f50a5a2dafd in asio::io_service::run (this=0x558b97e170c0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f50a5a2e251 in gcomm::AsioProtonet::event_loop (this=0x558b97e17020, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f50a5a4c2ed in GCommConn::run (this=0x558b97e16af0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f50a5a4f399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f50a540e700 (LWP 20639)): #0 0x00007f50adb65965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f50a5a6ffd7 in gu::Lock::wait (cond=..., this=0x7f50a540dbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f50a5a70069 in galera::ServiceThd::thd_func (arg=0x558b97dd7878) at galera/src/galera_service_thd.cpp:30 #3 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f50a65ba700 (LWP 20638)): #0 0x00007f50adb65d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000558b96205cab in safe_cond_timedwait (cond=0x558b977e5440 , mp=0x558b977e5380 , abstime=0x7f50a65b9d70, file=0x558b9668f360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000558b96206c65 in inline_mysql_cond_timedwait (that=0x558b977e5440 , mutex=0x558b977e5380 , abstime=0x7f50a65b9d70, src_file=0x558b9668f3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000558b962077c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000558b96198471 in pfs_spawn_thread (arg=0x558b97da0700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f50adb61dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f50abf1fead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f50adf72880 (LWP 20637)): #0 0x00007f50adb669d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000558b9620037a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000558b95a3cd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f50abf1520d in poll () from /lib64/libc.so.6 #5 0x0000558b955de96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000558b955ddc05 in mysqld_main (argc=162, argv=0x558b97c58d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000558b955d1da5 in main (argc=23, argv=0x7ffc01ed0888) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043A-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043A-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1008 galera_sr.GCF-1043A galera_sr.GCF-1018 galera_sr.mysql-wsrep-features#9 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1043B 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 19:33:38 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/GCF-1043B.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: DELETE FROM t1 INSERT INTO t1 VALUES (1,'y'),(2,'x') UPDATE t1 SET f2 = 'y' WHERE f1 = 1 OR f1 = 2; connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) - found 'core.21076' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043B-innodb/mysqld.1/data/core.21076 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 21076] [New LWP 21080] [New LWP 21081] [New LWP 21086] [New LWP 21084] [New LWP 21087] [New LWP 21099] [New LWP 21098] [New LWP 21100] [New LWP 21101] [New LWP 21106] [New LWP 21110] [New LWP 21113] [New LWP 21094] [New LWP 21107] [New LWP 21114] [New LWP 21102] [New LWP 21109] [New LWP 21089] [New LWP 21082] [New LWP 21108] [New LWP 21090] [New LWP 21097] [New LWP 21096] [New LWP 21093] [New LWP 21103] [New LWP 21083] [New LWP 21344] [New LWP 21091] [New LWP 21105] [New LWP 21104] [New LWP 21088] [New LWP 21085] [New LWP 21345] [New LWP 21092] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f3ed28379d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f3ed28379d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559e6631537a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559e65b51d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f3ed0be620d in poll () from /lib64/libc.so.6 #5 0x0000559e656f396a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559e656f2c05 in mysqld_main (argc=161, argv=0x559e6799ad20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559e656e6da5 in main (argc=23, argv=0x7ffc17f697d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f3ea5bfb700 (LWP 21092)): #0 0x00007f3ed2839d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000559e65e50255 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000559e65e505ab in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000559e66099751 in pfs_os_file_flush_func (file=..., src_file=0x559e667158b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559e6609c593 in fil_flush_low (space=0x559e67e6b080, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x0000559e660a8113 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x0000559e660a839a in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4580 #7 0x0000559e66023e64 in buf_dblwr_update (bpage=0x7f3ec008ba28, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:757 #8 0x0000559e6602b925 in buf_flush_write_complete (bpage=0x7f3ec008ba28, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x0000559e6601a3ef in buf_page_io_complete (bpage=0x7f3ec008ba28, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6135 #10 0x0000559e660a7e6d in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4467 #11 0x0000559e65f4d12f in io_handler_thread (arg=0x559e670d6440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f3ec0084700 (LWP 21345)): #0 0x00007f3ed2839d1d in fsync () from /lib64/libpthread.so.0 #1 0x0000559e65e50255 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x0000559e65e505ab in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x0000559e66099751 in pfs_os_file_flush_func (file=..., src_file=0x559e667158b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x0000559e6609c593 in fil_flush_low (space=0x559e67e6b420, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x0000559e660a8113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x0000559e65e2bc59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x0000559e65e2c6bb in log_write_up_to (lsn=5886626, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x0000559e65f97551 in trx_flush_log_if_needed_low (lsn=5886626) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x0000559e65f9759a in trx_flush_log_if_needed (lsn=5886626, trx=0x7f3ec08d94b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x0000559e65f993c3 in trx_commit_complete_for_mysql (trx=0x7f3ec08d94b8) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x0000559e65d7da6f in innobase_commit (hton=0x559e67c35880, thd=0x7f3e4c0920d0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #12 0x0000559e65b5789b in commit_one_phase_2 (thd=0x7f3e4c0920d0, all=true, trans=0x7f3e4c095618, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x0000559e65b5779d in ha_commit_one_phase (thd=0x7f3e4c0920d0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x0000559e65b56b19 in ha_commit_trans (thd=0x7f3e4c0920d0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x0000559e6599ca76 in trans_commit (thd=0x7f3e4c0920d0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x0000559e65d5b988 in Wsrep_storage_service::commit (this=0x7f3e4c0653e0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x0000559e663c485f in wsrep::transaction::certify_fragment (this=0x7f3e4c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #18 0x0000559e663c3fc3 in wsrep::transaction::streaming_step (this=0x7f3e4c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #19 0x0000559e663c137e in wsrep::transaction::after_row (this=0x7f3e4c0069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x0000559e65b69ba4 in wsrep::client_state::after_row (this=0x7f3e4c006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x0000559e65b52e87 in wsrep_after_row (thd=0x7f3e4c000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x0000559e65b64d26 in wsrep_after_row (thd=0x7f3e4c000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #23 0x0000559e65b65117 in handler::ha_write_row (this=0x7f3e58058428, buf=0x7f3e5805a010 "\375\001") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #24 0x0000559e657db18d in write_record (thd=0x7f3e4c000b00, table=0x7f3e580577f0, info=0x7f3ec0081f30) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x0000559e657d854f in mysql_insert (thd=0x7f3e4c000b00, table_list=0x7f3e4c01fa20, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:1067 #26 0x0000559e6582188a in mysql_execute_command (thd=0x7f3e4c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4751 #27 0x0000559e6582d675 in mysql_parse (thd=0x7f3e4c000b00, rawbuf=0x7f3e4c01f928 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7f3ec00830b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #28 0x0000559e6582cca8 in wsrep_mysql_parse (thd=0x7f3e4c000b00, rawbuf=0x7f3e4c01f928 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", length=37, parser_state=0x7f3ec00830b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #29 0x0000559e65818b65 in dispatch_command (command=COM_QUERY, thd=0x7f3e4c000b00, packet=0x7f3e4c0177f1 "INSERT INTO t1 VALUES (1,'y'),(2,'x')", packet_length=37, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #30 0x0000559e65817599 in do_command (thd=0x7f3e4c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #31 0x0000559e65987c50 in do_handle_one_connection (connect=0x559e67feb0a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #32 0x0000559e65987994 in handle_one_connection (arg=0x559e67feb0a0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #33 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67f44a50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #34 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #35 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f3ec8591700 (LWP 21085)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631a9a9 in safe_cond_wait (cond=0x7f3eb8016878, mp=0x7f3eb80167c8, file=0x559e66483a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559e65a70d06 in inline_mysql_cond_wait (that=0x7f3eb8016878, mutex=0x7f3eb80167c8, src_file=0x559e66483d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559e65a7399e in Wsrep_thd_queue::pop_front (this=0x7f3eb8016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559e65a7277e in wsrep_post_rollback_process (post_rollbacker=0x7f3eb8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000559e65a62f01 in start_wsrep_THD (arg=0x559e67b53760) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f3ea7bff700 (LWP 21088)): #0 0x00007f3ed2403644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559e65e4eba6 in LinuxAIOHandler::collect (this=0x7f3ea7bfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559e65e4f0cf in LinuxAIOHandler::poll (this=0x7f3ea7bfeb00, m1=0x7f3ea7bfeba8, m2=0x7f3ea7bfebb0, request=0x7f3ea7bfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559e65e4f232 in os_aio_linux_handler (global_segment=0, m1=0x7f3ea7bfeba8, m2=0x7f3ea7bfebb0, request=0x7f3ea7bfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559e65e53d56 in os_aio_handler (segment=0, m1=0x7f3ea7bfeba8, m2=0x7f3ea7bfebb0, request=0x7f3ea7bfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559e660a7bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559e65f4d12f in io_handler_thread (arg=0x559e670d6420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f3e8b7fe700 (LWP 21104)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b889 in os_event::wait (this=0x559e67c61490) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559e65e5b30f in os_event::wait_low (this=0x559e67c61490, reset_sig_count=20829) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559e65e5b680 in os_event_wait_low (event=0x559e67c61490, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559e65f475ff in srv_resume_thread (slot=0x559e66e985b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559e65f4b203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f3e8affd700 (LWP 21105)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b889 in os_event::wait (this=0x559e67c61510) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559e65e5b30f in os_event::wait_low (this=0x559e67c61510, reset_sig_count=18215) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559e65e5b680 in os_event_wait_low (event=0x559e67c61510, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559e65f475ff in srv_resume_thread (slot=0x559e66e985f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559e65f4b203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f3ea63fc700 (LWP 21091)): #0 0x00007f3ed2403644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559e65e4eba6 in LinuxAIOHandler::collect (this=0x7f3ea63fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559e65e4f0cf in LinuxAIOHandler::poll (this=0x7f3ea63fbb00, m1=0x7f3ea63fbba8, m2=0x7f3ea63fbbb0, request=0x7f3ea63fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559e65e4f232 in os_aio_linux_handler (global_segment=3, m1=0x7f3ea63fbba8, m2=0x7f3ea63fbbb0, request=0x7f3ea63fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559e65e53d56 in os_aio_handler (segment=3, m1=0x7f3ea63fbba8, m2=0x7f3ea63fbbb0, request=0x7f3ea63fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559e660a7bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559e65f4d12f in io_handler_thread (arg=0x559e670d6438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f3ec8056700 (LWP 21344)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631acab in safe_cond_timedwait (cond=0x559e6705afa0 , mp=0x559e6705d9a0 , abstime=0x7f3ec8055c60, file=0x559e663d5348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559e656e7725 in inline_mysql_cond_timedwait (that=0x559e6705afa0 , mutex=0x559e6705d9a0 , abstime=0x7f3ec8055c60, src_file=0x559e663d73b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559e656ec4fb in cache_thread (thd=0x7f3e58000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000559e656ec7a9 in one_thread_per_connection_end (thd=0x7f3e58000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000559e65987cda in do_handle_one_connection (connect=0x559e67ea8700) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000559e65987994 in handle_one_connection (arg=0x559e67ea8700) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67ad59a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f3ec247c700 (LWP 21083)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3eca71b867 in gu::Lock::wait (cond=..., this=0x7f3ec247b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f3eca71e3ab in RecvBuf::front (this=this@entry=0x559e67b597f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f3eca71bf46 in gcomm_recv (backend=, msg=0x559e67b21780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f3eca70feec in core_msg_recv (backend=backend@entry=0x559e67b218c8, recv_msg=recv_msg@entry=0x559e67b21780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f3eca713164 in gcs_core_recv (conn=0x559e67b21720, recv_act=recv_act@entry=0x7f3ec247bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f3eca71a3d8 in gcs_recv_thread (arg=0x559e67b214a0) at gcs/src/gcs.cpp:1398 #7 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f3e8bfff700 (LWP 21103)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b889 in os_event::wait (this=0x559e67c61410) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559e65e5b30f in os_event::wait_low (this=0x559e67c61410, reset_sig_count=24482) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559e65e5b680 in os_event_wait_low (event=0x559e67c61410, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559e65f475ff in srv_resume_thread (slot=0x559e66e98580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559e65f4b203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f3ea53fa700 (LWP 21093)): #0 0x00007f3ed2403644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559e65e4eba6 in LinuxAIOHandler::collect (this=0x7f3ea53f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559e65e4f0cf in LinuxAIOHandler::poll (this=0x7f3ea53f9b00, m1=0x7f3ea53f9ba8, m2=0x7f3ea53f9bb0, request=0x7f3ea53f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559e65e4f232 in os_aio_linux_handler (global_segment=5, m1=0x7f3ea53f9ba8, m2=0x7f3ea53f9bb0, request=0x7f3ea53f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559e65e53d56 in os_aio_handler (segment=5, m1=0x7f3ea53f9ba8, m2=0x7f3ea53f9bb0, request=0x7f3ea53f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559e660a7bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559e65f4d12f in io_handler_thread (arg=0x559e670d6448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f3e9b7fe700 (LWP 21096)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b1a4 in os_event::timed_wait (this=0x559e67e39ae0, abstime=0x7f3e9b7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559e65e5b4bb in os_event::wait_time_low (this=0x559e67e39ae0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559e65e5b65b in os_event_wait_time_low (event=0x559e67e39ae0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559e65e28314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f3e9affd700 (LWP 21097)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b1a4 in os_event::timed_wait (this=0x559e67c61590, abstime=0x7f3e9affcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559e65e5b4bb in os_event::wait_time_low (this=0x559e67c61590, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559e65e5b65b in os_event_wait_time_low (event=0x559e67c61590, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559e65f49808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f3ea6bfd700 (LWP 21090)): #0 0x00007f3ed2403644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559e65e4eba6 in LinuxAIOHandler::collect (this=0x7f3ea6bfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559e65e4f0cf in LinuxAIOHandler::poll (this=0x7f3ea6bfcb00, m1=0x7f3ea6bfcba8, m2=0x7f3ea6bfcbb0, request=0x7f3ea6bfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559e65e4f232 in os_aio_linux_handler (global_segment=2, m1=0x7f3ea6bfcba8, m2=0x7f3ea6bfcbb0, request=0x7f3ea6bfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559e65e53d56 in os_aio_handler (segment=2, m1=0x7f3ea6bfcba8, m2=0x7f3ea6bfcbb0, request=0x7f3ea6bfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559e660a7bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559e65f4d12f in io_handler_thread (arg=0x559e670d6430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f3e897fa700 (LWP 21108)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b889 in os_event::wait (this=0x559e67c61790) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559e65e5b30f in os_event::wait_low (this=0x559e67c61790, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559e65e5b680 in os_event_wait_low (event=0x559e67c61790, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559e660108b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f3ec2c7d700 (LWP 21082)): #0 0x00007f3ed0bf1483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f3eca6e26f8 in asio::detail::epoll_reactor::run (this=0x559e67b59f90, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f3eca6e6df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559e67b59e30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f3eca6e70a2 in asio::detail::task_io_service::run (this=0x559e67b59e30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f3eca6feafd in asio::io_service::run (this=0x559e67b58f30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f3eca6ff251 in gcomm::AsioProtonet::event_loop (this=0x559e67b58e90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f3eca71d2ed in GCommConn::run (this=0x559e67b59610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f3eca720399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f3ea73fe700 (LWP 21089)): #0 0x00007f3ed2403644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559e65e4eba6 in LinuxAIOHandler::collect (this=0x7f3ea73fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559e65e4f0cf in LinuxAIOHandler::poll (this=0x7f3ea73fdb00, m1=0x7f3ea73fdba8, m2=0x7f3ea73fdbb0, request=0x7f3ea73fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559e65e4f232 in os_aio_linux_handler (global_segment=1, m1=0x7f3ea73fdba8, m2=0x7f3ea73fdbb0, request=0x7f3ea73fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559e65e53d56 in os_aio_handler (segment=1, m1=0x7f3ea73fdba8, m2=0x7f3ea73fdbb0, request=0x7f3ea73fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559e660a7bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559e65f4d12f in io_handler_thread (arg=0x559e670d6428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f3e88ff9700 (LWP 21109)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631a9a9 in safe_cond_wait (cond=0x7f3e88ff8c70, mp=0x7f3e88ff8cb0, file=0x559e665e7860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559e65d72a13 in inline_mysql_cond_wait (that=0x7f3e88ff8c70, mutex=0x7f3e88ff8cb0, src_file=0x559e665e93a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559e65d77632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67ea2650) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f3e98ff9700 (LWP 21102)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b889 in os_event::wait (this=0x559e67c61390) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559e65e5b30f in os_event::wait_low (this=0x559e67c61390, reset_sig_count=5149) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559e65e5b680 in os_event_wait_low (event=0x559e67c61390, reset_sig_count=5149) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559e65f475ff in srv_resume_thread (slot=0x559e66e98548 , sig_count=5149, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559e65f4b610 in srv_purge_coordinator_suspend (slot=0x559e66e98548 , rseg_history_len=14) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x0000559e65f4b8a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f3ec80a1700 (LWP 21114)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631acab in safe_cond_timedwait (cond=0x559e6705afa0 , mp=0x559e6705d9a0 , abstime=0x7f3ec80a0c60, file=0x559e663d5348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559e656e7725 in inline_mysql_cond_timedwait (that=0x559e6705afa0 , mutex=0x559e6705d9a0 , abstime=0x7f3ec80a0c60, src_file=0x559e663d73b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559e656ec4fb in cache_thread (thd=0x7f3e60000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000559e656ec7a9 in one_thread_per_connection_end (thd=0x7f3e60000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000559e65987cda in do_handle_one_connection (connect=0x559e67fd0f70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000559e65987994 in handle_one_connection (arg=0x559e67fd0f70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67ff29e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f3e89ffb700 (LWP 21107)): #0 0x00007f3ed2839e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559e65e5bb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559e66002755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f3ea4bf9700 (LWP 21094)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b1a4 in os_event::timed_wait (this=0x559e67c61710, abstime=0x7f3ea4bf8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559e65e5b4bb in os_event::wait_time_low (this=0x559e67c61710, time_in_usec=346000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559e65e5b65b in os_event_wait_time_low (event=0x559e67c61710, time_in_usec=346000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559e66031c57 in pc_sleep_if_needed (next_loop_time=1549297958580, sig_count=2, cur_time=1549297958234) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x0000559e66032b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f3ec80ec700 (LWP 21113)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631a9a9 in safe_cond_wait (cond=0x559e6705afe0 , mp=0x559e6705e360 , file=0x559e663e4b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559e657183ce in inline_mysql_cond_wait (that=0x559e6705afe0 , mutex=0x559e6705e360 , src_file=0x559e663e56b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000559e65719c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67f63180) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f3ec8137700 (LWP 21110)): #0 0x00007f3ed0b2a40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000559e656e6e24 in my_sigwait (set=0x7f3ec8136cf0, sig=0x7f3ec8136cb4, code=0x7f3ec8136cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000559e656ed464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67c24fe0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f3e8a7fc700 (LWP 21106)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b889 in os_event::wait (this=0x559e67c61690) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559e65e5b30f in os_event::wait_low (this=0x559e67c61690, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559e65e5b680 in os_event_wait_low (event=0x559e67c61690, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559e66027dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f3e997fa700 (LWP 21101)): #0 0x00007f3ed2839e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559e65e5bb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559e65f4ab67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x0000559e65f4ac97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f3e99ffb700 (LWP 21100)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b1a4 in os_event::timed_wait (this=0x559e67e01440, abstime=0x7f3e99ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559e65e5b4bb in os_event::wait_time_low (this=0x559e67e01440, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559e65e5b65b in os_event_wait_time_low (event=0x559e67e01440, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559e65fa9aad in ib_wqueue_timedwait (wq=0x559e67dfafe0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559e660ebf11 in fts_optimize_thread (arg=0x559e67dfafe0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f3e9bfff700 (LWP 21098)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b1a4 in os_event::timed_wait (this=0x559e67c61610, abstime=0x7f3e9bffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559e65e5b4bb in os_event::wait_time_low (this=0x559e67c61610, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559e65e5b65b in os_event_wait_time_low (event=0x559e67c61610, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559e65f492e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f3e9a7fc700 (LWP 21099)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e65e5b1a4 in os_event::timed_wait (this=0x559e67e6bb00, abstime=0x7f3e9a7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559e65e5b4bb in os_event::wait_time_low (this=0x559e67e6bb00, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559e65e5b65b in os_event_wait_time_low (event=0x559e67e6bb00, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559e6609152c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f3ec14d6700 (LWP 21087)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631acab in safe_cond_timedwait (cond=0x559e678e7ca0 , mp=0x559e678e7be0 , abstime=0x7f3ec14d5cb0, file=0x559e66760130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559e66184add in inline_mysql_cond_timedwait (that=0x559e678e7ca0 , mutex=0x559e678e7be0 , abstime=0x7f3ec14d5cb0, src_file=0x559e66760178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559e66185001 in my_service_thread_sleep (control=0x559e66e9c900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559e6617949a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67c19710) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f3ec85dc700 (LWP 21084)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631a9a9 in safe_cond_wait (cond=0x7f3eb0015628, mp=0x7f3eb0015578, file=0x559e66483a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559e65a70d06 in inline_mysql_cond_wait (that=0x7f3eb0015628, mutex=0x7f3eb0015578, src_file=0x559e66483d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559e65a7399e in Wsrep_thd_queue::pop_front (this=0x7f3eb0015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559e65a723be in wsrep_rollback_process (rollbacker=0x7f3eb0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000559e65a62f01 in start_wsrep_THD (arg=0x559e67b69270) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f3ec8546700 (LWP 21086)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3eca635132 in fifo_lock_get (q=q@entry=0x559e67b47b20) at galerautils/src/gu_fifo.c:262 #2 0x00007f3eca6357a1 in gu_fifo_get_head (q=0x559e67b47b20, err=0x7f3ec854506c) at galerautils/src/gu_fifo.c:330 #3 0x00007f3eca7170cc in gcs_recv (conn=0x559e67b214a0, action=0x7f3ec85450b0) at gcs/src/gcs.cpp:2024 #4 0x00007f3eca74416c in galera::GcsActionSource::process (this=0x559e67b20e80, recv_ctx=0x7f3ec85454d0, exit_loop=@0x7f3ec854529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f3eca763033 in galera::ReplicatorSMM::async_recv (this=0x559e67b191b0, recv_ctx=0x7f3ec85454d0) at galera/src/replicator_smm.cpp:383 #6 0x0000559e663ce20e in wsrep::wsrep_provider_v26::run_applier (this=0x559e67aa0280, applier_ctx=0x7f3ec85454d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x0000559e65a712b4 in wsrep_replication_process (thd=0x7f3ea8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000559e65a62f01 in start_wsrep_THD (arg=0x559e67b71b50) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f3ec96de700 (LWP 21081)): #0 0x00007f3ed2836965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f3eca740fd7 in gu::Lock::wait (cond=..., this=0x7f3ec96ddbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f3eca741069 in galera::ServiceThd::thd_func (arg=0x559e67b198c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f3ecb28b700 (LWP 21080)): #0 0x00007f3ed2836d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559e6631acab in safe_cond_timedwait (cond=0x559e678fa440 , mp=0x559e678fa380 , abstime=0x7f3ecb28ad70, file=0x559e667a4360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559e6631bc65 in inline_mysql_cond_timedwait (that=0x559e678fa440 , mutex=0x559e678fa380 , abstime=0x7f3ecb28ad70, src_file=0x559e667a43a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559e6631c7c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559e662ad471 in pfs_spawn_thread (arg=0x559e67ae26f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f3ed2832dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f3ed0bf0ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f3ed2c43880 (LWP 21076)): #0 0x00007f3ed28379d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559e6631537a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559e65b51d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f3ed0be620d in poll () from /lib64/libc.so.6 #5 0x0000559e656f396a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559e656f2c05 in mysqld_main (argc=161, argv=0x559e6799ad20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559e656e6da5 in main (argc=23, argv=0x7ffc17f697d8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.21121' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043B-innodb/mysqld.2/data/core.21121 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 21121] [New LWP 21123] [New LWP 21124] [New LWP 21127] [New LWP 21129] [New LWP 21126] [New LWP 21128] [New LWP 21316] [New LWP 21299] [New LWP 21304] [New LWP 21318] [New LWP 21301] [New LWP 21309] [New LWP 21321] [New LWP 21305] [New LWP 21311] [New LWP 21322] [New LWP 21312] [New LWP 21324] [New LWP 21314] [New LWP 21313] [New LWP 21319] [New LWP 21315] [New LWP 21320] [New LWP 21317] [New LWP 21323] [New LWP 21327] [New LWP 21302] [New LWP 21300] [New LWP 21303] [New LWP 21310] [New LWP 21306] [New LWP 21125] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f72da09a9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f72da09a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056084489637a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005608440d2d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f72d844920d in poll () from /lib64/libc.so.6 #5 0x0000560843c7496a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000560843c73c05 in mysqld_main (argc=162, argv=0x560847c37d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000560843c67da5 in main (argc=23, argv=0x7ffd82d92368) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f72d0840700 (LWP 21125)): #0 0x00007f72d8454483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f72d1f456f8 in asio::detail::epoll_reactor::run (this=0x560847df2f70, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f72d1f49df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x560847df2e10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f72d1f4a0a2 in asio::detail::task_io_service::run (this=0x560847df2e10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f72d1f61afd in asio::io_service::run (this=0x560847df1b30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f72d1f62251 in gcomm::AsioProtonet::event_loop (this=0x560847df1a90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f72d1f802ed in GCommConn::run (this=0x560847df25a0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f72d1f83399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f7297fa8700 (LWP 21306)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc1a4 in os_event::timed_wait (this=0x560847f033f0, abstime=0x7f7297fa7af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005608443dc4bb in os_event::wait_time_low (this=0x560847f033f0, time_in_usec=683000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005608443dc65b in os_event_wait_time_low (event=0x560847f033f0, time_in_usec=683000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005608445b2c57 in pc_sleep_if_needed (next_loop_time=1549297965422, sig_count=2, cur_time=1549297964739) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x00005608445b3b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f72975a2700 (LWP 21310)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc1a4 in os_event::timed_wait (this=0x560847f03270, abstime=0x7f72975a1ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005608443dc4bb in os_event::wait_time_low (this=0x560847f03270, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005608443dc65b in os_event_wait_time_low (event=0x560847f03270, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005608444ca808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f72997ab700 (LWP 21303)): #0 0x00007f72d9c66644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005608443cfba6 in LinuxAIOHandler::collect (this=0x7f72997aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005608443d00cf in LinuxAIOHandler::poll (this=0x7f72997aab00, m1=0x7f72997aaba8, m2=0x7f72997aabb0, request=0x7f72997aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005608443d0232 in os_aio_linux_handler (global_segment=3, m1=0x7f72997aaba8, m2=0x7f72997aabb0, request=0x7f72997aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005608443d4d56 in os_aio_handler (segment=3, m1=0x7f72997aaba8, m2=0x7f72997aabb0, request=0x7f72997aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000560844628bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005608444ce12f in io_handler_thread (arg=0x560845657438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f729afae700 (LWP 21300)): #0 0x00007f72d9c66644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005608443cfba6 in LinuxAIOHandler::collect (this=0x7f729afadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005608443d00cf in LinuxAIOHandler::poll (this=0x7f729afadb00, m1=0x7f729afadba8, m2=0x7f729afadbb0, request=0x7f729afadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005608443d0232 in os_aio_linux_handler (global_segment=0, m1=0x7f729afadba8, m2=0x7f729afadbb0, request=0x7f729afadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005608443d4d56 in os_aio_handler (segment=0, m1=0x7f729afadba8, m2=0x7f729afadbb0, request=0x7f729afadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000560844628bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005608444ce12f in io_handler_thread (arg=0x560845657420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f7299fac700 (LWP 21302)): #0 0x00007f72d9c66644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005608443cfba6 in LinuxAIOHandler::collect (this=0x7f7299fabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005608443d00cf in LinuxAIOHandler::poll (this=0x7f7299fabb00, m1=0x7f7299fabba8, m2=0x7f7299fabbb0, request=0x7f7299fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005608443d0232 in os_aio_linux_handler (global_segment=2, m1=0x7f7299fabba8, m2=0x7f7299fabbb0, request=0x7f7299fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005608443d4d56 in os_aio_handler (segment=2, m1=0x7f7299fabba8, m2=0x7f7299fabbb0, request=0x7f7299fabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000560844628bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005608444ce12f in io_handler_thread (arg=0x560845657430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f729b163700 (LWP 21327)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056084489bcab in safe_cond_timedwait (cond=0x5608455dbfa0 , mp=0x5608455de9a0 , abstime=0x7f729b162c60, file=0x560844956348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000560843c68725 in inline_mysql_cond_timedwait (that=0x5608455dbfa0 , mutex=0x5608455de9a0 , abstime=0x7f729b162c60, src_file=0x5608449583b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000560843c6d4fb in cache_thread (thd=0x7f72a4000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000560843c6d7a9 in one_thread_per_connection_end (thd=0x7f72a4000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000560843f08cda in do_handle_one_connection (connect=0x56084822b200) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000560843f08994 in handle_one_connection (arg=0x56084822b200) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000056084482e471 in pfs_spawn_thread (arg=0x56084828a360) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f72c005f700 (LWP 21323)): #0 0x00007f72d838d40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000560843c67e24 in my_sigwait (set=0x7f72c005ecf0, sig=0x7f72c005ecb4, code=0x7f72c005ecb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000560843c6e464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000056084482e471 in pfs_spawn_thread (arg=0x560847ec6f10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f7286ffd700 (LWP 21317)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc889 in os_event::wait (this=0x560847f030f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005608443dc30f in os_event::wait_low (this=0x560847f030f0, reset_sig_count=23776) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005608443dc680 in os_event_wait_low (event=0x560847f030f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005608444c85ff in srv_resume_thread (slot=0x560845419580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005608444cc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f72857fa700 (LWP 21320)): #0 0x00007f72da09ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005608443dcb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000560844583755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f7287fff700 (LWP 21315)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc889 in os_event::wait (this=0x560847f03070) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005608443dc30f in os_event::wait_low (this=0x560847f03070, reset_sig_count=5299) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005608443dc680 in os_event_wait_low (event=0x560847f03070, reset_sig_count=5299) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005608444c85ff in srv_resume_thread (slot=0x560845419548 , sig_count=5299, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005608444cc610 in srv_purge_coordinator_suspend (slot=0x560845419548 , rseg_history_len=9) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x00005608444cc8a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f7285ffb700 (LWP 21319)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc889 in os_event::wait (this=0x560847f03370) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005608443dc30f in os_event::wait_low (this=0x560847f03370, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005608443dc680 in os_event_wait_low (event=0x560847f03370, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005608445a8dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f729559e700 (LWP 21313)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc1a4 in os_event::timed_wait (this=0x5608480aadd0, abstime=0x7f729559db60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005608443dc4bb in os_event::wait_time_low (this=0x5608480aadd0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005608443dc65b in os_event_wait_time_low (event=0x5608480aadd0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056084452aaad in ib_wqueue_timedwait (wq=0x5608480aad20, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056084466cf11 in fts_optimize_thread (arg=0x5608480aad20) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f7294d9d700 (LWP 21314)): #0 0x00007f72da09ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005608443dcb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005608444cbb67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005608444cbc97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f729b1ae700 (LWP 21324)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056084489b9a9 in safe_cond_wait (cond=0x5608455dbfe0 , mp=0x5608455df360 , file=0x560844965b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000560843c993ce in inline_mysql_cond_wait (that=0x5608455dbfe0 , mutex=0x5608455df360 , src_file=0x5608449666b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000560843c9ac60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056084482e471 in pfs_spawn_thread (arg=0x560848199aa0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f7295d9f700 (LWP 21312)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc1a4 in os_event::timed_wait (this=0x56084810d3a0, abstime=0x7f7295d9ed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005608443dc4bb in os_event::wait_time_low (this=0x56084810d3a0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005608443dc65b in os_event_wait_time_low (event=0x56084810d3a0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056084461252c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f7263fff700 (LWP 21322)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056084489b9a9 in safe_cond_wait (cond=0x7f7263ffec70, mp=0x7f7263ffecb0, file=0x560844b68860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005608442f3a13 in inline_mysql_cond_wait (that=0x7f7263ffec70, mutex=0x7f7263ffecb0, src_file=0x560844b6a3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005608442f8632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000056084482e471 in pfs_spawn_thread (arg=0x5608480bdc80) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f72965a0700 (LWP 21311)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc1a4 in os_event::timed_wait (this=0x560847f032f0, abstime=0x7f729659fd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005608443dc4bb in os_event::wait_time_low (this=0x560847f032f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005608443dc65b in os_event_wait_time_low (event=0x560847f032f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005608444ca2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f72987a9700 (LWP 21305)): #0 0x00007f72d9c66644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005608443cfba6 in LinuxAIOHandler::collect (this=0x7f72987a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005608443d00cf in LinuxAIOHandler::poll (this=0x7f72987a8b00, m1=0x7f72987a8ba8, m2=0x7f72987a8bb0, request=0x7f72987a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005608443d0232 in os_aio_linux_handler (global_segment=5, m1=0x7f72987a8ba8, m2=0x7f72987a8bb0, request=0x7f72987a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005608443d4d56 in os_aio_handler (segment=5, m1=0x7f72987a8ba8, m2=0x7f72987a8bb0, request=0x7f72987a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000560844628bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005608444ce12f in io_handler_thread (arg=0x560845657448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f7284ff9700 (LWP 21321)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc889 in os_event::wait (this=0x560847f03470) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005608443dc30f in os_event::wait_low (this=0x560847f03470, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005608443dc680 in os_event_wait_low (event=0x560847f03470, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005608445918b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f7296da1700 (LWP 21309)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc1a4 in os_event::timed_wait (this=0x5608480db380, abstime=0x7f7296da0d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005608443dc4bb in os_event::wait_time_low (this=0x5608480db380, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005608443dc65b in os_event_wait_time_low (event=0x5608480db380, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005608443a9314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f729a7ad700 (LWP 21301)): #0 0x00007f72d9c66644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005608443cfba6 in LinuxAIOHandler::collect (this=0x7f729a7acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005608443d00cf in LinuxAIOHandler::poll (this=0x7f729a7acb00, m1=0x7f729a7acba8, m2=0x7f729a7acbb0, request=0x7f729a7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005608443d0232 in os_aio_linux_handler (global_segment=1, m1=0x7f729a7acba8, m2=0x7f729a7acbb0, request=0x7f729a7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005608443d4d56 in os_aio_handler (segment=1, m1=0x7f729a7acba8, m2=0x7f729a7acbb0, request=0x7f729a7acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000560844628bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005608444ce12f in io_handler_thread (arg=0x560845657428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f72867fc700 (LWP 21318)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc889 in os_event::wait (this=0x560847f03170) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005608443dc30f in os_event::wait_low (this=0x560847f03170, reset_sig_count=20151) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005608443dc680 in os_event_wait_low (event=0x560847f03170, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005608444c85ff in srv_resume_thread (slot=0x5608454195b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005608444cc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f7298faa700 (LWP 21304)): #0 0x00007f72d9c66644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005608443cfba6 in LinuxAIOHandler::collect (this=0x7f7298fa9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005608443d00cf in LinuxAIOHandler::poll (this=0x7f7298fa9b00, m1=0x7f7298fa9ba8, m2=0x7f7298fa9bb0, request=0x7f7298fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005608443d0232 in os_aio_linux_handler (global_segment=4, m1=0x7f7298fa9ba8, m2=0x7f7298fa9bb0, request=0x7f7298fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005608443d4d56 in os_aio_handler (segment=4, m1=0x7f7298fa9ba8, m2=0x7f7298fa9bb0, request=0x7f7298fa9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000560844628bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005608444ce12f in io_handler_thread (arg=0x560845657440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f72c0860700 (LWP 21299)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056084489bcab in safe_cond_timedwait (cond=0x560845e68ca0 , mp=0x560845e68be0 , abstime=0x7f72c085fcb0, file=0x560844ce1130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000560844705add in inline_mysql_cond_timedwait (that=0x560845e68ca0 , mutex=0x560845e68be0 , abstime=0x7f72c085fcb0, src_file=0x560844ce1178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000560844706001 in my_service_thread_sleep (control=0x56084541d900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005608446fa49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056084482e471 in pfs_spawn_thread (arg=0x560847ebb6c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f72877fe700 (LWP 21316)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005608443dc889 in os_event::wait (this=0x560847f031f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005608443dc30f in os_event::wait_low (this=0x560847f031f0, reset_sig_count=16279) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005608443dc680 in os_event_wait_low (event=0x560847f031f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005608444c85ff in srv_resume_thread (slot=0x5608454195f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005608444cc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f72c2430700 (LWP 21128)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056084489b9a9 in safe_cond_wait (cond=0x7f72ac016938, mp=0x7f72ac016888, file=0x560844a04a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000560843ff1d06 in inline_mysql_cond_wait (that=0x7f72ac016938, mutex=0x7f72ac016888, src_file=0x560844a04d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000560843ff499e in Wsrep_thd_queue::pop_front (this=0x7f72ac016830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000560843ff377e in wsrep_post_rollback_process (post_rollbacker=0x7f72ac000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000560843fe3f01 in start_wsrep_THD (arg=0x560847e13670) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f72c2c7c700 (LWP 21126)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f72d1f7e867 in gu::Lock::wait (cond=..., this=0x7f72c2c7b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f72d1f813ab in RecvBuf::front (this=this@entry=0x560847df2788, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f72d1f7ef46 in gcomm_recv (backend=, msg=0x560847dbe690, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f72d1f72eec in core_msg_recv (backend=backend@entry=0x560847dbe7d8, recv_msg=recv_msg@entry=0x560847dbe690, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f72d1f76164 in gcs_core_recv (conn=0x560847dbe630, recv_act=recv_act@entry=0x7f72c2c7bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f72d1f7d3d8 in gcs_recv_thread (arg=0x560847dbe3b0) at gcs/src/gcs.cpp:1398 #7 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f72c23e5700 (LWP 21129)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f72d1e98132 in fifo_lock_get (q=q@entry=0x560847de4a30) at galerautils/src/gu_fifo.c:262 #2 0x00007f72d1e987a1 in gu_fifo_get_head (q=0x560847de4a30, err=0x7f72c23e406c) at galerautils/src/gu_fifo.c:330 #3 0x00007f72d1f7a0cc in gcs_recv (conn=0x560847dbe3b0, action=0x7f72c23e40b0) at gcs/src/gcs.cpp:2024 #4 0x00007f72d1fa716c in galera::GcsActionSource::process (this=0x560847deadd0, recv_ctx=0x7f72c23e44d0, exit_loop=@0x7f72c23e429e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f72d1fc6033 in galera::ReplicatorSMM::async_recv (this=0x560847db6160, recv_ctx=0x7f72c23e44d0) at galera/src/replicator_smm.cpp:383 #6 0x000056084494f20e in wsrep::wsrep_provider_v26::run_applier (this=0x560847d2bd80, applier_ctx=0x7f72c23e44d0) at /home/stepan/galera/git/10.4/worker[1] Trying to dump core for [mysqltest - pid: 22477, winpid: 22477] worker[1] Trying to dump core for [mysqld.1 - pid: 22172, winpid: 22172] worker[1] Trying to dump core for [mysqld.2 - pid: 22213, winpid: 22213] server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x0000560843ff22b4 in wsrep_replication_process (thd=0x7f72b0000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000560843fe3f01 in start_wsrep_THD (arg=0x560847e14660) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f72c247b700 (LWP 21127)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056084489b9a9 in safe_cond_wait (cond=0x7f72b8016878, mp=0x7f72b80167c8, file=0x560844a04a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000560843ff1d06 in inline_mysql_cond_wait (that=0x7f72b8016878, mutex=0x7f72b80167c8, src_file=0x560844a04d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000560843ff499e in Wsrep_thd_queue::pop_front (this=0x7f72b8016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000560843ff33be in wsrep_rollback_process (rollbacker=0x7f72b8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000560843fe3f01 in start_wsrep_THD (arg=0x560847e14b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f72d1942700 (LWP 21124)): #0 0x00007f72da099965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f72d1fa3fd7 in gu::Lock::wait (cond=..., this=0x7f72d1941bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f72d1fa4069 in galera::ServiceThd::thd_func (arg=0x560847db6878) at galera/src/galera_service_thd.cpp:30 #3 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f72d2aee700 (LWP 21123)): #0 0x00007f72da099d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056084489bcab in safe_cond_timedwait (cond=0x560845e7b440 , mp=0x560845e7b380 , abstime=0x7f72d2aedd70, file=0x560844d25360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056084489cc65 in inline_mysql_cond_timedwait (that=0x560845e7b440 , mutex=0x560845e7b380 , abstime=0x7f72d2aedd70, src_file=0x560844d253a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056084489d7c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056084482e471 in pfs_spawn_thread (arg=0x560847d7f700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f72da095dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f72d8453ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f72da4a6880 (LWP 21121)): #0 0x00007f72da09a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056084489637a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005608440d2d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f72d844920d in poll () from /lib64/libc.so.6 #5 0x0000560843c7496a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000560843c73c05 in mysqld_main (argc=162, argv=0x560847c37d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000560843c67da5 in main (argc=23, argv=0x7ffd82d92368) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043B-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1043B-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1043B Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.GCF-1051 'innodb' [ pass ] 4042 galera_sr.GCF-1060 'innodb' [ pass ] 256980 galera_sr.GCF-561 'innodb' [ pass ] 6663 galera_sr.GCF-571 'innodb' [ pass ] 5191 galera_sr.GCF-572 'innodb' [ pass ] 4284 galera_sr.GCF-580 'innodb' [ pass ] 4774 galera_sr.GCF-585 'innodb' [ pass ] 18873 galera_sr.GCF-597 'innodb' [ pass ] 5213 galera_sr.GCF-620 'innodb' [ pass ] 3713 galera_sr.GCF-623 'innodb' [ pass ] 5416 galera_sr.GCF-627 'innodb' [ pass ] 6101 galera_sr.GCF-845 'innodb' [ pass ] 5908 galera_sr.GCF-851 'innodb' [ pass ] 4040 galera_sr.GCF-867 'innodb' [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-04 19:39:48 line 2019-02-04 19:39:30 0 [Warning] WSREP: Failed to report last committed a08f5377-289a-11e9-9217-f64b147f7b76:696, -110 (Connection timed out) ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.1.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-867-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-867-innodb/' galera_sr.GCF-889 'innodb' [ pass ] 3999 galera_sr.GCF-900 'innodb' [ pass ] 4076 galera_sr.galera-features#56 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 19:56:09 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera-features#56.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); CREATE TABLE t1 (f1 INTEGER AUTO_INCREMENT PRIMARY KEY, f2 INTEGER) Engine=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; connection node_2; SET GLOBAL wsrep_slave_threads = 4; SET SESSION wsrep_trx_fragment_size = 1; connection node_1; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1a; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_2; INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) - found 'core.22173' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera-features#56-innodb/mysqld.1/data/core.22173 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 22173] [New LWP 22178] [New LWP 22175] [New LWP 22174] [New LWP 22176] [New LWP 22192] [New LWP 22177] [New LWP 22179] [New LWP 22200] [New LWP 22181] [New LWP 22207] [New LWP 22182] [New LWP 22194] [New LWP 22202] [New LWP 22203] [New LWP 22424] [New LWP 22204] [New LWP 22183] [New LWP 22196] [New LWP 22184] [New LWP 22201] [New LWP 22198] [New LWP 22185] [New LWP 22197] [New LWP 22188] [New LWP 22208] [New LWP 22191] [New LWP 22190] [New LWP 22195] [New LWP 22186] [New LWP 22180] [New LWP 22193] [New LWP 22199] [New LWP 22187] [New LWP 22460] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f45a15e39d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f45a15e39d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055fa72cd137a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055fa7250dd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f459f99220d in poll () from /lib64/libc.so.6 #5 0x000055fa720af96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055fa720aec05 in mysqld_main (argc=161, argv=0x55fa75466d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055fa720a2da5 in main (argc=23, argv=0x7fffcb71c3b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f4590072700 (LWP 22460)): #0 0x00007f45a15e5d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055fa7280c255 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055fa7280c5ab in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055fa72a55751 in pfs_os_file_flush_func (file=..., src_file=0x55fa730d18b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055fa72a58593 in fil_flush_low (space=0x55fa7593f030, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x000055fa72a64113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x000055fa727e7c59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055fa727e86bb in log_write_up_to (lsn=9932196, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055fa72953551 in trx_flush_log_if_needed_low (lsn=9932196) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000055fa7295359a in trx_flush_log_if_needed (lsn=9932196, trx=0x7f457b4035e0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000055fa729553c3 in trx_commit_complete_for_mysql (trx=0x7f457b4035e0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000055fa72739a6f in innobase_commit (hton=0x55fa757097b0, thd=0x7f451c0a2920, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #12 0x000055fa7251389b in commit_one_phase_2 (thd=0x7f451c0a2920, all=true, trans=0x7f451c0a5e68, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000055fa7251379d in ha_commit_one_phase (thd=0x7f451c0a2920, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000055fa72512b19 in ha_commit_trans (thd=0x7f451c0a2920, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000055fa72358a76 in trans_commit (thd=0x7f451c0a2920) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000055fa72717988 in Wsrep_storage_service::commit (this=0x7f451c0b1400, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x000055fa72d8085f in wsrep::transaction::certify_fragment (this=0x7f451c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #18 0x000055fa72d7ffc3 in wsrep::transaction::streaming_step (this=0x7f451c0069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #19 0x000055fa72d7d37e in wsrep::transaction::after_row (this=0x7f451c0069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x000055fa72525ba4 in wsrep::client_state::after_row (this=0x7f451c006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x000055fa7250ee87 in wsrep_after_row (thd=0x7f451c000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x000055fa72520d26 in wsrep_after_row (thd=0x7f451c000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #23 0x000055fa72521117 in handler::ha_write_row (this=0x7f451c0602c8, buf=0x7f451c07c080 "\375\025\063") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #24 0x000055fa7219718d in write_record (thd=0x7f451c000b00, table=0x7f451c05f690, info=0x7f451c022a30) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x000055fa7219c48c in select_insert::send_data (this=0x7f451c0229e8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x000055fa72254b47 in end_send (join=0x7f451c022a98, join_tab=0x7f451c0bab38, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x000055fa72391b89 in JOIN_CACHE::generate_full_extensions (this=0x7f451c0bb018, rec_ptr=0x7f451c146808 ",\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x000055fa723918a4 in JOIN_CACHE::join_matching_records (this=0x7f451c0bb018, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x000055fa72391270 in JOIN_CACHE::join_records (this=0x7f451c0bb018, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x000055fa7239140a in JOIN_CACHE::join_records (this=0x7f451c0baee8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x000055fa7239140a in JOIN_CACHE::join_records (this=0x7f451c025708, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x000055fa722517fb in sub_select_cache (join=0x7f451c022a98, join_tab=0x7f451c0ba028, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x000055fa72251a08 in sub_select (join=0x7f451c022a98, join_tab=0x7f451c0b9c78, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x000055fa722511d3 in do_select (join=0x7f451c022a98, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x000055fa72229cf1 in JOIN::exec_inner (this=0x7f451c022a98) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x000055fa7222913a in JOIN::exec (this=0x7f451c022a98) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x000055fa7222a42c in mysql_select (thd=0x7f451c000b00, tables=0x7f451c0206f8, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f451c0229e8, unit=0x7f451c0049b8, select_lex=0x7f451c0201d8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x000055fa7221bd2d in handle_select (thd=0x7f451c000b00, lex=0x7f451c0048f0, result=0x7f451c0229e8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x000055fa721de13c in mysql_execute_command (thd=0x7f451c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4856 #40 0x000055fa721e9675 in mysql_parse (thd=0x7f451c000b00, rawbuf=0x7f451c01f928 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f45900710b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #41 0x000055fa721e8ca8 in wsrep_mysql_parse (thd=0x7f451c000b00, rawbuf=0x7f451c01f928 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f45900710b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #42 0x000055fa721d4b65 in dispatch_command (command=COM_QUERY, thd=0x7f451c000b00, packet=0x7f451c0177f1 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #43 0x000055fa721d3599 in do_command (thd=0x7f451c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x000055fa72343c50 in do_handle_one_connection (connect=0x55fa75a12830) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #45 0x000055fa72343994 in handle_one_connection (arg=0x55fa75a12830) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #46 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa75a5b520) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f456ffff700 (LWP 22187)): #0 0x00007f45a15e5d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055fa7280c255 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055fa7280c5ab in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055fa72a55751 in pfs_os_file_flush_func (file=..., src_file=0x55fa730d18b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055fa72a58593 in fil_flush_low (space=0x55fa7593ec90, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x000055fa72a64113 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x000055fa72a6439a in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4580 #7 0x000055fa729dfe64 in buf_dblwr_update (bpage=0x7f457abe9678, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:757 #8 0x000055fa729e7925 in buf_flush_write_complete (bpage=0x7f457abe9678, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000055fa729d63ef in buf_page_io_complete (bpage=0x7f457abe9678, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6135 #10 0x000055fa72a63e6d in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4467 #11 0x000055fa7290912f in io_handler_thread (arg=0x55fa73a92448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f455dffb700 (LWP 22199)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72817889 in os_event::wait (this=0x55fa75735560) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fa7281730f in os_event::wait_low (this=0x55fa75735560, reset_sig_count=18697) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fa72817680 in os_event_wait_low (event=0x55fa75735560, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fa729035ff in srv_resume_thread (slot=0x55fa738545f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055fa72907203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f456d5f5700 (LWP 22193)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa728171a4 in os_event::timed_wait (this=0x55fa7593f710, abstime=0x7f456d5f4d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fa728174bb in os_event::wait_time_low (this=0x55fa7593f710, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fa7281765b in os_event_wait_time_low (event=0x55fa7593f710, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fa72a4d52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f45983f4700 (LWP 22180)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4599505067 in gu::Lock::wait (cond=..., this=0x7f45983f13f0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4599515c4c in galera::Monitor::enter (this=this@entry=0x55fa755e6210, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f45995093d8 in galera::ReplicatorSMM::commit_order_enter_remote (this=this@entry=0x55fa755e51b0, trx=...) at galera/src/replicator_smm.cpp:1344 #4 0x00007f459952908b in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:657 #5 0x000055fa72d8a771 in wsrep::wsrep_provider_v26::commit_order_enter (this=0x55fa7556c280, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:725 #6 0x000055fa72d7de66 in wsrep::transaction::before_commit (this=0x7f45800069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:440 #7 0x000055fa7240e6a8 in wsrep::client_state::before_commit (this=0x7f4580006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:391 #8 0x000055fa7240ada1 in wsrep_before_commit (thd=0x7f4580000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:195 #9 0x000055fa7240bf9f in Wsrep_high_priority_service::append_fragment_and_commit (this=0x7f45983f34d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:262 #10 0x000055fa72d73a63 in apply_fragment (high_priority_service=..., streaming_applier=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:67 #11 0x000055fa72d744c3 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:241 #12 0x000055fa72d77731 in wsrep::server_state::on_apply (this=0x55fa755af660, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:923 #13 0x000055fa72d8bddb in wsrep::high_priority_service::apply (this=0x7f45983f34d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #14 0x000055fa72d8968b in (anonymous namespace)::apply_cb (ctx=0x7f45983f34d0, wsh=0x7f45983f23e0, flags=0, buf=0x7f45983f2420, meta=0x7f45983f26b0, exit_loop=0x7f45983f25ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:460 #15 0x00007f45994d63b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7f4528081c80, recv_ctx=recv_ctx@entry=0x7f45983f34d0, apply_cb=0x55fa72d89468 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f45983f25ef: false) at galera/src/trx_handle.cpp:418 #16 0x00007f4599508d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55fa755e51b0, recv_ctx=recv_ctx@entry=0x7f45983f34d0, ts=...) at galera/src/replicator_smm.cpp:489 #17 0x00007f459950ea40 in galera::ReplicatorSMM::process_trx (this=0x55fa755e51b0, recv_ctx=0x7f45983f34d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #18 0x00007f45994efc6b in galera::GcsActionSource::process_writeset (this=this@entry=0x55fa755ece80, recv_ctx=0x7f45983f34d0, act=..., exit_loop=@0x7f45983f329e: false) at galera/src/gcs_action_source.cpp:62 #19 0x00007f45994efe45 in galera::GcsActionSource::dispatch (this=this@entry=0x55fa755ece80, recv_ctx=recv_ctx@entry=0x7f45983f34d0, act=..., exit_loop=@0x7f45983f329e: false) at galera/src/gcs_action_source.cpp:109 #20 0x00007f45994f021d in galera::GcsActionSource::process (this=0x55fa755ece80, recv_ctx=0x7f45983f34d0, exit_loop=@0x7f45983f329e: false) at galera/src/gcs_action_source.cpp:182 #21 0x00007f459950f033 in galera::ReplicatorSMM::async_recv (this=0x55fa755e51b0, recv_ctx=0x7f45983f34d0) at galera/src/replicator_smm.cpp:383 #22 0x000055fa72d8a20e in wsrep::wsrep_provider_v26::run_applier (this=0x55fa7556c280, applier_ctx=0x7f45983f34d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #23 0x000055fa7242d2b4 in wsrep_replication_process (thd=0x7f4580000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #24 0x000055fa7241ef01 in start_wsrep_THD (arg=0x55fa756474a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #25 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #26 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f45789a9700 (LWP 22186)): #0 0x00007f45a11af644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fa7280aba6 in LinuxAIOHandler::collect (this=0x7f45789a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fa7280b0cf in LinuxAIOHandler::poll (this=0x7f45789a8b00, m1=0x7f45789a8ba8, m2=0x7f45789a8bb0, request=0x7f45789a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fa7280b232 in os_aio_linux_handler (global_segment=4, m1=0x7f45789a8ba8, m2=0x7f45789a8bb0, request=0x7f45789a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fa7280fd56 in os_aio_handler (segment=4, m1=0x7f45789a8ba8, m2=0x7f45789a8bb0, request=0x7f45789a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055fa72a63bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055fa7290912f in io_handler_thread (arg=0x55fa73a92440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f455ffff700 (LWP 22195)): #0 0x00007f45a15e5e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055fa72817b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055fa72906b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x000055fa72906c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f456e5f7700 (LWP 22190)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa728171a4 in os_event::timed_wait (this=0x55fa7590d6f0, abstime=0x7f456e5f6d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fa728174bb in os_event::wait_time_low (this=0x55fa7590d6f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fa7281765b in os_event_wait_time_low (event=0x55fa7590d6f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fa727e4314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f456ddf6700 (LWP 22191)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa728171a4 in os_event::timed_wait (this=0x55fa757355e0, abstime=0x7f456ddf5ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fa728174bb in os_event::wait_time_low (this=0x55fa757355e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fa7281765b in os_event_wait_time_low (event=0x55fa757355e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fa72905808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f4590108700 (LWP 22208)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4599505067 in gu::Lock::wait (cond=..., this=0x7f45901045d0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4599515c4c in galera::Monitor::enter (this=this@entry=0x55fa755e6210, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f459950a717 in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x55fa755e51b0, trx=...) at galera/src/replicator_smm.cpp:1282 #4 0x00007f4599529063 in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:652 #5 0x000055fa72d8a771 in wsrep::wsrep_provider_v26::commit_order_enter (this=0x55fa7556c280, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:725 #6 0x000055fa72d7de66 in wsrep::transaction::before_commit (this=0x7f4530073250) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:440 #7 0x000055fa7240e6a8 in wsrep::client_state::before_commit (this=0x7f45300731e0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:391 #8 0x000055fa727166eb in wsrep_before_commit (thd=0x7f453006d360, all=true) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:195 #9 0x000055fa72717966 in Wsrep_storage_service::commit (this=0x7f453033f290, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:171 #10 0x000055fa72d8085f in wsrep::transaction::certify_fragment (this=0x7f45300069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #11 0x000055fa72d7ffc3 in wsrep::transaction::streaming_step (this=0x7f45300069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #12 0x000055fa72d7d37e in wsrep::transaction::after_row (this=0x7f45300069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #13 0x000055fa72525ba4 in wsrep::client_state::after_row (this=0x7f4530006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #14 0x000055fa7250ee87 in wsrep_after_row (thd=0x7f4530000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #15 0x000055fa72520d26 in wsrep_after_row (thd=0x7f4530000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #16 0x000055fa72521117 in handler::ha_write_row (this=0x7f453016b578, buf=0x7f4530195f70 "\375\033#") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #17 0x000055fa7219718d in write_record (thd=0x7f4530000b00, table=0x7f453016a940, info=0x7f4530018200) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #18 0x000055fa7219c48c in select_insert::send_data (this=0x7f45300181b8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #19 0x000055fa72254b47 in end_send (join=0x7f4530018268, join_tab=0x7f453006a1d8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #20 0x000055fa72391b89 in JOIN_CACHE::generate_full_extensions (this=0x7f453006a588, rec_ptr=0x7f45302f30e4 "8\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #21 0x000055fa723918a4 in JOIN_CACHE::join_matching_records (this=0x7f453006a588, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #22 0x000055fa72391270 in JOIN_CACHE::join_records (this=0x7f453006a588, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #23 0x000055fa7239140a in JOIN_CACHE::join_records (this=0x7f453001b008, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #24 0x000055fa7239140a in JOIN_CACHE::join_records (this=0x7f453001aed8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #25 0x000055fa722517fb in sub_select_cache (join=0x7f4530018268, join_tab=0x7f45300696c8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #26 0x000055fa72251a08 in sub_select (join=0x7f4530018268, join_tab=0x7f4530069318, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #27 0x000055fa722511d3 in do_select (join=0x7f4530018268, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #28 0x000055fa72229cf1 in JOIN::exec_inner (this=0x7f4530018268) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #29 0x000055fa7222913a in JOIN::exec (this=0x7f4530018268) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #30 0x000055fa7222a42c in mysql_select (thd=0x7f4530000b00, tables=0x7f4530015f48, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f45300181b8, unit=0x7f45300049b8, select_lex=0x7f4530015a28) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #31 0x000055fa7221bd2d in handle_select (thd=0x7f4530000b00, lex=0x7f45300048f0, result=0x7f45300181b8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #32 0x000055fa721de13c in mysql_execute_command (thd=0x7f4530000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4856 #33 0x000055fa721e9675 in mysql_parse (thd=0x7f4530000b00, rawbuf=0x7f4530015178 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f45901070b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #34 0x000055fa721e8ca8 in wsrep_mysql_parse (thd=0x7f4530000b00, rawbuf=0x7f4530015178 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7f45901070b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #35 0x000055fa721d4b65 in dispatch_command (command=COM_QUERY, thd=0x7f4530000b00, packet=0x7f4530028211 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #36 0x000055fa721d3599 in do_command (thd=0x7f4530000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #37 0x000055fa72343c50 in do_handle_one_connection (connect=0x55fa75a71920) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #38 0x000055fa72343994 in handle_one_connection (arg=0x55fa75a71920) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #39 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa75ac6a70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #40 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #41 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f456f7fe700 (LWP 22188)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa728171a4 in os_event::timed_wait (this=0x55fa75735760, abstime=0x7f456f7fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fa728174bb in os_event::wait_time_low (this=0x55fa75735760, time_in_usec=282000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fa7281765b in os_event_wait_time_low (event=0x55fa75735760, time_in_usec=282000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fa729edc57 in pc_sleep_if_needed (next_loop_time=1549299308739, sig_count=2, cur_time=1549299308457) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x000055fa729eeb90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f455effd700 (LWP 22197)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72817889 in os_event::wait (this=0x55fa75735460) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fa7281730f in os_event::wait_low (this=0x55fa75735460, reset_sig_count=24566) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fa72817680 in os_event_wait_low (event=0x55fa75735460, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fa729035ff in srv_resume_thread (slot=0x55fa73854580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055fa72907203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f45791aa700 (LWP 22185)): #0 0x00007f45a11af644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fa7280aba6 in LinuxAIOHandler::collect (this=0x7f45791a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fa7280b0cf in LinuxAIOHandler::poll (this=0x7f45791a9b00, m1=0x7f45791a9ba8, m2=0x7f45791a9bb0, request=0x7f45791a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fa7280b232 in os_aio_linux_handler (global_segment=3, m1=0x7f45791a9ba8, m2=0x7f45791a9bb0, request=0x7f45791a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fa7280fd56 in os_aio_handler (segment=3, m1=0x7f45791a9ba8, m2=0x7f45791a9bb0, request=0x7f45791a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055fa72a63bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055fa7290912f in io_handler_thread (arg=0x55fa73a92438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f455e7fc700 (LWP 22198)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72817889 in os_event::wait (this=0x55fa757354e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fa7281730f in os_event::wait_low (this=0x55fa757354e0, reset_sig_count=19702) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fa72817680 in os_event_wait_low (event=0x55fa757354e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fa729035ff in srv_resume_thread (slot=0x55fa738545b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055fa72907203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f455cff9700 (LWP 22201)): #0 0x00007f45a15e5e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055fa72817b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055fa729be755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f45799ab700 (LWP 22184)): #0 0x00007f45a11af644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fa7280aba6 in LinuxAIOHandler::collect (this=0x7f45799aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fa7280b0cf in LinuxAIOHandler::poll (this=0x7f45799aab00, m1=0x7f45799aaba8, m2=0x7f45799aabb0, request=0x7f45799aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fa7280b232 in os_aio_linux_handler (global_segment=2, m1=0x7f45799aaba8, m2=0x7f45799aabb0, request=0x7f45799aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fa7280fd56 in os_aio_handler (segment=2, m1=0x7f45799aaba8, m2=0x7f45799aabb0, request=0x7f45799aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055fa72a63bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055fa7290912f in io_handler_thread (arg=0x55fa73a92430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f455f7fe700 (LWP 22196)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72817889 in os_event::wait (this=0x55fa757353e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fa7281730f in os_event::wait_low (this=0x55fa757353e0, reset_sig_count=5168) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fa72817680 in os_event_wait_low (event=0x55fa757353e0, reset_sig_count=5168) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fa729035ff in srv_resume_thread (slot=0x55fa73854548 , sig_count=5168, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055fa72907610 in srv_purge_coordinator_suspend (slot=0x55fa73854548 , rseg_history_len=50) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x000055fa729078a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f457a1ac700 (LWP 22183)): #0 0x00007f45a11af644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fa7280aba6 in LinuxAIOHandler::collect (this=0x7f457a1abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fa7280b0cf in LinuxAIOHandler::poll (this=0x7f457a1abb00, m1=0x7f457a1abba8, m2=0x7f457a1abbb0, request=0x7f457a1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fa7280b232 in os_aio_linux_handler (global_segment=1, m1=0x7f457a1abba8, m2=0x7f457a1abbb0, request=0x7f457a1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fa7280fd56 in os_aio_handler (segment=1, m1=0x7f457a1abba8, m2=0x7f457a1abbb0, request=0x7f457a1abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055fa72a63bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055fa7290912f in io_handler_thread (arg=0x55fa73a92428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4598066700 (LWP 22204)): #0 0x00007f459f8d640a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055fa720a2e24 in my_sigwait (set=0x7f4598065cf0, sig=0x7f4598065cb4, code=0x7f4598065cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055fa720a9464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa756f7d00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f45900bd700 (LWP 22424)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72cd6cab in safe_cond_timedwait (cond=0x55fa73a16fa0 , mp=0x55fa73a199a0 , abstime=0x7f45900bcc60, file=0x55fa72d91348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fa720a3725 in inline_mysql_cond_timedwait (that=0x55fa73a16fa0 , mutex=0x55fa73a199a0 , abstime=0x7f45900bcc60, src_file=0x55fa72d933b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fa720a84fb in cache_thread (thd=0x7f4528000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055fa720a87a9 in one_thread_per_connection_end (thd=0x7f4528000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055fa72343cda in do_handle_one_connection (connect=0x55fa7564da20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x000055fa72343994 in handle_one_connection (arg=0x55fa7564da20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa755a19a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f45477fe700 (LWP 22203)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72cd69a9 in safe_cond_wait (cond=0x7f45477fdc70, mp=0x7f45477fdcb0, file=0x55fa72fa3860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fa7272ea13 in inline_mysql_cond_wait (that=0x7f45477fdc70, mutex=0x7f45477fdcb0, src_file=0x55fa72fa53a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fa72733632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa75976290) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4547fff700 (LWP 22202)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72817889 in os_event::wait (this=0x55fa757357e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fa7281730f in os_event::wait_low (this=0x55fa757357e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fa72817680 in os_event_wait_low (event=0x55fa757357e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fa729cc8b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f456cdf4700 (LWP 22194)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa728171a4 in os_event::timed_wait (this=0x55fa758d51f0, abstime=0x7f456cdf3b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fa728174bb in os_event::wait_time_low (this=0x55fa758d51f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fa7281765b in os_event_wait_time_low (event=0x55fa758d51f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fa72965aad in ib_wqueue_timedwait (wq=0x55fa758d5140, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055fa72aa7f11 in fts_optimize_thread (arg=0x55fa758d5140) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f457a9ad700 (LWP 22182)): #0 0x00007f45a11af644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fa7280aba6 in LinuxAIOHandler::collect (this=0x7f457a9acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fa7280b0cf in LinuxAIOHandler::poll (this=0x7f457a9acb00, m1=0x7f457a9acba8, m2=0x7f457a9acbb0, request=0x7f457a9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fa7280b232 in os_aio_linux_handler (global_segment=0, m1=0x7f457a9acba8, m2=0x7f457a9acbb0, request=0x7f457a9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fa7280fd56 in os_aio_handler (segment=0, m1=0x7f457a9acba8, m2=0x7f457a9acbb0, request=0x7f457a9acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055fa72a63bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055fa7290912f in io_handler_thread (arg=0x55fa73a92420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4590153700 (LWP 22207)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72cd69a9 in safe_cond_wait (cond=0x55fa73a16fe0 , mp=0x55fa73a1a360 , file=0x55fa72da0b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fa720d43ce in inline_mysql_cond_wait (that=0x55fa73a16fe0 , mutex=0x55fa73a1a360 , src_file=0x55fa72da16b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055fa720d5c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa75a36c50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f457bfff700 (LWP 22181)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72cd6cab in safe_cond_timedwait (cond=0x55fa742a3ca0 , mp=0x55fa742a3be0 , abstime=0x7f457bffecb0, file=0x55fa7311c130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fa72b40add in inline_mysql_cond_timedwait (that=0x55fa742a3ca0 , mutex=0x55fa742a3be0 , abstime=0x7f457bffecb0, src_file=0x55fa7311c178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fa72b41001 in my_service_thread_sleep (control=0x55fa73858900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055fa72b3549a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa756ed640) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f455d7fa700 (LWP 22200)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72817889 in os_event::wait (this=0x55fa757356e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fa7281730f in os_event::wait_low (this=0x55fa757356e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fa72817680 in os_event_wait_low (event=0x55fa757356e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fa729e3dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f459843f700 (LWP 22179)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72cd69a9 in safe_cond_wait (cond=0x7f457c015628, mp=0x7f457c015578, file=0x55fa72e3fa00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fa7242cd06 in inline_mysql_cond_wait (that=0x7f457c015628, mutex=0x7f457c015578, src_file=0x55fa72e3fd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fa7242f99e in Wsrep_thd_queue::pop_front (this=0x7f457c015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055fa7242e77e in wsrep_post_rollback_process (post_rollbacker=0x7f457c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055fa7241ef01 in start_wsrep_THD (arg=0x55fa75627760) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f459137a700 (LWP 22177)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45994c7867 in gu::Lock::wait (cond=..., this=0x7f4591379850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f45994ca3ab in RecvBuf::front (this=this@entry=0x55fa7562d7f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f45994c7f46 in gcomm_recv (backend=, msg=0x55fa755ed780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f45994bbeec in core_msg_recv (backend=backend@entry=0x55fa755ed8c8, recv_msg=recv_msg@entry=0x55fa755ed780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f45994bf164 in gcs_core_recv (conn=0x55fa755ed720, recv_act=recv_act@entry=0x7f4591379d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f45994c63d8 in gcs_recv_thread (arg=0x55fa755ed4a0) at gcs/src/gcs.cpp:1398 #7 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f456edf8700 (LWP 22192)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa728171a4 in os_event::timed_wait (this=0x55fa75735660, abstime=0x7f456edf7d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fa728174bb in os_event::wait_time_low (this=0x55fa75735660, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fa7281765b in os_event_wait_time_low (event=0x55fa75735660, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fa729052e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4591b7b700 (LWP 22176)): #0 0x00007f459f99d483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f459948e6f8 in asio::detail::epoll_reactor::run (this=0x55fa7562df10, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f4599492df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55fa7562de30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f45994930a2 in asio::detail::task_io_service::run (this=0x55fa7562de30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f45994aaafd in asio::io_service::run (this=0x55fa7562cf30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f45994ab251 in gcomm::AsioProtonet::event_loop (this=0x55fa7562ce90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f45994c92ed in GCommConn::run (this=0x55fa7562d610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f45994cc399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f459a037700 (LWP 22174)): #0 0x00007f45a15e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72cd6cab in safe_cond_timedwait (cond=0x55fa742b6440 , mp=0x55fa742b6380 , abstime=0x7f459a036d70, file=0x55fa73160360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fa72cd7c65 in inline_mysql_cond_timedwait (that=0x55fa742b6440 , mutex=0x55fa742b6380 , abstime=0x7f459a036d70, src_file=0x55fa731603a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fa72cd87c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055fa72c69471 in pfs_spawn_thread (arg=0x55fa755ae6f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4593fff700 (LWP 22175)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f45994ecfd7 in gu::Lock::wait (cond=..., this=0x7f4593ffebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f45994ed069 in galera::ServiceThd::thd_func (arg=0x55fa755e58c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f459848a700 (LWP 22178)): #0 0x00007f45a15e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fa72cd69a9 in safe_cond_wait (cond=0x7f4588016878, mp=0x7f45880167c8, file=0x55fa72e3fa00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fa7242cd06 in inline_mysql_cond_wait (that=0x7f4588016878, mutex=0x7f45880167c8, src_file=0x55fa72e3fd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fa7242f99e in Wsrep_thd_queue::pop_front (this=0x7f4588016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055fa7242e3be in wsrep_rollback_process (rollbacker=0x7f4588001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055fa7241ef01 in start_wsrep_THD (arg=0x55fa7563d290) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f45a15dedd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f459f99cead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f45a19ef880 (LWP 22173)): #0 0x00007f45a15e39d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055fa72cd137a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055fa7250dd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f459f99220d in poll () from /lib64/libc.so.6 #5 0x000055fa720af96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055fa720aec05 in mysqld_main (argc=161, argv=0x55fa75466d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055fa720a2da5 in main (argc=23, argv=0x7fffcb71c3b8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.22215' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera-features#56-innodb/mysqld.2/data/core.22215 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 22215] [New LWP 22221] [New LWP 22220] [New LWP 22391] [New LWP 22223] [New LWP 22387] [New LWP 22403] [New LWP 22390] [New LWP 22394] [New LWP 22405] [New LWP 22397] [New LWP 22479] [New LWP 22406] [New LWP 22399] [New LWP 22481] [New LWP 22218] [New LWP 22410] [New LWP 22222] [New LWP 22402] [New LWP 22217] [New LWP 22398] [New LWP 22407] [New LWP 22219] [New LWP 22388] [New LWP 22408] [New LWP 22480] [New LWP 22392] [New LWP 22389] [New LWP 22411] [New LWP 22393] [New LWP 22409] [New LWP 22400] [New LWP 22404] [New LWP 22412] [New LWP 22401] [New LWP 22415] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fd39604d9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fd39604d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005631c308c37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005631c28c8d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd3943fc20d in poll () from /lib64/libc.so.6 #5 0x00005631c246a96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005631c2469c05 in mysqld_main (argc=162, argv=0x5631c656ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005631c245dda5 in main (argc=23, argv=0x7ffdbf42c5c8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 36 (Thread 0x7fd38c0b8700 (LWP 22415)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd38df2e7f5 in gcs_replv (conn=0x5631c66f53b0, act_in=0x7fd38c0b4d40, act=act@entry=0x7fd38c0b4d20, scheduled=) at gcs/src/gcs.cpp:1843 #2 0x00007fd38df80b7e in galera::Gcs::replv (this=this@entry=0x5631c66ed868, actv=..., act=..., scheduled=scheduled@entry=true) at galera/src/galera_gcs.hpp:138 #3 0x00007fd38df761f1 in galera::ReplicatorSMM::replicate (this=0x5631c66ed160, trx=..., meta=0x7fd38c0b53c8) at galera/src/replicator_smm.cpp:661 #4 0x00007fd38df92932 in galera_certify (gh=, conn_id=, trx_handle=, flags=, meta=0x7fd38c0b53c8) at galera/src/wsrep_provider.cpp:535 #5 0x00005631c3145580 in wsrep::wsrep_provider_v26::certify (this=0x5631c6662d80, client_id=..., ws_handle=..., flags=0, ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:693 #6 0x00005631c313b6da in wsrep::transaction::certify_fragment (this=0x7fd368006b30, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1142 #7 0x00005631c313afc3 in wsrep::transaction::streaming_step (this=0x7fd368006b30, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #8 0x00005631c313837e in wsrep::transaction::after_row (this=0x7fd368006b30) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #9 0x00005631c28e0ba4 in wsrep::client_state::after_row (this=0x7fd368006ac0) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #10 0x00005631c28c9e87 in wsrep_after_row (thd=0x7fd368000c40) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #11 0x00005631c28dbd26 in wsrep_after_row (thd=0x7fd368000c40) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #12 0x00005631c28dc117 in handler::ha_write_row (this=0x7fd3681f2e18, buf=0x7fd3681f6570 "\375\342!") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #13 0x00005631c255218d in write_record (thd=0x7fd368000c40, table=0x7fd3681f21e0, info=0x7fd368033700) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #14 0x00005631c255748c in select_insert::send_data (this=0x7fd3680336b8, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #15 0x00005631c260fb47 in end_send (join=0x7fd368033768, join_tab=0x7fd36806e328, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #16 0x00005631c274cb89 in JOIN_CACHE::generate_full_extensions (this=0x7fd36806e808, rec_ptr=0x7fd368297548 ",") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #17 0x00005631c274c8a4 in JOIN_CACHE::join_matching_records (this=0x7fd36806e808, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #18 0x00005631c274c270 in JOIN_CACHE::join_records (this=0x7fd36806e808, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #19 0x00005631c274c40a in JOIN_CACHE::join_records (this=0x7fd36806e6d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #20 0x00005631c274c40a in JOIN_CACHE::join_records (this=0x7fd3680363d8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #21 0x00005631c260c7fb in sub_select_cache (join=0x7fd368033768, join_tab=0x7fd36806d818, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #22 0x00005631c260ca08 in sub_select (join=0x7fd368033768, join_tab=0x7fd36806d468, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #23 0x00005631c260c1d3 in do_select (join=0x7fd368033768, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #24 0x00005631c25e4cf1 in JOIN::exec_inner (this=0x7fd368033768) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #25 0x00005631c25e413a in JOIN::exec (this=0x7fd368033768) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #26 0x00005631c25e542c in mysql_select (thd=0x7fd368000c40, tables=0x7fd3680313c8, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7fd3680336b8, unit=0x7fd368004af8, select_lex=0x7fd368030ea8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #27 0x00005631c25d6d2d in handle_select (thd=0x7fd368000c40, lex=0x7fd368004a30, result=0x7fd3680336b8, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #28 0x00005631c259913c in mysql_execute_command (thd=0x7fd368000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4856 #29 0x00005631c25a4675 in mysql_parse (thd=0x7fd368000c40, rawbuf=0x7fd3680305f8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fd38c0b70b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #30 0x00005631c25a3ca8 in wsrep_mysql_parse (thd=0x7fd368000c40, rawbuf=0x7fd3680305f8 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=76, parser_state=0x7fd38c0b70b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #31 0x00005631c258fb65 in dispatch_command (command=COM_QUERY, thd=0x7fd368000c40, packet=0x7fd368041211 "INSERT INTO t1 (f2) SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4;", packet_length=77, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #32 0x00005631c258e599 in do_command (thd=0x7fd368000c40) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #33 0x00005631c26fec50 in do_handle_one_connection (connect=0x5631c6b66190) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #34 0x00005631c26fe994 in handle_one_connection (arg=0x5631c6b66190) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #35 0x00005631c3024471 in pfs_spawn_thread (arg=0x5631c6bc52f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #36 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #37 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fd350ff9700 (LWP 22401)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd21a4 in os_event::timed_wait (this=0x5631c69e5d60, abstime=0x7fd350ff8b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005631c2bd24bb in os_event::wait_time_low (this=0x5631c69e5d60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005631c2bd265b in os_event_wait_time_low (event=0x5631c69e5d60, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005631c2d20aad in ib_wqueue_timedwait (wq=0x5631c69e5cb0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005631c2e62f11 in fts_optimize_thread (arg=0x5631c69e5cb0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fd38c103700 (LWP 22412)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c30919a9 in safe_cond_wait (cond=0x5631c3dd1fe0 , mp=0x5631c3dd5360 , file=0x5631c315bb48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005631c248f3ce in inline_mysql_cond_wait (that=0x5631c3dd1fe0 , mutex=0x5631c3dd5360 , src_file=0x5631c315c6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005631c2490c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005631c3024471 in pfs_spawn_thread (arg=0x5631c6ad4a30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fd342ffd700 (LWP 22404)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd2889 in os_event::wait (this=0x5631c683e080) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005631c2bd230f in os_event::wait_low (this=0x5631c683e080, reset_sig_count=25547) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005631c2bd2680 in os_event_wait_low (event=0x5631c683e080, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005631c2cbe5ff in srv_resume_thread (slot=0x5631c3c0f580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005631c2cc2203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fd3517fa700 (LWP 22400)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd21a4 in os_event::timed_wait (this=0x5631c6a48330, abstime=0x7fd3517f9d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005631c2bd24bb in os_event::wait_time_low (this=0x5631c6a48330, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005631c2bd265b in os_event_wait_time_low (event=0x5631c6a48330, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005631c2e0852c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fd327fff700 (LWP 22409)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd2889 in os_event::wait (this=0x5631c683e400) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005631c2bd230f in os_event::wait_low (this=0x5631c683e400, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005631c2bd2680 in os_event_wait_low (event=0x5631c683e400, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005631c2d878b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fd353fff700 (LWP 22393)): #0 0x00007fd395c19644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005631c2bc5ba6 in LinuxAIOHandler::collect (this=0x7fd353ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005631c2bc60cf in LinuxAIOHandler::poll (this=0x7fd353ffeb00, m1=0x7fd353ffeba8, m2=0x7fd353ffebb0, request=0x7fd353ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005631c2bc6232 in os_aio_linux_handler (global_segment=5, m1=0x7fd353ffeba8, m2=0x7fd353ffebb0, request=0x7fd353ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005631c2bcad56 in os_aio_handler (segment=5, m1=0x7fd353ffeba8, m2=0x7fd353ffebb0, request=0x7fd353ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005631c2e1ebc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005631c2cc412f in io_handler_thread (arg=0x5631c3e4d448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fd38c14e700 (LWP 22411)): #0 0x00007fd39434040a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005631c245de24 in my_sigwait (set=0x7fd38c14dcf0, sig=0x7fd38c14dcb4, code=0x7fd38c14dcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005631c2464464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005631c3024471 in pfs_spawn_thread (arg=0x5631c6801d10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fd35a5ad700 (LWP 22389)): #0 0x00007fd395c19644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005631c2bc5ba6 in LinuxAIOHandler::collect (this=0x7fd35a5acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005631c2bc60cf in LinuxAIOHandler::poll (this=0x7fd35a5acb00, m1=0x7fd35a5acba8, m2=0x7fd35a5acbb0, request=0x7fd35a5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005631c2bc6232 in os_aio_linux_handler (global_segment=1, m1=0x7fd35a5acba8, m2=0x7fd35a5acbb0, request=0x7fd35a5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005631c2bcad56 in os_aio_handler (segment=1, m1=0x7fd35a5acba8, m2=0x7fd35a5acbb0, request=0x7fd35a5acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005631c2e1ebc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005631c2cc412f in io_handler_thread (arg=0x5631c3e4d428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fd358daa700 (LWP 22392)): #0 0x00007fd395c19644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005631c2bc5ba6 in LinuxAIOHandler::collect (this=0x7fd358da9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005631c2bc60cf in LinuxAIOHandler::poll (this=0x7fd358da9b00, m1=0x7fd358da9ba8, m2=0x7fd358da9bb0, request=0x7fd358da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005631c2bc6232 in os_aio_linux_handler (global_segment=4, m1=0x7fd358da9ba8, m2=0x7fd358da9bb0, request=0x7fd358da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005631c2bcad56 in os_aio_handler (segment=4, m1=0x7fd358da9ba8, m2=0x7fd358da9bb0, request=0x7fd358da9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005631c2e1ebc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005631c2cc412f in io_handler_thread (arg=0x5631c3e4d440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fd37c0d2700 (LWP 22480)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd38de4b132 in fifo_lock_get (q=q@entry=0x5631c671ba30) at galerautils/src/gu_fifo.c:262 #2 0x00007fd38de4b7a1 in gu_fifo_get_head (q=0x5631c671ba30, err=0x7fd37c0d106c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd38df2d0cc in gcs_recv (conn=0x5631c66f53b0, action=0x7fd37c0d10b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd38df5a16c in galera::GcsActionSource::process (this=0x5631c6725dd0, recv_ctx=0x7fd37c0d14d0, exit_loop=@0x7fd37c0d129e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd38df79033 in galera::ReplicatorSMM::async_recv (this=0x5631c66ed160, recv_ctx=0x7fd37c0d14d0) at galera/src/replicator_smm.cpp:383 #6 0x00005631c314520e in wsrep::wsrep_provider_v26::run_applier (this=0x5631c6662d80, applier_ctx=0x7fd37c0d14d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005631c27e82b4 in wsrep_replication_process (thd=0x7fd304000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005631c27d9f01 in start_wsrep_THD (arg=0x7fd36809c570) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fd340ff9700 (LWP 22408)): #0 0x00007fd39604fe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005631c2bd2b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005631c2d79755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fd35adae700 (LWP 22388)): #0 0x00007fd395c19644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005631c2bc5ba6 in LinuxAIOHandler::collect (this=0x7fd35adadb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005631c2bc60cf in LinuxAIOHandler::poll (this=0x7fd35adadb00, m1=0x7fd35adadba8, m2=0x7fd35adadbb0, request=0x7fd35adadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005631c2bc6232 in os_aio_linux_handler (global_segment=0, m1=0x7fd35adadba8, m2=0x7fd35adadbb0, request=0x7fd35adadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005631c2bcad56 in os_aio_handler (segment=0, m1=0x7fd35adadba8, m2=0x7fd35adadbb0, request=0x7fd35adadbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005631c2e1ebc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005631c2cc412f in io_handler_thread (arg=0x5631c3e4d420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fd37ec7c700 (LWP 22219)): #0 0x00007fd394407483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd38def86f8 in asio::detail::epoll_reactor::run (this=0x5631c672df70, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fd38defcdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5631c672de10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fd38defd0a2 in asio::detail::task_io_service::run (this=0x5631c672de10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fd38df14afd in asio::io_service::run (this=0x5631c672cb30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fd38df15251 in gcomm::AsioProtonet::event_loop (this=0x5631c672ca90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fd38df332ed in GCommConn::run (this=0x5631c672d5a0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fd38df36399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fd3417fa700 (LWP 22407)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd2889 in os_event::wait (this=0x5631c683e300) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005631c2bd230f in os_event::wait_low (this=0x5631c683e300, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005631c2bd2680 in os_event_wait_low (event=0x5631c683e300, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005631c2d9edde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fd351ffb700 (LWP 22398)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd21a4 in os_event::timed_wait (this=0x5631c683e200, abstime=0x7fd351ffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005631c2bd24bb in os_event::wait_time_low (this=0x5631c683e200, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005631c2bd265b in os_event_wait_time_low (event=0x5631c683e200, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005631c2cc0808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fd38eaa1700 (LWP 22217)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c3091cab in safe_cond_timedwait (cond=0x5631c4671440 , mp=0x5631c4671380 , abstime=0x7fd38eaa0d70, file=0x5631c351b360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005631c3092c65 in inline_mysql_cond_timedwait (that=0x5631c4671440 , mutex=0x5631c4671380 , abstime=0x7fd38eaa0d70, src_file=0x5631c351b3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005631c30937c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005631c3024471 in pfs_spawn_thread (arg=0x5631c66b6700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fd343fff700 (LWP 22402)): #0 0x00007fd39604fe3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005631c2bd2b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005631c2cc1b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005631c2cc1c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fd38c7a8700 (LWP 22222)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c30919a9 in safe_cond_wait (cond=0x7fd36c016938, mp=0x7fd36c016888, file=0x5631c31faa00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005631c27e7d06 in inline_mysql_cond_wait (that=0x7fd36c016938, mutex=0x7fd36c016888, src_file=0x5631c31fad20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005631c27ea99e in Wsrep_thd_queue::pop_front (this=0x7fd36c016830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005631c27e977e in wsrep_post_rollback_process (post_rollbacker=0x7fd36c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005631c27d9f01 in start_wsrep_THD (arg=0x5631c6745b60) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fd3277fe700 (LWP 22410)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c30919a9 in safe_cond_wait (cond=0x7fd3277fdc70, mp=0x7fd3277fdcb0, file=0x5631c335e860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005631c2ae9a13 in inline_mysql_cond_wait (that=0x7fd3277fdc70, mutex=0x7fd3277fdcb0, src_file=0x5631c33603a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005631c2aee632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x00005631c3024471 in pfs_spawn_thread (arg=0x5631c69f8c10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fd38d8f5700 (LWP 22218)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd38df56fd7 in gu::Lock::wait (cond=..., this=0x7fd38d8f4bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd38df57069 in galera::ServiceThd::thd_func (arg=0x5631c66ed878) at galera/src/galera_service_thd.cpp:30 #3 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fd37c087700 (LWP 22481)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd38de4b132 in fifo_lock_get (q=q@entry=0x5631c671ba30) at galerautils/src/gu_fifo.c:262 #2 0x00007fd38de4b7a1 in gu_fifo_get_head (q=0x5631c671ba30, err=0x7fd37c08606c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd38df2d0cc in gcs_recv (conn=0x5631c66f53b0, action=0x7fd37c0860b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd38df5a16c in galera::GcsActionSource::process (this=0x5631c6725dd0, recv_ctx=0x7fd37c0864d0, exit_loop=@0x7fd37c08629e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd38df79033 in galera::ReplicatorSMM::async_recv (this=0x5631c66ed160, recv_ctx=0x7fd37c0864d0) at galera/src/replicator_smm.cpp:383 #6 0x00005631c314520e in wsrep::wsrep_provider_v26::run_applier (this=0x5631c6662d80, applier_ctx=0x7fd37c0864d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005631c27e82b4 in wsrep_replication_process (thd=0x7fd308000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005631c27d9f01 in start_wsrep_THD (arg=0x7fd36806a850) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fd352ffd700 (LWP 22399)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd21a4 in os_event::timed_wait (this=0x5631c683e280, abstime=0x7fd352ffcd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005631c2bd24bb in os_event::wait_time_low (this=0x5631c683e280, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005631c2bd265b in os_event_wait_time_low (event=0x5631c683e280, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005631c2cc02e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fd341ffb700 (LWP 22406)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd2889 in os_event::wait (this=0x5631c683e100) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005631c2bd230f in os_event::wait_low (this=0x5631c683e100, reset_sig_count=21931) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005631c2bd2680 in os_event_wait_low (event=0x5631c683e100, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005631c2cbe5ff in srv_resume_thread (slot=0x5631c3c0f5b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005631c2cc2203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fd38c06d700 (LWP 22479)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd38de4b132 in fifo_lock_get (q=q@entry=0x5631c671ba30) at galerautils/src/gu_fifo.c:262 #2 0x00007fd38de4b7a1 in gu_fifo_get_head (q=0x5631c671ba30, err=0x7fd38c06c06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd38df2d0cc in gcs_recv (conn=0x5631c66f53b0, action=0x7fd38c06c0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd38df5a16c in galera::GcsActionSource::process (this=0x5631c6725dd0, recv_ctx=0x7fd38c06c4d0, exit_loop=@0x7fd38c06c29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd38df79033 in galera::ReplicatorSMM::async_recv (this=0x5631c66ed160, recv_ctx=0x7fd38c06c4d0) at galera/src/replicator_smm.cpp:383 #6 0x00005631c314520e in wsrep::wsrep_provider_v26::run_applier (this=0x5631c6662d80, applier_ctx=0x7fd38c06c4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005631c27e82b4 in wsrep_replication_process (thd=0x7fd310000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005631c27d9f01 in start_wsrep_THD (arg=0x7fd3680cf9b0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fd3527fc700 (LWP 22397)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd21a4 in os_event::timed_wait (this=0x5631c6a16310, abstime=0x7fd3527fbd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005631c2bd24bb in os_event::wait_time_low (this=0x5631c6a16310, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005631c2bd265b in os_event_wait_time_low (event=0x5631c6a16310, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005631c2b9f314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fd3427fc700 (LWP 22405)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd2889 in os_event::wait (this=0x5631c683e180) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005631c2bd230f in os_event::wait_low (this=0x5631c683e180, reset_sig_count=19561) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005631c2bd2680 in os_event_wait_low (event=0x5631c683e180, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005631c2cbe5ff in srv_resume_thread (slot=0x5631c3c0f5f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005631c2cc2203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fd3537fe700 (LWP 22394)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd21a4 in os_event::timed_wait (this=0x5631c683e380, abstime=0x7fd3537fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005631c2bd24bb in os_event::wait_time_low (this=0x5631c683e380, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005631c2bd265b in os_event_wait_time_low (event=0x5631c683e380, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005631c2da8c57 in pc_sleep_if_needed (next_loop_time=1549299315046, sig_count=2, cur_time=1549299314046) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x00005631c2da9b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fd359dac700 (LWP 22390)): #0 0x00007fd395c19644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005631c2bc5ba6 in LinuxAIOHandler::collect (this=0x7fd359dabb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005631c2bc60cf in LinuxAIOHandler::poll (this=0x7fd359dabb00, m1=0x7fd359dabba8, m2=0x7fd359dabbb0, request=0x7fd359dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005631c2bc6232 in os_aio_linux_handler (global_segment=2, m1=0x7fd359dabba8, m2=0x7fd359dabbb0, request=0x7fd359dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005631c2bcad56 in os_aio_handler (segment=2, m1=0x7fd359dabba8, m2=0x7fd359dabbb0, request=0x7fd359dabbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005631c2e1ebc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005631c2cc412f in io_handler_thread (arg=0x5631c3e4d430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fd3437fe700 (LWP 22403)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c2bd2889 in os_event::wait (this=0x5631c683e000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005631c2bd230f in os_event::wait_low (this=0x5631c683e000, reset_sig_count=5566) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005631c2bd2680 in os_event_wait_low (event=0x5631c683e000, reset_sig_count=5566) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005631c2cbe5ff in srv_resume_thread (slot=0x5631c3c0f548 , sig_count=5566, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005631c2cc2610 in srv_purge_coordinator_suspend (slot=0x5631c3c0f548 , rseg_history_len=14) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x00005631c2cc28a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fd35bfff700 (LWP 22387)): #0 0x00007fd39604cd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c3091cab in safe_cond_timedwait (cond=0x5631c465eca0 , mp=0x5631c465ebe0 , abstime=0x7fd35bffecb0, file=0x5631c34d7130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005631c2efbadd in inline_mysql_cond_timedwait (that=0x5631c465eca0 , mutex=0x5631c465ebe0 , abstime=0x7fd35bffecb0, src_file=0x5631c34d7178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005631c2efc001 in my_service_thread_sleep (control=0x5631c3c13900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005631c2ef049a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005631c3024471 in pfs_spawn_thread (arg=0x5631c67f64c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fd38c75d700 (LWP 22223)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd38de4b132 in fifo_lock_get (q=q@entry=0x5631c671ba30) at galerautils/src/gu_fifo.c:262 #2 0x00007fd38de4b7a1 in gu_fifo_get_head (q=0x5631c671ba30, err=0x7fd38c75c06c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd38df2d0cc in gcs_recv (conn=0x5631c66f53b0, action=0x7fd38c75c0b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd38df5a16c in galera::GcsActionSource::process (this=0x5631c6725dd0, recv_ctx=0x7fd38c75c4d0, exit_loop=@0x7fd38c75c29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd38df79033 in galera::ReplicatorSMM::async_recv (this=0x5631c66ed160, recv_ctx=0x7fd38c75c4d0) at galera/src/replicator_smm.cpp:383 #6 0x00005631c314520e in wsrep::wsrep_provider_v26::run_applier (this=0x5631c6662d80, applier_ctx=0x7fd38c75c4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005631c27e82b4 in wsrep_replication_process (thd=0x7fd364000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005631c27d9f01 in start_wsrep_THD (arg=0x5631c674e670) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fd3595ab700 (LWP 22391)): #0 0x00007fd395c19644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005631c2bc5ba6 in LinuxAIOHandler::collect (this=0x7fd3595aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005631c2bc60cf in LinuxAIOHandler::poll (this=0x7fd3595aab00, m1=0x7fd3595aaba8, m2=0x7fd3595aabb0, request=0x7fd3595aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005631c2bc6232 in os_aio_linux_handler (global_segment=3, m1=0x7fd3595aaba8, m2=0x7fd3595aabb0, request=0x7fd3595aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005631c2bcad56 in os_aio_handler (segment=3, m1=0x7fd3595aaba8, m2=0x7fd3595aabb0, request=0x7fd3595aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005631c2e1ebc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005631c2cc412f in io_handler_thread (arg=0x5631c3e4d438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fd37e47b700 (LWP 22220)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd38df31867 in gu::Lock::wait (cond=..., this=0x7fd37e47a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd38df343ab in RecvBuf::front (this=this@entry=0x5631c672d788, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fd38df31f46 in gcomm_recv (backend=, msg=0x5631c66f5690, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fd38df25eec in core_msg_recv (backend=backend@entry=0x5631c66f57d8, recv_msg=recv_msg@entry=0x5631c66f5690, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fd38df29164 in gcs_core_recv (conn=0x5631c66f5630, recv_act=recv_act@entry=0x7fd37e47ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fd38df303d8 in gcs_recv_thread (arg=0x5631c66f53b0) at gcs/src/gcs.cpp:1398 #7 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fd38c7f3700 (LWP 22221)): #0 0x00007fd39604c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005631c30919a9 in safe_cond_wait (cond=0x7fd374016878, mp=0x7fd3740167c8, file=0x5631c31faa00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005631c27e7d06 in inline_mysql_cond_wait (that=0x7fd374016878, mutex=0x7fd3740167c8, src_file=0x5631c31fad20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005631c27ea99e in Wsrep_thd_queue::pop_front (this=0x7fd374016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005631c27e93be in wsrep_rollback_process (rollbacker=0x7fd374001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005631c27d9f01 in start_wsrep_THD (arg=0x5631c674e9f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fd396048dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd394406ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fd396459880 (LWP 22215)): #0 0x00007fd39604d9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005631c308c37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005631c28c8d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd3943fc20d in poll () from /lib64/libc.so.6 #5 0x00005631c246a96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005631c2469c05 in mysqld_main (argc=162, argv=0x5631c656ed20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005631c245dda5 in main (argc=23, argv=0x7ffdbf42c5c8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera-features#56-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera-features#56-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera-features#56 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_bf_abort 'innodb' [ pass ] 18915 galera_sr.galera_sr_blob 'innodb' [ pass ] 5159 galera_sr.galera_sr_cc_master 'innodb' [ fail ] Test ended at 2019-02-04 19:57:17 CURRENT_TEST: galera_sr.galera_sr_cc_master Server [mysqld.2 - pid: 22713, winpid: 22713, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2019-02-04 19:56:45 41 [Note] WSREP: Stop replication 2019-02-04 19:56:45 41 [Note] WSREP: Server status change synced -> disconnecting 2019-02-04 19:56:45 41 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-04 19:56:45 41 [Note] WSREP: Closing send monitor... 2019-02-04 19:56:45 41 [Note] WSREP: Closed send monitor. 2019-02-04 19:56:45 41 [Note] WSREP: gcomm: terminating thread 2019-02-04 19:56:45 41 [Note] WSREP: gcomm: joining thread 2019-02-04 19:56:45 41 [Note] WSREP: gcomm: closing backend 2019-02-04 19:56:48 41 [Note] WSREP: (c67481db, 'tcp://0.0.0.0:16005') connection to peer c544dae1 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-04 19:56:48 41 [Note] WSREP: (c67481db, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16002 2019-02-04 19:56:50 41 [Note] WSREP: (c67481db, 'tcp://0.0.0.0:16005') reconnecting to c544dae1 (tcp://127.0.0.1:16002), attempt 0 2019-02-04 19:56:53 41 [Note] WSREP: (c67481db, 'tcp://0.0.0.0:16005') connection to peer c544dae1 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-04 19:56:55 41 [Note] WSREP: (c67481db, 'tcp://0.0.0.0:16005') connection established to c544dae1 tcp://127.0.0.1:16002 2019-02-04 19:56:55 41 [Note] WSREP: evs::proto(c67481db, LEAVING, view_id(REG,c544dae1,2)) suspecting node: c544dae1 2019-02-04 19:56:55 41 [Note] WSREP: evs::proto(c67481db, LEAVING, view_id(REG,c544dae1,2)) suspected node without join message, declaring inactive 2019-02-04 19:56:55 41 [Note] WSREP: view(view_id(NON_PRIM,c544dae1,2) memb { c67481db,0 } joined { } left { } partitioned { c544dae1,0 }) 2019-02-04 19:56:55 41 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-04 19:56:55 41 [Note] WSREP: view((empty)) 2019-02-04 19:56:55 41 [Note] WSREP: gcomm: closed 2019-02-04 19:56:55 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-04 19:56:55 0 [Note] WSREP: Writing down CC checksum: 2f3c0a2d 54793b57 53bbbfb2 19ff692a at offset 144 2019-02-04 19:56:55 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-04 19:56:55 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-04 19:56:55 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-04 19:56:55 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 64) 2019-02-04 19:56:55 0 [Note] WSREP: Received self-leave message. 2019-02-04 19:56:55 0 [Note] WSREP: Writing down CC checksum: 6b50c4fe ec84d8ca 6f23af8d 8bbf688c at offset 64 2019-02-04 19:56:55 1 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-04 19:56:55 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-04 19:56:55 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-04 19:56:55 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-04 19:56:55 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-04 19:56:55 1 [Note] WSREP: ####### drain monitors upto 64 2019-02-04 19:56:55 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-04 19:56:55 1 [Note] WSREP: ####### My UUID: c67481db-289d-11e9-beaf-1b91ec718868 2019-02-04 19:56:55 1 [Note] WSREP: ####### ST not required 2019-02-04 19:56:55 41 [Note] WSREP: recv_thread() joined. 2019-02-04 19:56:55 41 [Note] WSREP: Closing replication queue. 2019-02-04 19:56:55 41 [Note] WSREP: Closing slave action queue. 2019-02-04 19:56:55 1 [Note] WSREP: ================================================ View: id: c54fd4cd-289d-11e9-91c0-03784c7ca9b8:-1 seqno: -1 status: 1 prococol_version: 4 own_index: 0 final: 0 members 2019-02-04 19:56:55 1 [Note] WSREP: id: c67481db-289d-11e9-beaf-1b91ec718868 name: cnts7gl1.localdomain 2019-02-04 19:56:55 1 [Note] WSREP: ================================================= 2019-02-04 19:56:55 1 [Note] WSREP: Non-primary view 2019-02-04 19:56:55 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-04 19:56:55 1 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-04 19:56:55 1 [Note] WSREP: ####### drain monitors upto 64 2019-02-04 19:56:55 1 [Note] WSREP: ####### My UUID: c67481db-289d-11e9-beaf-1b91ec718868 2019-02-04 19:56:55 1 [Note] WSREP: ####### ST not required 2019-02-04 19:56:55 1 [Note] WSREP: ================================================ View: id: c54fd4cd-289d-11e9-91c0-03784c7ca9b8:-1 seqno: -1 status: 1 prococol_version: 4 own_index: -1 final: 1 members 2019-02-04 19:56:55 1 [Note] WSREP: ================================================= 2019-02-04 19:56:55 1 [Note] WSREP: Non-primary view 2019-02-04 19:56:55 1 [Note] WSREP: Server status change disconnecting -> disconnected 2019-02-04 19:56:55 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-04 19:56:55 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-04 19:56:55 1 [Note] WSREP: Applier thread exiting 0 2019-02-04 19:56:55 35 [Warning] WSREP: Failed to replicate rollback fragment for 264: 3 2019-02-04 19:56:57 3 [Note] WSREP: rollbacker thread exiting 2019-02-04 19:56:58 41 [Note] WSREP: Stop replication 2019-02-04 19:57:00 42 [Note] WSREP: rollbacker thread exiting 2019-02-04 19:57:00 41 [Note] WSREP: Start replication 2019-02-04 19:57:00 41 [Note] WSREP: Connecting with bootstrap option: 0 2019-02-04 19:57:00 41 [Note] WSREP: Setting GCS initial position to c54fd4cd-289d-11e9-91c0-03784c7ca9b8:64 2019-02-04 19:57:00 41 [Note] WSREP: protonet asio version 0 2019-02-04 19:57:00 41 [Note] WSREP: Using CRC-32C for message checksums. 2019-02-04 19:57:00 41 [Note] WSREP: backend: asio 2019-02-04 19:57:00 41 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-02-04 19:57:00 41 [Warning] WSREP: access file(/home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data//gvwstate.dat) failed(No such file or directory) 2019-02-04 19:57:00 41 [Note] WSREP: restore pc from disk failed 2019-02-04 19:57:00 41 [Note] WSREP: GMCast version 0 2019-02-04 19:57:00 41 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') listening at tcp://0.0.0.0:16005 2019-02-04 19:57:00 41 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') multicast: , ttl: 1 2019-02-04 19:57:00 41 [Note] WSREP: EVS version 1 2019-02-04 19:57:00 41 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '127.0.0.1:16002' 2019-02-04 19:57:00 41 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') connection established to c544dae1 tcp://127.0.0.1:16002 2019-02-04 19:57:00 41 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: 2019-02-04 19:57:00 41 [Note] WSREP: EVS version upgrade 0 -> 1 2019-02-04 19:57:00 41 [Note] WSREP: declaring c544dae1 at tcp://127.0.0.1:16002 stable 2019-02-04 19:57:00 41 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-02-04 19:57:00 41 [Note] WSREP: Node c544dae1 state prim 2019-02-04 19:57:00 41 [Note] WSREP: view(view_id(PRIM,c544dae1,4) memb { c544dae1,0 e379cbe8,0 } joined { } left { } partitioned { }) 2019-02-04 19:57:00 41 [Note] WSREP: save pc into disk 2019-02-04 19:57:01 41 [Note] WSREP: gcomm: connected 2019-02-04 19:57:01 41 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-02-04 19:57:01 41 [Note] WSREP: Shifting CLOSED -> OPEN (TO: -1) 2019-02-04 19:57:01 41 [Note] WSREP: Opened channel 'my_wsrep_cluster' 2019-02-04 19:57:01 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2019-02-04 19:57:01 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-02-04 19:57:01 0 [Note] WSREP: STATE EXCHANGE: sent state msg: e3d290bd-289d-11e9-9fb7-03a9aa5670ef 2019-02-04 19:57:01 0 [Note] WSREP: STATE EXCHANGE: got state msg: e3d290bd-289d-11e9-9fb7-03a9aa5670ef from 0 (cnts7gl1.localdomain) 2019-02-04 19:57:01 0 [Note] WSREP: STATE EXCHANGE: got state msg: e3d290bd-289d-11e9-9fb7-03a9aa5670ef from 1 (cnts7gl1.localdomain) 2019-02-04 19:57:01 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 3, members = 1/2 (joined/total), act_id = 65, last_appl. = 0, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = c54fd4cd-289d-11e9-91c0-03784c7ca9b8 2019-02-04 19:57:01 0 [Note] WSREP: Writing down CC checksum: 758f8131 f73af2b9 4f0ee5d6 d0a485d7 at offset 224 2019-02-04 19:57:01 0 [Note] WSREP: Flow-control interval: [23, 23] 2019-02-04 19:57:01 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-04 19:57:01 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 66) 2019-02-04 19:57:01 45 [Note] WSREP: Starting applier thread 45 2019-02-04 19:57:01 45 [Note] WSREP: ####### processing CC 66, local, ordered 2019-02-04 19:57:01 45 [Note] WSREP: ####### drain monitors upto 64 2019-02-04 19:57:01 45 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-02-04 19:57:01 45 [Note] WSREP: ####### My UUID: e379cbe8-289d-11e9-813c-222cef6ce6a2 2019-02-04 19:57:01 45 [Note] WSREP: Server cnts7gl1.localdomain connected to cluster at position c54fd4cd-289d-11e9-91c0-03784c7ca9b8:66 with ID e379cbe8-289d-11e9-813c-222cef6ce6a2 2019-02-04 19:57:01 45 [Note] WSREP: Server status change disconnected -> connected 2019-02-04 19:57:01 45 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-04 19:57:01 45 [Note] WSREP: State transfer required: Group state: c54fd4cd-289d-11e9-91c0-03784c7ca9b8:66 Local state: c54fd4cd-289d-11e9-91c0-03784c7ca9b8:64 2019-02-04 19:57:01 45 [Note] WSREP: Server status change connected -> joiner 2019-02-04 19:57:01 45 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-04 19:57:01 45 [Note] WSREP: You have configured 'rsync' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2019-02-04 19:57:01 45 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-02-04 19:57:01 0 [Note] WSREP: Service thread queue flushed. 2019-02-04 19:57:01 45 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-02-04 19:57:01 45 [Note] WSREP: ####### IST uuid:c54fd4cd-289d-11e9-91c0-03784c7ca9b8 f: 65, l: 66, p: 10 2019-02-04 19:57:01 45 [Note] WSREP: IST receiver addr using tcp://127.0.0.1:16006 2019-02-04 19:57:01 45 [Note] WSREP: Prepared IST receiver for 65-66, listening at: tcp://127.0.0.1:16006 2019-02-04 19:57:01 0 [Note] WSREP: Member 1.0 (cnts7gl1.localdomain) requested state transfer from '*any*'. Selected 0.0 (cnts7gl1.localdomain)(SYNCED) as donor. 2019-02-04 19:57:01 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 66) 2019-02-04 19:57:01 45 [Note] WSREP: Requesting state transfer: success, donor: 0 2019-02-04 19:57:01 45 [Note] WSREP: Receiving IST: 2 writesets, seqnos 65-66 2019-02-04 19:57:01 0 [Note] WSREP: ####### IST applying starts with 65 2019-02-04 19:57:01 0 [Note] WSREP: ####### IST current seqno initialized to 3 2019-02-04 19:57:01 0 [Note] WSREP: 0.0 (cnts7gl1.localdomain): State transfer to 1.0 (cnts7gl1.localdomain) complete. 2019-02-04 19:57:01 0 [Note] WSREP: Receiving IST... 0.0% ( 0/64 events) complete. 2019-02-04 19:57:01 0 [Note] WSREP: IST preload starting at 3 2019-02-04 19:57:01 0 [Note] WSREP: Service thread queue flushed. 2019-02-04 19:57:01 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:2, protocol version: 5 2019-02-04 19:57:01 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-04 19:57:01 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 66 last received: 5 2019-02-04 19:57:01 0 [Note] WSREP: Member 0.0 (cnts7gl1.localdomain) synced with group. 2019-02-04 19:57:01 45 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) at galera/src/replicator_smm.hpp:pop_front():320. Null event. 2019-02-04 19:57:01 45 [Note] WSREP: Closing send monitor... 2019-02-04 19:57:01 45 [Note] WSREP: Closed send monitor. 2019-02-04 19:57:01 45 [Note] WSREP: gcomm: terminating thread 2019-02-04 19:57:01 45 [Note] WSREP: gcomm: joining thread 2019-02-04 19:57:01 45 [Note] WSREP: gcomm: closing backend 2019-02-04 19:57:03 45 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') turning message relay requesting off 2019-02-04 19:57:04 45 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') connection to peer c544dae1 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-04 19:57:04 45 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16002 2019-02-04 19:57:06 45 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') reconnecting to c544dae1 (tcp://127.0.0.1:16002), attempt 0 2019-02-04 19:57:09 45 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') connection to peer c544dae1 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-04 19:57:11 45 [Note] WSREP: (e379cbe8, 'tcp://0.0.0.0:16005') connection established to c544dae1 tcp://127.0.0.1:16002 2019-02-04 19:57:12 45 [Note] WSREP: evs::proto(e379cbe8, LEAVING, view_id(REG,c544dae1,4)) suspecting node: c544dae1 2019-02-04 19:57:12 45 [Note] WSREP: evs::proto(e379cbe8, LEAVING, view_id(REG,c544dae1,4)) suspected node without join message, declaring inactive 2019-02-04 19:57:12 45 [Note] WSREP: view(view_id(NON_PRIM,c544dae1,4) memb { e379cbe8,0 } joined { } left { } partitioned { c544dae1,0 }) 2019-02-04 19:57:12 45 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-04 19:57:12 45 [Note] WSREP: view((empty)) 2019-02-04 19:57:12 45 [Note] WSREP: gcomm: closed 2019-02-04 19:57:12 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-04 19:57:12 0 [Note] WSREP: Writing down CC checksum: c5b1544b d07f41dd 0accd1e8 2804c1d9 at offset 144 2019-02-04 19:57:12 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-04 19:57:12 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-04 19:57:12 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-04 19:57:12 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 66) 2019-02-04 19:57:12 0 [Note] WSREP: Received self-leave message. 2019-02-04 19:57:12 0 [Note] WSREP: Writing down CC checksum: 6b50c4fe ec84d8ca 6f23af8d 8bbf688c at offset 64 2019-02-04 19:57:12 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-04 19:57:12 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-04 19:57:12 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-04 19:57:12 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-04 19:57:12 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-04 19:57:12 45 [Note] WSREP: recv_thread() joined. 2019-02-04 19:57:12 45 [Note] WSREP: Closing replication queue. 2019-02-04 19:57:12 45 [Note] WSREP: Closing slave action queue. mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. 190204 19:57:12 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.3-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63459 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f924400fc40 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f92606759a0 thread_stack 0x49000 mysys/stacktrace.c:269(my_print_stacktrace)[0x555949cbf28c] sql/signal_handler.cc:168(handle_fatal_signal)[0x5559494fb939] sigaction.c:0(__restore_rt)[0x7f926baa15d0] :0(__GI_raise)[0x7f9269d90207] :0(__GI_abort)[0x7f9269d918f8] :0(__assert_fail_base)[0x7f9269d89026] :0(__GI___assert_fail)[0x7f9269d890d2] src/replicator_str.cpp:945(galera::ReplicatorSMM::request_state_transfer(void*, wsrep_uuid const&, long, void const*, long))[0x7f92639dbdc7] src/replicator_smm.cpp:2733(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7f92639cae78] src/gcs_action_source.cpp:154(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f92639aaec9] src/gcs_action_source.cpp:188(galera::GcsActionSource::process(void*, bool&))[0x7f92639ab21d] src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7f92639ca033] src/wsrep_provider_v26.cpp:646(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x555949d7820e] sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x55594941b2b4] sql/wsrep_mysqld.cc:2770(start_wsrep_THD(void*))[0x55594940cf01] pthread_create.c:0(start_thread)[0x7f926ba99dd5] /lib64/libc.so.6(clone+0x6d)[0x7f9269e57ead] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 45 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file at /home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data/ ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > connection node_1; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2a; SET SESSION wsrep_sync_wait=0; SET GLOBAL wsrep_cluster_address = ''; SET SESSION wsrep_sync_wait = DEFAULT; connection node_2; INSERT INTO t1 VALUES (6); ERROR HY000: Lost connection to MySQL server during query connection node_1; SELECT COUNT(*) = 0 FROM mysql.wsrep_streaming_log; COUNT(*) = 0 1 connection node_2a; connection node_1; connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2b; - found 'core.22715' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.22715 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 22984] [New LWP 22987] [New LWP 22884] [New LWP 22717] [New LWP 22885] [New LWP 22882] [New LWP 22889] [New LWP 22897] [New LWP 22887] [New LWP 22893] [New LWP 22900] [New LWP 22888] [New LWP 22903] [New LWP 22896] [New LWP 22892] [New LWP 22899] [New LWP 22906] [New LWP 22895] [New LWP 22905] [New LWP 22982] [New LWP 22901] [New LWP 22883] [New LWP 22983] [New LWP 22904] [New LWP 22715] [New LWP 22907] [New LWP 22977] [New LWP 22902] [New LWP 22894] [New LWP 22898] [New LWP 22886] [New LWP 22718] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f926ba9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f926ba9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000555949cbf37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005559494fbd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9269d90207 in raise () from /lib64/libc.so.6 #5 0x00007f9269d918f8 in abort () from /lib64/libc.so.6 #6 0x00007f9269d89026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f9269d890d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007f92639dbdc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x55594d113160, recv_ctx=recv_ctx@entry=0x7f92606754d0, group_uuid=..., cc_seqno=cc_seqno@entry=66, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007f92639cae78 in galera::ReplicatorSMM::process_conf_change (this=0x55594d113160, recv_ctx=0x7f92606754d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007f92639aaec9 in galera::GcsActionSource::dispatch (this=this@entry=0x55594d147dd0, recv_ctx=recv_ctx@entry=0x7f92606754d0, act=..., exit_loop=@0x7f926067529e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007f92639ab21d in galera::GcsActionSource::process (this=0x55594d147dd0, recv_ctx=0x7f92606754d0, exit_loop=@0x7f926067529e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007f92639ca033 in galera::ReplicatorSMM::async_recv (this=0x55594d113160, recv_ctx=0x7f92606754d0) at galera/src/replicator_smm.cpp:383 #13 0x0000555949d7820e in wsrep::wsrep_provider_v26::run_applier (this=0x55594d088d80, applier_ctx=0x7f92606754d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #14 0x000055594941b2b4 in wsrep_replication_process (thd=0x7f924400fc40, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x000055594940cf01 in start_wsrep_THD (arg=0x7f91dc041700) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #16 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f9263346700 (LWP 22718)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f92639a7fd7 in gu::Lock::wait (cond=..., this=0x7f9263345bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f92639a8069 in galera::ServiceThd::thd_func (arg=0x55594d113878) at galera/src/galera_service_thd.cpp:30 #3 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f922affd700 (LWP 22886)): #0 0x00007f926b66a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005559497f8ba6 in LinuxAIOHandler::collect (this=0x7f922affcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005559497f90cf in LinuxAIOHandler::poll (this=0x7f922affcb00, m1=0x7f922affcba8, m2=0x7f922affcbb0, request=0x7f922affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005559497f9232 in os_aio_linux_handler (global_segment=3, m1=0x7f922affcba8, m2=0x7f922affcbb0, request=0x7f922affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005559497fdd56 in os_aio_handler (segment=3, m1=0x7f922affcba8, m2=0x7f922affcbb0, request=0x7f922affcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000555949a51bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005559498f712f in io_handler_thread (arg=0x55594aa80438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f92197fa700 (LWP 22898)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949805889 in os_event::wait (this=0x55594d260070) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055594980530f in os_event::wait_low (this=0x55594d260070, reset_sig_count=107) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555949805680 in os_event_wait_low (event=0x55594d260070, reset_sig_count=107) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005559498f15ff in srv_resume_thread (slot=0x55594a842548 , sig_count=107, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005559498f5610 in srv_purge_coordinator_suspend (slot=0x55594a842548 , rseg_history_len=31) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x00005559498f58a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f921b7fe700 (LWP 22894)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005559498051a4 in os_event::timed_wait (this=0x55594d2602f0, abstime=0x7f921b7fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005559498054bb in os_event::wait_time_low (this=0x55594d2602f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055594980565b in os_event_wait_time_low (event=0x55594d2602f0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005559498f32e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f9212ffd700 (LWP 22902)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949805889 in os_event::wait (this=0x55594d260370) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055594980530f in os_event::wait_low (this=0x55594d260370, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555949805680 in os_event_wait_low (event=0x55594d260370, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005559499d1dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f9250b23700 (LWP 22977)): #0 0x00007f9269e4d20d in poll () from /lib64/libc.so.6 #1 0x0000555949d3061e in vio_io_wait (vio=0x55594d5e74b0, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:976 #2 0x0000555949d2ec3e in vio_socket_io_wait (vio=0x55594d5e74b0, event=VIO_IO_EVENT_READ) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:118 #3 0x0000555949d2ee24 in vio_read (vio=0x55594d5e74b0, buf=0x7f91dc00a450 "\a", size=4) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:195 #4 0x00005559496648bc in my_real_read (net=0x7f91dc000df8, complen=0x7f9250b22810, header=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:962 #5 0x0000555949665143 in my_net_read_packet_reallen (net=0x7f91dc000df8, read_from_server=1 '\001', reallen=0x7f9250b22878) at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1243 #6 0x00005559496650f6 in my_net_read_packet (net=0x7f91dc000df8, read_from_server=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1227 #7 0x00005559491c0ed2 in do_command (thd=0x7f91dc000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1229 #8 0x0000555949331c50 in do_handle_one_connection (connect=0x55594d59d720) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #9 0x0000555949331994 in handle_one_connection (arg=0x55594d59d720) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #10 0x0000555949c57471 in pfs_spawn_thread (arg=0x55594d5e9260) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #11 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f9250bb9700 (LWP 22907)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949cc49a9 in safe_cond_wait (cond=0x55594aa04fe0 , mp=0x55594aa08360 , file=0x555949d8eb48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005559490c23ce in inline_mysql_cond_wait (that=0x55594aa04fe0 , mutex=0x55594aa08360 , src_file=0x555949d8f6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005559490c3c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000555949c57471 in pfs_spawn_thread (arg=0x55594d5d4dc0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f926beaa880 (LWP 22715)): #0 0x00007f9269e4d20d in poll () from /lib64/libc.so.6 #1 0x000055594909d96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #2 0x000055594909cc05 in mysqld_main (argc=162, argv=0x55594cf94d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #3 0x0000555949090da5 in main (argc=23, argv=0x7ffcee57d638) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 24 (Thread 0x7f9211ffb700 (LWP 22904)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949805889 in os_event::wait (this=0x55594d260470) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055594980530f in os_event::wait_low (this=0x55594d260470, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555949805680 in os_event_wait_low (event=0x55594d260470, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005559499ba8b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f92606c1700 (LWP 22983)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949cc49a9 in safe_cond_wait (cond=0x7f9248007f58, mp=0x7f9248007ea8, file=0x555949e2da00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055594941ad06 in inline_mysql_cond_wait (that=0x7f9248007f58, mutex=0x7f9248007ea8, src_file=0x555949e2dd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055594941d99e in Wsrep_thd_queue::pop_front (this=0x7f9248007e50) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055594941c77e in wsrep_post_rollback_process (post_rollbacker=0x7f9248001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055594940cf01 in start_wsrep_THD (arg=0x7f91dc040170) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f9250a04700 (LWP 22883)): #0 0x00007f926b66a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005559497f8ba6 in LinuxAIOHandler::collect (this=0x7f9250a03b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005559497f90cf in LinuxAIOHandler::poll (this=0x7f9250a03b00, m1=0x7f9250a03ba8, m2=0x7f9250a03bb0, request=0x7f9250a03be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005559497f9232 in os_aio_linux_handler (global_segment=0, m1=0x7f9250a03ba8, m2=0x7f9250a03bb0, request=0x7f9250a03be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005559497fdd56 in os_aio_handler (segment=0, m1=0x7f9250a03ba8, m2=0x7f9250a03bb0, request=0x7f9250a03be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000555949a51bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005559498f712f in io_handler_thread (arg=0x55594aa80420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f92137fe700 (LWP 22901)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949805889 in os_event::wait (this=0x55594d2601f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055594980530f in os_event::wait_low (this=0x55594d2601f0, reset_sig_count=272) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555949805680 in os_event_wait_low (event=0x55594d2601f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005559498f15ff in srv_resume_thread (slot=0x55594a8425f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005559498f5203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f9250b6e700 (LWP 22982)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949cc49a9 in safe_cond_wait (cond=0x7f9234006e48, mp=0x7f9234006d98, file=0x555949e2da00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055594941ad06 in inline_mysql_cond_wait (that=0x7f9234006e48, mutex=0x7f9234006d98, src_file=0x555949e2dd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055594941d99e in Wsrep_thd_queue::pop_front (this=0x7f9234006d40) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055594941c3be in wsrep_rollback_process (rollbacker=0x7f9234000c40, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055594940cf01 in start_wsrep_THD (arg=0x7f91dc040720) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f92117fa700 (LWP 22905)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949cc49a9 in safe_cond_wait (cond=0x7f92117f9c70, mp=0x7f92117f9cb0, file=0x555949f91860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055594971ca13 in inline_mysql_cond_wait (that=0x7f92117f9c70, mutex=0x7f92117f9cb0, src_file=0x555949f933a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000555949721632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x0000555949c57471 in pfs_spawn_thread (arg=0x55594d41afb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f921affd700 (LWP 22895)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005559498051a4 in os_event::timed_wait (this=0x55594d46a7e0, abstime=0x7f921affcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005559498054bb in os_event::wait_time_low (this=0x55594d46a7e0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055594980565b in os_event_wait_time_low (event=0x55594d46a7e0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555949a3b52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f9250c04700 (LWP 22906)): #0 0x00007f9269d9140a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000555949090e24 in my_sigwait (set=0x7f9250c03cf0, sig=0x7f9250c03cb4, code=0x7f9250c03cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000555949097464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000555949c57471 in pfs_spawn_thread (arg=0x55594d223f10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f9218ff9700 (LWP 22899)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949805889 in os_event::wait (this=0x55594d2600f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055594980530f in os_event::wait_low (this=0x55594d2600f0, reset_sig_count=364) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555949805680 in os_event_wait_low (event=0x55594d2600f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005559498f15ff in srv_resume_thread (slot=0x55594a842580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005559498f5203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f921bfff700 (LWP 22892)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005559498051a4 in os_event::timed_wait (this=0x55594d4387c0, abstime=0x7f921bffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005559498054bb in os_event::wait_time_low (this=0x55594d4387c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055594980565b in os_event_wait_time_low (event=0x55594d4387c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005559497d2314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f921a7fc700 (LWP 22896)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005559498051a4 in os_event::timed_wait (this=0x55594d408100, abstime=0x7f921a7fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005559498054bb in os_event::wait_time_low (this=0x55594d408100, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055594980565b in os_event_wait_time_low (event=0x55594d408100, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000555949953aad in ib_wqueue_timedwait (wq=0x55594d408050, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000555949a95f11 in fts_optimize_thread (arg=0x55594d408050) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f92127fc700 (LWP 22903)): #0 0x00007f926baa0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000555949805b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005559499ac755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f9229ffb700 (LWP 22888)): #0 0x00007f926b66a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005559497f8ba6 in LinuxAIOHandler::collect (this=0x7f9229ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005559497f90cf in LinuxAIOHandler::poll (this=0x7f9229ffab00, m1=0x7f9229ffaba8, m2=0x7f9229ffabb0, request=0x7f9229ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005559497f9232 in os_aio_linux_handler (global_segment=5, m1=0x7f9229ffaba8, m2=0x7f9229ffabb0, request=0x7f9229ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005559497fdd56 in os_aio_handler (segment=5, m1=0x7f9229ffaba8, m2=0x7f9229ffabb0, request=0x7f9229ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000555949a51bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005559498f712f in io_handler_thread (arg=0x55594aa80448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f9213fff700 (LWP 22900)): #0 0x00007f926ba9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949805889 in os_event::wait (this=0x55594d260170) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055594980530f in os_event::wait_low (this=0x55594d260170, reset_sig_count=310) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000555949805680 in os_event_wait_low (event=0x55594d260170, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005559498f15ff in srv_resume_thread (slot=0x55594a8425b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005559498f5203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f9228df4700 (LWP 22893)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005559498051a4 in os_event::timed_wait (this=0x55594d260270, abstime=0x7f9228df3ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005559498054bb in os_event::wait_time_low (this=0x55594d260270, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055594980565b in os_event_wait_time_low (event=0x55594d260270, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005559498f3808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f922a7fc700 (LWP 22887)): #0 0x00007f926b66a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005559497f8ba6 in LinuxAIOHandler::collect (this=0x7f922a7fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005559497f90cf in LinuxAIOHandler::poll (this=0x7f922a7fbb00, m1=0x7f922a7fbba8, m2=0x7f922a7fbbb0, request=0x7f922a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005559497f9232 in os_aio_linux_handler (global_segment=4, m1=0x7f922a7fbba8, m2=0x7f922a7fbbb0, request=0x7f922a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005559497fdd56 in os_aio_handler (segment=4, m1=0x7f922a7fbba8, m2=0x7f922a7fbbb0, request=0x7f922a7fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000555949a51bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005559498f712f in io_handler_thread (arg=0x55594aa80440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f9219ffb700 (LWP 22897)): #0 0x00007f926baa0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000555949805b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005559498f4b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005559498f4c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f92297fa700 (LWP 22889)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005559498051a4 in os_event::timed_wait (this=0x55594d2603f0, abstime=0x7f92297f9af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005559498054bb in os_event::wait_time_low (this=0x55594d2603f0, time_in_usec=602000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055594980565b in os_event_wait_time_low (event=0x55594d2603f0, time_in_usec=602000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005559499dbc57 in pc_sleep_if_needed (next_loop_time=1549299432767, sig_count=2, cur_time=1549299432165) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x00005559499dcb90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f9252056700 (LWP 22882)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949cc4cab in safe_cond_timedwait (cond=0x55594b291ca0 , mp=0x55594b291be0 , abstime=0x7f9252055cb0, file=0x55594a10a130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000555949b2eadd in inline_mysql_cond_timedwait (that=0x55594b291ca0 , mutex=0x55594b291be0 , abstime=0x7f9252055cb0, src_file=0x55594a10a178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000555949b2f001 in my_service_thread_sleep (control=0x55594a846900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000555949b2349a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000555949c57471 in pfs_spawn_thread (arg=0x55594d2186c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f922b7fe700 (LWP 22885)): #0 0x00007f926b66a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005559497f8ba6 in LinuxAIOHandler::collect (this=0x7f922b7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005559497f90cf in LinuxAIOHandler::poll (this=0x7f922b7fdb00, m1=0x7f922b7fdba8, m2=0x7f922b7fdbb0, request=0x7f922b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005559497f9232 in os_aio_linux_handler (global_segment=2, m1=0x7f922b7fdba8, m2=0x7f922b7fdbb0, request=0x7f922b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005559497fdd56 in os_aio_handler (segment=2, m1=0x7f922b7fdba8, m2=0x7f922b7fdbb0, request=0x7f922b7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000555949a51bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005559498f712f in io_handler_thread (arg=0x55594aa80430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f92644f2700 (LWP 22717)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000555949cc4cab in safe_cond_timedwait (cond=0x55594b2a4440 , mp=0x55594b2a4380 , abstime=0x7f92644f1d70, file=0x55594a14e360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000555949cc5c65 in inline_mysql_cond_timedwait (that=0x55594b2a4440 , mutex=0x55594b2a4380 , abstime=0x7f92644f1d70, src_file=0x55594a14e3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000555949cc67c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000555949c57471 in pfs_spawn_thread (arg=0x55594d0dc700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f922bfff700 (LWP 22884)): #0 0x00007f926b66a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005559497f8ba6 in LinuxAIOHandler::collect (this=0x7f922bffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005559497f90cf in LinuxAIOHandler::poll (this=0x7f922bffeb00, m1=0x7f922bffeba8, m2=0x7f922bffebb0, request=0x7f922bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005559497f9232 in os_aio_linux_handler (global_segment=1, m1=0x7f922bffeba8, m2=0x7f922bffebb0, request=0x7f922bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005559497fdd56 in os_aio_handler (segment=1, m1=0x7f922bffeba8, m2=0x7f922bffebb0, request=0x7f922bffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000555949a51bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005559498f712f in io_handler_thread (arg=0x55594aa80428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f926062b700 (LWP 22987)): #0 0x00007f926ba9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9263984bdb in gu::Lock::wait (this=this@entry=0x7f92606286d0, cond=..., date=...) at galerautils/src/gu_lock.hpp:62 #2 0x00007f92639cf976 in galera::Monitor::wait (this=this@entry=0x55594d114100, gtid=..., wait_until=...) at galera/src/monitor.hpp:355 #3 0x00007f92639c157f in galera::ReplicatorSMM::sync_wait (this=0x55594d113160, upto=, tout=, gtid=0x7f9260628b10) at galera/src/replicator_smm.cpp:1661 #4 0x00007f92639e0643 in galera_sync_wait (wsrep=, upto=, tout=, gtid=) at galera/src/wsrep_provider.cpp:998 #5 0x0000555949d78ec5 in wsrep::wsrep_provider_v26::causal_read (this=0x55594d088d80, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:801 #6 0x0000555949d643c5 in wsrep::server_state::causal_read (this=0x55594d0dd670, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:636 #7 0x0000555949d5ed1d in wsrep::client_state::sync_wait (this=0x7f924c0069a0, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/client_state.cpp:417 #8 0x000055594940570c in wsrep_sync_wait (thd=0x7f924c000b20, mask=1) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:1022 #9 0x00005559491c8734 in mysql_execute_command (thd=0x7f924c000b20) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:3776 #10 0x00005559491d7675 in mysql_parse (thd=0x7f924c000b20, rawbuf=0x7f924c016c08 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7f926062a0b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #11 0x00005559491d6ca8 in wsrep_mysql_parse (thd=0x7f924c000b20, rawbuf=0x7f924c016c08 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7f926062a0b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #12 0x00005559491c2b65 in dispatch_command (command=COM_QUERY, thd=0x7f924c000b20, packet=0x7f924c00a301 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", packet_length=102, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #13 0x00005559491c1599 in do_command (thd=0x7f924c000b20) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #14 0x0000555949331c50 in do_handle_one_connection (connect=0x55594d15f020) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #15 0x0000555949331994 in handle_one_connection (arg=0x55594d15f020) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #16 0x0000555949c57471 in pfs_spawn_thread (arg=0x55594d5e7870) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #17 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007f9269e57ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f9260676700 (LWP 22984)): #0 0x00007f926ba9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000555949cbf37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005559494fbd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9269d90207 in raise () from /lib64/libc.so.6 #5 0x00007f9269d918f8 in abort () from /lib64/libc.so.6 #6 0x00007f9269d89026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f9269d890d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007f92639dbdc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x55594d113160, recv_ctx=recv_ctx@entry=0x7f92606754d0, group_uuid=..., cc_seqno=cc_seqno@entry=66, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007f92639cae78 in galera::ReplicatorSMM::process_conf_change (this=0x55594d113160, recv_ctx=0x7f92606754d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007f92639aaec9 in galera::GcsActionSource::dispatch (this=this@entry=0x55594d147dd0, recv_ctx=recv_ctx@entry=0x7f92606754d0, act=..., exit_loop=@0x7f926067529e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007f92639ab21d in galera::GcsActionSource::process (this=0x55594d147dd0, recv_ctx=0x7f92606754d0, exit_loop=@0x7f926067529e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007f92639ca033 in galera::ReplicatorSMM::async_recv (this=0x55594d113160, recv_ctx=0x7f92606754d0) at galera/src/replicator_smm.cpp:383 #13 0x0000555949d7820e in wsrep::wsrep_provider_v26::run_applier (this=0x55594d088d80, applier_ctx=0x7f92606754d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #14 0x000055594941b2b4 in wsrep_replication_process (thd=0x7f924400fc40, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x000055594940cf01 in start_wsrep_THD (arg=0x7f91dc041700) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #16 0x00007f926ba99dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f9269e57ead in clone () from /lib64/libc.so.6 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_cc_master galera_sr.galera_sr_bf_abort galera_sr.galera_sr_blob 2019-02-04 19:56:55 35 [Warning] WSREP: Failed to replicate rollback fragment for 264: 3 2019-02-04 19:57:01 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-04 19:57:01 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 66 last received: 5 2019-02-04 19:57:01 45 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_cc_slave 'innodb' [ pass ] 22738 galera_sr.galeworker[1] Trying to dump core for [mysqltest - pid: 23744, winpid: 23744] worker[1] Trying to dump core for [mysqld.1 - pid: 23391, winpid: 23391] worker[1] Trying to dump core for [mysqld.2 - pid: 23432, winpid: 23432] ra_sr_concurrent 'innodb' [ pass ] 6092 galera_sr.galera_sr_conflict 'innodb' [ pass ] 4596 galera_sr.galera_sr_conflict_on_commit 'innodb' [ pass ] 4923 galera_sr.galera_sr_conflict_on_commit2 'innodb' [ pass ] 5141 galera_sr.galera_sr_conflict_with_rollback_master 'innodb' [ pass ] 4458 galera_sr.galera_sr_ddl_master 'innodb' [ fail ] Test ended at 2019-02-04 19:58:23 CURRENT_TEST: galera_sr.galera_sr_ddl_master --- /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_ddl_master.result 2019-01-23 14:59:37.381821400 +0300 +++ /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_ddl_master.reject 2019-02-04 19:58:22.773820700 +0300 @@ -44,5 +44,5 @@ 1 SELECT COUNT(*) = 0 FROM mysql.wsrep_streaming_log; COUNT(*) = 0 -1 +0 DROP TABLE t1; mysqltest: Result content mismatch - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_ddl_master-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_ddl_master-innodb/' galera_sr.galera_sr_ddl_schema 'innodb' [ pass ] 4927 galera_sr.galera_sr_ddl_slave 'innodb' [ pass ] 5156 galera_sr.galera_sr_ddl_unrelated 'innodb' [ pass ] 7038 galera_sr.galera_sr_dupkey_error 'innodb' [ pass ] 4899 galera_sr.galera_sr_fk_conflict 'innodb' [ pass ] 9811 galera_sr.galera_sr_insert_select 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 20:15:07 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr_insert_select.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) - found 'core.23392' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_insert_select-innodb/mysqld.1/data/core.23392 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 23392] [New LWP 23394] [New LWP 23393] [New LWP 23395] [New LWP 23397] [New LWP 23396] [New LWP 23400] [New LWP 23398] [New LWP 23399] [New LWP 23406] [New LWP 23404] [New LWP 23401] [New LWP 23409] [New LWP 23413] [New LWP 23405] [New LWP 23411] [New LWP 23410] [New LWP 23414] [New LWP 23412] [New LWP 23416] [New LWP 23417] [New LWP 23419] [New LWP 23420] [New LWP 23418] [New LWP 23422] [New LWP 23421] [New LWP 23641] [New LWP 23423] [New LWP 23426] [New LWP 23415] [New LWP 23403] [New LWP 23407] [New LWP 23402] [New LWP 23427] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007fd2831759d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fd2831759d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005564e0bc637a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005564e0402d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd28152420d in poll () from /lib64/libc.so.6 #5 0x00005564dffa496a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005564dffa3c05 in mysqld_main (argc=161, argv=0x5564e21fbd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005564dff97da5 in main (argc=23, argv=0x7ffc891ca958) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7fd27083a700 (LWP 23427)): #0 0x00007fd283177d1d in fsync () from /lib64/libpthread.so.0 #1 0x00005564e0701255 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x00005564e07015ab in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x00005564e094a751 in pfs_os_file_flush_func (file=..., src_file=0x5564e0fc68b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x00005564e094d593 in fil_flush_low (space=0x5564e26d4010, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x00005564e0959113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x00005564e06dcc59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x00005564e06dd6bb in log_write_up_to (lsn=11107185, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x00005564e0848551 in trx_flush_log_if_needed_low (lsn=11107185) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x00005564e084859a in trx_flush_log_if_needed (lsn=11107185, trx=0x7fd2712d9390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x00005564e084a3c3 in trx_commit_complete_for_mysql (trx=0x7fd2712d9390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x00005564e062ea6f in innobase_commit (hton=0x5564e249e7b0, thd=0x7fd210058480, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #12 0x00005564e040889b in commit_one_phase_2 (thd=0x7fd210058480, all=true, trans=0x7fd21005b9c8, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x00005564e040879d in ha_commit_one_phase (thd=0x7fd210058480, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x00005564e0407b19 in ha_commit_trans (thd=0x7fd210058480, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x00005564e024da76 in trans_commit (thd=0x7fd210058480) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x00005564e060c988 in Wsrep_storage_service::commit (this=0x7fd21036f7b0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x00005564e0c7585f in wsrep::transaction::certify_fragment (this=0x7fd2100069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #18 0x00005564e0c74fc3 in wsrep::transaction::streaming_step (this=0x7fd2100069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #19 0x00005564e0c7237e in wsrep::transaction::after_row (this=0x7fd2100069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x00005564e041aba4 in wsrep::client_state::after_row (this=0x7fd210006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x00005564e0403e87 in wsrep_after_row (thd=0x7fd210000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x00005564e0415d26 in wsrep_after_row (thd=0x7fd210000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #23 0x00005564e0416117 in handler::ha_write_row (this=0x7fd21028d188, buf=0x7fd21019a400 "\375\313&") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #24 0x00005564e008c18d in write_record (thd=0x7fd210000b00, table=0x7fd21028c550, info=0x7fd210018450) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x00005564e009148c in select_insert::send_data (this=0x7fd210018408, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #26 0x00005564e0149b47 in end_send (join=0x7fd2100184b8, join_tab=0x7fd2102786f8, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #27 0x00005564e0286b89 in JOIN_CACHE::generate_full_extensions (this=0x7fd210278d08, rec_ptr=0x7fd210325804 "\b\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #28 0x00005564e02868a4 in JOIN_CACHE::join_matching_records (this=0x7fd210278d08, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #29 0x00005564e0286270 in JOIN_CACHE::join_records (this=0x7fd210278d08, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #30 0x00005564e028640a in JOIN_CACHE::join_records (this=0x7fd210278bd8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #31 0x00005564e028640a in JOIN_CACHE::join_records (this=0x7fd210278aa8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #32 0x00005564e01467fb in sub_select_cache (join=0x7fd2100184b8, join_tab=0x7fd210277be8, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #33 0x00005564e0146a08 in sub_select (join=0x7fd2100184b8, join_tab=0x7fd210277838, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #34 0x00005564e01461d3 in do_select (join=0x7fd2100184b8, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #35 0x00005564e011ecf1 in JOIN::exec_inner (this=0x7fd2100184b8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #36 0x00005564e011e13a in JOIN::exec (this=0x7fd2100184b8) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #37 0x00005564e011f42c in mysql_select (thd=0x7fd210000b00, tables=0x7fd210016118, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3491498496, result=0x7fd210018408, unit=0x7fd2100049b8, select_lex=0x7fd210015a48) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #38 0x00005564e0110d2d in handle_select (thd=0x7fd210000b00, lex=0x7fd2100048f0, result=0x7fd210018408, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #39 0x00005564e00d313c in mysql_execute_command (thd=0x7fd210000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4856 #40 0x00005564e00de675 in mysql_parse (thd=0x7fd210000b00, rawbuf=0x7fd210015178 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fd2708390b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #41 0x00005564e00ddca8 in wsrep_mysql_parse (thd=0x7fd210000b00, rawbuf=0x7fd210015178 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=91, parser_state=0x7fd2708390b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #42 0x00005564e00c9b65 in dispatch_command (command=COM_QUERY, thd=0x7fd210000b00, packet=0x7fd210028211 "INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=91, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #43 0x00005564e00c8599 in do_command (thd=0x7fd210000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #44 0x00005564e0238c50 in do_handle_one_connection (connect=0x5564e2806900) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #45 0x00005564e0238994 in handle_one_connection (arg=0x5564e2806900) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #46 0x00005564e0b5e471 in pfs_spawn_thread (arg=0x5564e285ba50) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #47 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #48 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fd2537fe700 (LWP 23402)): #0 0x00007fd282d41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005564e06ffba6 in LinuxAIOHandler::collect (this=0x7fd2537fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005564e07000cf in LinuxAIOHandler::poll (this=0x7fd2537fdb00, m1=0x7fd2537fdba8, m2=0x7fd2537fdbb0, request=0x7fd2537fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005564e0700232 in os_aio_linux_handler (global_segment=1, m1=0x7fd2537fdba8, m2=0x7fd2537fdbb0, request=0x7fd2537fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564e0704d56 in os_aio_handler (segment=1, m1=0x7fd2537fdba8, m2=0x7fd2537fdbb0, request=0x7fd2537fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005564e0958bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005564e07fe12f in io_handler_thread (arg=0x5564e1987428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fd250ff9700 (LWP 23407)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c889 in os_event::wait (this=0x5564e2662820) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005564e070c30f in os_event::wait_low (this=0x5564e2662820, reset_sig_count=6192) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005564e070c680 in os_event_wait_low (event=0x5564e2662820, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005564e06dd1eb in log_write_up_to (lsn=11107206, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x00005564e08dd528 in buf_flush_write_block_low (bpage=0x7fd270aac440, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:962 #6 0x00005564e08ddef0 in buf_flush_page (buf_pool=0x5564e25afec0, bpage=0x7fd270aac440, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1164 #7 0x00005564e08de838 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=2, n_to_flush=5) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1388 #8 0x00005564e08debbb in buf_flush_page_and_try_neighbors (bpage=0x7fd270aac440, flush_type=BUF_FLUSH_LIST, n_to_flush=5, count=0x7fd250ff88c8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1460 #9 0x00005564e08df7d4 in buf_do_flush_list_batch (buf_pool=0x5564e25afec0, min_n=5, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1718 #10 0x00005564e08dfdcc in buf_flush_batch (buf_pool=0x5564e25afec0, flush_type=BUF_FLUSH_LIST, min_n=5, lsn_limit=18446744073709551615, n=0x7fd250ff8b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1788 #11 0x00005564e08e048c in buf_flush_do_batch (buf_pool=0x5564e25afec0, type=BUF_FLUSH_LIST, min_n=5, lsn_limit=18446744073709551615, n=0x7fd250ff8b70) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1946 #12 0x00005564e08e3187 in pc_flush_slot () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2749 #13 0x00005564e08e40d5 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3158 #14 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #15 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fd252ffd700 (LWP 23403)): #0 0x00007fd282d41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005564e06ffba6 in LinuxAIOHandler::collect (this=0x7fd252ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005564e07000cf in LinuxAIOHandler::poll (this=0x7fd252ffcb00, m1=0x7fd252ffcba8, m2=0x7fd252ffcbb0, request=0x7fd252ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005564e0700232 in os_aio_linux_handler (global_segment=2, m1=0x7fd252ffcba8, m2=0x7fd252ffcbb0, request=0x7fd252ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564e0704d56 in os_aio_handler (segment=2, m1=0x7fd252ffcba8, m2=0x7fd252ffcbb0, request=0x7fd252ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005564e0958bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005564e07fe12f in io_handler_thread (arg=0x5564e1987430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fd248ff9700 (LWP 23415)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c889 in os_event::wait (this=0x5564e24ca3c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005564e070c30f in os_event::wait_low (this=0x5564e24ca3c0, reset_sig_count=4990) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005564e070c680 in os_event_wait_low (event=0x5564e24ca3c0, reset_sig_count=4990) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005564e07f85ff in srv_resume_thread (slot=0x5564e1749548 , sig_count=4990, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005564e07fc610 in srv_purge_coordinator_suspend (slot=0x5564e1749548 , rseg_history_len=15) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x00005564e07fc8a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fd270885700 (LWP 23426)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e0bcb9a9 in safe_cond_wait (cond=0x5564e190bfe0 , mp=0x5564e190f360 , file=0x5564e0c95b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564dffc93ce in inline_mysql_cond_wait (that=0x5564e190bfe0 , mutex=0x5564e190f360 , src_file=0x5564e0c966b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005564dffcac60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005564e0b5e471 in pfs_spawn_thread (arg=0x5564e27cbc30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fd278074700 (LWP 23423)): #0 0x00007fd28146840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005564dff97e24 in my_sigwait (set=0x7fd278073cf0, sig=0x7fd278073cb4, code=0x7fd278073cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005564dff9e464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005564e0b5e471 in pfs_spawn_thread (arg=0x5564e248cd00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fd2707ef700 (LWP 23641)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e0bcbcab in safe_cond_timedwait (cond=0x5564e190bfa0 , mp=0x5564e190e9a0 , abstime=0x7fd2707eec60, file=0x5564e0c86348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005564dff98725 in inline_mysql_cond_timedwait (that=0x5564e190bfa0 , mutex=0x5564e190e9a0 , abstime=0x7fd2707eec60, src_file=0x5564e0c883b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005564dff9d4fb in cache_thread (thd=0x7fd208000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x00005564dff9d7a9 in one_thread_per_connection_end (thd=0x7fd208000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x00005564e0238cda in do_handle_one_connection (connect=0x5564e23e2a20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x00005564e0238994 in handle_one_connection (arg=0x5564e23e2a20) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x00005564e0b5e471 in pfs_spawn_thread (arg=0x5564e23369a0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fd2397fa700 (LWP 23421)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c889 in os_event::wait (this=0x5564e24ca7c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005564e070c30f in os_event::wait_low (this=0x5564e24ca7c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005564e070c680 in os_event_wait_low (event=0x5564e24ca7c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005564e08c18b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fd238ff9700 (LWP 23422)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e0bcb9a9 in safe_cond_wait (cond=0x7fd238ff8c70, mp=0x7fd238ff8cb0, file=0x5564e0e98860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564e0623a13 in inline_mysql_cond_wait (that=0x7fd238ff8c70, mutex=0x7fd238ff8cb0, src_file=0x5564e0e9a3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005564e0628632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x00005564e0b5e471 in pfs_spawn_thread (arg=0x5564e270b270) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fd23affd700 (LWP 23418)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c889 in os_event::wait (this=0x5564e24ca540) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005564e070c30f in os_event::wait_low (this=0x5564e24ca540, reset_sig_count=16127) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005564e070c680 in os_event_wait_low (event=0x5564e24ca540, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005564e07f85ff in srv_resume_thread (slot=0x5564e17495f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005564e07fc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fd239ffb700 (LWP 23420)): #0 0x00007fd283177e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005564e070cb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005564e08b3755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fd23a7fc700 (LWP 23419)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c889 in os_event::wait (this=0x5564e24ca6c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005564e070c30f in os_event::wait_low (this=0x5564e24ca6c0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005564e070c680 in os_event_wait_low (event=0x5564e24ca6c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005564e08d8dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fd23b7fe700 (LWP 23417)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c889 in os_event::wait (this=0x5564e24ca440) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005564e070c30f in os_event::wait_low (this=0x5564e24ca440, reset_sig_count=24179) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005564e070c680 in os_event_wait_low (event=0x5564e24ca440, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005564e07f85ff in srv_resume_thread (slot=0x5564e1749580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005564e07fc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fd23bfff700 (LWP 23416)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c889 in os_event::wait (this=0x5564e24ca4c0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005564e070c30f in os_event::wait_low (this=0x5564e24ca4c0, reset_sig_count=20601) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005564e070c680 in os_event_wait_low (event=0x5564e24ca4c0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005564e07f85ff in srv_resume_thread (slot=0x5564e17495b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005564e07fc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fd24a7fc700 (LWP 23412)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c1a4 in os_event::timed_wait (this=0x5564e26d46f0, abstime=0x7fd24a7fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005564e070c4bb in os_event::wait_time_low (this=0x5564e26d46f0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005564e070c65b in os_event_wait_time_low (event=0x5564e26d46f0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005564e094252c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fd2497fa700 (LWP 23414)): #0 0x00007fd283177e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005564e070cb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005564e07fbb67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005564e07fbc97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fd24affd700 (LWP 23410)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c1a4 in os_event::timed_wait (this=0x5564e24ca5c0, abstime=0x7fd24affcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005564e070c4bb in os_event::wait_time_low (this=0x5564e24ca5c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005564e070c65b in os_event_wait_time_low (event=0x5564e24ca5c0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005564e07fa808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fd24bfff700 (LWP 23411)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c1a4 in os_event::timed_wait (this=0x5564e24ca640, abstime=0x7fd24bffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005564e070c4bb in os_event::wait_time_low (this=0x5564e24ca640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005564e070c65b in os_event_wait_time_low (event=0x5564e24ca640, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005564e07fa2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fd251ffb700 (LWP 23405)): #0 0x00007fd282d41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005564e06ffba6 in LinuxAIOHandler::collect (this=0x7fd251ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005564e07000cf in LinuxAIOHandler::poll (this=0x7fd251ffab00, m1=0x7fd251ffaba8, m2=0x7fd251ffabb0, request=0x7fd251ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005564e0700232 in os_aio_linux_handler (global_segment=4, m1=0x7fd251ffaba8, m2=0x7fd251ffabb0, request=0x7fd251ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564e0704d56 in os_aio_handler (segment=4, m1=0x7fd251ffaba8, m2=0x7fd251ffabb0, request=0x7fd251ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005564e0958bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005564e07fe12f in io_handler_thread (arg=0x5564e1987440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fd249ffb700 (LWP 23413)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c1a4 in os_event::timed_wait (this=0x5564e266a1d0, abstime=0x7fd249ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005564e070c4bb in os_event::wait_time_low (this=0x5564e266a1d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005564e070c65b in os_event_wait_time_low (event=0x5564e266a1d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005564e085aaad in ib_wqueue_timedwait (wq=0x5564e266a120, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005564e099cf11 in fts_optimize_thread (arg=0x5564e266a120) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fd24b7fe700 (LWP 23409)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e070c1a4 in os_event::timed_wait (this=0x5564e26a26d0, abstime=0x7fd24b7fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005564e070c4bb in os_event::wait_time_low (this=0x5564e26a26d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005564e070c65b in os_event_wait_time_low (event=0x5564e26a26d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005564e06d9314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fd253fff700 (LWP 23401)): #0 0x00007fd282d41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005564e06ffba6 in LinuxAIOHandler::collect (this=0x7fd253ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005564e07000cf in LinuxAIOHandler::poll (this=0x7fd253ffeb00, m1=0x7fd253ffeba8, m2=0x7fd253ffebb0, request=0x7fd253ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005564e0700232 in os_aio_linux_handler (global_segment=0, m1=0x7fd253ffeba8, m2=0x7fd253ffebb0, request=0x7fd253ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564e0704d56 in os_aio_handler (segment=0, m1=0x7fd253ffeba8, m2=0x7fd253ffebb0, request=0x7fd253ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005564e0958bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005564e07fe12f in io_handler_thread (arg=0x5564e1987420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fd2527fc700 (LWP 23404)): #0 0x00007fd282d41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005564e06ffba6 in LinuxAIOHandler::collect (this=0x7fd2527fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005564e07000cf in LinuxAIOHandler::poll (this=0x7fd2527fbb00, m1=0x7fd2527fbba8, m2=0x7fd2527fbbb0, request=0x7fd2527fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005564e0700232 in os_aio_linux_handler (global_segment=3, m1=0x7fd2527fbba8, m2=0x7fd2527fbbb0, request=0x7fd2527fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564e0704d56 in os_aio_handler (segment=3, m1=0x7fd2527fbba8, m2=0x7fd2527fbbb0, request=0x7fd2527fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005564e0958bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005564e07fe12f in io_handler_thread (arg=0x5564e1987438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fd2517fa700 (LWP 23406)): #0 0x00007fd282d41644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005564e06ffba6 in LinuxAIOHandler::collect (this=0x7fd2517f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005564e07000cf in LinuxAIOHandler::poll (this=0x7fd2517f9b00, m1=0x7fd2517f9ba8, m2=0x7fd2517f9bb0, request=0x7fd2517f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005564e0700232 in os_aio_linux_handler (global_segment=5, m1=0x7fd2517f9ba8, m2=0x7fd2517f9bb0, request=0x7fd2517f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005564e0704d56 in os_aio_handler (segment=5, m1=0x7fd2517f9ba8, m2=0x7fd2517f9bb0, request=0x7fd2517f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005564e0958bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005564e07fe12f in io_handler_thread (arg=0x5564e1987448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fd278483700 (LWP 23399)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd27af73132 in fifo_lock_get (q=q@entry=0x5564e23a8b20) at galerautils/src/gu_fifo.c:262 #2 0x00007fd27af737a1 in gu_fifo_get_head (q=0x5564e23a8b20, err=0x7fd27848206c) at galerautils/src/gu_fifo.c:330 #3 0x00007fd27b0550cc in gcs_recv (conn=0x5564e23824a0, action=0x7fd2784820b0) at gcs/src/gcs.cpp:2024 #4 0x00007fd27b08216c in galera::GcsActionSource::process (this=0x5564e2381e80, recv_ctx=0x7fd2784824d0, exit_loop=@0x7fd27848229e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fd27b0a1033 in galera::ReplicatorSMM::async_recv (this=0x5564e237a1b0, recv_ctx=0x7fd2784824d0) at galera/src/replicator_smm.cpp:383 #6 0x00005564e0c7f20e in wsrep::wsrep_provider_v26::run_applier (this=0x5564e2301280, applier_ctx=0x7fd2784824d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x00005564e03222b4 in wsrep_replication_process (thd=0x7fd260000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x00005564e0313f01 in start_wsrep_THD (arg=0x5564e23dc4a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fd2784ce700 (LWP 23398)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e0bcb9a9 in safe_cond_wait (cond=0x7fd25c015628, mp=0x7fd25c015578, file=0x5564e0d34a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564e0321d06 in inline_mysql_cond_wait (that=0x7fd25c015628, mutex=0x7fd25c015578, src_file=0x5564e0d34d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005564e032499e in Wsrep_thd_queue::pop_front (this=0x7fd25c015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005564e032377e in wsrep_post_rollback_process (post_rollbacker=0x7fd25c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005564e0313f01 in start_wsrep_THD (arg=0x5564e23d2290) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fd271ed7700 (LWP 23400)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e0bcbcab in safe_cond_timedwait (cond=0x5564e2198ca0 , mp=0x5564e2198be0 , abstime=0x7fd271ed6cb0, file=0x5564e1011130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005564e0a35add in inline_mysql_cond_timedwait (that=0x5564e2198ca0 , mutex=0x5564e2198be0 , abstime=0x7fd271ed6cb0, src_file=0x5564e1011178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005564e0a36001 in my_service_thread_sleep (control=0x5564e174d900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005564e0a2a49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005564e0b5e471 in pfs_spawn_thread (arg=0x5564e2482640) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fd272e7d700 (LWP 23396)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd27b059867 in gu::Lock::wait (cond=..., this=0x7fd272e7c850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd27b05c3ab in RecvBuf::front (this=this@entry=0x5564e23c27f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fd27b059f46 in gcomm_recv (backend=, msg=0x5564e2382780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fd27b04deec in core_msg_recv (backend=backend@entry=0x5564e23828c8, recv_msg=recv_msg@entry=0x5564e2382780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fd27b051164 in gcs_core_recv (conn=0x5564e2382720, recv_act=recv_act@entry=0x7fd272e7cd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fd27b0583d8 in gcs_recv_thread (arg=0x5564e23824a0) at gcs/src/gcs.cpp:1398 #7 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fd278519700 (LWP 23397)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e0bcb9a9 in safe_cond_wait (cond=0x7fd268016878, mp=0x7fd2680167c8, file=0x5564e0d34a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005564e0321d06 in inline_mysql_cond_wait (that=0x7fd268016878, mutex=0x7fd2680167c8, src_file=0x5564e0d34d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005564e032499e in Wsrep_thd_queue::pop_front (this=0x7fd268016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005564e03233be in wsrep_rollback_process (rollbacker=0x7fd268001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005564e0313f01 in start_wsrep_THD (arg=0x5564e23dbcc0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fd27367e700 (LWP 23395)): #0 0x00007fd28152f483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fd27b0206f8 in asio::detail::epoll_reactor::run (this=0x5564e23c2f10, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fd27b024df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5564e23c2e30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fd27b0250a2 in asio::detail::task_io_service::run (this=0x5564e23c2e30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fd27b03cafd in asio::io_service::run (this=0x5564e23c1f30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fd27b03d251 in gcomm::AsioProtonet::event_loop (this=0x5564e23c1e90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fd27b05b2ed in GCommConn::run (this=0x5564e23c2610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fd27b05e399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fd27bbc9700 (LWP 23393)): #0 0x00007fd283174d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005564e0bcbcab in safe_cond_timedwait (cond=0x5564e21ab440 , mp=0x5564e21ab380 , abstime=0x7fd27bbc8d70, file=0x5564e1055360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005564e0bccc65 in inline_mysql_cond_timedwait (that=0x5564e21ab440 , mutex=0x5564e21ab380 , abstime=0x7fd27bbc8d70, src_file=0x5564e10553a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005564e0bcd7c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005564e0b5e471 in pfs_spawn_thread (arg=0x5564e23436f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fd27a01c700 (LWP 23394)): #0 0x00007fd283174965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd27b07efd7 in gu::Lock::wait (cond=..., this=0x7fd27a01bbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fd27b07f069 in galera::ServiceThd::thd_func (arg=0x5564e237a8c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007fd283170dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fd28152eead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fd283581880 (LWP 23392)): #0 0x00007fd2831759d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005564e0bc637a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005564e0402d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fd28152420d in poll () from /lib64/libc.so.6 #5 0x00005564dffa496a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x00005564dffa3c05 in mysqld_main (argc=161, argv=0x5564e21fbd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x00005564dff97da5 in main (argc=23, argv=0x7ffc891ca958) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.23434' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_insert_select-innodb/mysqld.2/data/core.23434 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 23434] [New LWP 23437] [New LWP 23438] [New LWP 23440] [New LWP 23441] [New LWP 23605] [New LWP 23604] [New LWP 23608] [New LWP 23606] [New LWP 23610] [New LWP 23607] [New LWP 23611] [New LWP 23616] [New LWP 23615] [New LWP 23617] [New LWP 23618] [New LWP 23622] [New LWP 23624] [New LWP 23623] [New LWP 23625] [New LWP 23727] [New LWP 23629] [New LWP 23609] [New LWP 23626] [New LWP 23442] [New LWP 23621] [New LWP 23619] [New LWP 23628] [New LWP 23627] [New LWP 23614] [New LWP 23620] [New LWP 23436] [New LWP 23439] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fe2192679d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fe2192679d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559fbfa9637a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559fbf2d2d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fe21761620d in poll () from /lib64/libc.so.6 #5 0x0000559fbee7496a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559fbee73c05 in mysqld_main (argc=162, argv=0x559fc194bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559fbee67da5 in main (argc=23, argv=0x7ffe2ffda288) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fe201b7a700 (LWP 23439)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe21114b867 in gu::Lock::wait (cond=..., this=0x7fe201b79850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fe21114e3ab in RecvBuf::front (this=this@entry=0x559fc1b0a788, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fe21114bf46 in gcomm_recv (backend=, msg=0x559fc1ad2690, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fe21113feec in core_msg_recv (backend=backend@entry=0x559fc1ad27d8, recv_msg=recv_msg@entry=0x559fc1ad2690, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fe211143164 in gcs_core_recv (conn=0x559fc1ad2630, recv_act=recv_act@entry=0x7fe201b79d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fe21114a3d8 in gcs_recv_thread (arg=0x559fc1ad23b0) at gcs/src/gcs.cpp:1398 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fe211cbb700 (LWP 23436)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbfa9bcab in safe_cond_timedwait (cond=0x559fc107b440 , mp=0x559fc107b380 , abstime=0x7fe211cbad70, file=0x559fbff25360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559fbfa9cc65 in inline_mysql_cond_timedwait (that=0x559fc107b440 , mutex=0x559fc107b380 , abstime=0x7fe211cbad70, src_file=0x559fbff253a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559fbfa9d7c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000559fbfa2e471 in pfs_spawn_thread (arg=0x559fc1a93700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fe1c6ffd700 (LWP 23620)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc889 in os_event::wait (this=0x559fc1c1afd0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559fbf5dc30f in os_event::wait_low (this=0x559fc1c1afd0, reset_sig_count=5029) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559fbf5dc680 in os_event_wait_low (event=0x559fc1c1afd0, reset_sig_count=5029) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559fbf6c85ff in srv_resume_thread (slot=0x559fc0619548 , sig_count=5029, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559fbf6cc610 in srv_purge_coordinator_suspend (slot=0x559fc0619548 , rseg_history_len=1) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x0000559fbf6cc8a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fe1d9df6700 (LWP 23614)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc1a4 in os_event::timed_wait (this=0x559fc1df3720, abstime=0x7fe1d9df5d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559fbf5dc4bb in os_event::wait_time_low (this=0x559fc1df3720, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559fbf5dc65b in os_event_wait_time_low (event=0x559fc1df3720, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559fbf5a9314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fe1b6ffd700 (LWP 23627)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbfa9b9a9 in safe_cond_wait (cond=0x7fe1b6ffcc70, mp=0x7fe1b6ffccb0, file=0x559fbfd68860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559fbf4f3a13 in inline_mysql_cond_wait (that=0x7fe1b6ffcc70, mutex=0x7fe1b6ffccb0, src_file=0x559fbfd6a3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559fbf4f8632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x0000559fbfa2e471 in pfs_spawn_thread (arg=0x559fc1dd6330) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fe210050700 (LWP 23628)): #0 0x00007fe21755a40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000559fbee67e24 in my_sigwait (set=0x7fe21004fcf0, sig=0x7fe21004fcb4, code=0x7fe21004fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000559fbee6e464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000559fbfa2e471 in pfs_spawn_thread (arg=0x559fc1bded10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fe1c77fe700 (LWP 23619)): #0 0x00007fe219269e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559fbf5dcb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559fbf6cbb67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x0000559fbf6cbc97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fe1c67fc700 (LWP 23621)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc889 in os_event::wait (this=0x559fc1c1b050) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559fbf5dc30f in os_event::wait_low (this=0x559fc1c1b050, reset_sig_count=23821) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559fbf5dc680 in os_event_wait_low (event=0x559fc1c1b050, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559fbf6c85ff in srv_resume_thread (slot=0x559fc0619580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559fbf6cc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fe210278700 (LWP 23442)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe211065132 in fifo_lock_get (q=q@entry=0x559fc1af8a30) at galerautils/src/gu_fifo.c:262 #2 0x00007fe2110657a1 in gu_fifo_get_head (q=0x559fc1af8a30, err=0x7fe21027706c) at galerautils/src/gu_fifo.c:330 #3 0x00007fe2111470cc in gcs_recv (conn=0x559fc1ad23b0, action=0x7fe2102770b0) at gcs/src/gcs.cpp:2024 #4 0x00007fe21117416c in galera::GcsActionSource::process (this=0x559fc1b02dd0, recv_ctx=0x7fe2102774d0, exit_loop=@0x7fe21027729e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fe211193033 in galera::ReplicatorSMM::async_recv (this=0x559fc1aca160, recv_ctx=0x7fe2102774d0) at galera/src/replicator_smm.cpp:383 #6 0x0000559fbfb4f20e in wsrep::wsrep_provider_v26::run_applier (this=0x559fc1a3fd80, applier_ctx=0x7fe2102774d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x0000559fbf1f22b4 in wsrep_replication_process (thd=0x7fe1e4000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000559fbf1e3f01 in start_wsrep_THD (arg=0x559fc1b2b670) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fe1b77fe700 (LWP 23626)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc889 in os_event::wait (this=0x559fc1c1b3d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559fbf5dc30f in os_event::wait_low (this=0x559fc1c1b3d0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559fbf5dc680 in os_event_wait_low (event=0x559fc1c1b3d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559fbf7918b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fe1dbfff700 (LWP 23609)): #0 0x00007fe218e33644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559fbf5cfba6 in LinuxAIOHandler::collect (this=0x7fe1dbffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559fbf5d00cf in LinuxAIOHandler::poll (this=0x7fe1dbffeb00, m1=0x7fe1dbffeba8, m2=0x7fe1dbffebb0, request=0x7fe1dbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559fbf5d0232 in os_aio_linux_handler (global_segment=4, m1=0x7fe1dbffeba8, m2=0x7fe1dbffebb0, request=0x7fe1dbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559fbf5d4d56 in os_aio_handler (segment=4, m1=0x7fe1dbffeba8, m2=0x7fe1dbffebb0, request=0x7fe1dbffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559fbf828bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559fbf6ce12f in io_handler_thread (arg=0x559fc0857440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fe200190700 (LWP 23629)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbfa9b9a9 in safe_cond_wait (cond=0x559fc07dbfe0 , mp=0x559fc07df360 , file=0x559fbfb65b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559fbee993ce in inline_mysql_cond_wait (that=0x559fc07dbfe0 , mutex=0x559fc07df360 , src_file=0x559fbfb666b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000559fbee9ac60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000559fbfa2e471 in pfs_spawn_thread (arg=0x559fc1bdf200) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fe2000fa700 (LWP 23727)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbfa9bcab in safe_cond_timedwait (cond=0x559fc07dbfa0 , mp=0x559fc07de9a0 , abstime=0x7fe2000f9c60, file=0x559fbfb56348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559fbee68725 in inline_mysql_cond_timedwait (that=0x559fc07dbfa0 , mutex=0x559fc07de9a0 , abstime=0x7fe2000f9c60, src_file=0x559fbfb583b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559fbee6d4fb in cache_thread (thd=0x7fe188000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000559fbee6d7a9 in one_thread_per_connection_end (thd=0x7fe188000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000559fbf108cda in do_handle_one_connection (connect=0x559fc1f8b8d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000559fbf108994 in handle_one_connection (arg=0x559fc1f8b8d0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x0000559fbfa2e471 in pfs_spawn_thread (arg=0x559fc1fa4560) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fe1b7fff700 (LWP 23625)): #0 0x00007fe219269e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000559fbf5dcb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000559fbf783755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fe1c57fa700 (LWP 23623)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc889 in os_event::wait (this=0x559fc1c1b150) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559fbf5dc30f in os_event::wait_low (this=0x559fc1c1b150, reset_sig_count=16962) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559fbf5dc680 in os_event_wait_low (event=0x559fc1c1b150, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559fbf6c85ff in srv_resume_thread (slot=0x559fc06195f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559fbf6cc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fe1c4ff9700 (LWP 23624)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc889 in os_event::wait (this=0x559fc1c1b2d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559fbf5dc30f in os_event::wait_low (this=0x559fc1c1b2d0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559fbf5dc680 in os_event_wait_low (event=0x559fc1c1b2d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559fbf7a8dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fe1c5ffb700 (LWP 23622)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc889 in os_event::wait (this=0x559fc1c1b0d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000559fbf5dc30f in os_event::wait_low (this=0x559fc1c1b0d0, reset_sig_count=19671) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000559fbf5dc680 in os_event_wait_low (event=0x559fc1c1b0d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000559fbf6c85ff in srv_resume_thread (slot=0x559fc06195b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000559fbf6cc203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fe1c7fff700 (LWP 23618)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc1a4 in os_event::timed_wait (this=0x559fc1dc3980, abstime=0x7fe1c7ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559fbf5dc4bb in os_event::wait_time_low (this=0x559fc1dc3980, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559fbf5dc65b in os_event_wait_time_low (event=0x559fc1dc3980, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559fbf72aaad in ib_wqueue_timedwait (wq=0x559fc1dc3850, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000559fbf86cf11 in fts_optimize_thread (arg=0x559fc1dc3850) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fe1d8df4700 (LWP 23617)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc1a4 in os_event::timed_wait (this=0x559fc1e25740, abstime=0x7fe1d8df3d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559fbf5dc4bb in os_event::wait_time_low (this=0x559fc1e25740, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559fbf5dc65b in os_event_wait_time_low (event=0x559fc1e25740, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559fbf81252c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fe1da5f7700 (LWP 23615)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc1a4 in os_event::timed_wait (this=0x559fc1c1b1d0, abstime=0x7fe1da5f6ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559fbf5dc4bb in os_event::wait_time_low (this=0x559fc1c1b1d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559fbf5dc65b in os_event_wait_time_low (event=0x559fc1c1b1d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559fbf6ca808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fe1d95f5700 (LWP 23616)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc1a4 in os_event::timed_wait (this=0x559fc1c1b250, abstime=0x7fe1d95f4d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559fbf5dc4bb in os_event::wait_time_low (this=0x559fc1c1b250, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559fbf5dc65b in os_event_wait_time_low (event=0x559fc1c1b250, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559fbf6ca2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fe1daffd700 (LWP 23611)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbf5dc1a4 in os_event::timed_wait (this=0x559fc1c1b350, abstime=0x7fe1daffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000559fbf5dc4bb in os_event::wait_time_low (this=0x559fc1c1b350, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000559fbf5dc65b in os_event_wait_time_low (event=0x559fc1c1b350, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000559fbf7b2c57 in pc_sleep_if_needed (next_loop_time=1549300454357, sig_count=2, cur_time=1549300453357) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x0000559fbf7b3b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fe1f1006700 (LWP 23607)): #0 0x00007fe218e33644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559fbf5cfba6 in LinuxAIOHandler::collect (this=0x7fe1f1005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559fbf5d00cf in LinuxAIOHandler::poll (this=0x7fe1f1005b00, m1=0x7fe1f1005ba8, m2=0x7fe1f1005bb0, request=0x7fe1f1005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559fbf5d0232 in os_aio_linux_handler (global_segment=2, m1=0x7fe1f1005ba8, m2=0x7fe1f1005bb0, request=0x7fe1f1005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559fbf5d4d56 in os_aio_handler (segment=2, m1=0x7fe1f1005ba8, m2=0x7fe1f1005bb0, request=0x7fe1f1005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559fbf828bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559fbf6ce12f in io_handler_thread (arg=0x559fc0857430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fe1db7fe700 (LWP 23610)): #0 0x00007fe218e33644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559fbf5cfba6 in LinuxAIOHandler::collect (this=0x7fe1db7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559fbf5d00cf in LinuxAIOHandler::poll (this=0x7fe1db7fdb00, m1=0x7fe1db7fdba8, m2=0x7fe1db7fdbb0, request=0x7fe1db7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559fbf5d0232 in os_aio_linux_handler (global_segment=5, m1=0x7fe1db7fdba8, m2=0x7fe1db7fdbb0, request=0x7fe1db7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559fbf5d4d56 in os_aio_handler (segment=5, m1=0x7fe1db7fdba8, m2=0x7fe1db7fdbb0, request=0x7fe1db7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559fbf828bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559fbf6ce12f in io_handler_thread (arg=0x559fc0857448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fe1f1807700 (LWP 23606)): #0 0x00007fe218e33644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559fbf5cfba6 in LinuxAIOHandler::collect (this=0x7fe1f1806b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559fbf5d00cf in LinuxAIOHandler::poll (this=0x7fe1f1806b00, m1=0x7fe1f1806ba8, m2=0x7fe1f1806bb0, request=0x7fe1f1806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559fbf5d0232 in os_aio_linux_handler (global_segment=1, m1=0x7fe1f1806ba8, m2=0x7fe1f1806bb0, request=0x7fe1f1806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559fbf5d4d56 in os_aio_handler (segment=1, m1=0x7fe1f1806ba8, m2=0x7fe1f1806bb0, request=0x7fe1f1806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559fbf828bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559fbf6ce12f in io_handler_thread (arg=0x559fc0857428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fe1f0805700 (LWP 23608)): #0 0x00007fe218e33644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559fbf5cfba6 in LinuxAIOHandler::collect (this=0x7fe1f0804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559fbf5d00cf in LinuxAIOHandler::poll (this=0x7fe1f0804b00, m1=0x7fe1f0804ba8, m2=0x7fe1f0804bb0, request=0x7fe1f0804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559fbf5d0232 in os_aio_linux_handler (global_segment=3, m1=0x7fe1f0804ba8, m2=0x7fe1f0804bb0, request=0x7fe1f0804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559fbf5d4d56 in os_aio_handler (segment=3, m1=0x7fe1f0804ba8, m2=0x7fe1f0804bb0, request=0x7fe1f0804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559fbf828bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559fbf6ce12f in io_handler_thread (arg=0x559fc0857438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fe1f385a700 (LWP 23604)): #0 0x00007fe219266d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbfa9bcab in safe_cond_timedwait (cond=0x559fc1068ca0 , mp=0x559fc1068be0 , abstime=0x7fe1f3859cb0, file=0x559fbfee1130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000559fbf905add in inline_mysql_cond_timedwait (that=0x559fc1068ca0 , mutex=0x559fc1068be0 , abstime=0x7fe1f3859cb0, src_file=0x559fbfee1178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000559fbf906001 in my_service_thread_sleep (control=0x559fc061d900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000559fbf8fa49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000559fbfa2e471 in pfs_spawn_thread (arg=0x559fc1bd34c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fe1f2008700 (LWP 23605)): #0 0x00007fe218e33644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000559fbf5cfba6 in LinuxAIOHandler::collect (this=0x7fe1f2007b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000559fbf5d00cf in LinuxAIOHandler::poll (this=0x7fe1f2007b00, m1=0x7fe1f2007ba8, m2=0x7fe1f2007bb0, request=0x7fe1f2007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000559fbf5d0232 in os_aio_linux_handler (global_segment=0, m1=0x7fe1f2007ba8, m2=0x7fe1f2007bb0, request=0x7fe1f2007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000559fbf5d4d56 in os_aio_handler (segment=0, m1=0x7fe1f2007ba8, m2=0x7fe1f2007bb0, request=0x7fe1f2007be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000559fbf828bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000559fbf6ce12f in io_handler_thread (arg=0x559fc0857420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fe2102c3700 (LWP 23441)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbfa9b9a9 in safe_cond_wait (cond=0x7fe1ec015628, mp=0x7fe1ec015578, file=0x559fbfc04a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559fbf1f1d06 in inline_mysql_cond_wait (that=0x7fe1ec015628, mutex=0x7fe1ec015578, src_file=0x559fbfc04d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559fbf1f499e in Wsrep_thd_queue::pop_front (this=0x7fe1ec015520) at /home/stepan/galera/git/10.4/worker[1] Trying to dump core for [mysqltest - pid: 24271, winpid: 24271] worker[1] Trying to dump core for [mysqld.1 - pid: 23974, winpid: 23974] worker[1] Trying to dump core for [mysqld.2 - pid: 24018, winpid: 24018] server/sql/wsrep_thd.h:69 #4 0x0000559fbf1f377e in wsrep_post_rollback_process (post_rollbacker=0x7fe1ec000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000559fbf1e3f01 in start_wsrep_THD (arg=0x559fc1b22b60) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fe21030e700 (LWP 23440)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000559fbfa9b9a9 in safe_cond_wait (cond=0x7fe1f8017b88, mp=0x7fe1f8017ad8, file=0x559fbfc04a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000559fbf1f1d06 in inline_mysql_cond_wait (that=0x7fe1f8017b88, mutex=0x7fe1f8017ad8, src_file=0x559fbfc04d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000559fbf1f499e in Wsrep_thd_queue::pop_front (this=0x7fe1f8017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000559fbf1f33be in wsrep_rollback_process (rollbacker=0x7fe1f8001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000559fbf1e3f01 in start_wsrep_THD (arg=0x559fc1b2cab0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fe20237b700 (LWP 23438)): #0 0x00007fe217621483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe2111126f8 in asio::detail::epoll_reactor::run (this=0x559fc1b0af70, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fe211116df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x559fc1b0ae10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fe2111170a2 in asio::detail::task_io_service::run (this=0x559fc1b0ae10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fe21112eafd in asio::io_service::run (this=0x559fc1b09b30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fe21112f251 in gcomm::AsioProtonet::event_loop (this=0x559fc1b09a90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fe21114d2ed in GCommConn::run (this=0x559fc1b0a5a0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fe211150399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fe210b0f700 (LWP 23437)): #0 0x00007fe219266965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe211170fd7 in gu::Lock::wait (cond=..., this=0x7fe210b0ebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fe211171069 in galera::ServiceThd::thd_func (arg=0x559fc1aca878) at galera/src/galera_service_thd.cpp:30 #3 0x00007fe219262dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe217620ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fe219673880 (LWP 23434)): #0 0x00007fe2192679d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000559fbfa9637a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000559fbf2d2d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fe21761620d in poll () from /lib64/libc.so.6 #5 0x0000559fbee7496a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000559fbee73c05 in mysqld_main (argc=162, argv=0x559fc194bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000559fbee67da5 in main (argc=23, argv=0x7ffe2ffda288) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_insert_select-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_insert_select-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_insert_select Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_kill_connection 'innodb' [ pass ] 4807 galera_sr.galera_sr_kill_query 'innodb' [ pass ] 7133 galera_sr.galera_sr_load_data 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 20:31:29 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr_load_data.log == connection node_2; connection node_1; SET SESSION wsrep_trx_fragment_size = 512; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; connection node_2; connection node_1; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) - found 'core.23975' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data-innodb/mysqld.1/data/core.23975 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 23975] [New LWP 23980] [New LWP 23979] [New LWP 23984] [New LWP 23983] [New LWP 23987] [New LWP 23986] [New LWP 23985] [New LWP 23999] [New LWP 24000] [New LWP 23989] [New LWP 24008] [New LWP 24005] [New LWP 24007] [New LWP 23982] [New LWP 24006] [New LWP 24009] [New LWP 24003] [New LWP 24012] [New LWP 23981] [New LWP 24236] [New LWP 23990] [New LWP 23998] [New LWP 23997] [New LWP 23988] [New LWP 23996] [New LWP 24002] [New LWP 23995] [New LWP 24004] [New LWP 23991] [New LWP 23993] [New LWP 24001] [New LWP 23992] [New LWP 24013] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f9c5fed49d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f9c5fed49d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056175d39c37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056175cbd8d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9c5e28320d in poll () from /lib64/libc.so.6 #5 0x000056175c77a96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056175c779c05 in mysqld_main (argc=161, argv=0x56175f876d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056175c76dda5 in main (argc=23, argv=0x7ffe454db698) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f9c4d5bd700 (LWP 24013)): #0 0x00007f9c5fed6d1d in fsync () from /lib64/libpthread.so.0 #1 0x000056175ced7255 in os_file_fsync_posix (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056175ced75ab in os_file_flush_func (file=19) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056175d120751 in pfs_os_file_flush_func (file=..., src_file=0x56175d79c8b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056175d123593 in fil_flush_low (space=0x56175fd47470, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x000056175d12f113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x000056175ceb2c59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000056175ceb36bb in log_write_up_to (lsn=12959376, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000056175d01e551 in trx_flush_log_if_needed_low (lsn=12959376) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #9 0x000056175d01e59a in trx_flush_log_if_needed (lsn=12959376, trx=0x7f9c4e0a7390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #10 0x000056175d0203c3 in trx_commit_complete_for_mysql (trx=0x7f9c4e0a7390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #11 0x000056175ce04a6f in innobase_commit (hton=0x56175fb117c0, thd=0x7f9be405b300, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #12 0x000056175cbde89b in commit_one_phase_2 (thd=0x7f9be405b300, all=true, trans=0x7f9be405e848, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #13 0x000056175cbde79d in ha_commit_one_phase (thd=0x7f9be405b300, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #14 0x000056175cbddb19 in ha_commit_trans (thd=0x7f9be405b300, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #15 0x000056175ca23a76 in trans_commit (thd=0x7f9be405b300) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #16 0x000056175cde2988 in Wsrep_storage_service::commit (this=0x7f9be4293260, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #17 0x000056175d44b85f in wsrep::transaction::certify_fragment (this=0x7f9be40069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #18 0x000056175d44afc3 in wsrep::transaction::streaming_step (this=0x7f9be40069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #19 0x000056175d44837e in wsrep::transaction::after_row (this=0x7f9be40069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #20 0x000056175cbf0ba4 in wsrep::client_state::after_row (this=0x7f9be4006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #21 0x000056175cbd9e87 in wsrep_after_row (thd=0x7f9be4000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #22 0x000056175cbebd26 in wsrep_after_row (thd=0x7f9be4000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #23 0x000056175cbec117 in handler::ha_write_row (this=0x7f9be416b8b8, buf=0x7f9be4267250 "\377\020<") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #24 0x000056175c86218d in write_record (thd=0x7f9be4000b00, table=0x7f9be4266610, info=0x7f9c4d5baaf0) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #25 0x000056175c89514f in read_sep_field (thd=0x7f9be4000b00, info=..., table_list=0x7f9be40153a0, fields_vars=..., set_fields=..., set_values=..., read_info=..., enclosed=..., skip_lines=0, ignore_check_option_errors=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:1152 #26 0x000056175c89370a in mysql_load (thd=0x7f9be4000b00, ex=0x7f9be4015318, table_list=0x7f9be40153a0, fields_vars=..., set_fields=..., set_values=..., handle_duplicates=DUP_ERROR, ignore=false, read_file_from_client=false) at /home/stepan/galera/git/10.4/server/sql/sql_load.cc:660 #27 0x000056175c8aa49d in mysql_execute_command (thd=0x7f9be4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5145 #28 0x000056175c8b4675 in mysql_parse (thd=0x7f9be4000b00, rawbuf=0x7f9be4015178 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=111, parser_state=0x7f9c4d5bc0b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #29 0x000056175c8b3ca8 in wsrep_mysql_parse (thd=0x7f9be4000b00, rawbuf=0x7f9be4015178 "LOAD DATA INFILE '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/galera_sr_load_data.csv' INTO TABLE t1", length=111, parser_state=0x7f9c4d5bc0b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #30 0x000056175c89fb65 in dispatch_command (command=COM_QUERY, thd=0x7f9be4000b00, packet=0x7f9be4028211 "", packet_length=112, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #31 0x000056175c89e599 in do_command (thd=0x7f9be4000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #32 0x000056175ca0ec50 in do_handle_one_connection (connect=0x56175feb1790) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #33 0x000056175ca0e994 in handle_one_connection (arg=0x56175feb1790) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #34 0x000056175d334471 in pfs_spawn_thread (arg=0x56175fecead0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #35 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #36 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f9c2e7fc700 (LWP 23992)): #0 0x00007f9c5fed6d1d in fsync () from /lib64/libpthread.so.0 #1 0x000056175ced7255 in os_file_fsync_posix (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000056175ced75ab in os_file_flush_func (file=15) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000056175d120751 in pfs_os_file_flush_func (file=..., src_file=0x56175d79c8b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000056175d123593 in fil_flush_low (space=0x56175fd470d0, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x000056175d12f113 in fil_flush (space_id=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x000056175d12f39a in fil_flush_file_spaces (purpose=FIL_TYPE_TABLESPACE) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4580 #7 0x000056175d0aae64 in buf_dblwr_update (bpage=0x7f9c4d88cef8, flush_type=BUF_FLUSH_LIST) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dblwr.cc:757 #8 0x000056175d0b2925 in buf_flush_write_complete (bpage=0x7f9c4d88cef8, dblwr=true) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:725 #9 0x000056175d0a13ef in buf_page_io_complete (bpage=0x7f9c4d88cef8, dblwr=true, evict=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:6135 #10 0x000056175d12ee6d in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4467 #11 0x000056175cfd412f in io_handler_thread (arg=0x56175e15d448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #12 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f9c25ffb700 (LWP 24001)): #0 0x00007f9c5e2dccf7 in __memcpy_ssse3 () from /lib64/libc.so.6 #1 0x000056175d3049d3 in copy_events_waits (dest=0x7f9c59de2fe8, source=0x7f9c5b07b990) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs_events_waits.cc:82 #2 0x000056175d304af9 in insert_events_waits_history_long (wait=0x7f9c5b07b990) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs_events_waits.cc:132 #3 0x000056175d337ee6 in end_mutex_wait_v1 (locker=0x7f9c25ffac70, rc=0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:3525 #4 0x000056175ce2db69 in PolicyMutex >::pfs_end (this=0x56175df1f3c0 , locker=0x7f9c25ffac70, ret=0) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:743 #5 0x000056175ce2bd32 in PolicyMutex >::enter (this=0x56175df1f3c0 , n_spins=30, n_delay=4, name=0x56175d721710 "/home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc", line=2777) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:599 #6 0x000056175cfd2b9f in srv_get_task_queue_length () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2777 #7 0x000056175cffc3bb in trx_purge_wait_for_workers_to_complete () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1270 #8 0x000056175cffc5e1 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1327 #9 0x000056175cfd24d6 in srv_do_purge (n_total_purged=0x7f9c25ffad98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2579 #10 0x000056175cfd2903 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2702 #11 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f9c2dffb700 (LWP 23993)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee21a4 in os_event::timed_wait (this=0x56175fb3d760, abstime=0x7f9c2dffaaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056175cee24bb in os_event::wait_time_low (this=0x56175fb3d760, time_in_usec=459000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056175cee265b in os_event_wait_time_low (event=0x56175fb3d760, time_in_usec=459000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056175d0b8c57 in pc_sleep_if_needed (next_loop_time=1549301429541, sig_count=2, cur_time=1549301429082) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x000056175d0b9b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f9c2effd700 (LWP 23991)): #0 0x00007f9c5faa0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056175ced5ba6 in LinuxAIOHandler::collect (this=0x7f9c2effcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056175ced60cf in LinuxAIOHandler::poll (this=0x7f9c2effcb00, m1=0x7f9c2effcba8, m2=0x7f9c2effcbb0, request=0x7f9c2effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056175ced6232 in os_aio_linux_handler (global_segment=4, m1=0x7f9c2effcba8, m2=0x7f9c2effcbb0, request=0x7f9c2effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056175cedad56 in os_aio_handler (segment=4, m1=0x7f9c2effcba8, m2=0x7f9c2effcbb0, request=0x7f9c2effcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000056175d12ebc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000056175cfd412f in io_handler_thread (arg=0x56175e15d440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f9c13fff700 (LWP 24004)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee2889 in os_event::wait (this=0x56175fc235b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056175cee230f in os_event::wait_low (this=0x56175fc235b0, reset_sig_count=1354) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056175cee2680 in os_event_wait_low (event=0x56175fc235b0, reset_sig_count=1354) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056175cfdc42e in sync_array_wait_event (arr=0x56175fb146e0, cell=@0x7f9c13ffce60: 0x56175fb14850) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x000056175ce2da9d in TTASEventMutex::enter (this=0x56175fc22ee0, max_spins=60, max_delay=4, filename=0x56175d776b68 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x000056175ce2bcf8 in PolicyMutex >::enter (this=0x56175fc22ee0, n_spins=30, n_delay=4, name=0x56175d776b68 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc", line=1092) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x000056175d0c1412 in buf_LRU_get_free_block (buf_pool=0x56175fc22ee0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0lru.cc:1092 #8 0x000056175d09e9ea in buf_page_init_for_read (err=0x7f9c13ffd884, mode=132, page_id=..., page_size=..., unzip=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:5326 #9 0x000056175d0c81c6 in buf_read_page_low (err=0x7f9c13ffd884, sync=false, type=192, mode=132, page_id=..., page_size=..., unzip=false, ignore_missing_space=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:150 #10 0x000056175d0c8f03 in buf_read_page_background (page_id=..., page_size=..., sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0rea.cc:456 #11 0x000056175d061bef in btr_cur_prefetch_siblings (block=0x7f9c4d8907c0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:3305 #12 0x000056175d06a3f7 in btr_cur_optimistic_delete_func (cursor=0x56175fd5eef8, flags=0, mtr=0x7f9c13ffe1a0) at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0cur.cc:5806 #13 0x000056175cf8f7d9 in row_purge_remove_clust_if_poss_low (node=0x56175fd5ee60, mode=2) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:168 #14 0x000056175cf8f9c8 in row_purge_remove_clust_if_poss (node=0x56175fd5ee60) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:219 #15 0x000056175cf9121b in row_purge_del_mark (node=0x56175fd5ee60) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:809 #16 0x000056175cf927a5 in row_purge_record_func (node=0x56175fd5ee60, undo_rec=0x56175fd5f338 "", thr=0x56175fd5eda8, updated_extern=false) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1194 #17 0x000056175cf92ae2 in row_purge (node=0x56175fd5ee60, undo_rec=0x56175fd5f338 "", thr=0x56175fd5eda8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1261 #18 0x000056175cf92e8d in row_purge_step (thr=0x56175fd5eda8) at /home/stepan/galera/git/10.4/server/storage/innobase/row/row0purge.cc:1347 #19 0x000056175cf151a8 in que_thr_step (thr=0x56175fd5eda8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1042 #20 0x000056175cf153d8 in que_run_threads_low (thr=0x56175fd5eda8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1104 #21 0x000056175cf155c8 in que_run_threads (thr=0x56175fd5eda8) at /home/stepan/galera/git/10.4/server/storage/innobase/que/que0que.cc:1144 #22 0x000056175cfd2061 in srv_task_execute () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2439 #23 0x000056175cfd2208 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2487 #24 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f9c2cff9700 (LWP 23995)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee21a4 in os_event::timed_wait (this=0x56175fd15b30, abstime=0x7f9c2cff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056175cee24bb in os_event::wait_time_low (this=0x56175fd15b30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056175cee265b in os_event_wait_time_low (event=0x56175fd15b30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056175ceaf314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f9c257fa700 (LWP 24002)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee2889 in os_event::wait (this=0x56175fb3d560) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056175cee230f in os_event::wait_low (this=0x56175fb3d560, reset_sig_count=16659) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056175cee2680 in os_event_wait_low (event=0x56175fb3d560, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056175cfce5ff in srv_resume_thread (slot=0x56175df1f5f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000056175cfd2203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f9c27fff700 (LWP 23996)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee21a4 in os_event::timed_wait (this=0x56175fb3d5e0, abstime=0x7f9c27ffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056175cee24bb in os_event::wait_time_low (this=0x56175fb3d5e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056175cee265b in os_event_wait_time_low (event=0x56175fb3d5e0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056175cfd0808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f9c4cc52700 (LWP 23988)): #0 0x00007f9c5faa0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056175ced5ba6 in LinuxAIOHandler::collect (this=0x7f9c4cc51b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056175ced60cf in LinuxAIOHandler::poll (this=0x7f9c4cc51b00, m1=0x7f9c4cc51ba8, m2=0x7f9c4cc51bb0, request=0x7f9c4cc51be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056175ced6232 in os_aio_linux_handler (global_segment=1, m1=0x7f9c4cc51ba8, m2=0x7f9c4cc51bb0, request=0x7f9c4cc51be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056175cedad56 in os_aio_handler (segment=1, m1=0x7f9c4cc51ba8, m2=0x7f9c4cc51bb0, request=0x7f9c4cc51be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000056175d12ebc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000056175cfd412f in io_handler_thread (arg=0x56175e15d428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f9c2d7fa700 (LWP 23997)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee21a4 in os_event::timed_wait (this=0x56175fb3d660, abstime=0x7f9c2d7f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056175cee24bb in os_event::wait_time_low (this=0x56175fb3d660, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056175cee265b in os_event_wait_time_low (event=0x56175fb3d660, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056175cfd02e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f9c277fe700 (LWP 23998)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee21a4 in os_event::timed_wait (this=0x56175fd47b50, abstime=0x7f9c277fdd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056175cee24bb in os_event::wait_time_low (this=0x56175fd47b50, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056175cee265b in os_event_wait_time_low (event=0x56175fd47b50, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056175d11852c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f9c2f7fe700 (LWP 23990)): #0 0x00007f9c5faa0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056175ced5ba6 in LinuxAIOHandler::collect (this=0x7f9c2f7fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056175ced60cf in LinuxAIOHandler::poll (this=0x7f9c2f7fdb00, m1=0x7f9c2f7fdba8, m2=0x7f9c2f7fdbb0, request=0x7f9c2f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056175ced6232 in os_aio_linux_handler (global_segment=3, m1=0x7f9c2f7fdba8, m2=0x7f9c2f7fdbb0, request=0x7f9c2f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056175cedad56 in os_aio_handler (segment=3, m1=0x7f9c2f7fdba8, m2=0x7f9c2f7fdbb0, request=0x7f9c2f7fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000056175d12ebc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000056175cfd412f in io_handler_thread (arg=0x56175e15d438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f9c4d527700 (LWP 24236)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175d3a1cab in safe_cond_timedwait (cond=0x56175e0e1fa0 , mp=0x56175e0e49a0 , abstime=0x7f9c4d526c60, file=0x56175d45c348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056175c76e725 in inline_mysql_cond_timedwait (that=0x56175e0e1fa0 , mutex=0x56175e0e49a0 , abstime=0x7f9c4d526c60, src_file=0x56175d45e3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056175c7734fb in cache_thread (thd=0x7f9be0000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000056175c7737a9 in one_thread_per_connection_end (thd=0x7f9be0000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000056175ca0ecda in do_handle_one_connection (connect=0x56175fa55d40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x000056175ca0e994 in handle_one_connection (arg=0x56175fa55d40) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000056175d334471 in pfs_spawn_thread (arg=0x56175fe20a10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f9c508f1700 (LWP 23981)): #0 0x00007f9c5e28e483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9c53c576f8 in asio::detail::epoll_reactor::run (this=0x56175fa35f90, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f9c53c5bdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x56175fa35e30, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f9c53c5c0a2 in asio::detail::task_io_service::run (this=0x56175fa35e30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f9c53c73afd in asio::io_service::run (this=0x56175fa34f30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f9c53c74251 in gcomm::AsioProtonet::event_loop (this=0x56175fa34e90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f9c53c922ed in GCommConn::run (this=0x56175fa35610) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f9c53c95399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f9c4d608700 (LWP 24012)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175d3a19a9 in safe_cond_wait (cond=0x56175e0e1fe0 , mp=0x56175e0e5360 , file=0x56175d46bb48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056175c79f3ce in inline_mysql_cond_wait (that=0x56175e0e1fe0 , mutex=0x56175e0e5360 , src_file=0x56175d46c6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000056175c7a0c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000056175d334471 in pfs_spawn_thread (arg=0x56175fe3ef40) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f9c24ff9700 (LWP 24003)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee2889 in os_event::wait (this=0x56175fb3d460) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056175cee230f in os_event::wait_low (this=0x56175fb3d460, reset_sig_count=24925) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056175cee2680 in os_event_wait_low (event=0x56175fb3d460, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056175cfce5ff in srv_resume_thread (slot=0x56175df1f580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000056175cfd2203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f9c4d653700 (LWP 24009)): #0 0x00007f9c5e1c740a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000056175c76de24 in my_sigwait (set=0x7f9c4d652cf0, sig=0x7f9c4d652cb4, code=0x7f9c4d652cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000056175c774464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000056175d334471 in pfs_spawn_thread (arg=0x56175fb00f30) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f9c12ffd700 (LWP 24006)): #0 0x00007f9c5fed6e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056175cee2b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056175d089755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f9c500f0700 (LWP 23982)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9c53c90867 in gu::Lock::wait (cond=..., this=0x7f9c500ef850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f9c53c933ab in RecvBuf::front (this=this@entry=0x56175fa357f8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f9c53c90f46 in gcomm_recv (backend=, msg=0x56175f9fd780, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f9c53c84eec in core_msg_recv (backend=backend@entry=0x56175f9fd8c8, recv_msg=recv_msg@entry=0x56175f9fd780, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f9c53c88164 in gcs_core_recv (conn=0x56175f9fd720, recv_act=recv_act@entry=0x7f9c500efd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f9c53c8f3d8 in gcs_recv_thread (arg=0x56175f9fd4a0) at gcs/src/gcs.cpp:1398 #7 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f9c127fc700 (LWP 24007)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee2889 in os_event::wait (this=0x56175fb3d7e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056175cee230f in os_event::wait_low (this=0x56175fb3d7e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056175cee2680 in os_event_wait_low (event=0x56175fb3d7e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056175d0978b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f9c137fe700 (LWP 24005)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee2889 in os_event::wait (this=0x56175fb3d6e0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000056175cee230f in os_event::wait_low (this=0x56175fb3d6e0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000056175cee2680 in os_event_wait_low (event=0x56175fb3d6e0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000056175d0aedde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f9c11ffb700 (LWP 24008)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175d3a19a9 in safe_cond_wait (cond=0x7f9c11ffac70, mp=0x7f9c11ffacb0, file=0x56175d66e860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056175cdf9a13 in inline_mysql_cond_wait (that=0x7f9c11ffac70, mutex=0x7f9c11ffacb0, src_file=0x56175d6703a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056175cdfe632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000056175d334471 in pfs_spawn_thread (arg=0x56175fd7e6d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f9c2ffff700 (LWP 23989)): #0 0x00007f9c5faa0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056175ced5ba6 in LinuxAIOHandler::collect (this=0x7f9c2fffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056175ced60cf in LinuxAIOHandler::poll (this=0x7f9c2fffeb00, m1=0x7f9c2fffeba8, m2=0x7f9c2fffebb0, request=0x7f9c2fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056175ced6232 in os_aio_linux_handler (global_segment=2, m1=0x7f9c2fffeba8, m2=0x7f9c2fffebb0, request=0x7f9c2fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056175cedad56 in os_aio_handler (segment=2, m1=0x7f9c2fffeba8, m2=0x7f9c2fffebb0, request=0x7f9c2fffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000056175d12ebc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000056175cfd412f in io_handler_thread (arg=0x56175e15d430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f9c267fc700 (LWP 24000)): #0 0x00007f9c5fed6e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000056175cee2b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000056175cfd1b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x000056175cfd1c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f9c26ffd700 (LWP 23999)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175cee21a4 in os_event::timed_wait (this=0x56175fcdd520, abstime=0x7f9c26ffcb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000056175cee24bb in os_event::wait_time_low (this=0x56175fcdd520, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000056175cee265b in os_event_wait_time_low (event=0x56175fcdd520, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000056175d030aad in ib_wqueue_timedwait (wq=0x56175fcdd470, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000056175d172f11 in fts_optimize_thread (arg=0x56175fcdd470) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f9c4f859700 (LWP 23985)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9c53baa132 in fifo_lock_get (q=q@entry=0x56175fa23b20) at galerautils/src/gu_fifo.c:262 #2 0x00007f9c53baa7a1 in gu_fifo_get_head (q=0x56175fa23b20, err=0x7f9c4f85806c) at galerautils/src/gu_fifo.c:330 #3 0x00007f9c53c8c0cc in gcs_recv (conn=0x56175f9fd4a0, action=0x7f9c4f8580b0) at gcs/src/gcs.cpp:2024 #4 0x00007f9c53cb916c in galera::GcsActionSource::process (this=0x56175f9fce80, recv_ctx=0x7f9c4f8584d0, exit_loop=@0x7f9c4f85829e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f9c53cd8033 in galera::ReplicatorSMM::async_recv (this=0x56175f9f51b0, recv_ctx=0x7f9c4f8584d0) at galera/src/replicator_smm.cpp:383 #6 0x000056175d45520e in wsrep::wsrep_provider_v26::run_applier (this=0x56175f97c280, applier_ctx=0x7f9c4f8584d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x000056175caf82b4 in wsrep_replication_process (thd=0x7f9c3c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000056175cae9f01 in start_wsrep_THD (arg=0x56175fa45060) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f9c4eca5700 (LWP 23986)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175d3a1cab in safe_cond_timedwait (cond=0x56175e96eca0 , mp=0x56175e96ebe0 , abstime=0x7f9c4eca4cb0, file=0x56175d7e7130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056175d20badd in inline_mysql_cond_timedwait (that=0x56175e96eca0 , mutex=0x56175e96ebe0 , abstime=0x7f9c4eca4cb0, src_file=0x56175d7e7178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056175d20c001 in my_service_thread_sleep (control=0x56175df23900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000056175d20049a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000056175d334471 in pfs_spawn_thread (arg=0x56175faf5650) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f9c4d453700 (LWP 23987)): #0 0x00007f9c5faa0644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000056175ced5ba6 in LinuxAIOHandler::collect (this=0x7f9c4d452b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000056175ced60cf in LinuxAIOHandler::poll (this=0x7f9c4d452b00, m1=0x7f9c4d452ba8, m2=0x7f9c4d452bb0, request=0x7f9c4d452be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000056175ced6232 in os_aio_linux_handler (global_segment=0, m1=0x7f9c4d452ba8, m2=0x7f9c4d452bb0, request=0x7f9c4d452be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000056175cedad56 in os_aio_handler (segment=0, m1=0x7f9c4d452ba8, m2=0x7f9c4d452bb0, request=0x7f9c4d452be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000056175d12ebc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000056175cfd412f in io_handler_thread (arg=0x56175e15d420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f9c4f8ef700 (LWP 23983)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175d3a19a9 in safe_cond_wait (cond=0x7f9c44017b88, mp=0x7f9c44017ad8, file=0x56175d50aa00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056175caf7d06 in inline_mysql_cond_wait (that=0x7f9c44017b88, mutex=0x7f9c44017ad8, src_file=0x56175d50ad20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056175cafa99e in Wsrep_thd_queue::pop_front (this=0x7f9c44017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056175caf93be in wsrep_rollback_process (rollbacker=0x7f9c44001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000056175cae9f01 in start_wsrep_THD (arg=0x56175fa45270) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f9c4f8a4700 (LWP 23984)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175d3a19a9 in safe_cond_wait (cond=0x7f9c38015628, mp=0x7f9c38015578, file=0x56175d50aa00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056175caf7d06 in inline_mysql_cond_wait (that=0x7f9c38015628, mutex=0x7f9c38015578, src_file=0x56175d50ad20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000056175cafa99e in Wsrep_thd_queue::pop_front (this=0x7f9c38015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000056175caf977e in wsrep_post_rollback_process (post_rollbacker=0x7f9c38000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000056175cae9f01 in start_wsrep_THD (arg=0x56175fa2f760) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f9c58928700 (LWP 23979)): #0 0x00007f9c5fed3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000056175d3a1cab in safe_cond_timedwait (cond=0x56175e981440 , mp=0x56175e981380 , abstime=0x7f9c58927d70, file=0x56175d82b360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000056175d3a2c65 in inline_mysql_cond_timedwait (that=0x56175e981440 , mutex=0x56175e981380 , abstime=0x7f9c58927d70, src_file=0x56175d82b3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000056175d3a37c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000056175d334471 in pfs_spawn_thread (arg=0x56175f9be6f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f9c52d75700 (LWP 23980)): #0 0x00007f9c5fed3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9c53cb5fd7 in gu::Lock::wait (cond=..., this=0x7f9c52d74bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f9c53cb6069 in galera::ServiceThd::thd_func (arg=0x56175f9f58c8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f9c5fecfdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9c5e28dead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f9c602e0880 (LWP 23975)): #0 0x00007f9c5fed49d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000056175d39c37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000056175cbd8d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f9c5e28320d in poll () from /lib64/libc.so.6 #5 0x000056175c77a96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056175c779c05 in mysqld_main (argc=161, argv=0x56175f876d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056175c76dda5 in main (argc=23, argv=0x7ffe454db698) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.24020' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data-innodb/mysqld.2/data/core.24020 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 24020] [New LWP 24188] [New LWP 24193] [New LWP 24194] [New LWP 24201] [New LWP 24025] [New LWP 24197] [New LWP 24203] [New LWP 24189] [New LWP 24198] [New LWP 24204] [New LWP 24199] [New LWP 24210] [New LWP 24200] [New LWP 24026] [New LWP 24023] [New LWP 24202] [New LWP 24205] [New LWP 24206] [New LWP 24208] [New LWP 24209] [New LWP 24215] [New LWP 24217] [New LWP 24223] [New LWP 24028] [New LWP 24191] [New LWP 24022] [New LWP 24024] [New LWP 24207] [New LWP 24187] [New LWP 24192] [New LWP 24027] [New LWP 24190] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fcc70a969d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fcc70a969d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055e4c0b9f37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055e4c03dbd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fcc6ee4520d in poll () from /lib64/libc.so.6 #5 0x000055e4bff7d96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055e4bff7cc05 in mysqld_main (argc=162, argv=0x55e4c2d59d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055e4bff70da5 in main (argc=23, argv=0x7ffc7403f578) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7fcc38805700 (LWP 24190)): #0 0x00007fcc70662644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055e4c06d8ba6 in LinuxAIOHandler::collect (this=0x7fcc38804b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055e4c06d90cf in LinuxAIOHandler::poll (this=0x7fcc38804b00, m1=0x7fcc38804ba8, m2=0x7fcc38804bb0, request=0x7fcc38804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055e4c06d9232 in os_aio_linux_handler (global_segment=2, m1=0x7fcc38804ba8, m2=0x7fcc38804bb0, request=0x7fcc38804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055e4c06ddd56 in os_aio_handler (segment=2, m1=0x7fcc38804ba8, m2=0x7fcc38804bb0, request=0x7fcc38804be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055e4c0931bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055e4c07d712f in io_handler_thread (arg=0x55e4c1960430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fcc682f3700 (LWP 24027)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c0ba49a9 in safe_cond_wait (cond=0x7fcc50016938, mp=0x7fcc50016888, file=0x55e4c0d0da00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055e4c02fad06 in inline_mysql_cond_wait (that=0x7fcc50016938, mutex=0x7fcc50016888, src_file=0x55e4c0d0dd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055e4c02fd99e in Wsrep_thd_queue::pop_front (this=0x7fcc50016830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055e4c02fc77e in wsrep_post_rollback_process (post_rollbacker=0x7fcc50000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055e4c02ecf01 in start_wsrep_THD (arg=0x55e4c2f359a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fcc337fe700 (LWP 24192)): #0 0x00007fcc70662644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055e4c06d8ba6 in LinuxAIOHandler::collect (this=0x7fcc337fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055e4c06d90cf in LinuxAIOHandler::poll (this=0x7fcc337fdb00, m1=0x7fcc337fdba8, m2=0x7fcc337fdbb0, request=0x7fcc337fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055e4c06d9232 in os_aio_linux_handler (global_segment=4, m1=0x7fcc337fdba8, m2=0x7fcc337fdbb0, request=0x7fcc337fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055e4c06ddd56 in os_aio_handler (segment=4, m1=0x7fcc337fdba8, m2=0x7fcc337fdbb0, request=0x7fcc337fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055e4c0931bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055e4c07d712f in io_handler_thread (arg=0x55e4c1960440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fcc3b059700 (LWP 24187)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c0ba4cab in safe_cond_timedwait (cond=0x55e4c2171ca0 , mp=0x55e4c2171be0 , abstime=0x7fcc3b058cb0, file=0x55e4c0fea130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055e4c0a0eadd in inline_mysql_cond_timedwait (that=0x55e4c2171ca0 , mutex=0x55e4c2171be0 , abstime=0x7fcc3b058cb0, src_file=0x55e4c0fea178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055e4c0a0f001 in my_service_thread_sleep (control=0x55e4c1726900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055e4c0a0349a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055e4c0b37471 in pfs_spawn_thread (arg=0x55e4c2fdd5d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fcc07fff700 (LWP 24207)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e5889 in os_event::wait (this=0x55e4c3025330) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055e4c06e530f in os_event::wait_low (this=0x55e4c3025330, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055e4c06e5680 in os_event_wait_low (event=0x55e4c3025330, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055e4c08b1dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fcc59b7a700 (LWP 24024)): #0 0x00007fcc6ee50483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fcc689416f8 in asio::detail::epoll_reactor::run (this=0x55e4c2f14f70, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007fcc68945df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55e4c2f14e10, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007fcc689460a2 in asio::detail::task_io_service::run (this=0x55e4c2f14e10, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007fcc6895dafd in asio::io_service::run (this=0x55e4c2f13b30) at asio/asio/impl/io_service.ipp:58 #5 0x00007fcc6895e251 in gcomm::AsioProtonet::event_loop (this=0x55e4c2f13a90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007fcc6897c2ed in GCommConn::run (this=0x55e4c2f145a0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007fcc6897f399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fcc694ea700 (LWP 24022)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c0ba4cab in safe_cond_timedwait (cond=0x55e4c2184440 , mp=0x55e4c2184380 , abstime=0x7fcc694e9d70, file=0x55e4c102e360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055e4c0ba5c65 in inline_mysql_cond_timedwait (that=0x55e4c2184440 , mutex=0x55e4c2184380 , abstime=0x7fcc694e9d70, src_file=0x55e4c102e3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055e4c0ba67c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055e4c0b37471 in pfs_spawn_thread (arg=0x55e4c2ea1700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fcc33fff700 (LWP 24191)): #0 0x00007fcc70662644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055e4c06d8ba6 in LinuxAIOHandler::collect (this=0x7fcc33ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055e4c06d90cf in LinuxAIOHandler::poll (this=0x7fcc33ffeb00, m1=0x7fcc33ffeba8, m2=0x7fcc33ffebb0, request=0x7fcc33ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055e4c06d9232 in os_aio_linux_handler (global_segment=3, m1=0x7fcc33ffeba8, m2=0x7fcc33ffebb0, request=0x7fcc33ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055e4c06ddd56 in os_aio_handler (segment=3, m1=0x7fcc33ffeba8, m2=0x7fcc33ffebb0, request=0x7fcc33ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055e4c0931bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055e4c07d712f in io_handler_thread (arg=0x55e4c1960438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fcc682a8700 (LWP 24028)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcc68894132 in fifo_lock_get (q=q@entry=0x55e4c2f06a30) at galerautils/src/gu_fifo.c:262 #2 0x00007fcc688947a1 in gu_fifo_get_head (q=0x55e4c2f06a30, err=0x7fcc682a706c) at galerautils/src/gu_fifo.c:330 #3 0x00007fcc689760cc in gcs_recv (conn=0x55e4c2ee03b0, action=0x7fcc682a70b0) at gcs/src/gcs.cpp:2024 #4 0x00007fcc689a316c in galera::GcsActionSource::process (this=0x55e4c2f0cdd0, recv_ctx=0x7fcc682a74d0, exit_loop=@0x7fcc682a729e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007fcc689c2033 in galera::ReplicatorSMM::async_recv (this=0x55e4c2ed8160, recv_ctx=0x7fcc682a74d0) at galera/src/replicator_smm.cpp:383 #6 0x000055e4c0c5820e in wsrep::wsrep_provider_v26::run_applier (this=0x55e4c2e4dd80, applier_ctx=0x7fcc682a74d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x000055e4c02fb2b4 in wsrep_replication_process (thd=0x7fcc44000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055e4c02ecf01 in start_wsrep_THD (arg=0x55e4c2f368f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fcc58145700 (LWP 24223)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c0ba4cab in safe_cond_timedwait (cond=0x55e4c18e4fa0 , mp=0x55e4c18e79a0 , abstime=0x7fcc58144c60, file=0x55e4c0c5f348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055e4bff71725 in inline_mysql_cond_timedwait (that=0x55e4c18e4fa0 , mutex=0x55e4c18e79a0 , abstime=0x7fcc58144c60, src_file=0x55e4c0c613b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055e4bff764fb in cache_thread (thd=0x7fcc3c000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055e4bff767a9 in one_thread_per_connection_end (thd=0x7fcc3c000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055e4c0211cda in do_handle_one_connection (connect=0x55e4c33626e0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x000055e4c0211994 in handle_one_connection (arg=0x55e4c33626e0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000055e4c0b37471 in pfs_spawn_thread (arg=0x55e4c33ac830) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fcc58190700 (LWP 24217)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c0ba49a9 in safe_cond_wait (cond=0x55e4c18e4fe0 , mp=0x55e4c18e8360 , file=0x55e4c0c6eb48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055e4bffa23ce in inline_mysql_cond_wait (that=0x55e4c18e4fe0 , mutex=0x55e4c18e8360 , src_file=0x55e4c0c6f6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055e4bffa3c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055e4c0b37471 in pfs_spawn_thread (arg=0x55e4c3399d80) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fcc68080700 (LWP 24215)): #0 0x00007fcc6ed8940a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055e4bff70e24 in my_sigwait (set=0x7fcc6807fcf0, sig=0x7fcc6807fcb4, code=0x7fcc6807fcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055e4bff77464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055e4c0b37471 in pfs_spawn_thread (arg=0x55e4c2fe8e20) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fcc06ffd700 (LWP 24209)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e5889 in os_event::wait (this=0x55e4c3025430) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055e4c06e530f in os_event::wait_low (this=0x55e4c3025430, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055e4c06e5680 in os_event_wait_low (event=0x55e4c3025430, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055e4c089a8b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fcc077fe700 (LWP 24208)): #0 0x00007fcc70a98e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055e4c06e5b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055e4c088c755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fcc1cff9700 (LWP 24206)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e5889 in os_event::wait (this=0x55e4c30250b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055e4c06e530f in os_event::wait_low (this=0x55e4c30250b0, reset_sig_count=24954) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055e4c06e5680 in os_event_wait_low (event=0x55e4c30250b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055e4c07d15ff in srv_resume_thread (slot=0x55e4c1722580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055e4c07d5203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fcc1d7fa700 (LWP 24205)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e5889 in os_event::wait (this=0x55e4c30251b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055e4c06e530f in os_event::wait_low (this=0x55e4c30251b0, reset_sig_count=17128) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055e4c06e5680 in os_event_wait_low (event=0x55e4c30251b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055e4c07d15ff in srv_resume_thread (slot=0x55e4c17225f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055e4c07d5203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fcc1effd700 (LWP 24202)): #0 0x00007fcc70a98e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055e4c06e5b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055e4c07d4b67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x000055e4c07d4c97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fcc5bffe700 (LWP 24023)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcc6899ffd7 in gu::Lock::wait (cond=..., this=0x7fcc5bffdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fcc689a0069 in galera::ServiceThd::thd_func (arg=0x55e4c2ed8878) at galera/src/galera_service_thd.cpp:30 #3 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fcc6833e700 (LWP 24026)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c0ba49a9 in safe_cond_wait (cond=0x7fcc48016878, mp=0x7fcc480167c8, file=0x55e4c0d0da00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055e4c02fad06 in inline_mysql_cond_wait (that=0x7fcc48016878, mutex=0x7fcc480167c8, src_file=0x55e4c0d0dd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055e4c02fd99e in Wsrep_thd_queue::pop_front (this=0x7fcc48016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055e4c02fc3be in wsrep_rollback_process (rollbacker=0x7fcc48001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055e4c02ecf01 in start_wsrep_THD (arg=0x55e4c2f36b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fcc31df6700 (LWP 24200)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e51a4 in os_event::timed_wait (this=0x55e4c322f7a0, abstime=0x7fcc31df5d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055e4c06e54bb in os_event::wait_time_low (this=0x55e4c322f7a0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055e4c06e565b in os_event_wait_time_low (event=0x55e4c322f7a0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055e4c091b52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fcc067fc700 (LWP 24210)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c0ba49a9 in safe_cond_wait (cond=0x7fcc067fbc70, mp=0x7fcc067fbcb0, file=0x55e4c0e71860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055e4c05fca13 in inline_mysql_cond_wait (that=0x7fcc067fbc70, mutex=0x7fcc067fbcb0, src_file=0x55e4c0e733a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055e4c0601632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000055e4c0b37471 in pfs_spawn_thread (arg=0x55e4c31dff70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fcc1ffff700 (LWP 24199)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e51a4 in os_event::timed_wait (this=0x55e4c30252b0, abstime=0x7fcc1fffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055e4c06e54bb in os_event::wait_time_low (this=0x55e4c30252b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055e4c06e565b in os_event_wait_time_low (event=0x55e4c30252b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055e4c07d32e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fcc1dffb700 (LWP 24204)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e5889 in os_event::wait (this=0x55e4c3025130) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055e4c06e530f in os_event::wait_low (this=0x55e4c3025130, reset_sig_count=20423) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055e4c06e5680 in os_event_wait_low (event=0x55e4c3025130, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055e4c07d15ff in srv_resume_thread (slot=0x55e4c17225b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055e4c07d5203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fcc30df4700 (LWP 24198)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e51a4 in os_event::timed_wait (this=0x55e4c3025230, abstime=0x7fcc30df3ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055e4c06e54bb in os_event::wait_time_low (this=0x55e4c3025230, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055e4c06e565b in os_event_wait_time_low (event=0x55e4c3025230, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055e4c07d3808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fcc39006700 (LWP 24189)): #0 0x00007fcc70662644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055e4c06d8ba6 in LinuxAIOHandler::collect (this=0x7fcc39005b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055e4c06d90cf in LinuxAIOHandler::poll (this=0x7fcc39005b00, m1=0x7fcc39005ba8, m2=0x7fcc39005bb0, request=0x7fcc39005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055e4c06d9232 in os_aio_linux_handler (global_segment=1, m1=0x7fcc39005ba8, m2=0x7fcc39005bb0, request=0x7fcc39005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055e4c06ddd56 in os_aio_handler (segment=1, m1=0x7fcc39005ba8, m2=0x7fcc39005bb0, request=0x7fcc39005be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055e4c0931bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055e4c07d712f in io_handler_thread (arg=0x55e4c1960428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fcc1e7fc700 (LWP 24203)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e5889 in os_event::wait (this=0x55e4c3025030) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055e4c06e530f in os_event::wait_low (this=0x55e4c3025030, reset_sig_count=5146) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055e4c06e5680 in os_event_wait_low (event=0x55e4c3025030, reset_sig_count=5146) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055e4c07d15ff in srv_resume_thread (slot=0x55e4c1722548 , sig_count=5146, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055e4c07d5610 in srv_purge_coordinator_suspend (slot=0x55e4c1722548 , rseg_history_len=34) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x000055e4c07d58a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fcc315f5700 (LWP 24197)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e51a4 in os_event::timed_wait (this=0x55e4c31fd780, abstime=0x7fcc315f4d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055e4c06e54bb in os_event::wait_time_low (this=0x55e4c31fd780, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055e4c06e565b in os_event_wait_time_low (event=0x55e4c31fd780, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055e4c06b2314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fcc59379700 (LWP 24025)): #0 0x00007fcc70a95965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcc6897a867 in gu::Lock::wait (cond=..., this=0x7fcc59378850) at galerautils/src/gu_lock.hpp:45 #2 0x00007fcc6897d3ab in RecvBuf::front (this=this@entry=0x55e4c2f14788, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007fcc6897af46 in gcomm_recv (backend=, msg=0x55e4c2ee0690, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007fcc6896eeec in core_msg_recv (backend=backend@entry=0x55e4c2ee07d8, recv_msg=recv_msg@entry=0x55e4c2ee0690, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007fcc68972164 in gcs_core_recv (conn=0x55e4c2ee0630, recv_act=recv_act@entry=0x7fcc59378d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007fcc689793d8 in gcs_recv_thread (arg=0x55e4c2ee03b0) at gcs/src/gcs.cpp:1398 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fcc1f7fe700 (LWP 24201)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e51a4 in os_event::timed_wait (this=0x55e4c31cd0c0, abstime=0x7fcc1f7fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055e4c06e54bb in os_event::wait_time_low (this=0x55e4c31cd0c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055e4c06e565b in os_event_wait_time_low (event=0x55e4c31cd0c0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055e4c0833aad in ib_wqueue_timedwait (wq=0x55e4c31cd010, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055e4c0975f11 in fts_optimize_thread (arg=0x55e4c31cd010) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fcc327fc700 (LWP 24194)): #0 0x00007fcc70a95d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055e4c06e51a4 in os_event::timed_wait (this=0x55e4c30253b0, abstime=0x7fcc327fbaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055e4c06e54bb in os_event::wait_time_low (this=0x55e4c30253b0, time_in_usec=999000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055e4c06e565b in os_event_wait_time_low (event=0x55e4c30253b0, time_in_usec=999000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055e4c08bbc57 in pc_sleep_if_needed (next_loop_time=1549301436559, sig_count=2, cur_time=1549301435560) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x000055e4c08bcb90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fcc32ffd700 (LWP 24193)): #0 0x00007fcc70662644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055e4c06d8ba6 in LinuxAIOHandler::collect (this=0x7fcc32ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055e4c06d90cf in LinuxAIOHandler::poll (this=0x7fcc32ffcb00, m1=0x7fcc32ffcba8, m2=0x7fcc32ffcbb0, request=0x7fcc32ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055e4c06d9232 in os_aio_linux_handler (global_segment=5, m1=0x7fcc32ffcba8, m2=0x7fcc32ffcbb0, request=0x7fcc32ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055e4c06ddd56 in os_aio_handler (segment=5, m1=0x7fcc32ffcba8, m2=0x7fcc32ffcbb0, request=0x7fcc32ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055e4c0931bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055e4c07d712f in io_handler_thread (arg=0x55e4c1960448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fcc39807700 (LWP 24188)): #0 0x00007fcc70662644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055e4c06d8ba6 in LinuxAIOHandler::collect (this=0x7fcc39806b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055e4c06d90cf in LinuxAIOHandler::poll (this=0x7fcc39806b00, m1=0x7fcc39806ba8, m2=0x7fcc39806bb0, request=0x7fcc39806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055e4c06d9232 in os_aio_linux_handler (global_segment=0, m1=0x7fcc39806ba8, m2=0x7fcc39806bb0, request=0x7fcc39806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055e4c06ddd56 in os_aio_handler (segment=0, m1=0x7fcc39806ba8, m2=0x7fcc39806bb0, request=0x7fcc39806be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055e4c0931bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055e4c07d712f in io_handler_thread (arg=0x55e4c1960420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fcc70a91dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcc6ee4fead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fcc70ea2880 (LWP 24020)): #0 0x00007fcc70a969d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055e4c0b9f37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055e4c03dbd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fcc6ee4520d in poll () from /lib64/libc.so.6 #5 0x000055e4bff7d96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055e4bff7cc05 in mysqld_main (argc=162, argv=0x55e4c2d59d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055e4bff70da5 in main (argc=23, argv=0x7ffc7403f578) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_load_data galera_sr.galera_sr_kill_query galera_sr.galera_sr_kill_connection Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_myisam 'innodb' [ pass ] 2835 galera_sr.galera_sr_parallel_apply 'innodb' [ pass ] 5269 galera_sr.galera_sr_rollback 'innodb' [ pass ] 5766 galera_sr.galera_sr_rollback_retry 'innodb' [ pass ] 4059 galera_sr.galera_sr_rollback_savepoint 'innodb' [ pass ] 5441 galera_sr.galera_sr_rollback_statement 'innodb' [ fail ] Test ended at 2019-02-04 20:35:17 CURRENT_TEST: galera_sr.galera_sr_rollback_statement mysqltest: At line 34: query 'INSERT INTO t2 SELECT * FROM t1' succeeded - should have failed with errno 1062... The result from queries just before the failure was: connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 CHAR(255)) ENGINE=InnoDB; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3; ALTER TABLE t1 CHANGE f1 f1 INTEGER; ALTER TABLE t1 DROP PRIMARY KEY; INSERT INTO t1 VALUES (1, 'abc'); SET SESSION wsrep_trx_fragment_size = 1; INSERT INTO t2 SELECT * FROM t1; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_rollback_statement-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_rollback_statement-innodb/' galera_sr.galera_sr_shutdown_master 'innodb' [ pass ] 24403 galera_sr.galera_sr_shutdown_slave 'innodb' [ pass ] 25043 galera_sr.galera_sr_table_contents 'innodb' [ fail ] Test ended at 2019-02-04 20:36:21 CURRENT_TEST: galera_sr.galera_sr_table_contents cp: cannot stat 'std_data/binlog-header.log': No such file or directory mysqltest: In included file "./suite/galera/include/galera_dump_sr_table.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/galera_sr_table_contents.test at line 21: At line 20: exec of 'cp std_data/binlog-header.log /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/sr_binlog.log' failed, error: 256, status: 1, errno: 11 Output from before failure: Warnings: Warning 1287 ' INTO FROM...' instead The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t1 VALUES (1),(2),(3); CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t2 VALUES (1),(2),(3); SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; Start of Simple Insert INSERT INTO t1 VALUES (4); Warnings: Warning 1287 ' INTO FROM...' instead - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_table_contents-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_table_contents-innodb/' galera_sr.galera_sr_transaction_replay 'innodb' [ pass ] 5376 galera_sr.galera_sr_unit_statements 'innodb' [ pass ] 4276 galera_sr.galera_sr_ws_size 'innodb' [ pass ] 13058 galera_sr.galera_sr_ws_size2 'innodb' [ pass ] 9234 galera_sr.galera_var_ignore_apply_errors_sr 'innodb' [ pass ] 4321 galera_sr.mysql-wsrep#215 'innodb' [ pass ] 9929 galera_sr.mysql-wsrep-features#138 'innodb' [ pass ] 4839 galera_sr.mysql-wsrep-features#14 'innodb' [worker[1] Trying to dump core for [mysqltest - pid: 28284, winpid: 28284] worker[1] Trying to dump core for [mysqld.1 - pid: 28032, winpid: 28032] worker[1] Trying to dump core for [mysqld.2 - pid: 28073, winpid: 28073] pass ] 4601 galera_sr.mysql-wsrep-features#148 'innodb' [ pass ] 8813 MTR's internal check of the test case 'galera_sr.mysql-wsrep-features#148' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16001 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.result 2019-02-04 20:37:29.114820700 +0300 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.reject 2019-02-04 20:37:38.201820700 +0300 @@ -1098,4 +1098,4 @@ user_variables 1.0 DISABLED INFORMATION SCHEMA 100403.0 NULL NULL Sergey Vojtovich User-defined variables GPL OFF Gamma 1.0 partition 1.0 DISABLED STORAGE ENGINE 100403.0 NULL NULL Mikael Ronstrom, MySQL AB Partition Storage Engine Helper GPL OFF Stable 1.0 VARIABLE_NAME VARIABLE_VALUE -DEBUG_SYNC ON - current signal: '' +DEBUG_SYNC ON - current signal: 'signal.wsrep_apply_cb' mysqltest: Result length mismatch not ok galera_sr.mysql-wsrep-features#15 'innodb' [ pass ] 4171 galera_sr.mysql-wsrep-features#165 'innodb' [ fail ] Test ended at 2019-02-04 20:38:15 CURRENT_TEST: galera_sr.mysql-wsrep-features#165 mysqltest: In included file "/home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.test at line 27: At line 101: query 'SELECT * FROM t1' failed: 1205: Lock wait timeout exceeded; try restarting transaction The result from queries just before the failure was: < snip > connection node_1; SET DEBUG_SYNC = 'now WAIT_FOR wait'; connection node_1; SET DEBUG_SYNC = 'now SIGNAL continue'; connection node_1c; UPDATE t1 SET f2 = 'x' WHERE f1 = 3; connection node_1a; COMMIT; connection node_1b; ERROR 40001: Deadlock found when trying to get lock; try restarting transaction connection node_1; SELECT * FROM t1; f1 f2 1 a 2 a 3 x 4 a 5 a connection node_2; SELECT * FROM t1; More results from queries before failure can be found in /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysql-wsrep-features#165.log - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#165-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#165-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 2019-02-04 20:38:05 3 [Warning] WSREP: Event 3 Write_rows_v1 apply failed: 121, seqno 12 galera_sr.mysql-wsrep-features#22 'innodb' [ pass ] 5105 galera_sr.mysql-wsrep-features#27 'innodb' [ pass ] 6631 galera_sr.mysql-wsrep-features#35 'innodb' [ pass ] 4404 galera_sr.mysql-wsrep-features#93 'innodb' [ pass ] 3779 galera_sr.mysql-wsrep-features#96 'innodb' [ pass ] 6150 galera_sr.galera_sr_kill_all_norecovery 'innodb' [ pass ] 19093 MTR's internal check of the test case 'galera_sr.galera_sr_kill_all_norecovery' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16001 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.result 2019-02-04 20:39:12.527820700 +0300 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.reject 2019-02-04 20:39:31.837820700 +0300 @@ -78,7 +78,7 @@ AUTOCOMMIT ON AUTOMATIC_SP_PRIVILEGES ON AUTO_INCREMENT_INCREMENT 2 -AUTO_INCREMENT_OFFSET 2 +AUTO_INCREMENT_OFFSET 1 BACK_LOG 80 BASEDIR /home/stepan/galera/git/10.4/server BIG_TABLES OFF mysqltest: Result content mismatch not ok galera_sr.galera_sr_kill_slave 'innodb' [ pass ] 26182 galera_sr.galera_sr_mysqldump_sst 'innodb' [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-04 20:44:39 line 2019-02-04 20:43:15 13 [Warning] WSREP: no THD for trx: 51 2019-02-04 20:43:16 13 [Warning] WSREP: no THD for trx: 51 ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.2.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' galera_sr.galera_sr_small_gcache 'innodb' [ fail ] timeout after 900 seconds Test ended at 2019-02-04 21:00:49 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr_small_gcache.log == connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=InnoDB; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); SET SESSION wsrep_trx_fragment_size = 1; CREATE TABLE t1 (f1 INTEGER) ENGINE=InnoDB; INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4; == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.1.' (111) - found 'core.28033' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.1/data/core.28033 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 28033] [New LWP 28060] [New LWP 28040] [New LWP 28061] [New LWP 28039] [New LWP 28051] [New LWP 28067] [New LWP 28041] [New LWP 28053] [New LWP 28068] [New LWP 28046] [New LWP 28034] [New LWP 28043] [New LWP 28054] [New LWP 28038] [New LWP 28037] [New LWP 28063] [New LWP 28035] [New LWP 28058] [New LWP 28045] [New LWP 28044] [New LWP 28064] [New LWP 28042] [New LWP 28050] [New LWP 28062] [New LWP 28047] [New LWP 28048] [New LWP 28052] [New LWP 28057] [New LWP 28056] [New LWP 28059] [New LWP 28036] [New LWP 28286] [New LWP 28055] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f418dffe9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f418dffe9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f3c1a5f37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f3c129bd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f418c3ad20d in poll () from /lib64/libc.so.6 #5 0x000055f3c0e3d96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f3c0e3cc05 in mysqld_main (argc=161, argv=0x55f3c3ebfd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f3c0e30da5 in main (argc=23, argv=0x7ffd01a9e738) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 34 (Thread 0x7f41557fa700 (LWP 28055)): #0 0x00007f418e000d1d in fsync () from /lib64/libpthread.so.0 #1 0x000055f3c159a255 in os_file_fsync_posix (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2458 #2 0x000055f3c159a5ab in os_file_flush_func (file=20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2559 #3 0x000055f3c17e3751 in pfs_os_file_flush_func (file=..., src_file=0x55f3c1e5f8b0 "/home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc", src_line=829) at /home/stepan/galera/git/10.4/server/storage/innobase/include/os0file.ic:406 #4 0x000055f3c17e6593 in fil_flush_low (space=0x55f3c4390460, metadata=false) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:829 #5 0x000055f3c17f2113 in fil_flush (space_id=4294967280) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4507 #6 0x000055f3c1575c59 in log_write_flush_to_disk_low () at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:873 #7 0x000055f3c15766bb in log_write_up_to (lsn=6942171, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1111 #8 0x000055f3c15767eb in log_buffer_sync_in_background (flush=true) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:1156 #9 0x000055f3c1693c3c in srv_sync_log_buffer_in_background () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1984 #10 0x000055f3c16941ff in srv_master_do_active_tasks () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2151 #11 0x000055f3c1694d5d in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2358 #12 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f417c084700 (LWP 28286)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a5889 in os_event::wait (this=0x55f3c431eb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f3c15a530f in os_event::wait_low (this=0x55f3c431eb60, reset_sig_count=6257) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f3c15a5680 in os_event_wait_low (event=0x55f3c431eb60, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f3c15761eb in log_write_up_to (lsn=6943515, flush_to_disk=true, rotate_key=false) at /home/stepan/galera/git/10.4/server/storage/innobase/log/log0log.cc:989 #5 0x000055f3c16e1551 in trx_flush_log_if_needed_low (lsn=6943515) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1192 #6 0x000055f3c16e159a in trx_flush_log_if_needed (lsn=6943515, trx=0x7f417c8d8390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1214 #7 0x000055f3c16e33c3 in trx_commit_complete_for_mysql (trx=0x7f417c8d8390) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1733 #8 0x000055f3c14c7a6f in innobase_commit (hton=0x55f3c415a830, thd=0x7f411014a8b0, commit_trx=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4622 #9 0x000055f3c12a189b in commit_one_phase_2 (thd=0x7f411014a8b0, all=true, trans=0x7f411014ddf8, is_real_trans=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1740 #10 0x000055f3c12a179d in ha_commit_one_phase (thd=0x7f411014a8b0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1720 #11 0x000055f3c12a0b19 in ha_commit_trans (thd=0x7f411014a8b0, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1533 #12 0x000055f3c10e6a76 in trans_commit (thd=0x7f411014a8b0) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #13 0x000055f3c14a5988 in Wsrep_storage_service::commit (this=0x7f4110179af0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_storage_service.cc:173 #14 0x000055f3c1b0e85f in wsrep::transaction::certify_fragment (this=0x7f41100069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1159 #15 0x000055f3c1b0dfc3 in wsrep::transaction::streaming_step (this=0x7f41100069f0, lock=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:1033 #16 0x000055f3c1b0b37e in wsrep::transaction::after_row (this=0x7f41100069f0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:247 #17 0x000055f3c12b3ba4 in wsrep::client_state::after_row (this=0x7f4110006980) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/client_state.hpp:305 #18 0x000055f3c129ce87 in wsrep_after_row (thd=0x7f4110000b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:123 #19 0x000055f3c12aed26 in wsrep_after_row (thd=0x7f4110000b00) at /home/stepan/galera/git/10.4/server/sql/handler.cc:6439 #20 0x000055f3c12af117 in handler::ha_write_row (this=0x7f411007c558, buf=0x7f4110077760 "\375\001") at /home/stepan/galera/git/10.4/server/sql/handler.cc:6470 #21 0x000055f3c0f2518d in write_record (thd=0x7f4110000b00, table=0x7f411007b920, info=0x7f4110016e38) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:2026 #22 0x000055f3c0f2a48c in select_insert::send_data (this=0x7f4110016df0, values=...) at /home/stepan/galera/git/10.4/server/sql/sql_insert.cc:3872 #23 0x000055f3c0fe2b47 in end_send (join=0x7f4110016ea0, join_tab=0x7f4110087018, end_of_records=false) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:20574 #24 0x000055f3c111fb89 in JOIN_CACHE::generate_full_extensions (this=0x7f41100873c8, rec_ptr=0x7f411010a5a0 "t\001") at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2400 #25 0x000055f3c111f8a4 in JOIN_CACHE::join_matching_records (this=0x7f41100873c8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2292 #26 0x000055f3c111f270 in JOIN_CACHE::join_records (this=0x7f41100873c8, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2088 #27 0x000055f3c111f40a in JOIN_CACHE::join_records (this=0x7f4110019c40, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #28 0x000055f3c111f40a in JOIN_CACHE::join_records (this=0x7f4110019b10, skip_last=false) at /home/stepan/galera/git/10.4/server/sql/sql_join_cache.cc:2134 #29 0x000055f3c0fdf7fb in sub_select_cache (join=0x7f4110016ea0, join_tab=0x7f4110086508, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19170 #30 0x000055f3c0fdfa08 in sub_select (join=0x7f4110016ea0, join_tab=0x7f4110086158, end_of_records=true) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:19341 #31 0x000055f3c0fdf1d3 in do_select (join=0x7f4110016ea0, procedure=0x0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:18932 #32 0x000055f3c0fb7cf1 in JOIN::exec_inner (this=0x7f4110016ea0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4094 #33 0x000055f3c0fb713a in JOIN::exec (this=0x7f4110016ea0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:3888 #34 0x000055f3c0fb842c in mysql_select (thd=0x7f4110000b00, tables=0x7f4110014b00, wild_num=0, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489925888, result=0x7f4110016df0, unit=0x7f41100049b8, select_lex=0x7f41100145e0) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:4293 #35 0x000055f3c0fa9d2d in handle_select (thd=0x7f4110000b00, lex=0x7f41100048f0, result=0x7f4110016df0, setup_tables_done_option=1073741824) at /home/stepan/galera/git/10.4/server/sql/sql_select.cc:385 #36 0x000055f3c0f6c13c in mysql_execute_command (thd=0x7f4110000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:4856 #37 0x000055f3c0f77675 in mysql_parse (thd=0x7f4110000b00, rawbuf=0x7f4110013e68 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7f417c0830b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8117 #38 0x000055f3c0f76ca8 in wsrep_mysql_parse (thd=0x7f4110000b00, rawbuf=0x7f4110013e68 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", length=71, parser_state=0x7f417c0830b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7929 #39 0x000055f3c0f62b65 in dispatch_command (command=COM_QUERY, thd=0x7f4110000b00, packet=0x7f411000a451 "INSERT INTO t1 SELECT 1 FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4", packet_length=71, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1787 #40 0x000055f3c0f61599 in do_command (thd=0x7f4110000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #41 0x000055f3c10d1c50 in do_handle_one_connection (connect=0x55f3c44844c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1398 #42 0x000055f3c10d1994 in handle_one_connection (arg=0x55f3c44844c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #43 0x000055f3c19f7471 in pfs_spawn_thread (arg=0x55f3c4488390) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #44 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #45 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f417ec7d700 (LWP 28036)): #0 0x00007f418c3b8483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f4185ea96f8 in asio::detail::epoll_reactor::run (this=0x55f3c407ee50, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f4185eaddf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55f3c407db60, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f4185eae0a2 in asio::detail::task_io_service::run (this=0x55f3c407db60, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f4185ec5afd in asio::io_service::run (this=0x55f3c407eb30) at asio/asio/impl/io_service.ipp:58 #5 0x00007f4185ec6251 in gcomm::AsioProtonet::event_loop (this=0x55f3c407ea90, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f4185ee42ed in GCommConn::run (this=0x55f3c407e480) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f4185ee7399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f4146ffd700 (LWP 28059)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a5889 in os_event::wait (this=0x55f3c4186550) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f3c15a530f in os_event::wait_low (this=0x55f3c4186550, reset_sig_count=16584) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f3c15a5680 in os_event_wait_low (event=0x55f3c4186550, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f3c16915ff in srv_resume_thread (slot=0x55f3c25e25f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055f3c1695203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f4154ff9700 (LWP 28056)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a5889 in os_event::wait (this=0x55f3c41863d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f3c15a530f in os_event::wait_low (this=0x55f3c41863d0, reset_sig_count=5134) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f3c15a5680 in os_event_wait_low (event=0x55f3c41863d0, reset_sig_count=5134) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f3c16915ff in srv_resume_thread (slot=0x55f3c25e2548 , sig_count=5134, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055f3c1695610 in srv_purge_coordinator_suspend (slot=0x55f3c25e2548 , rseg_history_len=12) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x000055f3c16958a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f4147fff700 (LWP 28057)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a5889 in os_event::wait (this=0x55f3c41864d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f3c15a530f in os_event::wait_low (this=0x55f3c41864d0, reset_sig_count=21340) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f3c15a5680 in os_event_wait_low (event=0x55f3c41864d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f3c16915ff in srv_resume_thread (slot=0x55f3c25e25b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055f3c1695203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4157fff700 (LWP 28052)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a51a4 in os_event::timed_wait (this=0x55f3c4186650, abstime=0x7f4157ffed20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f3c15a54bb in os_event::wait_time_low (this=0x55f3c4186650, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f3c15a565b in os_event_wait_time_low (event=0x55f3c4186650, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f3c16932e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f415cdf9700 (LWP 28048)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a51a4 in os_event::timed_wait (this=0x55f3c4186750, abstime=0x7f415cdf8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f3c15a54bb in os_event::wait_time_low (this=0x55f3c4186750, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f3c15a565b in os_event_wait_time_low (event=0x55f3c4186750, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f3c177bc57 in pc_sleep_if_needed (next_loop_time=1549303190194, sig_count=2, cur_time=1549303189194) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x000055f3c177cb90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f415d5fa700 (LWP 28047)): #0 0x00007f418dbca644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f3c1598ba6 in LinuxAIOHandler::collect (this=0x7f415d5f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f3c15990cf in LinuxAIOHandler::poll (this=0x7f415d5f9b00, m1=0x7f415d5f9ba8, m2=0x7f415d5f9bb0, request=0x7f415d5f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f3c1599232 in os_aio_linux_handler (global_segment=5, m1=0x7f415d5f9ba8, m2=0x7f415d5f9bb0, request=0x7f415d5f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f3c159dd56 in os_aio_handler (segment=5, m1=0x7f415d5f9ba8, m2=0x7f415d5f9bb0, request=0x7f415d5f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055f3c17f1bc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055f3c169712f in io_handler_thread (arg=0x55f3c2820448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f41457fa700 (LWP 28062)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a5889 in os_event::wait (this=0x55f3c41867d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f3c15a530f in os_event::wait_low (this=0x55f3c41867d0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f3c15a5680 in os_event_wait_low (event=0x55f3c41867d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f3c175a8b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f41577fe700 (LWP 28050)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a51a4 in os_event::timed_wait (this=0x55f3c435eb20, abstime=0x7f41577fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f3c15a54bb in os_event::wait_time_low (this=0x55f3c435eb20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f3c15a565b in os_event_wait_time_low (event=0x55f3c435eb20, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f3c1572314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f415fdff700 (LWP 28042)): #0 0x00007f418dbca644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f3c1598ba6 in LinuxAIOHandler::collect (this=0x7f415fdfeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f3c15990cf in LinuxAIOHandler::poll (this=0x7f415fdfeb00, m1=0x7f415fdfeba8, m2=0x7f415fdfebb0, request=0x7f415fdfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f3c1599232 in os_aio_linux_handler (global_segment=0, m1=0x7f415fdfeba8, m2=0x7f415fdfebb0, request=0x7f415fdfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f3c159dd56 in os_aio_handler (segment=0, m1=0x7f415fdfeba8, m2=0x7f415fdfebb0, request=0x7f415fdfebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055f3c17f1bc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055f3c169712f in io_handler_thread (arg=0x55f3c2820420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f41840ff700 (LWP 28064)): #0 0x00007f418c2f140a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055f3c0e30e24 in my_sigwait (set=0x7f41840fecf0, sig=0x7f41840fecb4, code=0x7f41840fecb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055f3c0e37464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055f3c19f7471 in pfs_spawn_thread (arg=0x55f3c4149fa0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f415edfd700 (LWP 28044)): #0 0x00007f418dbca644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f3c1598ba6 in LinuxAIOHandler::collect (this=0x7f415edfcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f3c15990cf in LinuxAIOHandler::poll (this=0x7f415edfcb00, m1=0x7f415edfcba8, m2=0x7f415edfcbb0, request=0x7f415edfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f3c1599232 in os_aio_linux_handler (global_segment=2, m1=0x7f415edfcba8, m2=0x7f415edfcbb0, request=0x7f415edfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f3c159dd56 in os_aio_handler (segment=2, m1=0x7f415edfcba8, m2=0x7f415edfcbb0, request=0x7f415edfcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055f3c17f1bc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055f3c169712f in io_handler_thread (arg=0x55f3c2820430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f415e5fc700 (LWP 28045)): #0 0x00007f418dbca644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f3c1598ba6 in LinuxAIOHandler::collect (this=0x7f415e5fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f3c15990cf in LinuxAIOHandler::poll (this=0x7f415e5fbb00, m1=0x7f415e5fbba8, m2=0x7f415e5fbbb0, request=0x7f415e5fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f3c1599232 in os_aio_linux_handler (global_segment=3, m1=0x7f415e5fbba8, m2=0x7f415e5fbbb0, request=0x7f415e5fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f3c159dd56 in os_aio_handler (segment=3, m1=0x7f415e5fbba8, m2=0x7f415e5fbbb0, request=0x7f415e5fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055f3c17f1bc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055f3c169712f in io_handler_thread (arg=0x55f3c2820438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f41477fe700 (LWP 28058)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a5889 in os_event::wait (this=0x55f3c4186450) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f3c15a530f in os_event::wait_low (this=0x55f3c4186450, reset_sig_count=25090) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f3c15a5680 in os_event_wait_low (event=0x55f3c4186450, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f3c16915ff in srv_resume_thread (slot=0x55f3c25e2580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055f3c1695203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f41858a6700 (LWP 28035)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4185f07fd7 in gu::Lock::wait (cond=..., this=0x7f41858a5bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4185f08069 in galera::ServiceThd::thd_func (arg=0x55f3c403e7d8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4144ff9700 (LWP 28063)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c1a649a9 in safe_cond_wait (cond=0x7f4144ff8c70, mp=0x7f4144ff8cb0, file=0x55f3c1d31860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f3c14bca13 in inline_mysql_cond_wait (that=0x7f4144ff8c70, mutex=0x7f4144ff8cb0, src_file=0x55f3c1d333a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f3c14c1632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x000055f3c19f7471 in pfs_spawn_thread (arg=0x55f3c43c7680) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f417e47c700 (LWP 28037)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4185ee2867 in gu::Lock::wait (cond=..., this=0x7f417e47b850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4185ee53ab in RecvBuf::front (this=this@entry=0x55f3c407e668, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f4185ee2f46 in gcomm_recv (backend=, msg=0x55f3c4046720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f4185ed6eec in core_msg_recv (backend=backend@entry=0x55f3c4046868, recv_msg=recv_msg@entry=0x55f3c4046720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f4185eda164 in gcs_core_recv (conn=0x55f3c40466c0, recv_act=recv_act@entry=0x7f417e47bd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f4185ee13d8 in gcs_recv_thread (arg=0x55f3c4046440) at gcs/src/gcs.cpp:1398 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f41847a4700 (LWP 28038)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c1a649a9 in safe_cond_wait (cond=0x7f4174017b88, mp=0x7f4174017ad8, file=0x55f3c1bcda00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f3c11bad06 in inline_mysql_cond_wait (that=0x7f4174017b88, mutex=0x7f4174017ad8, src_file=0x55f3c1bcdd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f3c11bd99e in Wsrep_thd_queue::pop_front (this=0x7f4174017a80) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f3c11bc3be in wsrep_rollback_process (rollbacker=0x7f4174001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055f3c11acf01 in start_wsrep_THD (arg=0x55f3c408e220) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4155ffb700 (LWP 28054)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a51a4 in os_event::timed_wait (this=0x55f3c4326510, abstime=0x7f4155ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f3c15a54bb in os_event::wait_time_low (this=0x55f3c4326510, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f3c15a565b in os_event_wait_time_low (event=0x55f3c4326510, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f3c16f3aad in ib_wqueue_timedwait (wq=0x55f3c4326460, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055f3c1835f11 in fts_optimize_thread (arg=0x55f3c4326460) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f415f5fe700 (LWP 28043)): #0 0x00007f418dbca644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f3c1598ba6 in LinuxAIOHandler::collect (this=0x7f415f5fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f3c15990cf in LinuxAIOHandler::poll (this=0x7f415f5fdb00, m1=0x7f415f5fdba8, m2=0x7f415f5fdbb0, request=0x7f415f5fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f3c1599232 in os_aio_linux_handler (global_segment=1, m1=0x7f415f5fdba8, m2=0x7f415f5fdbb0, request=0x7f415f5fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f3c159dd56 in os_aio_handler (segment=1, m1=0x7f415f5fdba8, m2=0x7f415f5fdbb0, request=0x7f415f5fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055f3c17f1bc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055f3c169712f in io_handler_thread (arg=0x55f3c2820428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f4186a52700 (LWP 28034)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c1a64cab in safe_cond_timedwait (cond=0x55f3c3044440 , mp=0x55f3c3044380 , abstime=0x7f4186a51d70, file=0x55f3c1eee360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f3c1a65c65 in inline_mysql_cond_timedwait (that=0x55f3c3044440 , mutex=0x55f3c3044380 , abstime=0x7f4186a51d70, src_file=0x55f3c1eee3a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f3c1a667c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055f3c19f7471 in pfs_spawn_thread (arg=0x55f3c40076f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f415ddfb700 (LWP 28046)): #0 0x00007f418dbca644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055f3c1598ba6 in LinuxAIOHandler::collect (this=0x7f415ddfab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055f3c15990cf in LinuxAIOHandler::poll (this=0x7f415ddfab00, m1=0x7f415ddfaba8, m2=0x7f415ddfabb0, request=0x7f415ddfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055f3c1599232 in os_aio_linux_handler (global_segment=4, m1=0x7f415ddfaba8, m2=0x7f415ddfabb0, request=0x7f415ddfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055f3c159dd56 in os_aio_handler (segment=4, m1=0x7f415ddfaba8, m2=0x7f415ddfabb0, request=0x7f415ddfabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055f3c17f1bc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055f3c169712f in io_handler_thread (arg=0x55f3c2820440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4184069700 (LWP 28068)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c1a64cab in safe_cond_timedwait (cond=0x55f3c27a4fa0 , mp=0x55f3c27a79a0 , abstime=0x7f4184068c60, file=0x55f3c1b1f348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f3c0e31725 in inline_mysql_cond_timedwait (that=0x55f3c27a4fa0 , mutex=0x55f3c27a79a0 , abstime=0x7f4184068c60, src_file=0x55f3c1b213b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f3c0e364fb in cache_thread (thd=0x7f4118000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055f3c0e367a9 in one_thread_per_connection_end (thd=0x7f4118000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055f3c10d1cda in do_handle_one_connection (connect=0x55f3c44fa740) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x000055f3c10d1994 in handle_one_connection (arg=0x55f3c44fa740) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x000055f3c19f7471 in pfs_spawn_thread (arg=0x55f3c4517ac0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f41567fc700 (LWP 28053)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a51a4 in os_event::timed_wait (this=0x55f3c4390b40, abstime=0x7f41567fbd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f3c15a54bb in os_event::wait_time_low (this=0x55f3c4390b40, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f3c15a565b in os_event_wait_time_low (event=0x55f3c4390b40, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f3c17db52c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f417d4d6700 (LWP 28041)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c1a64cab in safe_cond_timedwait (cond=0x55f3c3031ca0 , mp=0x55f3c3031be0 , abstime=0x7f417d4d5cb0, file=0x55f3c1eaa130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055f3c18ceadd in inline_mysql_cond_timedwait (that=0x55f3c3031ca0 , mutex=0x55f3c3031be0 , abstime=0x7f417d4d5cb0, src_file=0x55f3c1eaa178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055f3c18cf001 in my_service_thread_sleep (control=0x55f3c25e6900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055f3c18c349a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055f3c19f7471 in pfs_spawn_thread (arg=0x55f3c413e6c0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f41840b4700 (LWP 28067)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c1a649a9 in safe_cond_wait (cond=0x55f3c27a4fe0 , mp=0x55f3c27a8360 , file=0x55f3c1b2eb48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f3c0e623ce in inline_mysql_cond_wait (that=0x55f3c27a4fe0 , mutex=0x55f3c27a8360 , src_file=0x55f3c1b2f6b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055f3c0e63c60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055f3c19f7471 in pfs_spawn_thread (arg=0x55f3c4487ef0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4156ffd700 (LWP 28051)): #0 0x00007f418dffdd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a51a4 in os_event::timed_wait (this=0x55f3c41865d0, abstime=0x7f4156ffcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055f3c15a54bb in os_event::wait_time_low (this=0x55f3c41865d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055f3c15a565b in os_event_wait_time_low (event=0x55f3c41865d0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055f3c1693808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4184759700 (LWP 28039)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c1a649a9 in safe_cond_wait (cond=0x7f4168015628, mp=0x7f4168015578, file=0x55f3c1bcda00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055f3c11bad06 in inline_mysql_cond_wait (that=0x7f4168015628, mutex=0x7f4168015578, src_file=0x55f3c1bcdd20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055f3c11bd99e in Wsrep_thd_queue::pop_front (this=0x7f4168015520) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055f3c11bc77e in wsrep_post_rollback_process (post_rollbacker=0x7f4168000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055f3c11acf01 in start_wsrep_THD (arg=0x55f3c40785d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4145ffb700 (LWP 28061)): #0 0x00007f418e000e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055f3c15a5b8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055f3c174c755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f418470e700 (LWP 28040)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4185dfc132 in fifo_lock_get (q=q@entry=0x55f3c406c9e0) at galerautils/src/gu_fifo.c:262 #2 0x00007f4185dfc7a1 in gu_fifo_get_head (q=0x55f3c406c9e0, err=0x7f418470d06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f4185ede0cc in gcs_recv (conn=0x55f3c4046440, action=0x7f418470d0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f4185f0b16c in galera::GcsActionSource::process (this=0x55f3c4046090, recv_ctx=0x7f418470d4d0, exit_loop=@0x7f418470d29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f4185f2a033 in galera::ReplicatorSMM::async_recv (this=0x55f3c403e0c0, recv_ctx=0x7f418470d4d0) at galera/src/replicator_smm.cpp:383 #6 0x000055f3c1b1820e in wsrep::wsrep_provider_v26::run_applier (this=0x55f3c3fc5280, applier_ctx=0x7f418470d4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x000055f3c11bb2b4 in wsrep_replication_process (thd=0x7f416c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055f3c11acf01 in start_wsrep_THD (arg=0x55f3c4096b00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f41467fc700 (LWP 28060)): #0 0x00007f418dffd965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055f3c15a5889 in os_event::wait (this=0x55f3c41866d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055f3c15a530f in os_event::wait_low (this=0x55f3c41866d0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055f3c15a5680 in os_event_wait_low (event=0x55f3c41866d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055f3c1771dde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f418dff9dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f418c3b7ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f418e40a880 (LWP 28033)): #0 0x00007f418dffe9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055f3c1a5f37a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055f3c129bd65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f418c3ad20d in poll () from /lib64/libc.so.6 #5 0x000055f3c0e3d96a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055f3c0e3cc05 in mysqld_main (argc=161, argv=0x55f3c3ebfd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055f3c0e30da5 in main (argc=23, argv=0x7ffd01a9e738) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - found 'core.28075' (5/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_small_gcache-innodb/mysqld.2/data/core.28075 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 28075] [New LWP 28087] [New LWP 28077] [New LWP 28088] [New LWP 28078] [New LWP 28250] [New LWP 28089] [New LWP 28251] [New LWP 28091] [New LWP 28252] [New LWP 28253] [New LWP 28256] [New LWP 28254] [New LWP 28263] [New LWP 28262] [New LWP 28259] [New LWP 28274] [New LWP 28264] [New LWP 28261] [New LWP 28268] [New LWP 28265] [New LWP 28271] [New LWP 28266] [New LWP 28272] [New LWP 28273] [New LWP 28269] [New LWP 28270] [New LWP 28277] [New LWP 28267] [New LWP 28255] [New LWP 28260] [New LWP 28249] [New LWP 28090] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f4da144a9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f4da144a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556b38e6937a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556b386a5d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4d9f7f920d in poll () from /lib64/libc.so.6 #5 0x0000556b3824796a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000556b38246c05 in mysqld_main (argc=162, argv=0x556b3a75bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000556b3823ada5 in main (argc=23, argv=0x7fff1157abf8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f4d984a6700 (LWP 28090)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b38e6e9a9 in safe_cond_wait (cond=0x7f4d7c016938, mp=0x7f4d7c016888, file=0x556b38fd7a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556b385c4d06 in inline_mysql_cond_wait (that=0x7f4d7c016938, mutex=0x7f4d7c016888, src_file=0x556b38fd7d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556b385c799e in Wsrep_thd_queue::pop_front (this=0x7f4d7c016830) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000556b385c677e in wsrep_post_rollback_process (post_rollbacker=0x7f4d7c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x0000556b385b6f01 in start_wsrep_THD (arg=0x556b3a93b260) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f4d8385a700 (LWP 28249)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b38e6ecab in safe_cond_timedwait (cond=0x556b3a43bca0 , mp=0x556b3a43bbe0 , abstime=0x7f4d83859cb0, file=0x556b392b4130 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556b38cd8add in inline_mysql_cond_timedwait (that=0x556b3a43bca0 , mutex=0x556b3a43bbe0 , abstime=0x7f4d83859cb0, src_file=0x556b392b4178 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556b38cd9001 in my_service_thread_sleep (control=0x556b399f0900 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000556b38ccd49a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000556b38e01471 in pfs_spawn_thread (arg=0x556b3a9e3580) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f4d667fc700 (LWP 28260)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af1a4 in os_event::timed_wait (this=0x556b3aa2b250, abstime=0x7f4d667fbba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556b389af4bb in os_event::wait_time_low (this=0x556b3aa2b250, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556b389af65b in os_event_wait_time_low (event=0x556b3aa2b250, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556b38a9d808 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f4d677fe700 (LWP 28255)): #0 0x00007f4da1016644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556b389a2ba6 in LinuxAIOHandler::collect (this=0x7f4d677fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556b389a30cf in LinuxAIOHandler::poll (this=0x7f4d677fdb00, m1=0x7f4d677fdba8, m2=0x7f4d677fdbb0, request=0x7f4d677fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556b389a3232 in os_aio_linux_handler (global_segment=5, m1=0x7f4d677fdba8, m2=0x7f4d677fdbb0, request=0x7f4d677fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556b389a7d56 in os_aio_handler (segment=5, m1=0x7f4d677fdba8, m2=0x7f4d677fdbb0, request=0x7f4d677fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000556b38bfbbc1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000556b38aa112f in io_handler_thread (arg=0x556b39c2a448 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f4d55ffb700 (LWP 28267)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af889 in os_event::wait (this=0x556b3aa2b150) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556b389af30f in os_event::wait_low (this=0x556b3aa2b150, reset_sig_count=20382) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556b389af680 in os_event_wait_low (event=0x556b3aa2b150, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556b38a9b5ff in srv_resume_thread (slot=0x556b399ec5b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000556b38a9f203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4d9012d700 (LWP 28277)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b38e6ecab in safe_cond_timedwait (cond=0x556b39baefa0 , mp=0x556b39bb19a0 , abstime=0x7f4d9012cc60, file=0x556b38f29348 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556b3823b725 in inline_mysql_cond_timedwait (that=0x556b39baefa0 , mutex=0x556b39bb19a0 , abstime=0x7f4d9012cc60, src_file=0x556b38f2b3b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556b382404fb in cache_thread (thd=0x7f4d70000c40) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x0000556b382407a9 in one_thread_per_connection_end (thd=0x7f4d70000c40, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x0000556b384dbcda in do_handle_one_connection (connect=0x556b3ad68700) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1413 #6 0x0000556b384db994 in handle_one_connection (arg=0x556b3ad68700) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1301 #7 0x0000556b38e01471 in pfs_spawn_thread (arg=0x556b3adb2850) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f4d3ffff700 (LWP 28270)): #0 0x00007f4da144ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000556b389afb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000556b38b56755 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f4d54ff9700 (LWP 28269)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af889 in os_event::wait (this=0x556b3aa2b350) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556b389af30f in os_event::wait_low (this=0x556b3aa2b350, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556b389af680 in os_event_wait_low (event=0x556b3aa2b350, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556b38b7bdde in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4d9804c700 (LWP 28273)): #0 0x00007f4d9f73d40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x0000556b3823ae24 in my_sigwait (set=0x7f4d9804bcf0, sig=0x7f4d9804bcb4, code=0x7f4d9804bcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x0000556b38241464 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000556b38e01471 in pfs_spawn_thread (arg=0x556b3a9eedd0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4d3effd700 (LWP 28272)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b38e6e9a9 in safe_cond_wait (cond=0x7f4d3effcc70, mp=0x7f4d3effccb0, file=0x556b3913b860 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556b388c6a13 in inline_mysql_cond_wait (that=0x7f4d3effcc70, mutex=0x7f4d3effccb0, src_file=0x556b3913d3a0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=289) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556b388cb632 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:289 #4 0x0000556b38e01471 in pfs_spawn_thread (arg=0x556b3abe5f90) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f4d567fc700 (LWP 28266)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af889 in os_event::wait (this=0x556b3aa2b0d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556b389af30f in os_event::wait_low (this=0x556b3aa2b0d0, reset_sig_count=24488) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556b389af680 in os_event_wait_low (event=0x556b3aa2b0d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556b38a9b5ff in srv_resume_thread (slot=0x556b399ec580 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000556b38a9f203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f4d3f7fe700 (LWP 28271)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af889 in os_event::wait (this=0x556b3aa2b450) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556b389af30f in os_event::wait_low (this=0x556b3aa2b450, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556b389af680 in os_event_wait_low (event=0x556b3aa2b450, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556b38b648b3 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3092 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f4d56ffd700 (LWP 28265)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af889 in os_event::wait (this=0x556b3aa2b050) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556b389af30f in os_event::wait_low (this=0x556b3aa2b050, reset_sig_count=5171) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556b389af680 in os_event_wait_low (event=0x556b3aa2b050, reset_sig_count=5171) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556b38a9b5ff in srv_resume_thread (slot=0x556b399ec548 , sig_count=5171, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000556b38a9f610 in srv_purge_coordinator_suspend (slot=0x556b399ec548 , rseg_history_len=31) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x0000556b38a9f8a6 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f4d557fa700 (LWP 28268)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af889 in os_event::wait (this=0x556b3aa2b1d0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000556b389af30f in os_event::wait_low (this=0x556b3aa2b1d0, reset_sig_count=17593) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000556b389af680 in os_event_wait_low (event=0x556b3aa2b1d0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000556b38a9b5ff in srv_resume_thread (slot=0x556b399ec5f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x0000556b38a9f203 in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f4d657fa700 (LWP 28261)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af1a4 in os_event::timed_wait (this=0x556b3aa2b2d0, abstime=0x7f4d657f9d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556b389af4bb in os_event::wait_time_low (this=0x556b3aa2b2d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556b389af65b in os_event_wait_time_low (event=0x556b3aa2b2d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556b38a9d2e6 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f4d577fe700 (LWP 28264)): #0 0x00007f4da144ce3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000556b389afb8b in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000556b38a9eb67 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x0000556b38a9ec97 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4d90178700 (LWP 28274)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b38e6e9a9 in safe_cond_wait (cond=0x556b39baefe0 , mp=0x556b39bb2360 , file=0x556b38f38b48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556b3826c3ce in inline_mysql_cond_wait (that=0x556b39baefe0 , mutex=0x556b39bb2360 , src_file=0x556b38f396b0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000556b3826dc60 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000556b38e01471 in pfs_spawn_thread (arg=0x556b3ad9fda0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4d65ffb700 (LWP 28259)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af1a4 in os_event::timed_wait (this=0x556b3ac037a0, abstime=0x7f4d65ffad40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556b389af4bb in os_event::wait_time_low (this=0x556b3ac037a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556b389af65b in os_event_wait_time_low (event=0x556b3ac037a0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556b3897c314 in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f4d64ff9700 (LWP 28262)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af1a4 in os_event::timed_wait (this=0x556b3ac357c0, abstime=0x7f4d64ff8d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556b389af4bb in os_event::wait_time_low (this=0x556b3ac357c0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556b389af65b in os_event_wait_time_low (event=0x556b3ac357c0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556b38be552c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4d57fff700 (LWP 28263)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af1a4 in os_event::timed_wait (this=0x556b3abd30e0, abstime=0x7f4d57ffeb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556b389af4bb in os_event::wait_time_low (this=0x556b3abd30e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556b389af65b in os_event_wait_time_low (event=0x556b3abd30e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556b38afdaad in ib_wqueue_timedwait (wq=0x556b3abd3030, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000556b38c3ff11 in fts_optimize_thread (arg=0x556b3abd3030) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f4d67fff700 (LWP 28254)): #0 0x00007f4da1016644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556b389a2ba6 in LinuxAIOHandler::collect (this=0x7f4d67ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556b389a30cf in LinuxAIOHandler::poll (this=0x7f4d67ffeb00, m1=0x7f4d67ffeba8, m2=0x7f4d67ffebb0, request=0x7f4d67ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556b389a3232 in os_aio_linux_handler (global_segment=4, m1=0x7f4d67ffeba8, m2=0x7f4d67ffebb0, request=0x7f4d67ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556b389a7d56 in os_aio_handler (segment=4, m1=0x7f4d67ffeba8, m2=0x7f4d67ffebb0, request=0x7f4d67ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000556b38bfbbc1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000556b38aa112f in io_handler_thread (arg=0x556b39c2a440 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f4d66ffd700 (LWP 28256)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b389af1a4 in os_event::timed_wait (this=0x556b3aa2b3d0, abstime=0x7f4d66ffcaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000556b389af4bb in os_event::wait_time_low (this=0x556b3aa2b3d0, time_in_usec=795000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000556b389af65b in os_event_wait_time_low (event=0x556b3aa2b3d0, time_in_usec=795000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000556b38b85c57 in pc_sleep_if_needed (next_loop_time=1549303195436, sig_count=2, cur_time=1549303194641) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x0000556b38b86b90 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4d80a05700 (LWP 28253)): #0 0x00007f4da1016644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556b389a2ba6 in LinuxAIOHandler::collect (this=0x7f4d80a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556b389a30cf in LinuxAIOHandler::poll (this=0x7f4d80a04b00, m1=0x7f4d80a04ba8, m2=0x7f4d80a04bb0, request=0x7f4d80a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556b389a3232 in os_aio_linux_handler (global_segment=3, m1=0x7f4d80a04ba8, m2=0x7f4d80a04bb0, request=0x7f4d80a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556b389a7d56 in os_aio_handler (segment=3, m1=0x7f4d80a04ba8, m2=0x7f4d80a04bb0, request=0x7f4d80a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000556b38bfbbc1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000556b38aa112f in io_handler_thread (arg=0x556b39c2a438 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4d81206700 (LWP 28252)): #0 0x00007f4da1016644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556b389a2ba6 in LinuxAIOHandler::collect (this=0x7f4d81205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556b389a30cf in LinuxAIOHandler::poll (this=0x7f4d81205b00, m1=0x7f4d81205ba8, m2=0x7f4d81205bb0, request=0x7f4d81205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556b389a3232 in os_aio_linux_handler (global_segment=2, m1=0x7f4d81205ba8, m2=0x7f4d81205bb0, request=0x7f4d81205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556b389a7d56 in os_aio_handler (segment=2, m1=0x7f4d81205ba8, m2=0x7f4d81205bb0, request=0x7f4d81205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000556b38bfbbc1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000556b38aa112f in io_handler_thread (arg=0x556b39c2a430 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f4d9845b700 (LWP 28091)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4d99248132 in fifo_lock_get (q=q@entry=0x556b3a9089f0) at galerautils/src/gu_fifo.c:262 #2 0x00007f4d992487a1 in gu_fifo_get_head (q=0x556b3a9089f0, err=0x7f4d9845a06c) at galerautils/src/gu_fifo.c:330 #3 0x00007f4d9932a0cc in gcs_recv (conn=0x556b3a8e2450, action=0x7f4d9845a0b0) at gcs/src/gcs.cpp:2024 #4 0x00007f4d9935716c in galera::GcsActionSource::process (this=0x556b3a8e20a0, recv_ctx=0x7f4d9845a4d0, exit_loop=@0x7f4d9845a29e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f4d99376033 in galera::ReplicatorSMM::async_recv (this=0x556b3a8da0d0, recv_ctx=0x7f4d9845a4d0) at galera/src/replicator_smm.cpp:383 #6 0x0000556b38f2220e in wsrep::wsrep_provider_v26::run_applier (this=0x556b3a861280, applier_ctx=0x7f4d9845a4d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #7 0x0000556b385c52b4 in wsrep_replication_process (thd=0x7f4d78000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x0000556b385b6f01 in start_wsrep_THD (arg=0x556b3a93b610) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #9 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f4d81a07700 (LWP 28251)): #0 0x00007f4da1016644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556b389a2ba6 in LinuxAIOHandler::collect (this=0x7f4d81a06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556b389a30cf in LinuxAIOHandler::poll (this=0x7f4d81a06b00, m1=0x7f4d81a06ba8, m2=0x7f4d81a06bb0, request=0x7f4d81a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556b389a3232 in os_aio_linux_handler (global_segment=1, m1=0x7f4d81a06ba8, m2=0x7f4d81a06bb0, request=0x7f4d81a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556b389a7d56 in os_aio_handler (segment=1, m1=0x7f4d81a06ba8, m2=0x7f4d81a06bb0, request=0x7f4d81a06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000556b38bfbbc1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000556b38aa112f in io_handler_thread (arg=0x556b39c2a428 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f4d984f1700 (LWP 28089)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b38e6e9a9 in safe_cond_wait (cond=0x7f4d88016878, mp=0x7f4d880167c8, file=0x556b38fd7a00 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x0000556b385c4d06 in inline_mysql_cond_wait (that=0x7f4d88016878, mutex=0x7f4d880167c8, src_file=0x556b38fd7d20 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x0000556b385c799e in Wsrep_thd_queue::pop_front (this=0x7f4d88016770) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x0000556b385c63be in wsrep_rollback_process (rollbacker=0x7f4d88001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x0000556b385b6f01 in start_wsrep_THD (arg=0x556b3a93caa0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4d82208700 (LWP 28250)): #0 0x00007f4da1016644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000556b389a2ba6 in LinuxAIOHandler::collect (this=0x7f4d82207b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000556b389a30cf in LinuxAIOHandler::poll (this=0x7f4d82207b00, m1=0x7f4d82207ba8, m2=0x7f4d82207bb0, request=0x7f4d82207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000556b389a3232 in os_aio_linux_handler (global_segment=0, m1=0x7f4d82207ba8, m2=0x7f4d82207bb0, request=0x7f4d82207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000556b389a7d56 in os_aio_handler (segment=0, m1=0x7f4d82207ba8, m2=0x7f4d82207bb0, request=0x7f4d82207be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x0000556b38bfbbc1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x0000556b38aa112f in io_handler_thread (arg=0x556b39c2a420 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4d98cf2700 (LWP 28078)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4d99353fd7 in gu::Lock::wait (cond=..., this=0x7f4d98cf1bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4d99354069 in galera::ServiceThd::thd_func (arg=0x556b3a8da7e8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4d91b7b700 (LWP 28088)): #0 0x00007f4da1449965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4d9932e867 in gu::Lock::wait (cond=..., this=0x7f4d91b7a850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4d993313ab in RecvBuf::front (this=this@entry=0x556b3a91a678, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f4d9932ef46 in gcomm_recv (backend=, msg=0x556b3a8e2730, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f4d99322eec in core_msg_recv (backend=backend@entry=0x556b3a8e2878, recv_msg=recv_msg@entry=0x556b3a8e2730, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f4d99326164 in gcs_core_recv (conn=0x556b3a8e26d0, recv_act=recv_act@entry=0x7f4d91b7ad70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f4d9932d3d8 in gcs_recv_thread (arg=0x556b3a8e2450) at gcs/src/gcs.cpp:1398 #7 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4d99e9e700 (LWP 28077)): #0 0x00007f4da1449d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000556b38e6ecab in safe_cond_timedwait (cond=0x556b3a44e440 , mp=0x556b3a44e380 , abstime=0x7f4d99e9dd70, file=0x556b392f8360 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000556b38e6fc65 in inline_mysql_cond_timedwait (that=0x556b3a44e440 , mutex=0x556b3a44e380 , abstime=0x7f4d99e9dd70, src_file=0x556b392f83a8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000556b38e707c8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000556b38e01471 in pfs_spawn_thread (arg=0x556b3a8a3700) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f4d9237c700 (LWP 28087)): #0 0x00007f4d9f804483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f4d992f56f8 in asio::detail::epoll_reactor::run (this=0x556b3a91af40, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f4d992f9df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x556b3a91ade0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f4d992fa0a2 in asio::detail::task_io_service::run (this=0x556b3a91ade0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f4d99311afd in asio::io_service::run (this=0x556b3a91ab40) at asio/asio/impl/io_service.ipp:58 #5 0x00007f4d99312251 in gcomm::AsioProtonet::event_loop (this=0x556b3a91aaa0, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f4d993302ed in GCommConn::run (this=0x556b3a91a490) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f4d99333399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f4da1445dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f4d9f803ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4da1856880 (LWP 28075)): #0 0x00007f4da144a9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000556b38e6937a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x0000556b386a5d65 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4d9f7f920d in poll () from /lib64/libc.so.6 #5 0x0000556b3824796a in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x0000556b38246c05 in mysqld_main (argc=162, argv=0x556b3a75bd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x0000556b3823ada5 in main (argc=23, argv=0x7fff1157abf8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 - deleting it, already saved 5 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_small_gcache-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_small_gcache-innodb/' -------------------------------------------------------------------------- The servers were restarted 25 times Spent 1263.776 of 11833 seconds executing testcases mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/stepan/galera/git/10.4/server/mysql-test/var/log/warnings' for details. Check of testcase failed for: galera_sr.mysql-wsrep-features#148 galera_sr.galera_sr_kill_all_norecovery Completed: Failed 18/83 tests, 78.31% were successful. Failing test(s): galera_sr.galera_sr_large_fragment galera_sr.GCF-437 galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_many_fragments galera_sr.mysql-wsrep-features#8 galera_sr.GCF-1043A galera_sr.GCF-1043B galera_sr.GCF-867 galera_sr.galera-features#56 galera_sr.galera_sr_cc_master galera_sr.galera_sr_ddl_master galera_sr.galera_sr_insert_select galera_sr.galera_sr_load_data galera_sr.galera_sr_rollback_statement galera_sr.galera_sr_table_contents galera_sr.mysql-wsrep-features#165 galera_sr.galera_sr_mysqldump_sst galera_sr.galera_sr_small_gcache The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html Errors/warnings were found in logfiles during server shutdown after running the following sequence(s) of tests: galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_kill_all_pcrecovery galera_sr.galera_sr_many_fragments galera_sr.mysql-wsrep-features#8 galera_sr.GCF-1008 galera_sr.GCF-1043A galera_sr.GCF-1018 galera_sr.mysql-wsrep-features#9 galera_sr.GCF-1043B galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera-features#56 galera_sr.galera_sr_cc_master galera_sr.galera_sr_bf_abort galera_sr.galera_sr_blob galera_sr.galera_sr_fk_conflict galera_sr.galera_sr_ddl_slave galera_sr.galera_sr_dupkey_error galera_sr.galera_sr_ddl_unrelated galera_sr.galera_sr_ddl_schema galera_sr.galera_sr_insert_select galera_sr.galera_sr_load_data galera_sr.galera_sr_kill_query galera_sr.galera_sr_kill_connection galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 mysql-test-run: *** ERROR: there were failing test cases