Checking supported features... MariaDB Version 10.4.3-MariaDB-debug - SSL connections supported - binaries are debug compiled Using suites: galera_sr Collecting tests... Installing system database... Using parallel: 4 worker[1] Using MTR_BUILD_THREAD 302, with reserved ports 16040..16059 worker[4] Using MTR_BUILD_THREAD 303, with reserved ports 16060..16079 worker[2] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 worker[3] Using MTR_BUILD_THREAD 301, with reserved ports 16020..16039 ============================================================================== TEST WORKER RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- galera_sr.GCF-574 [ disabled ] CTAS is not supported together with SR galera_sr.galera_sr_sbr [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.mysql-wsrep-features#29 [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.galera_sr_gtid 'innodb' w2 [ pass ] 2188 galera_sr.galera_sr_log_bin 'innodb' w1 [ pass ] 2898 galera_sr.mysql-wsrep-features#32 'innodb' w3 [ pass ] 3500 galera_sr.mysql-wsrep-features#136 'innodb' w2 [ pass ] 2105 galera_sr.galera_sr_v1_row_events 'innodb' w1 [ pass ] 2162 galera_sr.GCF-623 'innodb' w2 [ pass ] 2198 galera_sr.galera_sr_conflict_with_rollback_master 'innodb' w1 [ pass ] 2217 galera_sr.GCF-627 'innodb' w2 [ pass ] 2186 galera_sr.galera_sr_ddl_master 'innodb' w1 [ pass ] 2348 galera_sr.galera_sr_ddl_schema 'innodb' w1 [ pass ] 2152 galera_sr.GCF-845 'innodb' w2 [ pass ] 4172 galera_sr.galera_sr_ddl_slave 'innodb' w1 [ pass ] 2347 galera_sr.GCF-851 'innodb' w2 [ pass ] 2139 galera_sr.galera_sr_ddl_unrelated 'innodb' w1 [ pass ] 2287 galera_sr.galera_sr_dupkey_error 'innodb' w1 [ pass ] 2335 galera_sr.GCF-867 'innodb' w2 [ pass ] 3874 galera_sr.GCF-889 'innodb' w2 [ pass ] 2156 galera_sr.GCF-1018B 'innodb' w3 [ pass ] 38199 galera_sr.GCF-900 'innodb' w2 [ pass ] 2335 galera_sr.galera_sr_fk_conflict 'innodb' w1 [ pass ] 5991 galera_sr.GCF-437 'innodb' w3 [ fail ] Test ended at 2019-02-11 02:39:24 CURRENT_TEST: galera_sr.GCF-437 mysqltest: At line 13: query 'INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5' succeeded - should have failed with errno 1534... The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=MyISAM; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 2 * 1024 * 1024; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-437-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-437-innodb/' galera_sr.galera_sr_large_fragment 'innodb' w4 [ pass ] 91684 galera_sr.galera_sr_kill_all_nobootstrap 'innodb' w3 [ pass ] 13796 galera_sr.galera_sr_parallel_apply 'innodb' w4 [ pass ] 2349 galera_sr.galera_sr_rollback 'innodb' w4 [ pass ] 2568 galera_sr.galera_sr_rollback_retry 'innodb' w4 [ pass ] 2357 galera_sr.galera_sr_rollback_savepoint 'innodb' w4 [ pass ] 2465 galera_sr.galera_sr_kill_all_pcrecovery 'innodb' w3 [ pass ] 18933 galera_sr.galera_sr_rollback_statement 'innodb' w4 [ fail ] Test ended at 2019-02-11 02:40:18 CURRENT_TEST: galera_sr.galera_sr_rollback_statement mysqltest: At line 34: query 'INSERT INTO t2 SELECT * FROM t1' succeeded - should have failed with errno 1062... The result from queries just before the failure was: connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 CHAR(255)) ENGINE=InnoDB; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3; ALTER TABLE t1 CHANGE f1 f1 INTEGER; ALTER TABLE t1 DROP PRIMARY KEY; INSERT INTO t1 VALUES (1, 'abc'); SET SESSION wsrep_trx_fragment_size = 1; INSERT INTO t2 SELECT * FROM t1; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_rollback_statement-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_rollback_statement-innodb/' galera_sr.galera_sr_shutdown_master 'innodb' w4 [ pass ] 21200 galera_sr.galera_sr_shutdown_slave 'innodb' w4 [ pass ] 20758 galera_sr.galera_sr_table_contents 'innodb' w4 [ fail ] Test ended at 2019-02-11 02:41:09 CURRENT_TEST: galera_sr.galera_sr_table_contents cp: cannot stat 'std_data/binlog-header.log': No such file or directory mysqltest: In included file "./suite/galera/include/galera_dump_sr_table.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/galera_sr_table_contents.test at line 21: At line 20: exec of 'cp std_data/binlog-header.log /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/sr_binlog.log' failed, error: 256, status: 1, errno: 11 Output from before failure: Warnings: Warning 1287 ' INTO FROM...' instead The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t1 VALUES (1),(2),(3); CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t2 VALUES (1),(2),(3); SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; Start of Simple Insert INSERT INTO t1 VALUES (4); Warnings: Warning 1287 ' INTO FROM...' instead - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_table_contents-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_table_contents-innodb/' galera_sr.galera_sr_transaction_replay 'innodb' w4 [ pass ] 2578 galera_sr.galera_sr_unit_statements 'innodb' w4 [ pass ] 2283 galera_sr.galera_sr_ws_size 'innodb' w4 [ pass ] 3307 galera_sr.galera_sr_ws_size2 'innodb' w4 [ pass ] 4432 galera_sr.galera_var_ignore_apply_errors_sr 'innodb' w4 [ pass ] 2329 galera_sr.mysql-wsrep#215 'innodb' w4 [ pass ] 3183 galera_sr.mysql-wsrep-features#138 'innodb' w4 [ pass ] 2374 galera_sr.mysql-wsrep-features#14 'innodb' w4 [ pass ] 2244 galera_sr.mysql-wsrep-features#148 'innodb' w4 [ pass ] 2818 MTR's internal check of the test case 'galera_sr.mysql-wsrep-features#148' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16061 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.result 2019-02-11 02:41:41.499745800 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_2.reject 2019-02-11 02:41:44.542745800 +0200 @@ -1099,4 +1099,4 @@ user_variables 1.0 DISABLED INFORMATION SCHEMA 100403.0 NULL NULL Sergey Vojtovich User-defined variables GPL OFF Stable 1.0 partition 1.0 DISABLED STORAGE ENGINE 100403.0 NULL NULL Mikael Ronstrom, MySQL AB Partition Storage Engine Helper GPL OFF Stable 1.0 VARIABLE_NAME VARIABLE_VALUE -DEBUG_SYNC ON - current signal: '' +DEBUG_SYNC ON - current signal: 'signal.wsrep_apply_cb' mysqltest: Result length mismatch not ok galera_sr.galera_sr_insert_select 'innodb' w1 [ pass ] 191280 galera_sr.galera_sr_kill_connection 'innodb' w1 [ pass ] 2593 galera_sr.galera_sr_kill_query 'innodb' w1 [ pass ] 3173 galera_sr.mysql-wsrep-features#15 'innodb' w4 [ pass ] 2240 galera_sr.mysql-wsrep-features#165 'innodb' w4 [ fail ] Test ended at 2019-02-11 02:42:16 CURRENT_TEST: galera_sr.mysql-wsrep-features#165 mysqltest: In included file "/home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.test at line 27: At line 101: query 'SELECT * FROM t1' failed: 1205: Lock wait timeout exceeded; try restarting transaction The result from queries just before the failure was: < snip > connection node_1; SET DEBUG_SYNC = 'now WAIT_FOR wait'; connection node_1; SET DEBUG_SYNC = 'now SIGNAL continue'; connection node_1c; UPDATE t1 SET f2 = 'x' WHERE f1 = 3; connection node_1a; COMMIT; connection node_1b; ERROR 40001: Deadlock found when trying to get lock; try restarting transaction connection node_1; SELECT * FROM t1; f1 f2 1 a 2 a 3 x 4 a 5 a connection node_2; SELECT * FROM t1; More results from queries before failure can be found in /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/mysql-wsrep-features#165.log - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.mysql-wsrep-features#165-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#165-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 2019-02-11 2:42:05 3 [Warning] WSREP: Event 3 Write_rows_v1 apply failed: 121, seqno 12 galera_sr.mysql-wsrep-features#22 'innodb' w4 [ pass ] 2468 galera_sr.mysql-wsrep-features#27 'innodb' w4 [ pass ] 4352 galera_sr.mysql-wsrep-features#35 'innodb' w4 [ pass ] 2451 galera_sr.mysql-wsrep-features#93 'innodb' w4 [ pass ] 2301 galera_sr.mysql-wsrep-features#96 'innodb' w4 [ pass ] 2514 galera_sr.galera_sr_kill_all_norecovery 'innodb' w4 [ pass ] 13810 MTR's internal check of the test case 'galera_sr.galera_sr_kill_all_norecovery' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_1.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_1.result'. mysqltest: Connecting to server localhost:16060 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/4/mysqld.1.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_1.result 2019-02-11 02:42:54.036745800 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/4/tmp/check-mysqld_1.reject 2019-02-11 02:43:08.095745800 +0200 @@ -77,7 +77,7 @@ ARIA_USED_FOR_TEMP_TABLES ON AUTOCOMMIT ON AUTOMATIC_SP_PRIVILEGES ON -AUTO_INCREMENT_OFFSET 1 +AUTO_INCREMENT_OFFSET 2 BACK_LOG 80 BASEDIR /home/stepan/galera/git/10.4/server BIG_TABLES OFF mysqltest: Result content mismatch not ok galera_sr.galera_sr_kill_slave 'innodb' w4 [ pass ] 21347 galera_sr.galera_sr_load_data 'innodb' w1 [ pass ] 132932 galera_sr.galera_sr_myisam 'innodb' w1 [ pass ] 2148 galera_sr.galera_sr_mysqldump_sst 'innodb' w4 [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-11 02:44:49 line 2019-02-11 2:44:37 14 [Warning] WSREP: no THD for trx: 51 ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/4/log/mysqld.2.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/4/log/galera_sr.galera_sr_mysqldump_sst-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' galera_sr.galera_sr_many_fragments 'innodb' w1 [ pass ] 207628 galera_sr.galera-features#56 'innodb' w2 [ pass ] 543158 galera_sr.galera_sr_bf_abort 'innodb' w2 [ pass ] 4824 galera_sr.galera_sr_blob 'innodb' w2 [ pass ] 2356 galera_sr.galera_sr_small_gcache 'innodb' w4 [ pass ] 188362 galera_sr.mysql-wsrep-features#9 'innodb' w1 [ pass ] 19689 galera_sr.GCF-1018 'innodb' w1 [ pass ] 2591 galera_sr.galera_sr_cc_master 'innodb' w2 [ fail ] Test ended at 2019-02-11 02:48:25 CURRENT_TEST: galera_sr.galera_sr_cc_master Server [mysqld.2 - pid: 40674, winpid: 40674, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2019-02-11 2:47:55 10157 [Note] WSREP: Stop replication 2019-02-11 2:47:55 10157 [Note] WSREP: Server status change synced -> disconnecting 2019-02-11 2:47:55 10157 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 2:47:55 10157 [Note] WSREP: Closing send monitor... 2019-02-11 2:47:55 10157 [Note] WSREP: Closed send monitor. 2019-02-11 2:47:55 10157 [Note] WSREP: gcomm: terminating thread 2019-02-11 2:47:55 10157 [Note] WSREP: gcomm: joining thread 2019-02-11 2:47:55 10157 [Note] WSREP: gcomm: closing backend 2019-02-11 2:47:59 10157 [Note] WSREP: (51a3e14f, 'tcp://0.0.0.0:16005') connection to peer 512e1540 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-11 2:47:59 10157 [Note] WSREP: (51a3e14f, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16002 2019-02-11 2:48:00 10157 [Note] WSREP: (51a3e14f, 'tcp://0.0.0.0:16005') reconnecting to 512e1540 (tcp://127.0.0.1:16002), attempt 0 2019-02-11 2:48:04 10157 [Note] WSREP: (51a3e14f, 'tcp://0.0.0.0:16005') connection to peer 512e1540 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-11 2:48:05 10157 [Note] WSREP: evs::proto(51a3e14f, LEAVING, view_id(REG,512e1540,2)) suspecting node: 512e1540 2019-02-11 2:48:05 10157 [Note] WSREP: evs::proto(51a3e14f, LEAVING, view_id(REG,512e1540,2)) suspected node without join message, declaring inactive 2019-02-11 2:48:05 10157 [Note] WSREP: view(view_id(NON_PRIM,512e1540,2) memb { 51a3e14f,0 } joined { } left { } partitioned { 512e1540,0 }) 2019-02-11 2:48:05 10157 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-11 2:48:05 10157 [Note] WSREP: view((empty)) 2019-02-11 2:48:05 10157 [Note] WSREP: gcomm: closed 2019-02-11 2:48:05 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-11 2:48:05 0 [Note] WSREP: Writing down CC checksum: 1656b64e a89af623 e347ef63 3450fcd9 at offset 144 2019-02-11 2:48:05 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-11 2:48:05 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 2:48:05 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-11 2:48:05 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 30230) 2019-02-11 2:48:05 0 [Note] WSREP: Received self-leave message. 2019-02-11 2:48:05 0 [Note] WSREP: Writing down CC checksum: 7595294b cd6e1b18 34a1d2fa 884489c7 at offset 64 2019-02-11 2:48:05 124 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-11 2:48:05 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-11 2:48:05 124 [Note] WSREP: ####### drain monitors upto 30230 2019-02-11 2:48:05 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 2:48:05 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-11 2:48:05 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-11 2:48:05 124 [Note] WSREP: ####### My UUID: 51a3e14f-2d95-11e9-b48b-f22c1dd24056 2019-02-11 2:48:05 124 [Note] WSREP: ####### ST not required 2019-02-11 2:48:05 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-11 2:48:05 10157 [Note] WSREP: recv_thread() joined. 2019-02-11 2:48:05 10157 [Note] WSREP: Closing replication queue. 2019-02-11 2:48:05 10157 [Note] WSREP: Closing slave action queue. 2019-02-11 2:48:05 124 [Note] WSREP: ================================================ View: id: 512f3d14-2d95-11e9-bd23-562e3bf7a872:-1 seqno: -1 status: 1 prococol_version: 4 own_index: 0 final: 0 members 2019-02-11 2:48:05 124 [Note] WSREP: id: 51a3e14f-2d95-11e9-b48b-f22c1dd24056 name: cnt7glr11.localdomain 2019-02-11 2:48:05 124 [Note] WSREP: ================================================= 2019-02-11 2:48:05 124 [Note] WSREP: Non-primary view 2019-02-11 2:48:05 124 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 2:48:05 124 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-11 2:48:05 124 [Note] WSREP: ####### drain monitors upto 30230 2019-02-11 2:48:05 124 [Note] WSREP: ####### My UUID: 51a3e14f-2d95-11e9-b48b-f22c1dd24056 2019-02-11 2:48:05 124 [Note] WSREP: ####### ST not required 2019-02-11 2:48:05 124 [Note] WSREP: ================================================ View: id: 512f3d14-2d95-11e9-bd23-562e3bf7a872:-1 seqno: -1 status: 1 prococol_version: 4 own_index: -1 final: 1 members 2019-02-11 2:48:05 124 [Note] WSREP: ================================================= 2019-02-11 2:48:05 124 [Note] WSREP: Non-primary view 2019-02-11 2:48:05 124 [Note] WSREP: Server status change disconnecting -> disconnected 2019-02-11 2:48:05 124 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 2:48:05 124 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 2:48:05 124 [Note] WSREP: Applier thread exiting 0 2019-02-11 2:48:05 10151 [Warning] WSREP: Failed to replicate rollback fragment for 20938: 3 2019-02-11 2:48:07 0 [Note] WSREP: rollbacker thread exiting 2019-02-11 2:48:08 10157 [Note] WSREP: Stop replication 2019-02-11 2:48:10 10159 [Note] WSREP: rollbacker thread exiting 2019-02-11 2:48:10 10157 [Note] WSREP: Start replication 2019-02-11 2:48:10 10157 [Note] WSREP: Connecting with bootstrap option: 0 2019-02-11 2:48:10 10157 [Note] WSREP: Setting GCS initial position to 512f3d14-2d95-11e9-bd23-562e3bf7a872:30230 2019-02-11 2:48:10 10157 [Note] WSREP: protonet asio version 0 2019-02-11 2:48:10 10157 [Note] WSREP: Using CRC-32C for message checksums. 2019-02-11 2:48:10 10157 [Note] WSREP: backend: asio 2019-02-11 2:48:10 10157 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-02-11 2:48:10 10157 [Warning] WSREP: access file(/home/stepan/galera/git/10.4/server/mysql-test/var/2/mysqld.2/data//gvwstate.dat) failed(No such file or directory) 2019-02-11 2:48:10 10157 [Note] WSREP: restore pc from disk failed 2019-02-11 2:48:10 10157 [Note] WSREP: GMCast version 0 2019-02-11 2:48:10 10157 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') listening at tcp://0.0.0.0:16005 2019-02-11 2:48:10 10157 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') multicast: , ttl: 1 2019-02-11 2:48:10 10157 [Note] WSREP: EVS version 1 2019-02-11 2:48:10 10157 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '127.0.0.1:16002' 2019-02-11 2:48:10 10157 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') connection established to 512e1540 tcp://127.0.0.1:16002 2019-02-11 2:48:10 10157 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: 2019-02-11 2:48:11 10157 [Note] WSREP: EVS version upgrade 0 -> 1 2019-02-11 2:48:11 10157 [Note] WSREP: declaring 512e1540 at tcp://127.0.0.1:16002 stable 2019-02-11 2:48:11 10157 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-02-11 2:48:11 10157 [Note] WSREP: Node 512e1540 state prim 2019-02-11 2:48:11 10157 [Note] WSREP: view(view_id(PRIM,512e1540,4) memb { 512e1540,0 b48ed775,0 } joined { } left { } partitioned { }) 2019-02-11 2:48:11 10157 [Note] WSREP: save pc into disk 2019-02-11 2:48:11 10157 [Note] WSREP: gcomm: connected 2019-02-11 2:48:11 10157 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-02-11 2:48:11 10157 [Note] WSREP: Shifting CLOSED -> OPEN (TO: -1) 2019-02-11 2:48:11 10157 [Note] WSREP: Opened channel 'my_wsrep_cluster' 2019-02-11 2:48:11 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2019-02-11 2:48:11 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-02-11 2:48:11 0 [Note] WSREP: STATE EXCHANGE: sent state msg: b4dde347-2d96-11e9-987f-d6ec9f4f3653 2019-02-11 2:48:11 0 [Note] WSREP: STATE EXCHANGE: got state msg: b4dde347-2d96-11e9-987f-d6ec9f4f3653 from 0 (cnt7glr11.localdomain) 2019-02-11 2:48:11 0 [Note] WSREP: STATE EXCHANGE: got state msg: b4dde347-2d96-11e9-987f-d6ec9f4f3653 from 1 (cnt7glr11.localdomain) 2019-02-11 2:48:11 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 3, members = 1/2 (joined/total), act_id = 30231, last_appl. = 30134, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 512f3d14-2d95-11e9-bd23-562e3bf7a872 2019-02-11 2:48:11 0 [Note] WSREP: Writing down CC checksum: bcd27265 da1d9272 36d5c006 3ece4b77 at offset 232 2019-02-11 2:48:11 0 [Note] WSREP: Flow-control interval: [23, 23] 2019-02-11 2:48:11 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 2:48:11 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 30232) 2019-02-11 2:48:11 10162 [Note] WSREP: Starting applier thread 10162 2019-02-11 2:48:11 10162 [Note] WSREP: ####### processing CC 30232, local, ordered 2019-02-11 2:48:11 10162 [Note] WSREP: ####### drain monitors upto 30230 2019-02-11 2:48:11 10162 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-02-11 2:48:11 10162 [Note] WSREP: ####### My UUID: b48ed775-2d96-11e9-9e04-0712bf9d38c4 2019-02-11 2:48:11 10162 [Note] WSREP: Server cnt7glr11.localdomain connected to cluster at position 512f3d14-2d95-11e9-bd23-562e3bf7a872:30232 with ID b48ed775-2d96-11e9-9e04-0712bf9d38c4 2019-02-11 2:48:11 10162 [Note] WSREP: Server status change disconnected -> connected 2019-02-11 2:48:11 10162 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 2:48:11 10162 [Note] WSREP: State transfer required: Group state: 512f3d14-2d95-11e9-bd23-562e3bf7a872:30232 Local state: 512f3d14-2d95-11e9-bd23-562e3bf7a872:30230 2019-02-11 2:48:11 10162 [Note] WSREP: Server status change connected -> joiner 2019-02-11 2:48:11 10162 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-11 2:48:11 10162 [Note] WSREP: You have configured 'rsync' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2019-02-11 2:48:11 10162 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-02-11 2:48:11 0 [Note] WSREP: Service thread queue flushed. 2019-02-11 2:48:11 10162 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-02-11 2:48:11 10162 [Note] WSREP: ####### IST uuid:512f3d14-2d95-11e9-bd23-562e3bf7a872 f: 30231, l: 30232, p: 10 2019-02-11 2:48:11 10162 [Note] WSREP: IST receiver addr using tcp://127.0.0.1:16006 2019-02-11 2:48:11 10162 [Note] WSREP: Prepared IST receiver for 30231-30232, listening at: tcp://127.0.0.1:16006 2019-02-11 2:48:11 0 [Note] WSREP: Member 1.0 (cnt7glr11.localdomain) requested state transfer from '*any*'. Selected 0.0 (cnt7glr11.localdomain)(SYNCED) as donor. 2019-02-11 2:48:11 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 30232) 2019-02-11 2:48:11 10162 [Note] WSREP: Requesting state transfer: success, donor: 0 2019-02-11 2:48:11 10162 [Note] WSREP: Receiving IST: 2 writesets, seqnos 30231-30232 2019-02-11 2:48:11 0 [Note] WSREP: ####### IST applying starts with 30231 2019-02-11 2:48:11 0 [Note] WSREP: ####### IST current seqno initialized to 30131 2019-02-11 2:48:11 0 [Note] WSREP: Receiving IST... 0.0% ( 0/102 events) complete. 2019-02-11 2:48:11 0 [Note] WSREP: IST preload starting at 30131 2019-02-11 2:48:11 0 [Note] WSREP: Service thread queue flushed. 2019-02-11 2:48:11 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:30130, protocol version: 5 2019-02-11 2:48:11 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-11 2:48:11 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 30232 last received: 30171 2019-02-11 2:48:11 0 [Note] WSREP: 0.0 (cnt7glr11.localdomain): State transfer to 1.0 (cnt7glr11.localdomain) complete. 2019-02-11 2:48:11 10162 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) at galera/src/replicator_smm.hpp:pop_front():320. Null event. 2019-02-11 2:48:11 0 [Note] WSREP: Member 0.0 (cnt7glr11.localdomain) synced with group. 2019-02-11 2:48:11 10162 [Note] WSREP: Closing send monitor... 2019-02-11 2:48:11 10162 [Note] WSREP: Closed send monitor. 2019-02-11 2:48:11 10162 [Note] WSREP: gcomm: terminating thread 2019-02-11 2:48:11 10162 [Note] WSREP: gcomm: joining thread 2019-02-11 2:48:11 10162 [Note] WSREP: gcomm: closing backend 2019-02-11 2:48:14 10162 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') turning message relay requesting off 2019-02-11 2:48:15 10162 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') connection to peer 512e1540 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-11 2:48:15 10162 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16002 2019-02-11 2:48:16 10162 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') reconnecting to 512e1540 (tcp://127.0.0.1:16002), attempt 0 2019-02-11 2:48:19 10162 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') connection to peer 512e1540 with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-11 2:48:21 10162 [Note] WSREP: (b48ed775, 'tcp://0.0.0.0:16005') connection established to 512e1540 tcp://127.0.0.1:16002 2019-02-11 2:48:21 10162 [Note] WSREP: evs::proto(b48ed775, LEAVING, view_id(REG,512e1540,4)) suspecting node: 512e1540 2019-02-11 2:48:21 10162 [Note] WSREP: evs::proto(b48ed775, LEAVING, view_id(REG,512e1540,4)) suspected node without join message, declaring inactive 2019-02-11 2:48:21 10162 [Note] WSREP: view(view_id(NON_PRIM,512e1540,4) memb { b48ed775,0 } joined { } left { } partitioned { 512e1540,0 }) 2019-02-11 2:48:21 10162 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-11 2:48:21 10162 [Note] WSREP: view((empty)) 2019-02-11 2:48:21 10162 [Note] WSREP: gcomm: closed 2019-02-11 2:48:21 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-11 2:48:21 0 [Note] WSREP: Writing down CC checksum: 87da2db1 7272eed9 99efccaa 1156cfe2 at offset 144 2019-02-11 2:48:21 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-11 2:48:21 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 2:48:21 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-11 2:48:21 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 30232) 2019-02-11 2:48:21 0 [Note] WSREP: Received self-leave message. 2019-02-11 2:48:21 0 [Note] WSREP: Writing down CC checksum: 7595294b cd6e1b18 34a1d2fa 884489c7 at offset 64 2019-02-11 2:48:21 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-11 2:48:21 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-11 2:48:21 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-11 2:48:21 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-11 2:48:21 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-11 2:48:21 10162 [Note] WSREP: recv_thread() joined. 2019-02-11 2:48:21 10162 [Note] WSREP: Closing replication queue. 2019-02-11 2:48:21 10162 [Note] WSREP: Closing slave action queue. mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. 190211 2:48:21 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.3-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63454 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4a1c010780 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f4a382049a0 thread_stack 0x49000 /home/stepan/galera/git/10.4/server/sql/mysqld(my_print_stacktrace+0x40)[0x55fb385d7700] mysys/stacktrace.c:269(my_print_stacktrace)[0x55fb37e1525b] sigaction.c:0(__restore_rt)[0x7f4a402df5d0] :0(__GI_raise)[0x7f4a3e5ce207] :0(__GI_abort)[0x7f4a3e5cf8f8] :0(__assert_fail_base)[0x7f4a3e5c7026] :0(__GI___assert_fail)[0x7f4a3e5c70d2] src/replicator_str.cpp:945(galera::ReplicatorSMM::request_state_transfer(void*, wsrep_uuid const&, long, void const*, long))[0x7f4a33ce9dc7] src/replicator_smm.cpp:2733(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7f4a33cd8e78] src/gcs_action_source.cpp:154(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f4a33cb8ec9] src/gcs_action_source.cpp:188(galera::GcsActionSource::process(void*, bool&))[0x7f4a33cb921d] src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7f4a33cd8033] src/wsrep_provider_v26.cpp:690(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x55fb386916fc] sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x55fb37d347ba] sql/wsrep_mysqld.cc:2759(start_wsrep_THD(void*))[0x55fb37d26322] pthread_create.c:0(start_thread)[0x7f4a402d7dd5] /lib64/libc.so.6(clone+0x6d)[0x7f4a3e695ead] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 10162 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file at /home/stepan/galera/git/10.4/server/mysql-test/var/2/mysqld.2/data/ ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > connection node_1; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2a; SET SESSION wsrep_sync_wait=0; SET GLOBAL wsrep_cluster_address = ''; SET SESSION wsrep_sync_wait = DEFAULT; connection node_2; INSERT INTO t1 VALUES (6); ERROR HY000: Lost connection to MySQL server during query connection node_1; SELECT COUNT(*) = 0 FROM mysql.wsrep_streaming_log; COUNT(*) = 0 1 connection node_2a; connection node_1; connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2b; - found 'core.40676' (0/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.40676 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 66579] [New LWP 66582] [New LWP 41117] [New LWP 40689] [New LWP 41119] [New LWP 41179] [New LWP 41123] [New LWP 41118] [New LWP 41183] [New LWP 41173] [New LWP 41121] [New LWP 41184] [New LWP 41174] [New LWP 40676] [New LWP 41124] [New LWP 41187] [New LWP 41178] [New LWP 40690] [New LWP 41182] [New LWP 41185] [New LWP 41120] [New LWP 65831] [New LWP 41186] [New LWP 41122] [New LWP 41188] [New LWP 41176] [New LWP 41177] [New LWP 66578] [New LWP 41180] [New LWP 41175] [New LWP 66577] [New LWP 41181] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f4a402dc9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f4a402dc9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055fb385d77ee in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055fb37e15687 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4a3e5ce207 in raise () from /lib64/libc.so.6 #5 0x00007f4a3e5cf8f8 in abort () from /lib64/libc.so.6 #6 0x00007f4a3e5c7026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f4a3e5c70d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007f4a33ce9dc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x55fb3bd1c1f0, recv_ctx=recv_ctx@entry=0x7f4a382044d0, group_uuid=..., cc_seqno=cc_seqno@entry=30232, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007f4a33cd8e78 in galera::ReplicatorSMM::process_conf_change (this=0x55fb3bd1c1f0, recv_ctx=0x7f4a382044d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007f4a33cb8ec9 in galera::GcsActionSource::dispatch (this=this@entry=0x55fb3bd4ee60, recv_ctx=recv_ctx@entry=0x7f4a382044d0, act=..., exit_loop=@0x7f4a3820429e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007f4a33cb921d in galera::GcsActionSource::process (this=0x55fb3bd4ee60, recv_ctx=0x7f4a382044d0, exit_loop=@0x7f4a3820429e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007f4a33cd8033 in galera::ReplicatorSMM::async_recv (this=0x55fb3bd1c1f0, recv_ctx=0x7f4a382044d0) at galera/src/replicator_smm.cpp:383 #13 0x000055fb386916fc in wsrep::wsrep_provider_v26::run_applier (this=0x55fb3bc91e50, applier_ctx=0x7f4a382044d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #14 0x000055fb37d347ba in wsrep_replication_process (thd=0x7f4a1c010780, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x000055fb37d26322 in start_wsrep_THD (arg=0x7f49b81cea70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #16 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f49ecff9700 (LWP 41181)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811f4bf in os_event::wait (this=0x55fb3be675b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb3811ef45 in os_event::wait_low (this=0x55fb3be675b0, reset_sig_count=122264) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb3811f2b6 in os_event_wait_low (event=0x55fb3be675b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb3820b28b in srv_resume_thread (slot=0x55fb39159780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb3820ee8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f4a381ba700 (LWP 66577)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb385dce1d in safe_cond_wait (cond=0x7f4a200168e8, mp=0x7f4a20016838, file=0x55fb38746fe0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb37d3420c in inline_mysql_cond_wait (that=0x7f4a200168e8, mutex=0x7f4a20016838, src_file=0x55fb38747300 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fb37d36ea4 in Wsrep_thd_queue::pop_front (this=0x7f4a200167e0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055fb37d358c4 in wsrep_rollback_process (rollbacker=0x7f4a20001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055fb37d26322 in start_wsrep_THD (arg=0x7f49b818dbb0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f49ef7fe700 (LWP 41175)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811edda in os_event::timed_wait (this=0x55fb3be677b0, abstime=0x7f49ef7fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb3811f0f1 in os_event::wait_time_low (this=0x55fb3be677b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb3811f291 in os_event_wait_time_low (event=0x55fb3be677b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb3820cf72 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f49ed7fa700 (LWP 41180)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811f4bf in os_event::wait (this=0x55fb3be67630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb3811ef45 in os_event::wait_low (this=0x55fb3be67630, reset_sig_count=115839) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb3811f2b6 in os_event_wait_low (event=0x55fb3be67630, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb3820b28b in srv_resume_thread (slot=0x55fb391597b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb3820ee8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4a15370700 (LWP 66578)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb385dce1d in safe_cond_wait (cond=0x7f49a810b5d8, mp=0x7f49a810b528, file=0x55fb38746fe0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb37d3420c in inline_mysql_cond_wait (that=0x7f49a810b5d8, mutex=0x7f49a810b528, src_file=0x55fb38747300 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fb37d36ea4 in Wsrep_thd_queue::pop_front (this=0x7f49a810b4d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055fb37d35c84 in wsrep_post_rollback_process (post_rollbacker=0x7f49a8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055fb37d26322 in start_wsrep_THD (arg=0x7f49b8153e40) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f49ee7fc700 (LWP 41177)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811edda in os_event::timed_wait (this=0x55fb3c00f1e0, abstime=0x7f49ee7fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb3811f0f1 in os_event::wait_time_low (this=0x55fb3c00f1e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb3811f291 in os_event_wait_time_low (event=0x55fb3c00f1e0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb3826d793 in ib_wqueue_timedwait (wq=0x55fb3c00e470, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055fb383ae33c in fts_optimize_thread (arg=0x55fb3c00e470) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f49eeffd700 (LWP 41176)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811edda in os_event::timed_wait (this=0x55fb3c071670, abstime=0x7f49eeffcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb3811f0f1 in os_event::wait_time_low (this=0x55fb3c071670, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb3811f291 in os_event_wait_time_low (event=0x55fb3c071670, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb383545da in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4a153bb700 (LWP 41188)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb385dce1d in safe_cond_wait (cond=0x55fb3931c1e0 , mp=0x55fb3931f560 , file=0x55fb386a8148 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb379da53e in inline_mysql_cond_wait (that=0x55fb3931c1e0 , mutex=0x55fb3931f560 , src_file=0x55fb386a8cb0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055fb379dbdd0 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055fb3856f8f7 in pfs_spawn_thread (arg=0x55fb3c1db690) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4a02ffd700 (LWP 41122)): #0 0x00007f4a3fea8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb381127dc in LinuxAIOHandler::collect (this=0x7f4a02ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb38112d05 in LinuxAIOHandler::poll (this=0x7f4a02ffcb00, m1=0x7f4a02ffcba8, m2=0x7f4a02ffcbb0, request=0x7f4a02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb38112e68 in os_aio_linux_handler (global_segment=4, m1=0x7f4a02ffcba8, m2=0x7f4a02ffcbb0, request=0x7f4a02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb3811798c in os_aio_handler (segment=4, m1=0x7f4a02ffcba8, m2=0x7f4a02ffcbb0, request=0x7f4a02ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb3836a93d in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055fb38210da1 in io_handler_thread (arg=0x55fb39397640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f49e5ffb700 (LWP 41186)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb385dce1d in safe_cond_wait (cond=0x7f49e5ffac70, mp=0x7f49e5ffacb0, file=0x55fb388ab0c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055fb3803660b in inline_mysql_cond_wait (that=0x7f49e5ffac70, mutex=0x7f49e5ffacb0, src_file=0x55fb388aca58 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055fb3803b247 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055fb3856f8f7 in pfs_spawn_thread (arg=0x55fb3c020dd0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f4a1528f700 (LWP 65831)): #0 0x00007f4a3e68b20d in poll () from /lib64/libc.so.6 #1 0x000055fb38648a92 in vio_io_wait (vio=0x55fb3c1ee140, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:976 #2 0x000055fb386470b2 in vio_socket_io_wait (vio=0x55fb3c1ee140, event=VIO_IO_EVENT_READ) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:118 #3 0x000055fb38647298 in vio_read (vio=0x55fb3c1ee140, buf=0x7f49b80cd3d0 "\a", size=4) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:195 #4 0x000055fb37f7e244 in my_real_read (net=0x7f49b8000df8, complen=0x7f4a1528e810, header=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:962 #5 0x000055fb37f7eacb in my_net_read_packet_reallen (net=0x7f49b8000df8, read_from_server=1 '\001', reallen=0x7f4a1528e878) at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1243 #6 0x000055fb37f7ea7e in my_net_read_packet (net=0x7f49b8000df8, read_from_server=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1227 #7 0x000055fb37ada196 in do_command (thd=0x7f49b8000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1229 #8 0x000055fb37c4b0b9 in do_handle_one_connection (connect=0x55fb3c0b9160) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #9 0x000055fb37c4adfd in handle_one_connection (arg=0x55fb3c0b9160) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #10 0x000055fb3856f8f7 in pfs_spawn_thread (arg=0x55fb3bcd8b10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #11 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f4a03fff700 (LWP 41120)): #0 0x00007f4a3fea8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb381127dc in LinuxAIOHandler::collect (this=0x7f4a03ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb38112d05 in LinuxAIOHandler::poll (this=0x7f4a03ffeb00, m1=0x7f4a03ffeba8, m2=0x7f4a03ffebb0, request=0x7f4a03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb38112e68 in os_aio_linux_handler (global_segment=2, m1=0x7f4a03ffeba8, m2=0x7f4a03ffebb0, request=0x7f4a03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb3811798c in os_aio_handler (segment=2, m1=0x7f4a03ffeba8, m2=0x7f4a03ffebb0, request=0x7f4a03ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb3836a93d in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055fb38210da1 in io_handler_thread (arg=0x55fb39397630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f49e67fc700 (LWP 41185)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811f4bf in os_event::wait (this=0x55fb3be67930) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb3811ef45 in os_event::wait_low (this=0x55fb3be67930, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb3811f2b6 in os_event_wait_low (event=0x55fb3be67930, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb382d3fbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f49e7fff700 (LWP 41182)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811f4bf in os_event::wait (this=0x55fb3be676b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb3811ef45 in os_event::wait_low (this=0x55fb3be676b0, reset_sig_count=115212) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb3811f2b6 in os_event_wait_low (event=0x55fb3be676b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb3820b28b in srv_resume_thread (slot=0x55fb391597f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb3820ee8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f4a2b97d700 (LWP 40690)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4a33cb5fd7 in gu::Lock::wait (cond=..., this=0x7f4a2b97cbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4a33cb6069 in galera::ServiceThd::thd_func (arg=0x55fb3bd1c908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4a017fa700 (LWP 41178)): #0 0x00007f4a402dee3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055fb3811f7c1 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055fb3820e7f3 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055fb3820e923 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4a15406700 (LWP 41187)): #0 0x00007f4a3e5cf40a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055fb379a8f94 in my_sigwait (set=0x7f4a15405cf0, sig=0x7f4a15405cb4, code=0x7f4a15405cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055fb379af5d4 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055fb3856f8f7 in pfs_spawn_thread (arg=0x55fb3c0e66d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f4a01ffb700 (LWP 41124)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811edda in os_event::timed_wait (this=0x55fb3be678b0, abstime=0x7f4a01ffaaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb3811f0f1 in os_event::wait_time_low (this=0x55fb3be678b0, time_in_usec=795000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb3811f291 in os_event_wait_time_low (event=0x55fb3be678b0, time_in_usec=795000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb382f51b5 in pc_sleep_if_needed (next_loop_time=1549846103113, sig_count=2, cur_time=1549846102318) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055fb382f60ee in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4a406e8880 (LWP 40676)): #0 0x00007f4a3e68b20d in poll () from /lib64/libc.so.6 #1 0x000055fb379b5ada in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #2 0x000055fb379b4d75 in mysqld_main (argc=162, argv=0x55fb3bb9dd20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #3 0x000055fb379a8f15 in main (argc=23, argv=0x7ffe970b1ae8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 13 (Thread 0x7f49effff700 (LWP 41174)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811edda in os_event::timed_wait (this=0x55fb3be67730, abstime=0x7f49efffeba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb3811f0f1 in os_event::wait_time_low (this=0x55fb3be67730, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb3811f291 in os_event_wait_time_low (event=0x55fb3be67730, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb3820d494 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f49e6ffd700 (LWP 41184)): #0 0x00007f4a402dee3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055fb3811f7c1 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055fb382c5ed4 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4a037fe700 (LWP 41121)): #0 0x00007f4a3fea8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb381127dc in LinuxAIOHandler::collect (this=0x7f4a037fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb38112d05 in LinuxAIOHandler::poll (this=0x7f4a037fdb00, m1=0x7f4a037fdba8, m2=0x7f4a037fdbb0, request=0x7f4a037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb38112e68 in os_aio_linux_handler (global_segment=3, m1=0x7f4a037fdba8, m2=0x7f4a037fdbb0, request=0x7f4a037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb3811798c in os_aio_handler (segment=3, m1=0x7f4a037fdba8, m2=0x7f4a037fdbb0, request=0x7f4a037fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb3836a93d in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055fb38210da1 in io_handler_thread (arg=0x55fb39397638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4a00ff9700 (LWP 41173)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811edda in os_event::timed_wait (this=0x55fb3c03f5b0, abstime=0x7f4a00ff8d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055fb3811f0f1 in os_event::wait_time_low (this=0x55fb3c03f5b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055fb3811f291 in os_event_wait_time_low (event=0x55fb3c03f5b0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055fb380ebc4e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f49e77fe700 (LWP 41183)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811f4bf in os_event::wait (this=0x55fb3be67830) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb3811ef45 in os_event::wait_low (this=0x55fb3be67830, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb3811f2b6 in os_event_wait_low (event=0x55fb3be67830, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb382eb349 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f4a15206700 (LWP 41118)): #0 0x00007f4a3fea8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb381127dc in LinuxAIOHandler::collect (this=0x7f4a15205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb38112d05 in LinuxAIOHandler::poll (this=0x7f4a15205b00, m1=0x7f4a15205ba8, m2=0x7f4a15205bb0, request=0x7f4a15205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb38112e68 in os_aio_linux_handler (global_segment=0, m1=0x7f4a15205ba8, m2=0x7f4a15205bb0, request=0x7f4a15205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb3811798c in os_aio_handler (segment=0, m1=0x7f4a15205ba8, m2=0x7f4a15205bb0, request=0x7f4a15205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb3836a93d in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055fb38210da1 in io_handler_thread (arg=0x55fb39397620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f4a027fc700 (LWP 41123)): #0 0x00007f4a3fea8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb381127dc in LinuxAIOHandler::collect (this=0x7f4a027fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb38112d05 in LinuxAIOHandler::poll (this=0x7f4a027fbb00, m1=0x7f4a027fbba8, m2=0x7f4a027fbbb0, request=0x7f4a027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb38112e68 in os_aio_linux_handler (global_segment=5, m1=0x7f4a027fbba8, m2=0x7f4a027fbbb0, request=0x7f4a027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb3811798c in os_aio_handler (segment=5, m1=0x7f4a027fbba8, m2=0x7f4a027fbbb0, request=0x7f4a027fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb3836a93d in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055fb38210da1 in io_handler_thread (arg=0x55fb39397648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f49edffb700 (LWP 41179)): #0 0x00007f4a402db965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb3811f4bf in os_event::wait (this=0x55fb3be67530) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055fb3811ef45 in os_event::wait_low (this=0x55fb3be67530, reset_sig_count=29725) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055fb3811f2b6 in os_event_wait_low (event=0x55fb3be67530, reset_sig_count=29725) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055fb3820b28b in srv_resume_thread (slot=0x55fb39159748 , sig_count=29725, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055fb3820f29c in srv_purge_coordinator_suspend (slot=0x55fb39159748 , rseg_history_len=8) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055fb3820f532 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4a14a05700 (LWP 41119)): #0 0x00007f4a3fea8644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055fb381127dc in LinuxAIOHandler::collect (this=0x7f4a14a04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055fb38112d05 in LinuxAIOHandler::poll (this=0x7f4a14a04b00, m1=0x7f4a14a04ba8, m2=0x7f4a14a04bb0, request=0x7f4a14a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055fb38112e68 in os_aio_linux_handler (global_segment=1, m1=0x7f4a14a04ba8, m2=0x7f4a14a04bb0, request=0x7f4a14a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055fb3811798c in os_aio_handler (segment=1, m1=0x7f4a14a04ba8, m2=0x7f4a14a04bb0, request=0x7f4a14a04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055fb3836a93d in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055fb38210da1 in io_handler_thread (arg=0x55fb39397628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4a38d30700 (LWP 40689)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb385dd11f in safe_cond_timedwait (cond=0x55fb39bbb640 , mp=0x55fb39bbb580 , abstime=0x7f4a38d2fd70, file=0x55fb38a64e20 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fb385de0d9 in inline_mysql_cond_timedwait (that=0x55fb39bbb640 , mutex=0x55fb39bbb580 , abstime=0x7f4a38d2fd70, src_file=0x55fb38a64e68 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fb385dec3c in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055fb3856f8f7 in pfs_spawn_thread (arg=0x55fb3bce57d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4a16858700 (LWP 41117)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055fb385dd11f in safe_cond_timedwait (cond=0x55fb39ba8ea0 , mp=0x55fb39ba8de0 , abstime=0x7f4a16857cb0, file=0x55fb38a20bf0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055fb38446f63 in inline_mysql_cond_timedwait (that=0x55fb39ba8ea0 , mutex=0x55fb39ba8de0 , abstime=0x7f4a16857cb0, src_file=0x55fb38a20c38 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055fb38447487 in my_service_thread_sleep (control=0x55fb3915db00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055fb3843b920 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055fb3856f8f7 in pfs_spawn_thread (arg=0x55fb3be1f680) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f4a152da700 (LWP 66582)): #0 0x00007f4a402dbd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4a33c92bdb in gu::Lock::wait (this=this@entry=0x7f4a152d76f0, cond=..., date=...) at galerautils/src/gu_lock.hpp:62 #2 0x00007f4a33cdd976 in galera::Monitor::wait (this=this@entry=0x55fb3bd1d190, gtid=..., wait_until=...) at galera/src/monitor.hpp:355 #3 0x00007f4a33ccf57f in galera::ReplicatorSMM::sync_wait (this=0x55fb3bd1c1f0, upto=, tout=, gtid=0x7f4a152d7b30) at galera/src/replicator_smm.cpp:1661 #4 0x00007f4a33cee643 in galera_sync_wait (wsrep=, upto=, tout=, gtid=) at galera/src/wsrep_provider.cpp:998 #5 0x000055fb386923b3 in wsrep::wsrep_provider_v26::causal_read (this=0x55fb3bc91e50, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:845 #6 0x000055fb3867c907 in wsrep::server_state::causal_read (this=0x55fb3bce6740, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:648 #7 0x000055fb38677191 in wsrep::client_state::sync_wait (this=0x7f4a24006980, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/client_state.cpp:417 #8 0x000055fb37d1ebca in wsrep_sync_wait (thd=0x7f4a24000b20, mask=1) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:1022 #9 0x000055fb37ae1a44 in mysql_execute_command (thd=0x7f4a24000b20) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:3779 #10 0x000055fb37af0985 in mysql_parse (thd=0x7f4a24000b20, rawbuf=0x7f4a240192b8 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7f4a152d90d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #11 0x000055fb37aeffb8 in wsrep_mysql_parse (thd=0x7f4a24000b20, rawbuf=0x7f4a240192b8 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7f4a152d90d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #12 0x000055fb37adbe75 in dispatch_command (command=COM_QUERY, thd=0x7f4a24000b20, packet=0x7f4a2400a3a1 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", packet_length=102, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #13 0x000055fb37ada85d in do_command (thd=0x7f4a24000b20) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #14 0x000055fb37c4b0b9 in do_handle_one_connection (connect=0x55fb3c0b9160) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #15 0x000055fb37c4adfd in handle_one_connection (arg=0x55fb3c0b9160) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #16 0x000055fb3856f8f7 in pfs_spawn_thread (arg=0x55fb3bcd8b10) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #17 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4a38205700 (LWP 66579)): #0 0x00007f4a402dc9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055fb385d77ee in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055fb37e15687 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f4a3e5ce207 in raise () from /lib64/libc.so.6 #5 0x00007f4a3e5cf8f8 in abort () from /lib64/libc.so.6 #6 0x00007f4a3e5c7026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f4a3e5c70d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007f4a33ce9dc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x55fb3bd1c1f0, recv_ctx=recv_ctx@entry=0x7f4a382044d0, group_uuid=..., cc_seqno=cc_seqno@entry=30232, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007f4a33cd8e78 in galera::ReplicatorSMM::process_conf_change (this=0x55fb3bd1c1f0, recv_ctx=0x7f4a382044d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007f4a33cb8ec9 in galera::GcsActionSource::dispatch (this=this@entry=0x55fb3bd4ee60, recv_ctx=recv_ctx@entry=0x7f4a382044d0, act=..., exit_loop=@0x7f4a3820429e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007f4a33cb921d in galera::GcsActionSource::process (this=0x55fb3bd4ee60, recv_ctx=0x7f4a382044d0, exit_loop=@0x7f4a3820429e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007f4a33cd8033 in galera::ReplicatorSMM::async_recv (this=0x55fb3bd1c1f0, recv_ctx=0x7f4a382044d0) at galera/src/replicator_smm.cpp:383 #13 0x000055fb386916fc in wsrep::wsrep_provider_v26::run_applier (this=0x55fb3bc91e50, applier_ctx=0x7f4a382044d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #14 0x000055fb37d347ba in wsrep_replication_process (thd=0x7f4a1c010780, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x000055fb37d26322 in start_wsrep_THD (arg=0x7f49b81cea70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #16 0x00007f4a402d7dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f4a3e695ead in clone () from /lib64/libc.so.6 worker[3] Trying to dump core for [mysqltest - pid: 72533, winpid: 72533] worker[3] Trying to dump core for [mysqld.1 - pid: 72068, winpid: 72068] worker[3] Trying to dump core for [mysqld.2 - pid: 72143, winpid: 72143] Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.40676 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/2/log/galera_sr.galera_sr_cc_master-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera_sr_cc_master galera_sr.GCF-623 galera_sr.galera_sr_blob galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera_sr_bf_abort galera_sr.galera-features#56 galera_sr.GCF-845 2019-02-11 2:48:05 10151 [Warning] WSREP: Failed to replicate rollback fragment for 20938: 3 2019-02-11 2:48:11 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-11 2:48:11 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 30232 last received: 30171 2019-02-11 2:48:11 10162 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_cc_slave 'innodb' w2 [ pass ] 18065 galera_sr.galera_sr_concurrent 'innodb' w2 [ pass ] 2580 galera_sr.galera_sr_conflict 'innodb' w2 [ pass ] 2355 galera_sr.galera_sr_conflict_on_commit 'innodb' w2 [ pass ] 2347 galera_sr.galera_sr_conflict_on_commit2 'innodb' w2 [ pass ] 3339 galera_sr.galera_sr_load_data_splitting 'innodb' w3 [ fail ] Test ended at 2019-02-11 02:50:23 CURRENT_TEST: galera_sr.galera_sr_load_data_splitting --- /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_load_data_splitting.result 2019-01-23 13:58:38.972427700 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_load_data_splitting.reject 2019-02-11 02:50:22.164745800 +0200 @@ -1,9 +1,19 @@ +connection node_2; +connection node_1; SET SESSION wsrep_trx_fragment_size = 512; SET GLOBAL wsrep_load_data_splitting = TRUE; +Warnings: +Warning 1287 '@@wsrep_load_data_splitting' is deprecated and will be removed in a future release CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; +connection node_2; +connection node_1; +connection node_2; SELECT COUNT(*) = 95000 FROM t1; COUNT(*) = 95000 1 wsrep_last_committed_diff -1 +0 +connection node_1; +Warnings: +Warning 1287 '@@wsrep_load_data_splitting' is deprecated and will be removed in a future release DROP TABLE t1; mysqltest: Result length mismatch - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.galera_sr_load_data_splitting-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/' galera_sr.GCF-1043B 'innodb' w2 [ pass ] 138473 galera_sr.GCF-1051 'innodb' w2 [ pass ] 2316 galera_sr.GCF-561 'innodb' w2 [ pass ] 2662 galera_sr.GCF-571 'innodb' w2 [ pass ] 2399 galera_sr.GCF-572 'innodb' w2 [ pass ] 2289 galera_sr.GCF-580 'innodb' w2 [ pass ] 2261 galera_sr.GCF-585 'innodb' w2 [ pass ] 6785 galera_sr.GCF-1043A 'innodb' w1 [ pass ] 223617 galera_sr.GCF-597 'innodb' w2 [ pass ] 2346 galera_sr.GCF-620 'innodb' w2 [ pass ] 2139 galera_sr.mysql-wsrep-features#8 'innodb' w4 [ pass ] 257718 galera_sr.GCF-1008 'innodb' w4 [ pass ] 4263 galera_sr.GCF-1060 'innodb' w3 [ fail ] timeout after 900 seconds Test ended at 2019-02-11 03:06:24 Test case timeout after 900 seconds == /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/GCF-1060.log == connection node_2; connection node_1; connect node_1a, 127.0.0.1, root, , test, $NODE_MYPORT_1; Running a concurrent test with the following queries: TRUNCATE TABLE t1 INSERT INTO t1 VALUE (1,'x'),(2,'x'),(3,'x') INSERT INTO t1 VALUE (4, 'z'); connection node_1; DROP TABLE IF EXISTS t1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_1a; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; connection node_2; SET SESSION wsrep_sync_wait = 0; SET SESSION wsrep_trx_fragment_size = 1;;; == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) == /home/stepan/galera/git/10.4/server/mysql-test/var/3/tmp/analyze-timeout-mysqld.2.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/3/mysqld.' (111) - found 'core.72069' (1/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1060-innodb/mysqld.1/data/core.72069 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 72069] [New LWP 72074] [New LWP 72070] [New LWP 72073] [New LWP 72111] [New LWP 72075] [New LWP 72085] [New LWP 72088] [New LWP 72089] [New LWP 72094] [New LWP 72090] [New LWP 72107] [New LWP 72092] [New LWP 72108] [New LWP 72119] [New LWP 72118] [New LWP 72130] [New LWP 72129] [New LWP 72131] [New LWP 72611] [New LWP 72612] [New LWP 72117] [New LWP 72110] [New LWP 72140] [New LWP 72139] [New LWP 72071] [New LWP 72136] [New LWP 72076] [New LWP 72128] [New LWP 72091] [New LWP 72116] [New LWP 72093] [New LWP 72120] [New LWP 72109] [New LWP 72072] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.1 --def'. Program terminated with signal 6, Aborted. #0 0x00007f84296359d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f84296359d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055d1854be7ee in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055d184cfc687 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f84279e420d in poll () from /lib64/libc.so.6 #5 0x000055d18489cada in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055d18489bd75 in mysqld_main (argc=161, argv=0x55d1878b1d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055d18488ff15 in main (argc=23, argv=0x7ffdad152d48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 35 (Thread 0x7f8419b7b700 (LWP 72072)): #0 0x00007f84279ef483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f84214e06f8 in asio::detail::epoll_reactor::run (this=0x55d187a78fc0, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f84214e4df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x55d187a78ee0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f84214e50a2 in asio::detail::task_io_service::run (this=0x55d187a78ee0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f84214fcafd in asio::io_service::run (this=0x55d187a77fe0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f84214fd251 in gcomm::AsioProtonet::event_loop (this=0x55d187a77f40, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f842151b2ed in GCommConn::run (this=0x55d187a786c0) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f842151e399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f83f55f5700 (LWP 72109)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d185005dda in os_event::timed_wait (this=0x55d187b808b0, abstime=0x7f83f55f4d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d1850060f1 in os_event::wait_time_low (this=0x55d187b808b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d185006291 in os_event_wait_time_low (event=0x55d187b808b0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d1850f3f72 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f83e5ffb700 (LWP 72120)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1850064bf in os_event::wait (this=0x55d187b80730) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d185005f45 in os_event::wait_low (this=0x55d187b80730, reset_sig_count=355) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d1850062b6 in os_event_wait_low (event=0x55d187b80730, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d1850f228b in srv_resume_thread (slot=0x55d1860407b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d1850f5e8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f83f7fff700 (LWP 72093)): #0 0x00007f8429201644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d184ff97dc in LinuxAIOHandler::collect (this=0x7f83f7ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d184ff9d05 in LinuxAIOHandler::poll (this=0x7f83f7ffeb00, m1=0x7f83f7ffeba8, m2=0x7f83f7ffebb0, request=0x7f83f7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d184ff9e68 in os_aio_linux_handler (global_segment=5, m1=0x7f83f7ffeba8, m2=0x7f83f7ffebb0, request=0x7f83f7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d184ffe98c in os_aio_handler (segment=5, m1=0x7f83f7ffeba8, m2=0x7f83f7ffebb0, request=0x7f83f7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d18525193d in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d1850f7da1 in io_handler_thread (arg=0x55d18627e648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f83e7fff700 (LWP 72116)): #0 0x00007f8429637e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055d1850067c1 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055d1850f57f3 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x000055d1850f5923 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f84011aa700 (LWP 72091)): #0 0x00007f8429201644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d184ff97dc in LinuxAIOHandler::collect (this=0x7f84011a9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d184ff9d05 in LinuxAIOHandler::poll (this=0x7f84011a9b00, m1=0x7f84011a9ba8, m2=0x7f84011a9bb0, request=0x7f84011a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d184ff9e68 in os_aio_linux_handler (global_segment=3, m1=0x7f84011a9ba8, m2=0x7f84011a9bb0, request=0x7f84011a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d184ffe98c in os_aio_handler (segment=3, m1=0x7f84011a9ba8, m2=0x7f84011a9bb0, request=0x7f84011a9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d18525193d in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d1850f7da1 in io_handler_thread (arg=0x55d18627e638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f83e57fa700 (LWP 72128)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1850064bf in os_event::wait (this=0x55d187b80930) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d185005f45 in os_event::wait_low (this=0x55d187b80930, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d1850062b6 in os_event_wait_low (event=0x55d187b80930, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d1851d2349 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f8420446700 (LWP 72076)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f8421433132 in fifo_lock_get (q=q@entry=0x55d187a5ebd0) at galerautils/src/gu_fifo.c:262 #2 0x00007f84214337a1 in gu_fifo_get_head (q=0x55d187a5ebd0, err=0x7f842044506c) at galerautils/src/gu_fifo.c:330 #3 0x00007f84215150cc in gcs_recv (conn=0x55d187a38550, action=0x7f84204450b0) at gcs/src/gcs.cpp:2024 #4 0x00007f842154216c in galera::GcsActionSource::process (this=0x55d187a37f30, recv_ctx=0x7f84204454d0, exit_loop=@0x7f842044529e: false) at galera/src/gcs_action_source.cpp:161 #5 0x00007f8421561033 in galera::ReplicatorSMM::async_recv (this=0x55d187a30280, recv_ctx=0x7f84204454d0) at galera/src/replicator_smm.cpp:383 #6 0x000055d1855786fc in wsrep::wsrep_provider_v26::run_applier (this=0x55d1879b7360, applier_ctx=0x7f84204454d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #7 0x000055d184c1b7ba in wsrep_replication_process (thd=0x7f8410000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #8 0x000055d184c0d322 in start_wsrep_THD (arg=0x55d187a92550) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #9 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f84181d4700 (LWP 72136)): #0 0x00007f842792840a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055d18488ff94 in my_sigwait (set=0x7f84181d3cf0, sig=0x7f84181d3cb4, code=0x7f84181d3cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055d1848965d4 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d187b43e20) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f841bfff700 (LWP 72071)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f842153efd7 in gu::Lock::wait (cond=..., this=0x7f841bffebf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f842153f069 in galera::ServiceThd::thd_func (arg=0x55d187a30998) at galera/src/galera_service_thd.cpp:30 #3 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f8418189700 (LWP 72139)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c3e1d in safe_cond_wait (cond=0x55d1862031e0 , mp=0x55d186206560 , file=0x55d18558f148 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d1848c153e in inline_mysql_cond_wait (that=0x55d1862031e0 , mutex=0x55d186206560 , src_file=0x55d18558fcb0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055d1848c2dd0 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d187b44310) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f841813e700 (LWP 72140)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c411f in safe_cond_timedwait (cond=0x55d1862031a0 , mp=0x55d186205ba0 , abstime=0x7f841813dc60, file=0x55d18557f948 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d184890895 in inline_mysql_cond_timedwait (that=0x55d1862031a0 , mutex=0x55d186205ba0 , abstime=0x7f841813dc60, src_file=0x55d1855819b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d18489566b in cache_thread (thd=0x7f83b0000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055d184895919 in one_thread_per_connection_end (thd=0x7f83b0000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055d184b32143 in do_handle_one_connection (connect=0x55d187eef8c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055d184b31dfd in handle_one_connection (arg=0x55d187eef8c0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d187f11590) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f83f6df8700 (LWP 72110)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d185005dda in os_event::timed_wait (this=0x55d187d8ae20, abstime=0x7f83f6df7d60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d1850060f1 in os_event::wait_time_low (this=0x55d187d8ae20, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d185006291 in os_event_wait_time_low (event=0x55d187d8ae20, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d18523b5da in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f83e77fe700 (LWP 72117)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1850064bf in os_event::wait (this=0x55d187b80630) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d185005f45 in os_event::wait_low (this=0x55d187b80630, reset_sig_count=91) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d1850062b6 in os_event_wait_low (event=0x55d187b80630, reset_sig_count=91) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d1850f228b in srv_resume_thread (slot=0x55d186040748 , sig_count=91, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d1850f629c in srv_purge_coordinator_suspend (slot=0x55d186040748 , rseg_history_len=7) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x000055d1850f6532 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f84180a8700 (LWP 72612)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c411f in safe_cond_timedwait (cond=0x55d1862031a0 , mp=0x55d186205ba0 , abstime=0x7f84180a7c60, file=0x55d18557f948 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d184890895 in inline_mysql_cond_timedwait (that=0x55d1862031a0 , mutex=0x55d186205ba0 , abstime=0x7f84180a7c60, src_file=0x55d1855819b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d18489566b in cache_thread (thd=0x7f83a8000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055d184895919 in one_thread_per_connection_end (thd=0x7f83a8000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055d184b32143 in do_handle_one_connection (connect=0x55d187e17cf0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055d184b31dfd in handle_one_connection (arg=0x55d187e17cf0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d187e77090) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f84180f3700 (LWP 72611)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c411f in safe_cond_timedwait (cond=0x55d1862031a0 , mp=0x55d186205ba0 , abstime=0x7f84180f2c60, file=0x55d18557f948 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d184890895 in inline_mysql_cond_timedwait (that=0x55d1862031a0 , mutex=0x55d186205ba0 , abstime=0x7f84180f2c60, src_file=0x55d1855819b8 "/home/stepan/galera/git/10.4/server/sql/mysqld.cc", src_line=2675) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d18489566b in cache_thread (thd=0x7f83ac000b00) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2674 #4 0x000055d184895919 in one_thread_per_connection_end (thd=0x7f83ac000b00, put_in_cache=true) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:2761 #5 0x000055d184b32143 in do_handle_one_connection (connect=0x55d187b44a80) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1414 #6 0x000055d184b31dfd in handle_one_connection (arg=0x55d187b44a80) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #7 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d1879eca70) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #8 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f83cf7fe700 (LWP 72131)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c3e1d in safe_cond_wait (cond=0x7f83cf7fdc70, mp=0x7f83cf7fdcb0, file=0x55d1857920c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d184f1d60b in inline_mysql_cond_wait (that=0x7f83cf7fdc70, mutex=0x7f83cf7fdcb0, src_file=0x55d185793a58 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d184f22247 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d187dc1970) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f83e4ff9700 (LWP 72129)): #0 0x00007f8429637e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055d1850067c1 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055d1851aced4 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f83cffff700 (LWP 72130)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1850064bf in os_event::wait (this=0x55d187b80a30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d185005f45 in os_event::wait_low (this=0x55d187b80a30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d1850062b6 in os_event_wait_low (event=0x55d187b80a30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d1851bafbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f83e6ffd700 (LWP 72118)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1850064bf in os_event::wait (this=0x55d187b806b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d185005f45 in os_event::wait_low (this=0x55d187b806b0, reset_sig_count=380) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d1850062b6 in os_event_wait_low (event=0x55d187b806b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d1850f228b in srv_resume_thread (slot=0x55d186040780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d1850f5e8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f83e67fc700 (LWP 72119)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1850064bf in os_event::wait (this=0x55d187b807b0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055d185005f45 in os_event::wait_low (this=0x55d187b807b0, reset_sig_count=312) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055d1850062b6 in os_event_wait_low (event=0x55d187b807b0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055d1850f228b in srv_resume_thread (slot=0x55d1860407f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x000055d1850f5e8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f83f5df6700 (LWP 72108)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d185005dda in os_event::timed_wait (this=0x55d187b80830, abstime=0x7f83f5df5ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d1850060f1 in os_event::wait_time_low (this=0x55d187b80830, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d185006291 in os_event_wait_time_low (event=0x55d187b80830, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d1850f4494 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f84009a9700 (LWP 72092)): #0 0x00007f8429201644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d184ff97dc in LinuxAIOHandler::collect (this=0x7f84009a8b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d184ff9d05 in LinuxAIOHandler::poll (this=0x7f84009a8b00, m1=0x7f84009a8ba8, m2=0x7f84009a8bb0, request=0x7f84009a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d184ff9e68 in os_aio_linux_handler (global_segment=4, m1=0x7f84009a8ba8, m2=0x7f84009a8bb0, request=0x7f84009a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d184ffe98c in os_aio_handler (segment=4, m1=0x7f84009a8ba8, m2=0x7f84009a8bb0, request=0x7f84009a8be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d18525193d in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d1850f7da1 in io_handler_thread (arg=0x55d18627e640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f83f65f7700 (LWP 72107)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d185005dda in os_event::timed_wait (this=0x55d187d58d60, abstime=0x7f83f65f6d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d1850060f1 in os_event::wait_time_low (this=0x55d187d58d60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d185006291 in os_event_wait_time_low (event=0x55d187d58d60, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d184fd2c4e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f84019ab700 (LWP 72090)): #0 0x00007f8429201644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d184ff97dc in LinuxAIOHandler::collect (this=0x7f84019aab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d184ff9d05 in LinuxAIOHandler::poll (this=0x7f84019aab00, m1=0x7f84019aaba8, m2=0x7f84019aabb0, request=0x7f84019aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d184ff9e68 in os_aio_linux_handler (global_segment=2, m1=0x7f84019aaba8, m2=0x7f84019aabb0, request=0x7f84019aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d184ffe98c in os_aio_handler (segment=2, m1=0x7f84019aaba8, m2=0x7f84019aabb0, request=0x7f84019aabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d18525193d in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d1850f7da1 in io_handler_thread (arg=0x55d18627e630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f83f77fe700 (LWP 72094)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d185005dda in os_event::timed_wait (this=0x55d187b809b0, abstime=0x7f83f77fdaf0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d1850060f1 in os_event::wait_time_low (this=0x55d187b809b0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d185006291 in os_event_wait_time_low (event=0x55d187b809b0, time_in_usec=1000000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d1851dc1b5 in pc_sleep_if_needed (next_loop_time=1549847131275, sig_count=2, cur_time=1549847130275) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x000055d1851dd0ee in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f84021ac700 (LWP 72089)): #0 0x00007f8429201644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d184ff97dc in LinuxAIOHandler::collect (this=0x7f84021abb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d184ff9d05 in LinuxAIOHandler::poll (this=0x7f84021abb00, m1=0x7f84021abba8, m2=0x7f84021abbb0, request=0x7f84021abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d184ff9e68 in os_aio_linux_handler (global_segment=1, m1=0x7f84021abba8, m2=0x7f84021abbb0, request=0x7f84021abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d184ffe98c in os_aio_handler (segment=1, m1=0x7f84021abba8, m2=0x7f84021abbb0, request=0x7f84021abbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d18525193d in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d1850f7da1 in io_handler_thread (arg=0x55d18627e628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f84029ad700 (LWP 72088)): #0 0x00007f8429201644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055d184ff97dc in LinuxAIOHandler::collect (this=0x7f84029acb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055d184ff9d05 in LinuxAIOHandler::poll (this=0x7f84029acb00, m1=0x7f84029acba8, m2=0x7f84029acbb0, request=0x7f84029acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055d184ff9e68 in os_aio_linux_handler (global_segment=0, m1=0x7f84029acba8, m2=0x7f84029acbb0, request=0x7f84029acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055d184ffe98c in os_aio_handler (segment=0, m1=0x7f84029acba8, m2=0x7f84029acbb0, request=0x7f84029acbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x000055d18525193d in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x000055d1850f7da1 in io_handler_thread (arg=0x55d18627e620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f8403fff700 (LWP 72085)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c411f in safe_cond_timedwait (cond=0x55d186a8fea0 , mp=0x55d186a8fde0 , abstime=0x7f8403ffecb0, file=0x55d185907bf0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d18532df63 in inline_mysql_cond_timedwait (that=0x55d186a8fea0 , mutex=0x55d186a8fde0 , abstime=0x7f8403ffecb0, src_file=0x55d185907c38 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d18532e487 in my_service_thread_sleep (control=0x55d186044b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055d185322920 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d187b386f0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f8420491700 (LWP 72075)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c3e1d in safe_cond_wait (cond=0x7f8408017b68, mp=0x7f8408017ab8, file=0x55d18562dfe0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d184c1b20c in inline_mysql_cond_wait (that=0x7f8408017b68, mutex=0x7f8408017ab8, src_file=0x55d18562e300 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d184c1dea4 in Wsrep_thd_queue::pop_front (this=0x7f8408017a60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055d184c1cc84 in wsrep_post_rollback_process (post_rollbacker=0x7f8408001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055d184c0d322 in start_wsrep_THD (arg=0x55d187a88340) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f83f4df4700 (LWP 72111)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d185005dda in os_event::timed_wait (this=0x55d187d20ab0, abstime=0x7f83f4df3b60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055d1850060f1 in os_event::wait_time_low (this=0x55d187d20ab0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055d185006291 in os_event_wait_time_low (event=0x55d187d20ab0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055d185154793 in ib_wqueue_timedwait (wq=0x55d187d20980, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055d18529533c in fts_optimize_thread (arg=0x55d187d20980) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f841937a700 (LWP 72073)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f8421519867 in gu::Lock::wait (cond=..., this=0x7f8419379850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f842151c3ab in RecvBuf::front (this=this@entry=0x55d187a788a8, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f8421519f46 in gcomm_recv (backend=, msg=0x55d187a38830, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f842150deec in core_msg_recv (backend=backend@entry=0x55d187a38978, recv_msg=recv_msg@entry=0x55d187a38830, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f8421511164 in gcs_core_recv (conn=0x55d187a387d0, recv_act=recv_act@entry=0x7f8419379d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f84215183d8 in gcs_recv_thread (arg=0x55d187a38550) at gcs/src/gcs.cpp:1398 #7 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f8422089700 (LWP 72070)): #0 0x00007f8429634d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c411f in safe_cond_timedwait (cond=0x55d186aa2640 , mp=0x55d186aa2580 , abstime=0x7f8422088d70, file=0x55d18594be20 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055d1854c50d9 in inline_mysql_cond_timedwait (that=0x55d186aa2640 , mutex=0x55d186aa2580 , abstime=0x7f8422088d70, src_file=0x55d18594be68 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055d1854c5c3c in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055d1854568f7 in pfs_spawn_thread (arg=0x55d1879f9730) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f84204dc700 (LWP 72074)): #0 0x00007f8429634965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055d1854c3e1d in safe_cond_wait (cond=0x7f8404015608, mp=0x7f8404015558, file=0x55d18562dfe0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055d184c1b20c in inline_mysql_cond_wait (that=0x7f8404015608, mutex=0x7f8404015558, src_file=0x55d18562e300 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055d184c1dea4 in Wsrep_thd_queue::pop_front (this=0x7f8404015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055d184c1c8c4 in wsrep_rollback_process (rollbacker=0x7f8404000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055d184c0d322 in start_wsrep_THD (arg=0x55d187a91d70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f8429630dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f84279eeead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f8429a41880 (LWP 72069)): #0 0x00007f84296359d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055d1854be7ee in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055d184cfc687 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f84279e420d in poll () from /lib64/libc.so.6 #5 0x000055d18489cada in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000055d18489bd75 in mysqld_main (argc=161, argv=0x55d1878b1d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000055d18488ff15 in main (argc=23, argv=0x7ffdad152d48) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1060-innodb/mysqld.1/data/core.72069 - found 'core.72145' (2/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1060-innodb/mysqld.2/data/core.72145 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 72145] [New LWP 72197] [New LWP 72148] [New LWP 72194] [New LWP 72460] [New LWP 72195] [New LWP 72461] [New LWP 72466] [New LWP 72510] [New LWP 72147] [New LWP 72462] [New LWP 72518] [New LWP 72512] [New LWP 72464] [New LWP 72196] [New LWP 72519] [New LWP 72465] [New LWP 72467] [New LWP 72522] [New LWP 72515] [New LWP 72513] [New LWP 72524] [New LWP 72516] [New LWP 72525] [New LWP 72514] [New LWP 72520] [New LWP 72463] [New LWP 72528] [New LWP 72517] [New LWP 72523] [New LWP 72521] [New LWP 72153] [New LWP 72511] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f427f6549d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f427f6549d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000562869f8e7ee in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005628697cc687 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f427da0320d in poll () from /lib64/libc.so.6 #5 0x000056286936cada in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056286936bd75 in mysqld_main (argc=162, argv=0x56286c700d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056286935ff15 in main (argc=23, argv=0x7ffc2217a028) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 33 (Thread 0x7f4236ffd700 (LWP 72511)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad5dda in os_event::timed_wait (this=0x56286c9d8500, abstime=0x7f4236ffcba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562869ad60f1 in os_event::wait_time_low (this=0x56286c9d8500, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562869ad6291 in os_event_wait_time_low (event=0x56286c9d8500, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562869bc4494 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1836 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f4274a78700 (LWP 72153)): #0 0x00007f427da0e483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f42774ff6f8 in asio::detail::epoll_reactor::run (this=0x56286c8c8000, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f4277503df0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x56286c8c7ea0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f42775040a2 in asio::detail::task_io_service::run (this=0x56286c8c7ea0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f427751bafd in asio::io_service::run (this=0x56286c8c6bc0) at asio/asio/impl/io_service.ipp:58 #5 0x00007f427751c251 in gcomm::AsioProtonet::event_loop (this=0x56286c8c6b20, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f427753a2ed in GCommConn::run (this=0x56286c8c7630) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f427753d399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f4225ffb700 (LWP 72521)): #0 0x00007f427f656e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000562869ad67c1 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000562869c7ced4 in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f4224ff9700 (LWP 72523)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869f93e1d in safe_cond_wait (cond=0x7f4224ff8c70, mp=0x7f4224ff8cb0, file=0x56286a2620c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005628699ed60b in inline_mysql_cond_wait (that=0x7f4224ff8c70, mutex=0x7f4224ff8cb0, src_file=0x56286a263a58 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005628699f2247 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x0000562869f268f7 in pfs_spawn_thread (arg=0x56286cb923b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f4227fff700 (LWP 72517)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad64bf in os_event::wait (this=0x56286c9d8380) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562869ad5f45 in os_event::wait_low (this=0x56286c9d8380, reset_sig_count=323) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562869ad62b6 in os_event_wait_low (event=0x56286c9d8380, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562869bc228b in srv_resume_thread (slot=0x56286ab10780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562869bc5e8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4264706700 (LWP 72528)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4277576067 in gu::Lock::wait (cond=..., this=0x7f4264704320) at galerautils/src/gu_lock.hpp:45 #2 0x00007f4277586c4c in galera::Monitor::enter (this=this@entry=0x56286c880250, obj=...) at galera/src/monitor.hpp:156 #3 0x00007f427757b717 in galera::ReplicatorSMM::commit_order_enter_local (this=this@entry=0x56286c87f1f0, trx=...) at galera/src/replicator_smm.cpp:1282 #4 0x00007f427759a063 in galera_commit_order_enter (gh=, ws_handle=, meta=) at galera/src/wsrep_provider.cpp:652 #5 0x000056286a048c5f in wsrep::wsrep_provider_v26::commit_order_enter (this=0x56286c7f4e50, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:769 #6 0x000056286a03cfde in wsrep::transaction::after_statement (this=0x7f424800ff10) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/transaction.cpp:758 #7 0x000056286a02d675 in wsrep::client_state::after_statement (this=0x7f424800fea0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/client_state.cpp:241 #8 0x000056286948e75c in wsrep_after_statement (thd=0x7f424800a040) at /home/stepan/galera/git/10.4/server/sql/wsrep_trans_observer.h:309 #9 0x00005628694a7111 in wsrep_mysql_parse (thd=0x7f424800a040, rawbuf=0x7f4248017388 "INSERT INTO t1 VALUE (4, 'z')", length=29, parser_state=0x7f42647050d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7956 #10 0x0000562869492e75 in dispatch_command (command=COM_QUERY, thd=0x7f424800a040, packet=0x7f4248028821 "INSERT INTO t1 VALUE (4, 'z');", packet_length=30, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #11 0x000056286949185d in do_command (thd=0x7f424800a040) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #12 0x00005628696020b9 in do_handle_one_connection (connect=0x56286cd4cd70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #13 0x0000562869601dfd in handle_one_connection (arg=0x56286cd4cd70) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #14 0x0000562869f268f7 in pfs_spawn_thread (arg=0x56286ccd66b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #15 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f4242ffd700 (LWP 72463)): #0 0x00007f427f220644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562869ac97dc in LinuxAIOHandler::collect (this=0x7f4242ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562869ac9d05 in LinuxAIOHandler::poll (this=0x7f4242ffcb00, m1=0x7f4242ffcba8, m2=0x7f4242ffcbb0, request=0x7f4242ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562869ac9e68 in os_aio_linux_handler (global_segment=2, m1=0x7f4242ffcba8, m2=0x7f4242ffcbb0, request=0x7f4242ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562869ace98c in os_aio_handler (segment=2, m1=0x7f4242ffcba8, m2=0x7f4242ffcbb0, request=0x7f4242ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562869d2193d in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562869bc7da1 in io_handler_thread (arg=0x56286ad4e630 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f42267fc700 (LWP 72520)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad64bf in os_event::wait (this=0x56286c9d8600) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562869ad5f45 in os_event::wait_low (this=0x56286c9d8600, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562869ad62b6 in os_event_wait_low (event=0x56286c9d8600, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562869ca2349 in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:833 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4235ffb700 (LWP 72514)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad5dda in os_event::timed_wait (this=0x56286cb802d0, abstime=0x7f4235ffab60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562869ad60f1 in os_event::wait_time_low (this=0x56286cb802d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562869ad6291 in os_event_wait_time_low (event=0x56286cb802d0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562869c24793 in ib_wqueue_timedwait (wq=0x56286cb7f7e0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x0000562869d6533c in fts_optimize_thread (arg=0x56286cb7f7e0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4264751700 (LWP 72525)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869f93e1d in safe_cond_wait (cond=0x56286acd31e0 , mp=0x56286acd6560 , file=0x56286a05f148 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056286939153e in inline_mysql_cond_wait (that=0x56286acd31e0 , mutex=0x56286acd6560 , src_file=0x56286a05fcb0 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x0000562869392dd0 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x0000562869f268f7 in pfs_spawn_thread (arg=0x56286cd3aeb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f4234ff9700 (LWP 72516)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad64bf in os_event::wait (this=0x56286c9d8300) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562869ad5f45 in os_event::wait_low (this=0x56286c9d8300, reset_sig_count=81) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562869ad62b6 in os_event_wait_low (event=0x56286c9d8300, reset_sig_count=81) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562869bc228b in srv_resume_thread (slot=0x56286ab10748 , sig_count=81, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562869bc629c in srv_purge_coordinator_suspend (slot=0x56286ab10748 , rseg_history_len=43) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2614 #6 0x0000562869bc6532 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2689 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f426479c700 (LWP 72524)): #0 0x00007f427d94740a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000056286935ff94 in my_sigwait (set=0x7f426479bcf0, sig=0x7f426479bcb4, code=0x7f426479bcb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005628693665d4 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x0000562869f268f7 in pfs_spawn_thread (arg=0x56286c99bf90) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f4237fff700 (LWP 72513)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad5dda in os_event::timed_wait (this=0x56286cbe2af0, abstime=0x7f4237ffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562869ad60f1 in os_event::wait_time_low (this=0x56286cbe2af0, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562869ad6291 in os_event_wait_time_low (event=0x56286cbe2af0, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562869d0b5da in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f42357fa700 (LWP 72515)): #0 0x00007f427f656e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000562869ad67c1 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x0000562869bc57f3 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2309 #3 0x0000562869bc5923 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2350 #4 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f42257fa700 (LWP 72522)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad64bf in os_event::wait (this=0x56286c9d8700) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562869ad5f45 in os_event::wait_low (this=0x56286c9d8700, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562869ad62b6 in os_event_wait_low (event=0x56286c9d8700, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562869c8afbe in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3097 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f4240ff9700 (LWP 72467)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad5dda in os_event::timed_wait (this=0x56286c9d8680, abstime=0x7f4240ff8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562869ad60f1 in os_event::wait_time_low (this=0x56286c9d8680, time_in_usec=807000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562869ad6291 in os_event_wait_time_low (event=0x56286c9d8680, time_in_usec=807000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562869cac1b5 in pc_sleep_if_needed (next_loop_time=1549847132477, sig_count=2, cur_time=1549847131670) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2593 #5 0x0000562869cad0ee in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3052 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4241ffb700 (LWP 72465)): #0 0x00007f427f220644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562869ac97dc in LinuxAIOHandler::collect (this=0x7f4241ffab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562869ac9d05 in LinuxAIOHandler::poll (this=0x7f4241ffab00, m1=0x7f4241ffaba8, m2=0x7f4241ffabb0, request=0x7f4241ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562869ac9e68 in os_aio_linux_handler (global_segment=4, m1=0x7f4241ffaba8, m2=0x7f4241ffabb0, request=0x7f4241ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562869ace98c in os_aio_handler (segment=4, m1=0x7f4241ffaba8, m2=0x7f4241ffabb0, request=0x7f4241ffabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562869d2193d in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562869bc7da1 in io_handler_thread (arg=0x56286ad4e640 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4226ffd700 (LWP 72519)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad64bf in os_event::wait (this=0x56286c9d8400) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562869ad5f45 in os_event::wait_low (this=0x56286c9d8400, reset_sig_count=291) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562869ad62b6 in os_event_wait_low (event=0x56286c9d8400, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562869bc228b in srv_resume_thread (slot=0x56286ab107b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562869bc5e8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f427422c700 (LWP 72196)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869f93e1d in safe_cond_wait (cond=0x7f425c016858, mp=0x7f425c0167a8, file=0x56286a0fdfe0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005628696eb20c in inline_mysql_cond_wait (that=0x7f425c016858, mutex=0x7f425c0167a8, src_file=0x56286a0fe300 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005628696edea4 in Wsrep_thd_queue::pop_front (this=0x7f425c016750) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005628696ecc84 in wsrep_post_rollback_process (post_rollbacker=0x7f425c001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005628696dd322 in start_wsrep_THD (arg=0x56286c8e8e90) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f42427fc700 (LWP 72464)): #0 0x00007f427f220644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562869ac97dc in LinuxAIOHandler::collect (this=0x7f42427fbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562869ac9d05 in LinuxAIOHandler::poll (this=0x7f42427fbb00, m1=0x7f42427fbba8, m2=0x7f42427fbbb0, request=0x7f42427fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562869ac9e68 in os_aio_linux_handler (global_segment=3, m1=0x7f42427fbba8, m2=0x7f42427fbbb0, request=0x7f42427fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562869ace98c in os_aio_handler (segment=3, m1=0x7f42427fbba8, m2=0x7f42427fbbb0, request=0x7f42427fbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562869d2193d in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562869bc7da1 in io_handler_thread (arg=0x56286ad4e638 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f42367fc700 (LWP 72512)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad5dda in os_event::timed_wait (this=0x56286c9d8580, abstime=0x7f42367fbd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562869ad60f1 in os_event::wait_time_low (this=0x56286c9d8580, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562869ad6291 in os_event_wait_time_low (event=0x56286c9d8580, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562869bc3f72 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1672 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f42277fe700 (LWP 72518)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad64bf in os_event::wait (this=0x56286c9d8480) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x0000562869ad5f45 in os_event::wait_low (this=0x56286c9d8480, reset_sig_count=289) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x0000562869ad62b6 in os_event_wait_low (event=0x56286c9d8480, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x0000562869bc228b in srv_resume_thread (slot=0x56286ab107f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:901 #5 0x0000562869bc5e8f in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2483 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f42437fe700 (LWP 72462)): #0 0x00007f427f220644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562869ac97dc in LinuxAIOHandler::collect (this=0x7f42437fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562869ac9d05 in LinuxAIOHandler::poll (this=0x7f42437fdb00, m1=0x7f42437fdba8, m2=0x7f42437fdbb0, request=0x7f42437fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562869ac9e68 in os_aio_linux_handler (global_segment=1, m1=0x7f42437fdba8, m2=0x7f42437fdbb0, request=0x7f42437fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562869ace98c in os_aio_handler (segment=1, m1=0x7f42437fdba8, m2=0x7f42437fdbb0, request=0x7f42437fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562869d2193d in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562869bc7da1 in io_handler_thread (arg=0x56286ad4e628 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f42780a8700 (LWP 72147)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869f9411f in safe_cond_timedwait (cond=0x56286b572640 , mp=0x56286b572580 , abstime=0x7f42780a7d70, file=0x56286a41be20 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562869f950d9 in inline_mysql_cond_timedwait (that=0x56286b572640 , mutex=0x56286b572580 , abstime=0x7f42780a7d70, src_file=0x56286a41be68 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562869f95c3c in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x0000562869f268f7 in pfs_spawn_thread (arg=0x56286c8487d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f42377fe700 (LWP 72510)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869ad5dda in os_event::timed_wait (this=0x56286cbb0a30, abstime=0x7f42377fdd40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x0000562869ad60f1 in os_event::wait_time_low (this=0x56286cbb0a30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x0000562869ad6291 in os_event_wait_time_low (event=0x56286cbb0a30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x0000562869aa2c4e in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f42417fa700 (LWP 72466)): #0 0x00007f427f220644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562869ac97dc in LinuxAIOHandler::collect (this=0x7f42417f9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562869ac9d05 in LinuxAIOHandler::poll (this=0x7f42417f9b00, m1=0x7f42417f9ba8, m2=0x7f42417f9bb0, request=0x7f42417f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562869ac9e68 in os_aio_linux_handler (global_segment=5, m1=0x7f42417f9ba8, m2=0x7f42417f9bb0, request=0x7f42417f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562869ace98c in os_aio_handler (segment=5, m1=0x7f42417f9ba8, m2=0x7f42417f9bb0, request=0x7f42417f9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562869d2193d in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562869bc7da1 in io_handler_thread (arg=0x56286ad4e648 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f4243fff700 (LWP 72461)): #0 0x00007f427f220644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x0000562869ac97dc in LinuxAIOHandler::collect (this=0x7f4243ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x0000562869ac9d05 in LinuxAIOHandler::poll (this=0x7f4243ffeb00, m1=0x7f4243ffeba8, m2=0x7f4243ffebb0, request=0x7f4243ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x0000562869ac9e68 in os_aio_linux_handler (global_segment=0, m1=0x7f4243ffeba8, m2=0x7f4243ffebb0, request=0x7f4243ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x0000562869ace98c in os_aio_handler (segment=0, m1=0x7f4243ffeba8, m2=0x7f4243ffebb0, request=0x7f4243ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5665 #5 0x0000562869d2193d in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4344 #6 0x0000562869bc7da1 in io_handler_thread (arg=0x56286ad4e620 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4274277700 (LWP 72195)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869f93e1d in safe_cond_wait (cond=0x7f4254015608, mp=0x7f4254015558, file=0x56286a0fdfe0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005628696eb20c in inline_mysql_cond_wait (that=0x7f4254015608, mutex=0x7f4254015558, src_file=0x56286a0fe300 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005628696edea4 in Wsrep_thd_queue::pop_front (this=0x7f4254015500) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005628696ec8c4 in wsrep_rollback_process (rollbacker=0x7f4254000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005628696dd322 in start_wsrep_THD (arg=0x56286c8e95e0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4265dee700 (LWP 72460)): #0 0x00007f427f653d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869f9411f in safe_cond_timedwait (cond=0x56286b55fea0 , mp=0x56286b55fde0 , abstime=0x7f4265dedcb0, file=0x56286a3d7bf0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x0000562869dfdf63 in inline_mysql_cond_timedwait (that=0x56286b55fea0 , mutex=0x56286b55fde0 , abstime=0x7f4265dedcb0, src_file=0x56286a3d7c38 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x0000562869dfe487 in my_service_thread_sleep (control=0x56286ab14b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x0000562869df2920 in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x0000562869f268f7 in pfs_spawn_thread (arg=0x56286c9904e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4267ffe700 (LWP 72194)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4277538867 in gu::Lock::wait (cond=..., this=0x7f4267ffd850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f427753b3ab in RecvBuf::front (this=this@entry=0x56286c8c7818, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f4277538f46 in gcomm_recv (backend=, msg=0x56286c887720, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f427752ceec in core_msg_recv (backend=backend@entry=0x56286c887868, recv_msg=recv_msg@entry=0x56286c887720, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f4277530164 in gcs_core_recv (conn=0x56286c8876c0, recv_act=recv_act@entry=0x7f4267ffdd70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f42775373d8 in gcs_recv_thread (arg=0x56286c887440) at gcs/src/gcs.cpp:1398 #7 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4276efc700 (LWP 72148)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f427755dfd7 in gu::Lock::wait (cond=..., this=0x7f4276efbbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f427755e069 in galera::ServiceThd::thd_func (arg=0x56286c87f908) at galera/src/galera_service_thd.cpp:30 #3 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f42741e1700 (LWP 72197)): #0 0x00007f427f653965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000562869f93e1d in safe_cond_wait (cond=0x7f424c000cd0, mp=0x7f424c000c20, file=0x56286a0c6240 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000056286960a84c in inline_mysql_cond_wait (that=0x7f424c000cd0, mutex=0x7f424c000c20, src_file=0x56286a0c6628 "/home/stepan/galera/git/10.4/server/sql/mdl.cc", src_line=1143) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000056286960bcd6 in MDL_wait::timed_wait (this=0x7f424c000c20, owner=0x7f424c000bd0, abs_timeout=0x7f42741dd4f0, set_status_on_timeout=false, wait_state_name=0x56286aaf8ff0 ) at /home/stepan/galera/git/10.4/server/sql/mdl.cc:1143 #4 0x000056286960d9d9 in MDL_context::acquire_lock (this=0x7f424c000c20, mdl_request=0x7f424c00fbf8, lock_wait_timeout=86400) at /home/stepan/galera/git/10.4/server/sql/mdl.cc:2326 #5 0x000056286960de8f in MDL_context::acquire_locks (this=0x7f424c000c20, mdl_requests=0x7f42741dd630, lock_wait_timeout=86400) at /home/stepan/galera/git/10.4/server/sql/mdl.cc:2447 #6 0x000056286940142d in lock_table_names (thd=0x7f424c000b00, options=..., tables_start=0x7f424c00f7d8, tables_end=0x0, lock_wait_timeout=86400, flags=0) at /home/stepan/galera/git/10.4/server/sql/sql_base.cc:4019 #7 0x00005628699b2617 in lock_table_names (thd=0x7f424c000b00, table_list=0x7f424c00f7d8, table_list_end=0x0, lock_wait_timeout=86400, flags=0) at /home/stepan/galera/git/10.4/server/sql/sql_base.h:247 #8 0x00005628699b318d in Sql_cmd_truncate_table::lock_table (this=0x7f424c00fdf0, thd=0x7f424c000b00, table_ref=0x7f424c00f7d8, hton_can_recreate=0x7f42741dd94a) at /home/stepan/galera/git/10.4/server/sql/sql_truncate.cc:316 #9 0x00005628699b3562 in Sql_cmd_truncate_table::truncate_table (this=0x7f424c00fdf0, thd=0x7f424c000b00, table_ref=0x7f424c00f7d8) at /home/stepan/galera/git/10.4/server/sql/sql_truncate.cc:424 #10 0x00005628699b37c6 in Sql_cmd_truncate_table::execute (this=0x7f424c00fdf0, thd=0x7f424c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_truncate.cc:506 #11 0x00005628694a27d7 in mysql_execute_command (thd=0x7f424c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:6309 #12 0x00005628694a7985 in mysql_parse (thd=0x7f424c000b00, rawbuf=0x7f424c07f01b "TRUNCATE TABLE t1", length=17, parser_state=0x7f42741de4b0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #13 0x000056286990fc47 in Query_log_event::do_apply_event (this=0x7f424c0845e0, rgi=0x7f424c0230d0, query_arg=0x7f424c07f01b "TRUNCATE TABLE t1", q_len_arg=17) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:5665 #14 0x000056286990ec6b in Query_log_event::do_apply_event (this=0x7f424c0845e0, rgi=0x7f424c0230d0) at /home/stepan/galera/git/10.4/server/sql/log_event.cc:5358 #15 0x00005628693aa7d9 in Log_event::apply_event (this=0x7f424c0845e0, rgi=0x7f424c0230d0) at /home/stepan/galera/git/10.4/server/sql/log_event.h:1481 #16 0x00005628696eaaf1 in wsrep_apply_events (thd=0x7f424c000b00, rli=0x7f424c017a60, events_buf=0x7f42680054d0, buf_len=0) at /home/stepan/galera/git/10.4/server/sql/wsrep_applier.cc:206 #17 0x00005628696cacba in Wsrep_high_priority_service::apply_toi (this=0x7f42741e04d0, ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:373 #18 0x000056286a031d03 in apply_toi (provider=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:312 #19 0x000056286a034c0f in wsrep::server_state::on_apply (this=0x56286c849740, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:933 #20 0x000056286a04a357 in wsrep::high_priority_service::apply (this=0x7f42741e04d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #21 0x000056286a047a3e in (anonymous namespace)::apply_cb (ctx=0x7f42741e04d0, wsh=0x7f42741df3e0, flags=69, buf=0x7f42741df420, meta=0x7f42741df6b0, exit_loop=0x7f42741df5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:489 #22 0x00007f42775473b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7f424c0833a0, recv_ctx=recv_ctx@entry=0x7f42741e04d0, apply_cb=0x56286a04781b <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f42741df5ef: false) at galera/src/trx_handle.cpp:418 #23 0x00007f4277579d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x56286c87f1f0, recv_ctx=recv_ctx@entry=0x7f42741e04d0, ts=...) at galera/src/replicator_smm.cpp:489 #24 0x00007f427757fa40 in galera::ReplicatorSMM::process_trx (this=0x56286c87f1f0, recv_ctx=0x7f42741e04d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #25 0x00007f4277560c6b in galera::GcsActionSource::process_writeset (this=this@entry=0x56286c8bfe60, recv_ctx=0x7f42741e04d0, act=..., exit_loop=@0x7f42741e029e: false) at galera/src/gcs_action_source.cpp:62 #26 0x00007f4277560e45 in galera::GcsActionSource::dispatch (this=this@entry=0x56286c8bfe60, recv_ctx=recv_ctx@entry=0x7f42741e04d0, act=..., exit_loop=@0x7f42741e029e: false) at galera/src/gcs_action_source.cpp:109 #27 0x00007f427756121d in galera::GcsActionSource::process (this=0x56286c8bfe60, recv_ctx=0x7f42741e04d0, exit_loop=@0x7f42741e029e: false) at galera/src/gcs_action_source.cpp:182 #28 0x00007f4277580033 in galera::ReplicatorSMM::async_recv (this=0x56286c87f1f0, recv_ctx=0x7f42741e04d0) at galera/src/replicator_smm.cpp:383 #29 0x000056286a0486fc in wsrep::wsrep_provider_v26::run_applier (this=0x56286c7f4e50, applier_ctx=0x7f42741e04d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:690 #30 0x00005628696eb7ba in wsrep_replication_process (thd=0x7f424c000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #31 0x00005628696dd322 in start_wsrep_THD (arg=0x56286c89fd70) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2757 #32 0x00007f427f64fdd5 in start_thread () from /lib64/libpthread.so.0 #33 0x00007f427da0dead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f427fa60880 (LWP 72145)): #0 0x00007f427f6549d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000562869f8e7ee in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005628697cc687 in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f427da0320d in poll () from /lib64/libc.so.6 #5 0x000056286936cada in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #6 0x000056286936bd75 in mysqld_main (argc=162, argv=0x56286c700d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #7 0x000056286935ff15 in main (argc=23, argv=0x7ffc2217a028) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1060-innodb/mysqld.2/data/core.72145 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/3/log/galera_sr.GCF-1060-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-1060-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-1060 Attempting backtrace. You can use the following information to find out Attempting backtrace. You can use the following information to find out -------------------------------------------------------------------------- The servers were restarted 18 times Spent 2319.618 of 1747 seconds executing testcases Check of testcase failed for: galera_sr.mysql-wsrep-features#148 galera_sr.galera_sr_kill_all_norecovery Completed: Failed 8/83 tests, 90.36% were successful. Failing test(s): galera_sr.GCF-437 galera_sr.galera_sr_rollback_statement galera_sr.galera_sr_table_contents galera_sr.mysql-wsrep-features#165 galera_sr.galera_sr_mysqldump_sst galera_sr.galera_sr_cc_master galera_sr.galera_sr_load_data_splitting galera_sr.GCF-1060 The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html Errors/warnings were found in logfiles during server shutdown after running the following sequence(s) of tests: galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.galera_sr_cc_master galera_sr.GCF-623 galera_sr.galera_sr_blob galera_sr.GCF-851 galera_sr.GCF-867 galera_sr.galera_sr_bf_abort galera_sr.galera-features#56 galera_sr.GCF-845 galera_sr.GCF-1060 mysql-test-run: *** ERROR: there were failing test cases