Checking supported features... MariaDB Version 10.4.3-MariaDB-debug - SSL connections supported - binaries are debug compiled Using suites: galera_sr Collecting tests... Installing system database... worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- galera_sr.GCF-574 [ disabled ] CTAS is not supported together with SR galera_sr.galera_sr_sbr [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.mysql-wsrep-features#29 [ disabled ] binlog_format=STATEMENT not supported with SR galera_sr.galera_sr_large_fragment 'innodb' [ pass ] 100487 galera_sr.mysql-wsrep-features#32 'innodb' [ pass ] 3654 galera_sr.galera_sr_gtid 'innodb' [ pass ] 2123 galera_sr.galera_sr_log_bin 'innodb' [ fail ] Test ended at 2019-02-06 13:04:23 CURRENT_TEST: galera_sr.galera_sr_log_bin mysqltest: At line 51: query 'COMMIT' failed: 2013: Lost connection to MySQL server during query The result from queries just before the failure was: < snip > RESET MASTER; SET GLOBAL wsrep_on=ON; SET AUTOCOMMIT=OFF; SET SESSION wsrep_trx_fragment_size = 1; START TRANSACTION; INSERT INTO t3 VALUES (1); connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2a; SET AUTOCOMMIT=OFF; START TRANSACTION; INSERT INTO t4 VALUES (1); connection node_1; INSERT INTO t1 VALUES (2); COMMIT; connection node_1a; INSERT INTO t2 VALUES (2); COMMIT; connection node_2; INSERT INTO t3 VALUES (2); COMMIT; More results from queries before failure can be found in /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr_log_bin.log Server [mysqld.2 - pid: 2035, winpid: 2035, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2019-02-06 13:04:12 0 [Warning] WSREP: option --wsrep-causal-reads is deprecated 2019-02-06 13:04:12 0 [Note] /home/stepan/galera/git/10.4/server/sql/mysqld (mysqld 10.4.3-MariaDB-debug-log) starting as process 2037 ... 2019-02-06 13:04:12 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4186) 2019-02-06 13:04:12 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000) 2019-02-06 13:04:12 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm_4.soinitial position: 00000000-0000-0000-0000-000000000000:-1 wsrep loader: [INFO] wsrep_load(): loading provider library '/usr/lib/libgalera_smm_4.so' wsrep loader: [INFO] wsrep_load(): Galera 26.4.0(r4463) by Codership Oy loaded successfully. 2019-02-06 13:04:12 0 [Note] WSREP: CRC-32C: using hardware acceleration. 2019-02-06 13:04:12 0 [Warning] WSREP: Could not open state file for reading: '/home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data//grastate.dat' 2019-02-06 13:04:12 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1 2019-02-06 13:04:12 0 [Note] WSREP: GCache DEBUG: opened preamble: Version: 0 UUID: 00000000-0000-0000-0000-000000000000 Seqno: -1 - -1 Offset: -1 Synced: 0 2019-02-06 13:04:12 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID. 2019-02-06 13:04:12 0 [Note] WSREP: Passing config to GCS: base_dir = /home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data/; base_host = 127.0.0.1; base_port = 16005; cert.log_conflicts = no; cert.optimistic_pa = yes; dbug = ; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT30S; evs.install_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT10S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.debug = 0; gcache.dir = /home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data//galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_pac 2019-02-06 13:04:12 0 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1 2019-02-06 13:04:12 0 [Note] WSREP: Start replication 2019-02-06 13:04:12 0 [Note] WSREP: Connecting with bootstrap option: 0 2019-02-06 13:04:12 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1 2019-02-06 13:04:12 0 [Note] WSREP: protonet asio version 0 2019-02-06 13:04:12 0 [Note] WSREP: Using CRC-32C for message checksums. 2019-02-06 13:04:12 0 [Note] WSREP: backend: asio 2019-02-06 13:04:12 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-02-06 13:04:12 0 [Warning] WSREP: access file(/home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data//gvwstate.dat) failed(No such file or directory) 2019-02-06 13:04:12 0 [Note] WSREP: restore pc from disk failed 2019-02-06 13:04:12 0 [Note] WSREP: GMCast version 0 2019-02-06 13:04:12 0 [Note] WSREP: (ef8f2227, 'tcp://0.0.0.0:16005') listening at tcp://0.0.0.0:16005 2019-02-06 13:04:12 0 [Note] WSREP: (ef8f2227, 'tcp://0.0.0.0:16005') multicast: , ttl: 1 2019-02-06 13:04:12 0 [Note] WSREP: EVS version 1 2019-02-06 13:04:12 0 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '127.0.0.1:16002' 2019-02-06 13:04:12 0 [Note] WSREP: (ef8f2227, 'tcp://0.0.0.0:16005') connection established to ef2db834 tcp://127.0.0.1:16002 2019-02-06 13:04:12 0 [Note] WSREP: (ef8f2227, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: 2019-02-06 13:04:13 0 [Note] WSREP: EVS version upgrade 0 -> 1 2019-02-06 13:04:13 0 [Note] WSREP: declaring ef2db834 at tcp://127.0.0.1:16002 stable 2019-02-06 13:04:13 0 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-02-06 13:04:13 0 [Note] WSREP: Node ef2db834 state prim 2019-02-06 13:04:13 0 [Note] WSREP: view(view_id(PRIM,ef2db834,2) memb { ef2db834,0 ef8f2227,0 } joined { } left { } partitioned { }) 2019-02-06 13:04:13 0 [Note] WSREP: save pc into disk 2019-02-06 13:04:13 0 [Note] WSREP: gcomm: connected 2019-02-06 13:04:13 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-02-06 13:04:13 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2019-02-06 13:04:13 0 [Note] WSREP: Opened channel 'my_wsrep_cluster' 2019-02-06 13:04:13 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2019-02-06 13:04:13 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-02-06 13:04:13 0 [Note] WSREP: STATE EXCHANGE: sent state msg: efde7215-29fe-11e9-8d6b-27cc073b198b 2019-02-06 13:04:13 0 [Note] WSREP: STATE EXCHANGE: got state msg: efde7215-29fe-11e9-8d6b-27cc073b198b from 0 (cnt7glr11.localdomain) 2019-02-06 13:04:13 0 [Note] WSREP: STATE EXCHANGE: got state msg: efde7215-29fe-11e9-8d6b-27cc073b198b from 1 (cnt7glr11.localdomain) 2019-02-06 13:04:13 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 1, members = 1/2 (joined/total), act_id = 1, last_appl. = 0, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = ef2e8234-29fe-11e9-9599-be8c46cd9a7c 2019-02-06 13:04:13 0 [Note] WSREP: Writing down CC checksum: 0c91dc32 95617d2a 955afe77 d82cf9bc at offset 224 2019-02-06 13:04:13 0 [Note] WSREP: Flow-control interval: [23, 23] 2019-02-06 13:04:13 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-06 13:04:13 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2) 2019-02-06 13:04:13 3 [Note] WSREP: Starting applier thread 3 2019-02-06 13:04:13 3 [Note] WSREP: ####### processing CC 2, local, ordered 2019-02-06 13:04:13 3 [Note] WSREP: ####### drain monitors upto -1 2019-02-06 13:04:13 3 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-02-06 13:04:13 3 [Note] WSREP: ####### My UUID: ef8f2227-29fe-11e9-9943-4b496ceccdb4 2019-02-06 13:04:13 3 [Note] WSREP: Server cnt7glr11.localdomain connected to cluster at position ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 with ID ef8f2227-29fe-11e9-9943-4b496ceccdb4 2019-02-06 13:04:13 3 [Note] WSREP: Server status change disconnected -> connected 2019-02-06 13:04:13 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:04:13 3 [Note] WSREP: State transfer required: Group state: ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 Local state: 00000000-0000-0000-0000-000000000000:-1 2019-02-06 13:04:13 3 [Note] WSREP: Server status change connected -> joiner 2019-02-06 13:04:13 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:04:13 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '127.0.0.2:16007' --datadir '/home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data/' --defaults-file '/home/stepan/galera/git/10.4/server/mysql-test/var/my.cnf' --defaults-group-suffix '.2' --parent '2037' --binlog 'mysqld-bin' --binlog-index 'mysqld-bin.index'' 2019-02-06 13:04:14 3 [Note] WSREP: Prepared SST request: rsync|127.0.0.2:16007/rsync_sst 2019-02-06 13:04:14 3 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-02-06 13:04:14 0 [Note] WSREP: Service thread queue flushed. 2019-02-06 13:04:14 3 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-02-06 13:04:14 3 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 2, p: 10 2019-02-06 13:04:14 3 [Note] WSREP: IST receiver addr using tcp://127.0.0.1:16006 2019-02-06 13:04:14 3 [Note] WSREP: Prepared IST receiver for 0-2, listening at: tcp://127.0.0.1:16006 2019-02-06 13:04:14 0 [Note] WSREP: Member 1.0 (cnt7glr11.localdomain) requested state transfer from '*any*'. Selected 0.0 (cnt7glr11.localdomain)(SYNCED) as donor. 2019-02-06 13:04:14 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2) 2019-02-06 13:04:14 3 [Note] WSREP: Requesting state transfer: success, donor: 0 2019-02-06 13:04:14 3 [Note] WSREP: Resetting GCache seqno map due to different histories. 2019-02-06 13:04:14 3 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0 -> ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 2019-02-06 13:04:15 0 [Note] WSREP: 0.0 (cnt7glr11.localdomain): State transfer to 1.0 (cnt7glr11.localdomain) complete. 2019-02-06 13:04:15 0 [Note] WSREP: Member 0.0 (cnt7glr11.localdomain) synced with group. 2019-02-06 13:04:15 0 [Note] WSREP: (ef8f2227, 'tcp://0.0.0.0:16005') turning message relay requesting off WSREP_SST: [INFO] Extracting binlog files: (20190206 13:04:16.092) mysqld-bin.000002 WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2183 (20190206 13:04:16.102) WSREP_SST: [INFO] Joiner cleanup done. (20190206 13:04:16.611) 2019-02-06 13:04:16 4 [Note] WSREP: SST received 2019-02-06 13:04:16 4 [Note] WSREP: Server status change joiner -> initializing 2019-02-06 13:04:16 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:04:16 0 [Note] Plugin 'partition' is disabled. 2019-02-06 13:04:16 0 [Note] Plugin 'SEQUENCE' is disabled. 2019-02-06 13:04:16 0 [Note] InnoDB: Using Linux native AIO 2019-02-06 13:04:16 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2019-02-06 13:04:16 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-02-06 13:04:16 0 [Note] InnoDB: Uses event mutexes 2019-02-06 13:04:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-02-06 13:04:16 0 [Note] InnoDB: Number of pools: 1 2019-02-06 13:04:16 0 [Note] InnoDB: Using SSE2 crc32 instructions 2019-02-06 13:04:16 0 [Note] mysqld: O_TMPFILE is not supported on /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2 (disabling future attempts) 2019-02-06 13:04:16 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M 2019-02-06 13:04:16 0 [Note] InnoDB: Completed initialization of buffer pool 2019-02-06 13:04:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2019-02-06 13:04:16 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=151278 2019-02-06 13:04:16 0 [Note] InnoDB: Starting final batch to recover 18 pages from redo log. 2019-02-06 13:04:17 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2019-02-06 13:04:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2019-02-06 13:04:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2019-02-06 13:04:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2019-02-06 13:04:17 0 [Note] InnoDB: Waiting for purge to start 2019-02-06 13:04:17 0 [Note] InnoDB: 10.4.3 started; log sequence number 186029; transaction id 47 2019-02-06 13:04:17 0 [Note] InnoDB: Loading buffer pool(s) from /home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data/ib_buffer_pool 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_MUTEXES' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_CMP' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled. 2019-02-06 13:04:17 0 [Note] InnoDB: Buffer pool(s) load completed at 190206 13:04:17 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'user_variables' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled. 2019-02-06 13:04:17 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled. 2019-02-06 13:04:17 0 [Warning] /home/stepan/galera/git/10.4/server/sql/mysqld: unknown option '--loose-pam-debug' 2019-02-06 13:04:17 0 [Warning] /home/stepan/galera/git/10.4/server/sql/mysqld: unknown option '--loose-aria' 2019-02-06 13:04:17 0 [Note] Recovering after a crash using mysqld-bin 2019-02-06 13:04:17 0 [Note] Starting crash recovery... 2019-02-06 13:04:17 0 [Note] Crash recovery finished. 2019-02-06 13:04:17 0 [Note] Server socket created on IP: '127.0.0.1'. 2019-02-06 13:04:17 0 [Note] WSREP: wsrep_init_schema_and_SR (nil) 2019-02-06 13:04:17 0 [Note] WSREP: Server initialized 2019-02-06 13:04:17 0 [Note] WSREP: Server status change initializing -> initialized 2019-02-06 13:04:17 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:04:17 4 [Note] WSREP: Server status change initialized -> joined 2019-02-06 13:04:17 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:04:17 0 [Note] Reading of all Master_info entries succeded 2019-02-06 13:04:17 0 [Note] Added new Master_info '' to hash table 2019-02-06 13:04:17 0 [Note] /home/stepan/galera/git/10.4/server/sql/mysqld: ready for connections. Version: '10.4.3-MariaDB-debug-log' socket: '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.sock' port: 16001 Source distribution 2019-02-06 13:04:17 4 [Note] WSREP: Recovered position from storage: ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 2019-02-06 13:04:17 4 [Note] WSREP: Recovered view from SST: id: ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 status: primary prococol_version: 4 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO final: no own_index: 1 members(2): 0: ef2db834-29fe-11e9-a856-1686a8c9fc09, cnt7glr11.localdomain 1: ef8f2227-29fe-11e9-9943-4b496ceccdb4, cnt7glr11.localdomain 2019-02-06 13:04:17 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:04:17 1 [Note] WSREP: Recovered cluster id ef2e8234-29fe-11e9-9599-be8c46cd9a7c 2019-02-06 13:04:17 4 [Note] WSREP: SST received: ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 2019-02-06 13:04:17 3 [Note] WSREP: Installed new state from SST: ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 2019-02-06 13:04:17 3 [Note] WSREP: Cert. index preload up to 2 2019-02-06 13:04:17 0 [Note] WSREP: ####### IST applying starts with 3 2019-02-06 13:04:17 0 [Note] WSREP: ####### IST current seqno initialized to 2 2019-02-06 13:04:17 0 [Note] WSREP: Receiving IST... 0.0% (0/1 events) complete. 2019-02-06 13:04:17 0 [Note] WSREP: IST preload starting at 2 2019-02-06 13:04:17 0 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-02-06 13:04:17 0 [Note] WSREP: ####### Adjusting cert position: 0 -> 2 2019-02-06 13:04:17 0 [Note] WSREP: Service thread queue flushed. 2019-02-06 13:04:17 0 [Note] WSREP: Lowest cert indnex boundary for CC from preload: 2 2019-02-06 13:04:17 0 [Note] WSREP: Min available from gcache for CC from preload: 2 2019-02-06 13:04:17 0 [Note] WSREP: Receiving IST...100.0% (1/1 events) complete. 2019-02-06 13:04:17 3 [Note] WSREP: IST received: ef2e8234-29fe-11e9-9599-be8c46cd9a7c:2 2019-02-06 13:04:17 0 [Note] WSREP: 1.0 (cnt7glr11.localdomain): State transfer from 0.0 (cnt7glr11.localdomain) complete. 2019-02-06 13:04:17 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 2) 2019-02-06 13:04:17 0 [Note] WSREP: Member 1.0 (cnt7glr11.localdomain) synced with group. 2019-02-06 13:04:17 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 2) 2019-02-06 13:04:17 3 [Note] WSREP: Server cnt7glr11.localdomain synced with group 2019-02-06 13:04:17 3 [Note] WSREP: Server status change joined -> synced 2019-02-06 13:04:17 3 [Note] WSREP: Synchronized with group, ready for connections 2019-02-06 13:04:17 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. mysqld: /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0rseg.cc:92: void trx_rseg_update_wsrep_checkpoint(trx_rsegf_t*, const XID*, mtr_t*): Assertion `xid_seqno > wsrep_seqno' failed. 190206 13:04:20 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.3-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=13 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63454 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f86ec000b00 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f870c3d09a0 thread_stack 0x49000 mysys/stacktrace.c:269(my_print_stacktrace)[0x5630a4b5507c] sql/signal_handler.cc:168(handle_fatal_signal)[0x5630a43913d3] sigaction.c:0(__restore_rt)[0x7f8714be35d0] :0(__GI_raise)[0x7f8712ed2207] :0(__GI_abort)[0x7f8712ed38f8] :0(__assert_fail_base)[0x7f8712ecb026] :0(__GI___assert_fail)[0x7f8712ecb0d2] trx/trx0rseg.cc:92(trx_rseg_update_wsrep_checkpoint(unsigned char*, xid_t const*, mtr_t*))[0x5630a47cd1ab] trx/trx0purge.cc:290(trx_purge_add_undo_to_history(trx_t const*, trx_undo_t*&, mtr_t*))[0x5630a47b1e5e] trx/trx0trx.cc:1095(trx_write_serialisation_history(trx_t*, mtr_t*))[0x5630a47d6fed] trx/trx0trx.cc:1523(trx_commit_low(trx_t*, mtr_t*))[0x5630a47d8acd] trx/trx0trx.cc:1570(trx_commit(trx_t*))[0x5630a47d8c1f] trx/trx0trx.cc:1708(trx_commit_for_mysql(trx_t*))[0x5630a47d9029] handler/ha_innodb.cc:4389(innobase_commit_low(trx_t*))[0x5630a45bccf2] handler/ha_innodb.cc:4511(innobase_commit_ordered_2(trx_t*, THD*))[0x5630a45bd0b2] handler/ha_innodb.cc:4571(innobase_commit_ordered(handlerton*, THD*, bool))[0x5630a45bd2e8] sql/log.cc:8860(TC_LOG::run_commit_ordered(THD*, bool))[0x5630a44c0752] sql/log.cc:8110(MYSQL_BIN_LOG::trx_group_commit_leader(MYSQL_BIN_LOG::group_commit_entry*))[0x5630a44be5eb] sql/log.cc:7700(MYSQL_BIN_LOG::write_transaction_to_binlog_events(MYSQL_BIN_LOG::group_commit_entry*))[0x5630a44bd220] sql/log.cc:7370(MYSQL_BIN_LOG::write_transaction_to_binlog(THD*, binlog_cache_mngr*, Log_event*, bool, bool, bool))[0x5630a44bc803] sql/log.cc:1787(binlog_flush_cache(THD*, binlog_cache_mngr*, Log_event*, bool, bool, bool))[0x5630a44ac99e] sql/log.cc:1893(binlog_commit_flush_xid_caches(THD*, binlog_cache_mngr*, bool, unsigned long long))[0x5630a44acfdc] sql/log.cc:9642(MYSQL_BIN_LOG::log_and_order(THD*, unsigned long long, bool, bool, bool))[0x5630a44c29d7] sql/handler.cc:1591(ha_commit_trans(THD*, bool))[0x5630a43968c4] sql/transaction.cc:313(trans_commit(THD*))[0x5630a41dc0ee] sql/wsrep_high_priority_service.cc:318(Wsrep_high_priority_service::commit(wsrep::ws_handle const&, wsrep::ws_meta const&))[0x5630a428fa49] src/server_state.cpp:193(apply_write_set(wsrep::server_state&, wsrep::high_priority_service&, wsrep::ws_handle const&, wsrep::ws_meta const&, wsrep::const_buffer const&))[0x5630a4bf7f96] src/server_state.cpp:925(wsrep::server_state::on_apply(wsrep::high_priority_service&, wsrep::ws_handle const&, wsrep::ws_meta const&, wsrep::const_buffer const&))[0x5630a4bfb521] wsrep/high_priority_service.hpp:47(wsrep::high_priority_service::apply(wsrep::ws_handle const&, wsrep::ws_meta const&, wsrep::const_buffer const&))[0x5630a4c0fbcb] src/wsrep_provider_v26.cpp:460((anonymous namespace)::apply_cb(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*))[0x5630a4c0d47b] src/trx_handle.cpp:414(galera::TrxHandleSlave::apply(void*, wsrep_cb_status (*)(void*, wsrep_ws_handle const*, unsigned int, wsrep_buf const*, wsrep_trx_meta const*, bool*), wsrep_trx_meta const&, bool&))[0x7f870cad33b1] src/replicator_smm.cpp:500(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&))[0x7f870cb05d92] src/replicator_smm.cpp:2137(galera::ReplicatorSMM::process_trx(void*, boost::shared_ptr const&))[0x7f870cb0ba40] src/gcs_action_source.cpp:63(galera::GcsActionSource::process_writeset(void*, gcs_action const&, bool&))[0x7f870caecc6b] src/gcs_action_source.cpp:109(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f870caece45] src/gcs_action_source.cpp:188(galera::GcsActionSource::process(void*, bool&))[0x7f870caed21d] src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7f870cb0c033] /home/stepan/galera/git/10.4/server/sql/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x30)[0x5630a4c0dffe] src/wsrep_provider_v26.cpp:646(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x5630a42b0932] sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x5630a42a257f] pthread_create.c:0(start_thread)[0x7f8714bdbdd5] /lib64/libc.so.6(clone+0x6d)[0x7f8712f99ead] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 3 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file at /home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data/ ----------SERVER LOG END------------- - found 'core.2037' (0/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_log_bin-innodb/mysqld.2/data/core.2037 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 2124] [New LWP 2422] [New LWP 2421] [New LWP 2427] [New LWP 2493] [New LWP 2473] [New LWP 2495] [New LWP 2475] [New LWP 2073] [New LWP 2477] [New LWP 2420] [New LWP 2122] [New LWP 2487] [New LWP 2423] [New LWP 2492] [New LWP 2424] [New LWP 2496] [New LWP 2485] [New LWP 2498] [New LWP 2502] [New LWP 2499] [New LWP 2083] [New LWP 2123] [New LWP 2639] [New LWP 2078] [New LWP 2121] [New LWP 2037] [New LWP 2494] [New LWP 2488] [New LWP 2486] [New LWP 2483] [New LWP 2476] [New LWP 2474] [New LWP 2425] [New LWP 2426] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007f8714be09d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007f8714be09d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005630a4b5516a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005630a43917ff in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f8712ed2207 in raise () from /lib64/libc.so.6 #5 0x00007f8712ed38f8 in abort () from /lib64/libc.so.6 #6 0x00007f8712ecb026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f8712ecb0d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00005630a47cd1ab in trx_rseg_update_wsrep_checkpoint (rseg_header=0x7f86ddcfc026 "", xid=0x5630a8149540, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0rseg.cc:92 #9 0x00005630a47b1e5e in trx_purge_add_undo_to_history (trx=0x7f86de459140, undo=@0x7f86de45a160: 0x5630a8379bc0, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:286 #10 0x00005630a47d6fed in trx_write_serialisation_history (trx=0x7f86de459140, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1092 #11 0x00005630a47d8acd in trx_commit_low (trx=0x7f86de459140, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1503 #12 0x00005630a47d8c1f in trx_commit (trx=0x7f86de459140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1570 #13 0x00005630a47d9029 in trx_commit_for_mysql (trx=0x7f86de459140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1706 #14 0x00005630a45bccf2 in innobase_commit_low (trx=0x7f86de459140) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4387 #15 0x00005630a45bd0b2 in innobase_commit_ordered_2 (trx=0x7f86de459140, thd=0x7f86ec000b00) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4509 #16 0x00005630a45bd2e8 in innobase_commit_ordered (hton=0x5630a8124ba0, thd=0x7f86ec000b00, all=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4569 #17 0x00005630a44c0752 in TC_LOG::run_commit_ordered (this=0x5630a58d2400 , thd=0x7f86ec000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:8859 #18 0x00005630a44be5eb in MYSQL_BIN_LOG::trx_group_commit_leader (this=0x5630a58d2400 , leader=0x7f870c3cdd20) at /home/stepan/galera/git/10.4/server/sql/log.cc:8109 #19 0x00005630a44bd220 in MYSQL_BIN_LOG::write_transaction_to_binlog_events (this=0x5630a58d2400 , entry=0x7f870c3cdd20) at /home/stepan/galera/git/10.4/server/sql/log.cc:7700 #20 0x00005630a44bc803 in MYSQL_BIN_LOG::write_transaction_to_binlog (this=0x5630a58d2400 , thd=0x7f86ec000b00, cache_mngr=0x7f86ec0550c0, end_ev=0x7f870c3cdec0, all=true, using_stmt_cache=true, using_trx_cache=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:7370 #21 0x00005630a44ac99e in binlog_flush_cache (thd=0x7f86ec000b00, cache_mngr=0x7f86ec0550c0, end_ev=0x7f870c3cdec0, all=true, using_stmt=true, using_trx=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:1787 #22 0x00005630a44acfdc in binlog_commit_flush_xid_caches (thd=0x7f86ec000b00, cache_mngr=0x7f86ec0550c0, all=true, xid=11) at /home/stepan/galera/git/10.4/server/sql/log.cc:1893 #23 0x00005630a44c29d7 in MYSQL_BIN_LOG::log_and_order (this=0x5630a58d2400 , thd=0x7f86ec000b00, xid=11, all=true, need_prepare_ordered=false, need_commit_ordered=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:9642 #24 0x00005630a43968c4 in ha_commit_trans (thd=0x7f86ec000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1591 #25 0x00005630a41dc0ee in trans_commit (thd=0x7f86ec000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #26 0x00005630a428fa49 in Wsrep_high_priority_service::commit (this=0x7f870c3d04d0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:318 #27 0x00005630a4bf7f96 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:194 #28 0x00005630a4bfb521 in wsrep::server_state::on_apply (this=0x5630a7fcf7d0, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:923 #29 0x00005630a4c0fbcb in wsrep::high_priority_service::apply (this=0x7f870c3d04d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #30 0x00005630a4c0d47b in (anonymous namespace)::apply_cb (ctx=0x7f870c3d04d0, wsh=0x7f870c3cf3e0, flags=65, buf=0x7f870c3cf420, meta=0x7f870c3cf6b0, exit_loop=0x7f870c3cf5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:460 #31 0x00007f870cad33b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7f86ec02e4e0, recv_ctx=recv_ctx@entry=0x7f870c3d04d0, apply_cb=0x5630a4c0d258 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f870c3cf5ef: false) at galera/src/trx_handle.cpp:418 #32 0x00007f870cb05d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5630a80052a0, recv_ctx=recv_ctx@entry=0x7f870c3d04d0, ts=...) at galera/src/replicator_smm.cpp:489 #33 0x00007f870cb0ba40 in galera::ReplicatorSMM::process_trx (this=0x5630a80052a0, recv_ctx=0x7f870c3d04d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #34 0x00007f870caecc6b in galera::GcsActionSource::process_writeset (this=this@entry=0x5630a8035de0, recv_ctx=0x7f870c3d04d0, act=..., exit_loop=@0x7f870c3d029e: false) at galera/src/gcs_action_source.cpp:62 #35 0x00007f870caece45 in galera::GcsActionSource::dispatch (this=this@entry=0x5630a8035de0, recv_ctx=recv_ctx@entry=0x7f870c3d04d0, act=..., exit_loop=@0x7f870c3d029e: false) at galera/src/gcs_action_source.cpp:109 #36 0x00007f870caed21d in galera::GcsActionSource::process (this=0x5630a8035de0, recv_ctx=0x7f870c3d04d0, exit_loop=@0x7f870c3d029e: false) at galera/src/gcs_action_source.cpp:182 #37 0x00007f870cb0c033 in galera::ReplicatorSMM::async_recv (this=0x5630a80052a0, recv_ctx=0x7f870c3d04d0) at galera/src/replicator_smm.cpp:383 #38 0x00005630a4c0dffe in wsrep::wsrep_provider_v26::run_applier (this=0x5630a7f7aee0, applier_ctx=0x7f870c3d04d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #39 0x00005630a42b0932 in wsrep_replication_process (thd=0x7f86ec000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #40 0x00005630a42a257f in start_wsrep_THD (arg=0x5630a8025e00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #41 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #42 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f86d6ffd700 (LWP 2426)): #0 0x00007f87147ac644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005630a468e950 in LinuxAIOHandler::collect (this=0x7f86d6ffcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005630a468ee79 in LinuxAIOHandler::poll (this=0x7f86d6ffcb00, m1=0x7f86d6ffcba8, m2=0x7f86d6ffcbb0, request=0x7f86d6ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005630a468efdc in os_aio_linux_handler (global_segment=5, m1=0x7f86d6ffcba8, m2=0x7f86d6ffcbb0, request=0x7f86d6ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005630a4693b00 in os_aio_handler (segment=5, m1=0x7f86d6ffcba8, m2=0x7f86d6ffcbb0, request=0x7f86d6ffcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005630a48e79b1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005630a478ced9 in io_handler_thread (arg=0x5630a5916688 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f86d77fe700 (LWP 2425)): #0 0x00007f87147ac644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005630a468e950 in LinuxAIOHandler::collect (this=0x7f86d77fdb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005630a468ee79 in LinuxAIOHandler::poll (this=0x7f86d77fdb00, m1=0x7f86d77fdba8, m2=0x7f86d77fdbb0, request=0x7f86d77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005630a468efdc in os_aio_linux_handler (global_segment=4, m1=0x7f86d77fdba8, m2=0x7f86d77fdbb0, request=0x7f86d77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005630a4693b00 in os_aio_handler (segment=4, m1=0x7f86d77fdba8, m2=0x7f86d77fdbb0, request=0x7f86d77fdbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005630a48e79b1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005630a478ced9 in io_handler_thread (arg=0x5630a5916680 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f86d5ffb700 (LWP 2474)): #0 0x00007f8714bdfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469af4e in os_event::timed_wait (this=0x5630a81507f0, abstime=0x7f86d5ffaba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005630a469b265 in os_event::wait_time_low (this=0x5630a81507f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005630a469b405 in os_event_wait_time_low (event=0x5630a81507f0, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005630a47895b2 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f86c3fff700 (LWP 2476)): #0 0x00007f8714bdfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469af4e in os_event::timed_wait (this=0x5630a835ad60, abstime=0x7f86c3ffed60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005630a469b265 in os_event::wait_time_low (this=0x5630a835ad60, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005630a469b405 in os_event_wait_time_low (event=0x5630a835ad60, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005630a48d131c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f86c2ffd700 (LWP 2483)): #0 0x00007f8714be2e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005630a469b935 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005630a478a911 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x00005630a478aa41 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f86c1ffb700 (LWP 2486)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469b633 in os_event::wait (this=0x5630a8150770) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005630a469b0b9 in os_event::wait_low (this=0x5630a8150770, reset_sig_count=30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005630a469b42a in os_event_wait_low (event=0x5630a8150770, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005630a47873a9 in srv_resume_thread (slot=0x5630a56d87f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005630a478afad in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f86c0ff9700 (LWP 2488)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469b633 in os_event::wait (this=0x5630a8150670) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005630a469b0b9 in os_event::wait_low (this=0x5630a8150670, reset_sig_count=41) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005630a469b42a in os_event_wait_low (event=0x5630a8150670, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005630a47873a9 in srv_resume_thread (slot=0x5630a56d8780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005630a478afad in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f86b2ffd700 (LWP 2494)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469b633 in os_event::wait (this=0x5630a81509f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005630a469b0b9 in os_event::wait_low (this=0x5630a81509f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005630a469b42a in os_event_wait_low (event=0x5630a81509f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005630a48506a2 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3099 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f8714fec880 (LWP 2037)): #0 0x00007f8712f8f20d in poll () from /lib64/libc.so.6 #1 0x00005630a3f31cea in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #2 0x00005630a3f30f85 in mysqld_main (argc=164, argv=0x5630a7e86d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #3 0x00005630a3f25125 in main (argc=25, argv=0x7ffe85afbcc8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 26 (Thread 0x7f86fd379700 (LWP 2121)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f870cac4867 in gu::Lock::wait (cond=..., this=0x7f86fd378850) at galerautils/src/gu_lock.hpp:45 #2 0x00007f870cac73ab in RecvBuf::front (this=this@entry=0x5630a803d738, timeout=...) at gcs/src/gcs_gcomm.cpp:124 #3 0x00007f870cac4f46 in gcomm_recv (backend=, msg=0x5630a800d7b0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673 #4 0x00007f870cab8eec in core_msg_recv (backend=backend@entry=0x5630a800d8f8, recv_msg=recv_msg@entry=0x5630a800d7b0, timeout=timeout@entry=9223372035999999999) at gcs/src/gcs_core.cpp:480 #5 0x00007f870cabc164 in gcs_core_recv (conn=0x5630a800d750, recv_act=recv_act@entry=0x7f86fd378d70, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1184 #6 0x00007f870cac33d8 in gcs_recv_thread (arg=0x5630a800d4d0) at gcs/src/gcs.cpp:1398 #7 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f86ffffe700 (LWP 2078)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f870cae9fd7 in gu::Lock::wait (cond=..., this=0x7f86ffffdbf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007f870caea069 in galera::ServiceThd::thd_func (arg=0x5630a80059b8) at galera/src/galera_service_thd.cpp:30 #3 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f86ddbbc700 (LWP 2639)): #0 0x00007f8712f8f20d in poll () from /lib64/libc.so.6 #1 0x00005630a4bc640e in vio_io_wait (vio=0x5630a84e3eb0, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:976 #2 0x00005630a4bc4a2e in vio_socket_io_wait (vio=0x5630a84e3eb0, event=VIO_IO_EVENT_READ) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:118 #3 0x00005630a4bc4c14 in vio_read (vio=0x5630a84e3eb0, buf=0x7f868c00a530 "\a", size=4) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:195 #4 0x00005630a44fa3bc in my_real_read (net=0x7f868c000df8, complen=0x7f86ddbbb810, header=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:962 #5 0x00005630a44fac43 in my_net_read_packet_reallen (net=0x7f868c000df8, read_from_server=1 '\001', reallen=0x7f86ddbbb878) at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1243 #6 0x00005630a44fabf6 in my_net_read_packet (net=0x7f868c000df8, read_from_server=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1227 #7 0x00005630a40563a6 in do_command (thd=0x7f868c000b00) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1229 #8 0x00005630a41c72c9 in do_handle_one_connection (connect=0x5630a845bec0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #9 0x00005630a41c700d in handle_one_connection (arg=0x5630a845bec0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #10 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a84df6e0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #11 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f870c41c700 (LWP 2123)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a4b5a799 in safe_cond_wait (cond=0x7f86f4017c68, mp=0x7f86f4017bb8, file=0x5630a4cc37c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005630a42b0384 in inline_mysql_cond_wait (that=0x7f86f4017c68, mutex=0x7f86f4017bb8, src_file=0x5630a4cc3ae0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005630a42b301c in Wsrep_thd_queue::pop_front (this=0x7f86f4017b60) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005630a42b1dfc in wsrep_post_rollback_process (post_rollbacker=0x7f86f4001d50, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x00005630a42a257f in start_wsrep_THD (arg=0x5630a805fab0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f86fdb7a700 (LWP 2083)): #0 0x00007f8712f9a483 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f870ca8b6f8 in asio::detail::epoll_reactor::run (this=0x5630a803df20, block=block@entry=true, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391 #2 0x00007f870ca8fdf0 in asio::detail::task_io_service::do_run_one (this=this@entry=0x5630a803ddc0, lock=..., this_thread=..., ec=...) at asio/asio/detail/impl/task_io_service.ipp:355 #3 0x00007f870ca900a2 in asio::detail::task_io_service::run (this=0x5630a803ddc0, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148 #4 0x00007f870caa7afd in asio::io_service::run (this=0x5630a803ce00) at asio/asio/impl/io_service.ipp:58 #5 0x00007f870caa8251 in gcomm::AsioProtonet::event_loop (this=0x5630a803cd60, period=...) at gcomm/src/asio_protonet.cpp:116 #6 0x00007f870cac62ed in GCommConn::run (this=0x5630a803d550) at gcs/src/gcs_gcomm.cpp:498 #7 0x00007f870cac9399 in GCommConn::run_fn (arg=) at gcs/src/gcs_gcomm.cpp:209 #8 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f86fc060700 (LWP 2499)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a4b5a799 in safe_cond_wait (cond=0x5630a589b1e0 , mp=0x5630a589e560 , file=0x5630a4c24928 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005630a3f5674e in inline_mysql_cond_wait (that=0x5630a589b1e0 , mutex=0x5630a589e560 , src_file=0x5630a4c25490 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005630a3f57fe0 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a8408ae0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f86ddc07700 (LWP 2502)): #0 0x00007f8714be24ed in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f8714bdddcb in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007f8714bddc98 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00005630a4b59f60 in safe_mutex_lock (mp=0x5630a58d3b20 , my_flags=0, file=0x5630a4df6850 "/home/stepan/galera/git/10.4/server/sql/log.cc", line=8057) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:293 #4 0x00005630a44a71a7 in inline_mysql_mutex_lock (that=0x5630a58d3b20 , src_file=0x5630a4df6850 "/home/stepan/galera/git/10.4/server/sql/log.cc", src_line=8057) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:695 #5 0x00005630a44be3ec in MYSQL_BIN_LOG::trx_group_commit_leader (this=0x5630a58d2400 , leader=0x7f86ddc03fa0) at /home/stepan/galera/git/10.4/server/sql/log.cc:8057 #6 0x00005630a44bd220 in MYSQL_BIN_LOG::write_transaction_to_binlog_events (this=0x5630a58d2400 , entry=0x7f86ddc03fa0) at /home/stepan/galera/git/10.4/server/sql/log.cc:7700 #7 0x00005630a44bc803 in MYSQL_BIN_LOG::write_transaction_to_binlog (this=0x5630a58d2400 , thd=0x7f86d800a040, cache_mngr=0x7f86d807efa0, end_ev=0x7f86ddc04140, all=true, using_stmt_cache=true, using_trx_cache=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:7370 #8 0x00005630a44ac99e in binlog_flush_cache (thd=0x7f86d800a040, cache_mngr=0x7f86d807efa0, end_ev=0x7f86ddc04140, all=true, using_stmt=true, using_trx=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:1787 #9 0x00005630a44acfdc in binlog_commit_flush_xid_caches (thd=0x7f86d800a040, cache_mngr=0x7f86d807efa0, all=true, xid=13) at /home/stepan/galera/git/10.4/server/sql/log.cc:1893 #10 0x00005630a44c29d7 in MYSQL_BIN_LOG::log_and_order (this=0x5630a58d2400 , thd=0x7f86d800a040, xid=13, all=true, need_prepare_ordered=false, need_commit_ordered=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:9642 #11 0x00005630a43968c4 in ha_commit_trans (thd=0x7f86d800a040, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1591 #12 0x00005630a41dc0ee in trans_commit (thd=0x7f86d800a040) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #13 0x00005630a4065b3e in mysql_execute_command (thd=0x7f86d800a040) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:5828 #14 0x00005630a406cb95 in mysql_parse (thd=0x7f86d800a040, rawbuf=0x7f86d8017388 "COMMIT", length=6, parser_state=0x7f86ddc060d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #15 0x00005630a406c1c8 in wsrep_mysql_parse (thd=0x7f86d800a040, rawbuf=0x7f86d8017388 "COMMIT", length=6, parser_state=0x7f86ddc060d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #16 0x00005630a4058085 in dispatch_command (command=COM_QUERY, thd=0x7f86d800a040, packet=0x7f86d8028821 "COMMIT", packet_length=6, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #17 0x00005630a4056a6d in do_command (thd=0x7f86d800a040) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #18 0x00005630a41c72c9 in do_handle_one_connection (connect=0x5630a845bec0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #19 0x00005630a41c700d in handle_one_connection (arg=0x5630a845bec0) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #20 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a84e3eb0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #21 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f86fc0ab700 (LWP 2498)): #0 0x00007f8712ed340a in sigwaitinfo () from /lib64/libc.so.6 #1 0x00005630a3f251a4 in my_sigwait (set=0x7f86fc0aacf0, sig=0x7f86fc0aacb4, code=0x7f86fc0aacb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x00005630a3f2b7e4 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a83c9aa0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f86c27fc700 (LWP 2485)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469b633 in os_event::wait (this=0x5630a8379b10) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005630a469b0b9 in os_event::wait_low (this=0x5630a8379b10, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005630a469b42a in os_event_wait_low (event=0x5630a8379b10, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005630a47951d8 in sync_array_wait_event (arr=0x5630a8127ae0, cell=@0x7f86c27fb5d0: 0x5630a8127c50) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005630a45e65f3 in TTASEventMutex::enter (this=0x5630a83799b8, max_spins=60, max_delay=4, filename=0x5630a4ee79d0 "/home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc", line=108) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:478 #6 0x00005630a45e484e in PolicyMutex >::enter (this=0x5630a83799b8, n_spins=30, n_delay=4, name=0x5630a4ee79d0 "/home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc", line=108) at /home/stepan/galera/git/10.4/server/storage/innobase/include/ib0mutex.h:595 #7 0x00005630a47b6438 in TrxUndoRsegsIterator::set_next (this=0x5630a56d9218 ) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:108 #8 0x00005630a47b462b in trx_purge_choose_next_log () at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:969 #9 0x00005630a47b4881 in trx_purge_get_next_rec (n_pages_handled=0x7f86c27fbca0, heap=0x5630a8375fa0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1042 #10 0x00005630a47b4aca in trx_purge_fetch_next_rec (roll_ptr=0x5630a8376030, n_pages_handled=0x7f86c27fbca0, heap=0x5630a8375fa0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1112 #11 0x00005630a47b4df1 in trx_purge_attach_undo_recs (n_purge_threads=4) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1185 #12 0x00005630a47b52d7 in trx_purge (n_purge_threads=4, truncate=false) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:1313 #13 0x00005630a478b280 in srv_do_purge (n_total_purged=0x7f86c27fbd98) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2579 #14 0x00005630a478b6ad in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2702 #15 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f86fc0f6700 (LWP 2496)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a4b5a799 in safe_cond_wait (cond=0x5630a58d3488 , mp=0x5630a58d33d8 , file=0x5630a4df5e48 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005630a44a76e4 in inline_mysql_cond_wait (that=0x5630a58d3488 , mutex=0x5630a58d33d8 , src_file=0x5630a4df6850 "/home/stepan/galera/git/10.4/server/sql/log.cc", src_line=9926) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x00005630a44c35ac in binlog_background_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/log.cc:9926 #4 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a83a5970) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f86d7fff700 (LWP 2424)): #0 0x00007f87147ac644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005630a468e950 in LinuxAIOHandler::collect (this=0x7f86d7ffeb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005630a468ee79 in LinuxAIOHandler::poll (this=0x7f86d7ffeb00, m1=0x7f86d7ffeba8, m2=0x7f86d7ffebb0, request=0x7f86d7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005630a468efdc in os_aio_linux_handler (global_segment=3, m1=0x7f86d7ffeba8, m2=0x7f86d7ffebb0, request=0x7f86d7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005630a4693b00 in os_aio_handler (segment=3, m1=0x7f86d7ffeba8, m2=0x7f86d7ffebb0, request=0x7f86d7ffebe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005630a48e79b1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005630a478ced9 in io_handler_thread (arg=0x5630a5916678 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f86b3fff700 (LWP 2492)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469b633 in os_event::wait (this=0x5630a81508f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005630a469b0b9 in os_event::wait_low (this=0x5630a81508f0, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005630a469b42a in os_event_wait_low (event=0x5630a81508f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005630a4867bce in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f86dca05700 (LWP 2423)): #0 0x00007f87147ac644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005630a468e950 in LinuxAIOHandler::collect (this=0x7f86dca04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005630a468ee79 in LinuxAIOHandler::poll (this=0x7f86dca04b00, m1=0x7f86dca04ba8, m2=0x7f86dca04bb0, request=0x7f86dca04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005630a468efdc in os_aio_linux_handler (global_segment=2, m1=0x7f86dca04ba8, m2=0x7f86dca04bb0, request=0x7f86dca04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005630a4693b00 in os_aio_handler (segment=2, m1=0x7f86dca04ba8, m2=0x7f86dca04bb0, request=0x7f86dca04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005630a48e79b1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005630a478ced9 in io_handler_thread (arg=0x5630a5916670 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f86c17fa700 (LWP 2487)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469b633 in os_event::wait (this=0x5630a81506f0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005630a469b0b9 in os_event::wait_low (this=0x5630a81506f0, reset_sig_count=42) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005630a469b42a in os_event_wait_low (event=0x5630a81506f0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005630a47873a9 in srv_resume_thread (slot=0x5630a56d87b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x00005630a478afad in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f870c467700 (LWP 2122)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a4b5a799 in safe_cond_wait (cond=0x7f86e8015708, mp=0x7f86e8015658, file=0x5630a4cc37c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005630a42b0384 in inline_mysql_cond_wait (that=0x7f86e8015708, mutex=0x7f86e8015658, src_file=0x5630a4cc3ae0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005630a42b301c in Wsrep_thd_queue::pop_front (this=0x7f86e8015600) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x00005630a42b1a3c in wsrep_rollback_process (rollbacker=0x7f86e8000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x00005630a42a257f in start_wsrep_THD (arg=0x5630a805e650) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f86df059700 (LWP 2420)): #0 0x00007f8714bdfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a4b5aa9b in safe_cond_timedwait (cond=0x5630a6127ee0 , mp=0x5630a6127e20 , abstime=0x7f86df058cb0, file=0x5630a4fa0370 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005630a49c48cd in inline_mysql_cond_timedwait (that=0x5630a6127ee0 , mutex=0x5630a6127e20 , abstime=0x7f86df058cb0, src_file=0x5630a4fa03b8 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005630a49c4df1 in my_service_thread_sleep (control=0x5630a56dcb00 , sleep_time=29000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x00005630a49b928a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a8108870) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f86c37fe700 (LWP 2477)): #0 0x00007f8714bdfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469af4e in os_event::timed_wait (this=0x5630a82f8360, abstime=0x7f86c37fdb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005630a469b265 in os_event::wait_time_low (this=0x5630a82f8360, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005630a469b405 in os_event_wait_time_low (event=0x5630a82f8360, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005630a47e9857 in ib_wqueue_timedwait (wq=0x5630a82f7af0, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x00005630a492bd01 in fts_optimize_thread (arg=0x5630a82f7af0) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f870d634700 (LWP 2073)): #0 0x00007f8714bdfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a4b5aa9b in safe_cond_timedwait (cond=0x5630a613a680 , mp=0x5630a613a5c0 , abstime=0x7f870d633d70, file=0x5630a4fe45a0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x00005630a4b5ba55 in inline_mysql_cond_timedwait (that=0x5630a613a680 , mutex=0x5630a613a5c0 , abstime=0x7f870d633d70, src_file=0x5630a4fe45e8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x00005630a4b5c5b8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a7fce860) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f86d4ff9700 (LWP 2475)): #0 0x00007f8714bdfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469af4e in os_event::timed_wait (this=0x5630a8150870, abstime=0x7f86d4ff8d20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005630a469b265 in os_event::wait_time_low (this=0x5630a8150870, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005630a469b405 in os_event_wait_time_low (event=0x5630a8150870, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005630a4789090 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f86b27fc700 (LWP 2495)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a4b5a799 in safe_cond_wait (cond=0x7f86b27fbc70, mp=0x7f86b27fbcb0, file=0x5630a4e278a0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x00005630a45b2513 in inline_mysql_cond_wait (that=0x7f86b27fbc70, mutex=0x7f86b27fbcb0, src_file=0x5630a4e293e0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x00005630a45b7132 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x00005630a4aed261 in pfs_spawn_thread (arg=0x5630a830a500) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f86d57fa700 (LWP 2473)): #0 0x00007f8714bdfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469af4e in os_event::timed_wait (this=0x5630a8328d40, abstime=0x7f86d57f9d40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x00005630a469b265 in os_event::wait_time_low (this=0x5630a8328d40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x00005630a469b405 in os_event_wait_time_low (event=0x5630a8328d40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x00005630a4667e6a in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f86b37fe700 (LWP 2493)): #0 0x00007f8714be2e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00005630a469b935 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x00005630a48424ff in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f86d67fc700 (LWP 2427)): #0 0x00007f8714bdf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005630a469b633 in os_event::wait (this=0x5630a823f110) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x00005630a469b0b9 in os_event::wait_low (this=0x5630a823f110, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x00005630a469b42a in os_event_wait_low (event=0x5630a823f110, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x00005630a47951d8 in sync_array_wait_event (arr=0x5630a8127ae0, cell=@0x7f86d67fb680: 0x5630a8127bd0) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0arr.cc:469 #5 0x00005630a4798a57 in rw_lock_sx_lock_func (lock=0x7f86ddc0e880, pass=2, file_name=0x5630a4f2bea0 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1143) at /home/stepan/galera/git/10.4/server/storage/innobase/sync/sync0rw.cc:837 #6 0x00005630a4868555 in pfs_rw_lock_sx_lock_func (lock=0x7f86ddc0e880, pass=2, file_name=0x5630a4f2bea0 "/home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc", line=1143) at /home/stepan/galera/git/10.4/server/storage/innobase/include/sync0rw.ic:686 #7 0x00005630a486cbde in buf_flush_page (buf_pool=0x5630a82360f0, bpage=0x7f86ddc0e7b0, flush_type=BUF_FLUSH_LIST, sync=false) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1143 #8 0x00005630a486d628 in buf_flush_try_neighbors (page_id=..., flush_type=BUF_FLUSH_LIST, n_flushed=18, n_to_flush=200) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1388 #9 0x00005630a486d9ab in buf_flush_page_and_try_neighbors (bpage=0x7f86ddc0e7b0, flush_type=BUF_FLUSH_LIST, n_to_flush=200, count=0x7f86d67fb8b8) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1460 #10 0x00005630a486e5c4 in buf_do_flush_list_batch (buf_pool=0x5630a82360f0, min_n=200, lsn_limit=18446744073709551615) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1718 #11 0x00005630a486ebbc in buf_flush_batch (buf_pool=0x5630a82360f0, flush_type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7f86d67fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1788 #12 0x00005630a486f27c in buf_flush_do_batch (buf_pool=0x5630a82360f0, type=BUF_FLUSH_LIST, min_n=200, lsn_limit=18446744073709551615, n=0x7f86d67fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:1946 #13 0x00005630a486f550 in buf_flush_lists (min_n=200, lsn_limit=18446744073709551615, n_processed=0x7f86d67fbbd0) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2052 #14 0x00005630a487329b in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3204 #15 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f86dda07700 (LWP 2421)): #0 0x00007f87147ac644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005630a468e950 in LinuxAIOHandler::collect (this=0x7f86dda06b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005630a468ee79 in LinuxAIOHandler::poll (this=0x7f86dda06b00, m1=0x7f86dda06ba8, m2=0x7f86dda06bb0, request=0x7f86dda06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005630a468efdc in os_aio_linux_handler (global_segment=0, m1=0x7f86dda06ba8, m2=0x7f86dda06bb0, request=0x7f86dda06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005630a4693b00 in os_aio_handler (segment=0, m1=0x7f86dda06ba8, m2=0x7f86dda06bb0, request=0x7f86dda06be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005630a48e79b1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005630a478ced9 in io_handler_thread (arg=0x5630a5916660 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f86dd206700 (LWP 2422)): #0 0x00007f87147ac644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00005630a468e950 in LinuxAIOHandler::collect (this=0x7f86dd205b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x00005630a468ee79 in LinuxAIOHandler::poll (this=0x7f86dd205b00, m1=0x7f86dd205ba8, m2=0x7f86dd205bb0, request=0x7f86dd205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x00005630a468efdc in os_aio_linux_handler (global_segment=1, m1=0x7f86dd205ba8, m2=0x7f86dd205bb0, request=0x7f86dd205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x00005630a4693b00 in os_aio_handler (segment=1, m1=0x7f86dd205ba8, m2=0x7f86dd205bb0, request=0x7f86dd205be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x00005630a48e79b1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x00005630a478ced9 in io_handler_thread (arg=0x5630a5916668 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f870c3d1700 (LWP 2124)): #0 0x00007f8714be09d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00005630a4b5516a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x00005630a43917ff in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007f8712ed2207 in raise () from /lib64/libc.so.6 #5 0x00007f8712ed38f8 in abort () from /lib64/libc.so.6 #6 0x00007f8712ecb026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007f8712ecb0d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00005630a47cd1ab in trx_rseg_update_wsrep_checkpoint (rseg_header=0x7f86ddcfc026 "", xid=0x5630a8149540, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0rseg.cc:92 #9 0x00005630a47b1e5e in trx_purge_add_undo_to_history (trx=0x7f86de459140, undo=@0x7f86de45a160: 0x5630a8379bc0, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0purge.cc:286 #10 0x00005630a47d6fed in trx_write_serialisation_history (trx=0x7f86de459140, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1092 #11 0x00005630a47d8acd in trx_commit_low (trx=0x7f86de459140, mtr=0x7f870c3cd2b0) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1503 #12 0x00005630a47d8c1f in trx_commit (trx=0x7f86de459140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1570 #13 0x00005630a47d9029 in trx_commit_for_mysql (trx=0x7f86de459140) at /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0trx.cc:1706 #14 0x00005630a45bccf2 in innobase_commit_low (trx=0x7f86de459140) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4387 #15 0x00005630a45bd0b2 in innobase_commit_ordered_2 (trx=0x7f86de459140, thd=0x7f86ec000b00) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4509 #16 0x00005630a45bd2e8 in innobase_commit_ordered (hton=0x5630a8124ba0, thd=0x7f86ec000b00, all=true) at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:4569 #17 0x00005630a44c0752 in TC_LOG::run_commit_ordered (this=0x5630a58d2400 , thd=0x7f86ec000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:8859 #18 0x00005630a44be5eb in MYSQL_BIN_LOG::trx_group_commit_leader (this=0x5630a58d2400 , leader=0x7f870c3cdd20) at /home/stepan/galera/git/10.4/server/sql/log.cc:8109 #19 0x00005630a44bd220 in MYSQL_BIN_LOG::write_transaction_to_binlog_events (this=0x5630a58d2400 , entry=0x7f870c3cdd20) at /home/stepan/galera/git/10.4/server/sql/log.cc:7700 #20 0x00005630a44bc803 in MYSQL_BIN_LOG::write_transaction_to_binlog (this=0x5630a58d2400 , thd=0x7f86ec000b00, cache_mngr=0x7f86ec0550c0, end_ev=0x7f870c3cdec0, all=true, using_stmt_cache=true, using_trx_cache=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:7370 #21 0x00005630a44ac99e in binlog_flush_cache (thd=0x7f86ec000b00, cache_mngr=0x7f86ec0550c0, end_ev=0x7f870c3cdec0, all=true, using_stmt=true, using_trx=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:1787 #22 0x00005630a44acfdc in binlog_commit_flush_xid_caches (thd=0x7f86ec000b00, cache_mngr=0x7f86ec0550c0, all=true, xid=11) at /home/stepan/galera/git/10.4/server/sql/log.cc:1893 #23 0x00005630a44c29d7 in MYSQL_BIN_LOG::log_and_order (this=0x5630a58d2400 , thd=0x7f86ec000b00, xid=11, all=true, need_prepare_ordered=false, need_commit_ordered=true) at /home/stepan/galera/git/10.4/server/sql/log.cc:9642 #24 0x00005630a43968c4 in ha_commit_trans (thd=0x7f86ec000b00, all=true) at /home/stepan/galera/git/10.4/server/sql/handler.cc:1591 #25 0x00005630a41dc0ee in trans_commit (thd=0x7f86ec000b00) at /home/stepan/galera/git/10.4/server/sql/transaction.cc:313 #26 0x00005630a428fa49 in Wsrep_high_priority_service::commit (this=0x7f870c3d04d0, ws_handle=..., ws_meta=...) at /home/stepan/galera/git/10.4/server/sql/wsrep_high_priority_service.cc:318 #27 0x00005630a4bf7f96 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:194 #28 0x00005630a4bfb521 in wsrep::server_state::on_apply (this=0x5630a7fcf7d0, high_priority_service=..., ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:923 #29 0x00005630a4c0fbcb in wsrep::high_priority_service::apply (this=0x7f870c3d04d0, ws_handle=..., ws_meta=..., data=...) at /home/stepan/galera/git/10.4/server/wsrep-lib/include/wsrep/high_priority_service.hpp:46 #30 0x00005630a4c0d47b in (anonymous namespace)::apply_cb (ctx=0x7f870c3d04d0, wsh=0x7f870c3cf3e0, flags=65, buf=0x7f870c3cf420, meta=0x7f870c3cf6b0, exit_loop=0x7f870c3cf5ef) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:460 #31 0x00007f870cad33b1 in galera::TrxHandleSlave::apply (this=this@entry=0x7f86ec02e4e0, recv_ctx=recv_ctx@entry=0x7f870c3d04d0, apply_cb=0x5630a4c0d258 <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, meta=..., exit_loop=exit_loop@entry=@0x7f870c3cf5ef: false) at galera/src/trx_handle.cpp:418 #32 0x00007f870cb05d92 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x5630a80052a0, recv_ctx=recv_ctx@entry=0x7f870c3d04d0, ts=...) at galera/src/replicator_smm.cpp:489 #33 0x00007f870cb0ba40 in galera::ReplicatorSMM::process_trx (this=0x5630a80052a0, recv_ctx=0x7f870c3d04d0, ts_ptr=...) at galera/src/replicator_smm.cpp:2116 #34 0x00007f870caecc6b in galera::GcsActionSource::process_writeset (this=this@entry=0x5630a8035de0, recv_ctx=0x7f870c3d04d0, act=..., exit_loop=@0x7f870c3d029e: false) at galera/src/gcs_action_source.cpp:62 #35 0x00007f870caece45 in galera::GcsActionSource::dispatch (this=this@entry=0x5630a8035de0, recv_ctx=recv_ctx@entry=0x7f870c3d04d0, act=..., exit_loop=@0x7f870c3d029e: false) at galera/src/gcs_action_source.cpp:109 #36 0x00007f870caed21d in galera::GcsActionSource::process (this=0x5630a8035de0, recv_ctx=0x7f870c3d04d0, exit_loop=@0x7f870c3d029e: false) at galera/src/gcs_action_source.cpp:182 #37 0x00007f870cb0c033 in galera::ReplicatorSMM::async_recv (this=0x5630a80052a0, recv_ctx=0x7f870c3d04d0) at galera/src/replicator_smm.cpp:383 #38 0x00005630a4c0dffe in wsrep::wsrep_provider_v26::run_applier (this=0x5630a7f7aee0, applier_ctx=0x7f870c3d04d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #39 0x00005630a42b0932 in wsrep_replication_process (thd=0x7f86ec000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #40 0x00005630a42a257f in start_wsrep_THD (arg=0x5630a8025e00) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #41 0x00007f8714bdbdd5 in start_thread () from /lib64/libpthread.so.0 #42 0x00007f8712f99ead in clone () from /lib64/libc.so.6 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_log_bin-innodb/mysqld.2/data/core.2037 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_log_bin-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_log_bin-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.galera_sr_log_bin mysqld: /home/stepan/galera/git/10.4/server/storage/innobase/trx/trx0rseg.cc:92: void trx_rseg_update_wsrep_checkpoint(trx_rsegf_t*, const XID*, mtr_t*): Assertion `xid_seqno > wsrep_seqno' failed. Attempting backtrace. You can use the following information to find out galera_sr.mysql-wsrep-features#136 'innodb' [ pass ] 2106 galera_sr.galera_sr_v1_row_events 'innodb' [ pass ] 2111 galera_sr.GCF-1018B 'innodb' [ pass ] 21402 galera_sr.GCF-437 'innodb' [ fail ] Test ended at 2019-02-06 13:06:03 CURRENT_TEST: galera_sr.GCF-437 mysqltest: At line 13: query 'INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5' succeeded - should have failed with errno 1534... The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE ten (f1 INTEGER) ENGINE=MyISAM; INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(512)) ENGINE=InnoDB; SET SESSION wsrep_trx_fragment_size = 2 * 1024 * 1024; INSERT INTO t1 (f2) SELECT REPEAT('x', 512) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-437-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.GCF-437-innodb/' galera_sr.galera_sr_kill_all_nobootstrap 'innodb' [ pass ] 13544 galera_sr.galera_sr_kill_all_pcrecovery 'innodb' [ pass ] 18615 galera_sr.galera_sr_load_data_splitting 'innodb' [ fail ] Test ended at 2019-02-06 13:12:36 CURRENT_TEST: galera_sr.galera_sr_load_data_splitting --- /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_load_data_splitting.result 2019-01-23 13:58:38.972427700 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_load_data_splitting.reject 2019-02-06 13:12:36.095265000 +0200 @@ -1,9 +1,19 @@ +connection node_2; +connection node_1; SET SESSION wsrep_trx_fragment_size = 512; SET GLOBAL wsrep_load_data_splitting = TRUE; +Warnings: +Warning 1287 '@@wsrep_load_data_splitting' is deprecated and will be removed in a future release CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; +connection node_2; +connection node_1; +connection node_2; SELECT COUNT(*) = 95000 FROM t1; COUNT(*) = 95000 1 wsrep_last_committed_diff -1 +0 +connection node_1; +Warnings: +Warning 1287 '@@wsrep_load_data_splitting' is deprecated and will be removed in a future release DROP TABLE t1; mysqltest: Result length mismatch - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_load_data_splitting-innodb/' galera_sr.galera_sr_many_fragments 'innodb' [ pass ] 149222 galera_sr.mysql-wsrep-features#8 'innodb' [ pass ] 210582 galera_sr.mysql-wsrep-features#9 'innodb' [ pass ] 19123 galera_sr.GCF-1008 'innodb' [ pass ] 4228 galera_sr.GCF-1018 'innodb' [ pass ] 2143 galera_sr.GCF-1043A 'innodb' [ pass ] 81329 galera_sr.GCF-1043B 'innodb' [ pass ] 65803 galera_sr.GCF-1051 'innodb' [ pass ] 2236 galera_sr.GCF-1060 'innodb' [ pass ] 19876 galera_sr.GCF-561 'innodb' [ pass ] 2397 galera_sr.GCF-571 'innodb' [ pass ] 2295 galera_sr.GCF-572 'innodb' [ pass ] 2221 galera_sr.GCF-580 'innodb' [ pass ] 2212 galera_sr.GCF-585 'innodb' [ pass ] 6348 galera_sr.GCF-597 'innodb' [ pass ] 2202 galera_sr.GCF-620 'innodb' [ pass ] 2152 galera_sr.GCF-623 'innodb' [ pass ] 2191 galera_sr.GCF-627 'innodb' [ pass ] 2254 galera_sr.GCF-845 'innodb' [ pass ] 4112 galera_sr.GCF-851 'innodb' [ pass ] 2094 galera_sr.GCF-867 'innodb' [ pass ] 2978 galera_sr.GCF-889 'innodb' [ pass ] 2092 galera_sr.GCF-900 'innodb' [ pass ] 2120 galera_sr.galera-features#56 'innodb' [ pass ] 246688 galera_sr.galera_sr_bf_abort 'innodb' [ pass ] 4065 galera_sr.galera_sr_blob 'innodb' [ pass ] 2233 galera_sr.galera_sr_cc_master 'innodb' [ fail ] Test ended at 2019-02-06 13:27:26 CURRENT_TEST: galera_sr.galera_sr_cc_master Server [mysqld.2 - pid: 42204, winpid: 42204, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 2019-02-06 13:26:59 20030 [Note] WSREP: Stop replication 2019-02-06 13:26:59 20030 [Note] WSREP: Server status change synced -> disconnecting 2019-02-06 13:26:59 20030 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:26:59 20030 [Note] WSREP: Closing send monitor... 2019-02-06 13:26:59 20030 [Note] WSREP: Closed send monitor. 2019-02-06 13:26:59 20030 [Note] WSREP: gcomm: terminating thread 2019-02-06 13:26:59 20030 [Note] WSREP: gcomm: joining thread 2019-02-06 13:26:59 20030 [Note] WSREP: gcomm: closing backend 2019-02-06 13:27:02 20030 [Note] WSREP: (1c5dff22, 'tcp://0.0.0.0:16005') connection to peer 1be2a2bb with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-06 13:27:02 20030 [Note] WSREP: (1c5dff22, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16002 2019-02-06 13:27:04 20030 [Note] WSREP: (1c5dff22, 'tcp://0.0.0.0:16005') reconnecting to 1be2a2bb (tcp://127.0.0.1:16002), attempt 0 2019-02-06 13:27:07 20030 [Note] WSREP: (1c5dff22, 'tcp://0.0.0.0:16005') connection to peer 1be2a2bb with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-06 13:27:09 20030 [Note] WSREP: (1c5dff22, 'tcp://0.0.0.0:16005') connection established to 1be2a2bb tcp://127.0.0.1:16002 2019-02-06 13:27:09 20030 [Note] WSREP: evs::proto(1c5dff22, LEAVING, view_id(REG,1be2a2bb,4)) suspecting node: 1be2a2bb 2019-02-06 13:27:09 20030 [Note] WSREP: evs::proto(1c5dff22, LEAVING, view_id(REG,1be2a2bb,4)) suspected node without join message, declaring inactive 2019-02-06 13:27:09 20030 [Note] WSREP: view(view_id(NON_PRIM,1be2a2bb,4) memb { 1c5dff22,0 } joined { } left { } partitioned { 1be2a2bb,0 }) 2019-02-06 13:27:09 20030 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-06 13:27:09 20030 [Note] WSREP: view((empty)) 2019-02-06 13:27:09 20030 [Note] WSREP: gcomm: closed 2019-02-06 13:27:09 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-06 13:27:09 0 [Note] WSREP: Writing down CC checksum: 2fdd2b69 4df59677 d2a140c6 7f7d3d83 at offset 144 2019-02-06 13:27:09 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-06 13:27:09 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-06 13:27:09 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-06 13:27:09 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 70313) 2019-02-06 13:27:09 0 [Note] WSREP: Received self-leave message. 2019-02-06 13:27:09 0 [Note] WSREP: Writing down CC checksum: 3655d7eb ce05f785 9342a558 54a981f1 at offset 64 2019-02-06 13:27:09 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-06 13:27:09 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-06 13:27:09 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-06 13:27:09 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-06 13:27:09 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-06 13:27:09 9997 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-06 13:27:09 9997 [Note] WSREP: ####### drain monitors upto 70313 2019-02-06 13:27:09 9997 [Note] WSREP: ####### My UUID: 1c5dff22-2a00-11e9-9663-7a70c55f359d 2019-02-06 13:27:09 9997 [Note] WSREP: ####### ST not required 2019-02-06 13:27:09 20030 [Note] WSREP: recv_thread() joined. 2019-02-06 13:27:09 20030 [Note] WSREP: Closing replication queue. 2019-02-06 13:27:09 20030 [Note] WSREP: Closing slave action queue. 2019-02-06 13:27:09 9997 [Note] WSREP: ================================================ View: id: 1be456f0-2a00-11e9-83ff-be2e9fa77ae4:-1 seqno: -1 status: 1 prococol_version: 4 own_index: 0 final: 0 members 2019-02-06 13:27:09 9997 [Note] WSREP: id: 1c5dff22-2a00-11e9-9663-7a70c55f359d name: cnt7glr11.localdomain 2019-02-06 13:27:09 9997 [Note] WSREP: ================================================= 2019-02-06 13:27:09 9997 [Note] WSREP: Non-primary view 2019-02-06 13:27:09 9997 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:27:09 9997 [Note] WSREP: ####### processing CC -1, local, ordered 2019-02-06 13:27:09 9997 [Note] WSREP: ####### drain monitors upto 70313 2019-02-06 13:27:09 9997 [Note] WSREP: ####### My UUID: 1c5dff22-2a00-11e9-9663-7a70c55f359d 2019-02-06 13:27:09 9997 [Note] WSREP: ####### ST not required 2019-02-06 13:27:09 9997 [Note] WSREP: ================================================ View: id: 1be456f0-2a00-11e9-83ff-be2e9fa77ae4:-1 seqno: -1 status: 1 prococol_version: 4 own_index: -1 final: 1 members 2019-02-06 13:27:09 9997 [Note] WSREP: ================================================= 2019-02-06 13:27:09 9997 [Note] WSREP: Non-primary view 2019-02-06 13:27:09 9997 [Note] WSREP: Server status change disconnecting -> disconnected 2019-02-06 13:27:09 9997 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:27:09 9997 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:27:09 9997 [Note] WSREP: Applier thread exiting 0 2019-02-06 13:27:09 20024 [Warning] WSREP: Failed to replicate rollback fragment for 33472: 3 2019-02-06 13:27:11 0 [Note] WSREP: rollbacker thread exiting 2019-02-06 13:27:11 20030 [Note] WSREP: Stop replication 2019-02-06 13:27:13 20031 [Note] WSREP: rollbacker thread exiting 2019-02-06 13:27:13 20030 [Note] WSREP: Start replication 2019-02-06 13:27:13 20030 [Note] WSREP: Connecting with bootstrap option: 0 2019-02-06 13:27:13 20030 [Note] WSREP: Setting GCS initial position to 1be456f0-2a00-11e9-83ff-be2e9fa77ae4:70313 2019-02-06 13:27:13 20030 [Note] WSREP: protonet asio version 0 2019-02-06 13:27:13 20030 [Note] WSREP: Using CRC-32C for message checksums. 2019-02-06 13:27:13 20030 [Note] WSREP: backend: asio 2019-02-06 13:27:13 20030 [Note] WSREP: gcomm thread scheduling priority set to other:0 2019-02-06 13:27:13 20030 [Warning] WSREP: access file(/home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data//gvwstate.dat) failed(No such file or directory) 2019-02-06 13:27:13 20030 [Note] WSREP: restore pc from disk failed 2019-02-06 13:27:13 20030 [Note] WSREP: GMCast version 0 2019-02-06 13:27:13 20030 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') listening at tcp://0.0.0.0:16005 2019-02-06 13:27:13 20030 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') multicast: , ttl: 1 2019-02-06 13:27:13 20030 [Note] WSREP: EVS version 1 2019-02-06 13:27:13 20030 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '127.0.0.1:16002' 2019-02-06 13:27:13 20030 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') connection established to 1be2a2bb tcp://127.0.0.1:16002 2019-02-06 13:27:13 20030 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: 2019-02-06 13:27:13 20030 [Note] WSREP: EVS version upgrade 0 -> 1 2019-02-06 13:27:13 20030 [Note] WSREP: declaring 1be2a2bb at tcp://127.0.0.1:16002 stable 2019-02-06 13:27:13 20030 [Note] WSREP: PC protocol upgrade 0 -> 1 2019-02-06 13:27:13 20030 [Note] WSREP: Node 1be2a2bb state prim 2019-02-06 13:27:13 20030 [Note] WSREP: view(view_id(PRIM,1be2a2bb,6) memb { 1be2a2bb,0 26384d3a,0 } joined { } left { } partitioned { }) 2019-02-06 13:27:13 20030 [Note] WSREP: save pc into disk 2019-02-06 13:27:14 20030 [Note] WSREP: gcomm: connected 2019-02-06 13:27:14 20030 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2019-02-06 13:27:14 20030 [Note] WSREP: Shifting CLOSED -> OPEN (TO: -1) 2019-02-06 13:27:14 20030 [Note] WSREP: Opened channel 'my_wsrep_cluster' 2019-02-06 13:27:14 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 2019-02-06 13:27:14 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2019-02-06 13:27:14 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 268743d7-2a02-11e9-b2b8-5be809cf85b1 2019-02-06 13:27:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: 268743d7-2a02-11e9-b2b8-5be809cf85b1 from 0 (cnt7glr11.localdomain) 2019-02-06 13:27:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: 268743d7-2a02-11e9-b2b8-5be809cf85b1 from 1 (cnt7glr11.localdomain) 2019-02-06 13:27:14 0 [Note] WSREP: Quorum results: version = 5, component = PRIMARY, conf_id = 5, members = 1/2 (joined/total), act_id = 70314, last_appl. = 70253, protocols = 1/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 1be456f0-2a00-11e9-83ff-be2e9fa77ae4 2019-02-06 13:27:14 0 [Note] WSREP: Writing down CC checksum: a5dddcc3 1ff0c3fa 66d5c583 a9e572b8 at offset 232 2019-02-06 13:27:14 0 [Note] WSREP: Flow-control interval: [23, 23] 2019-02-06 13:27:14 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-06 13:27:14 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 70315) 2019-02-06 13:27:14 20033 [Note] WSREP: Starting applier thread 20033 2019-02-06 13:27:14 20033 [Note] WSREP: ####### processing CC 70315, local, ordered 2019-02-06 13:27:14 20033 [Note] WSREP: ####### drain monitors upto 70313 2019-02-06 13:27:14 20033 [Note] WSREP: REPL Protocols: 10 (5, 3) 2019-02-06 13:27:14 20033 [Note] WSREP: ####### My UUID: 26384d3a-2a02-11e9-9260-dec0b4789562 2019-02-06 13:27:14 20033 [Note] WSREP: Server cnt7glr11.localdomain connected to cluster at position 1be456f0-2a00-11e9-83ff-be2e9fa77ae4:70315 with ID 26384d3a-2a02-11e9-9260-dec0b4789562 2019-02-06 13:27:14 20033 [Note] WSREP: Server status change disconnected -> connected 2019-02-06 13:27:14 20033 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:27:14 20033 [Note] WSREP: State transfer required: Group state: 1be456f0-2a00-11e9-83ff-be2e9fa77ae4:70315 Local state: 1be456f0-2a00-11e9-83ff-be2e9fa77ae4:70313 2019-02-06 13:27:14 20033 [Note] WSREP: Server status change connected -> joiner 2019-02-06 13:27:14 20033 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2019-02-06 13:27:14 20033 [Note] WSREP: You have configured 'rsync' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2019-02-06 13:27:14 20033 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:0 (proto: 10), state transfer needed: yes 2019-02-06 13:27:14 0 [Note] WSREP: Service thread queue flushed. 2019-02-06 13:27:14 20033 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:0, protocol version: 5 2019-02-06 13:27:14 20033 [Note] WSREP: ####### IST uuid:1be456f0-2a00-11e9-83ff-be2e9fa77ae4 f: 70314, l: 70315, p: 10 2019-02-06 13:27:14 20033 [Note] WSREP: IST receiver addr using tcp://127.0.0.1:16006 2019-02-06 13:27:14 20033 [Note] WSREP: Prepared IST receiver for 70314-70315, listening at: tcp://127.0.0.1:16006 2019-02-06 13:27:14 0 [Note] WSREP: Member 1.0 (cnt7glr11.localdomain) requested state transfer from '*any*'. Selected 0.0 (cnt7glr11.localdomain)(SYNCED) as donor. 2019-02-06 13:27:14 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 70315) 2019-02-06 13:27:14 20033 [Note] WSREP: Requesting state transfer: success, donor: 0 2019-02-06 13:27:14 20033 [Note] WSREP: Receiving IST: 2 writesets, seqnos 70314-70315 2019-02-06 13:27:14 0 [Note] WSREP: ####### IST applying starts with 70314 2019-02-06 13:27:14 0 [Note] WSREP: ####### IST current seqno initialized to 70241 2019-02-06 13:27:14 0 [Note] WSREP: Receiving IST... 0.0% ( 0/75 events) complete. 2019-02-06 13:27:14 0 [Note] WSREP: IST preload starting at 70241 2019-02-06 13:27:14 0 [Note] WSREP: Service thread queue flushed. 2019-02-06 13:27:14 0 [Note] WSREP: 0.0 (cnt7glr11.localdomain): State transfer to 1.0 (cnt7glr11.localdomain) complete. 2019-02-06 13:27:14 0 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:70240, protocol version: 5 2019-02-06 13:27:14 0 [Note] WSREP: Member 0.0 (cnt7glr11.localdomain) synced with group. 2019-02-06 13:27:14 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-06 13:27:14 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 70315 last received: 70254 2019-02-06 13:27:14 20033 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) at galera/src/replicator_smm.hpp:pop_front():320. Null event. 2019-02-06 13:27:14 20033 [Note] WSREP: Closing send monitor... 2019-02-06 13:27:14 20033 [Note] WSREP: Closed send monitor. 2019-02-06 13:27:14 20033 [Note] WSREP: gcomm: terminating thread 2019-02-06 13:27:14 20033 [Note] WSREP: gcomm: joining thread 2019-02-06 13:27:14 20033 [Note] WSREP: gcomm: closing backend 2019-02-06 13:27:16 20033 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') turning message relay requesting off 2019-02-06 13:27:17 20033 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') connection to peer 1be2a2bb with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-06 13:27:17 20033 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') turning message relay requesting on, nonlive peers: tcp://127.0.0.1:16002 2019-02-06 13:27:19 20033 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') reconnecting to 1be2a2bb (tcp://127.0.0.1:16002), attempt 0 2019-02-06 13:27:22 20033 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') connection to peer 1be2a2bb with addr tcp://127.0.0.1:16002 timed out, no messages seen in PT3S 2019-02-06 13:27:24 20033 [Note] WSREP: (26384d3a, 'tcp://0.0.0.0:16005') connection established to 1be2a2bb tcp://127.0.0.1:16002 2019-02-06 13:27:24 20033 [Note] WSREP: evs::proto(26384d3a, LEAVING, view_id(REG,1be2a2bb,6)) suspecting node: 1be2a2bb 2019-02-06 13:27:24 20033 [Note] WSREP: evs::proto(26384d3a, LEAVING, view_id(REG,1be2a2bb,6)) suspected node without join message, declaring inactive 2019-02-06 13:27:24 20033 [Note] WSREP: view(view_id(NON_PRIM,1be2a2bb,6) memb { 26384d3a,0 } joined { } left { } partitioned { 1be2a2bb,0 }) 2019-02-06 13:27:24 20033 [Note] WSREP: PC protocol downgrade 1 -> 0 2019-02-06 13:27:24 20033 [Note] WSREP: view((empty)) 2019-02-06 13:27:24 20033 [Note] WSREP: gcomm: closed 2019-02-06 13:27:24 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2019-02-06 13:27:24 0 [Note] WSREP: Writing down CC checksum: 2e4e9d38 8cb145dd d55606e5 8b929be4 at offset 144 2019-02-06 13:27:24 0 [Note] WSREP: Flow-control interval: [16, 16] 2019-02-06 13:27:24 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-06 13:27:24 0 [Note] WSREP: Received NON-PRIMARY. 2019-02-06 13:27:24 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 70315) 2019-02-06 13:27:24 0 [Note] WSREP: Received self-leave message. 2019-02-06 13:27:24 0 [Note] WSREP: Writing down CC checksum: 3655d7eb ce05f785 9342a558 54a981f1 at offset 64 2019-02-06 13:27:24 0 [Note] WSREP: Flow-control interval: [0, 0] 2019-02-06 13:27:24 0 [Note] WSREP: Trying to continue unpaused monitor 2019-02-06 13:27:24 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2019-02-06 13:27:24 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1) 2019-02-06 13:27:24 0 [Note] WSREP: RECV thread exiting 0: Success 2019-02-06 13:27:24 20033 [Note] WSREP: recv_thread() joined. 2019-02-06 13:27:24 20033 [Note] WSREP: Closing replication queue. 2019-02-06 13:27:24 20033 [Note] WSREP: Closing slave action queue. mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. 190206 13:27:24 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.4.3-MariaDB-debug-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=3 max_threads=153 thread_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63454 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fa8fc02e500 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa9807759a0 thread_stack 0x49000 mysys/stacktrace.c:269(my_print_stacktrace)[0x55914e60f07c] sql/signal_handler.cc:168(handle_fatal_signal)[0x55914de4b3d3] sigaction.c:0(__restore_rt)[0x7fa98bba15d0] :0(__GI_raise)[0x7fa989e90207] :0(__GI_abort)[0x7fa989e918f8] :0(__assert_fail_base)[0x7fa989e89026] :0(__GI___assert_fail)[0x7fa989e890d2] src/replicator_str.cpp:945(galera::ReplicatorSMM::request_state_transfer(void*, wsrep_uuid const&, long, void const*, long))[0x7fa983adbdc7] src/replicator_smm.cpp:2733(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7fa983acae78] src/gcs_action_source.cpp:154(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7fa983aaaec9] src/gcs_action_source.cpp:188(galera::GcsActionSource::process(void*, bool&))[0x7fa983aab21d] src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7fa983aca033] src/wsrep_provider_v26.cpp:646(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x55914e6c7ffe] sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x55914dd6a932] sql/wsrep_mysqld.cc:2770(start_wsrep_THD(void*))[0x55914dd5c57f] pthread_create.c:0(start_thread)[0x7fa98bb99dd5] /lib64/libc.so.6(clone+0x6d)[0x7fa989f57ead] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 20033 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file at /home/stepan/galera/git/10.4/server/mysql-test/var/mysqld.2/data/ ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > connection node_1; SELECT COUNT(*) > 0 FROM mysql.wsrep_streaming_log; COUNT(*) > 0 1 connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2a; SET SESSION wsrep_sync_wait=0; SET GLOBAL wsrep_cluster_address = ''; SET SESSION wsrep_sync_wait = DEFAULT; connection node_2; INSERT INTO t1 VALUES (6); ERROR HY000: Lost connection to MySQL server during query connection node_1; SELECT COUNT(*) = 0 FROM mysql.wsrep_streaming_log; COUNT(*) = 0 1 connection node_2a; connection node_1; connect node_2b, 127.0.0.1, root, , test, $NODE_MYPORT_2; connection node_2b; - found 'core.42205' (1/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace from coredump /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.42205 Core generated by '/home/stepan/galera/git/10.4/server/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 63830] [New LWP 42221] [New LWP 42515] [New LWP 42518] [New LWP 42517] [New LWP 42522] [New LWP 42571] [New LWP 42580] [New LWP 42572] [New LWP 42581] [New LWP 42578] [New LWP 42579] [New LWP 42584] [New LWP 63833] [New LWP 42587] [New LWP 42519] [New LWP 63828] [New LWP 42520] [New LWP 42585] [New LWP 42521] [New LWP 42583] [New LWP 42582] [New LWP 42574] [New LWP 42205] [New LWP 42576] [New LWP 63829] [New LWP 42222] [New LWP 42586] [New LWP 42577] [New LWP 42575] [New LWP 42516] [New LWP 42573] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-group-suffix=.2 --def'. Program terminated with signal 6, Aborted. #0 0x00007fa98bb9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #0 0x00007fa98bb9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055914e60f16a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055914de4b7ff in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa989e90207 in raise () from /lib64/libc.so.6 #5 0x00007fa989e918f8 in abort () from /lib64/libc.so.6 #6 0x00007fa989e89026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007fa989e890d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007fa983adbdc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x55914fdf42a0, recv_ctx=recv_ctx@entry=0x7fa9807754d0, group_uuid=..., cc_seqno=cc_seqno@entry=70315, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007fa983acae78 in galera::ReplicatorSMM::process_conf_change (this=0x55914fdf42a0, recv_ctx=0x7fa9807754d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007fa983aaaec9 in galera::GcsActionSource::dispatch (this=this@entry=0x55914ff48090, recv_ctx=recv_ctx@entry=0x7fa9807754d0, act=..., exit_loop=@0x7fa98077529e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007fa983aab21d in galera::GcsActionSource::process (this=0x55914ff48090, recv_ctx=0x7fa9807754d0, exit_loop=@0x7fa98077529e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007fa983aca033 in galera::ReplicatorSMM::async_recv (this=0x55914fdf42a0, recv_ctx=0x7fa9807754d0) at galera/src/replicator_smm.cpp:383 #13 0x000055914e6c7ffe in wsrep::wsrep_provider_v26::run_applier (this=0x55914fd69e50, applier_ctx=0x7fa9807754d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #14 0x000055914dd6a932 in wsrep_replication_process (thd=0x7fa8fc02e500, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x000055914dd5c57f in start_wsrep_THD (arg=0x7fa95001f5f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #16 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fa93b7fe700 (LWP 42573)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e154f4e in os_event::timed_wait (this=0x55915004ceb0, abstime=0x7fa93b7fdd20) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055914e155265 in os_event::wait_time_low (this=0x55915004ceb0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055914e155405 in os_event_wait_time_low (event=0x55915004ceb0, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055914e243090 in srv_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1674 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fa971406700 (LWP 42516)): #0 0x00007fa98b76a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055914e148950 in LinuxAIOHandler::collect (this=0x7fa971405b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055914e148e79 in LinuxAIOHandler::poll (this=0x7fa971405b00, m1=0x7fa971405ba8, m2=0x7fa971405bb0, request=0x7fa971405be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055914e148fdc in os_aio_linux_handler (global_segment=0, m1=0x7fa971405ba8, m2=0x7fa971405bb0, request=0x7fa971405be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055914e14db00 in os_aio_handler (segment=0, m1=0x7fa971405ba8, m2=0x7fa971405bb0, request=0x7fa971405be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055914e3a19b1 in fil_aio_wait (segment=0) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055914e246ed9 in io_handler_thread (arg=0x55914f3d0660 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fa93a7fc700 (LWP 42575)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e154f4e in os_event::timed_wait (this=0x5591501f5e40, abstime=0x7fa93a7fbb60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055914e155265 in os_event::wait_time_low (this=0x5591501f5e40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055914e155405 in os_event_wait_time_low (event=0x5591501f5e40, time_in_usec=5000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055914e2a3857 in ib_wqueue_timedwait (wq=0x5591501f5d10, wait_in_usecs=5000000) at /home/stepan/galera/git/10.4/server/storage/innobase/ut/ut0wqueue.cc:163 #5 0x000055914e3e5d01 in fts_optimize_thread (arg=0x5591501f5d10) at /home/stepan/galera/git/10.4/server/storage/innobase/fts/fts0opt.cc:2897 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fa9397fa700 (LWP 42577)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e155633 in os_event::wait (this=0x55915004cc30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055914e1550b9 in os_event::wait_low (this=0x55915004cc30, reset_sig_count=48602) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055914e15542a in os_event_wait_low (event=0x55915004cc30, reset_sig_count=48602) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055914e2413a9 in srv_resume_thread (slot=0x55914f192748 , sig_count=48602, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055914e2453ba in srv_purge_coordinator_suspend (slot=0x55914f192748 , rseg_history_len=22) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2616 #6 0x000055914e245650 in srv_purge_coordinator_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2691 #7 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fa9715bb700 (LWP 42586)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e614799 in safe_cond_wait (cond=0x55914f3551e0 , mp=0x55914f358560 , file=0x55914e6de928 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1167) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055914da1074e in inline_mysql_cond_wait (that=0x55914f3551e0 , mutex=0x55914f358560 , src_file=0x55914e6df490 "/home/stepan/galera/git/10.4/server/sql/slave.cc", src_line=519) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1167 #3 0x000055914da11fe0 in handle_slave_background (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/slave.cc:519 #4 0x000055914e5a7261 in pfs_spawn_thread (arg=0x5591503c8f80) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fa983446700 (LWP 42222)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa983aa7fd7 in gu::Lock::wait (cond=..., this=0x7fa983445bf0) at galerautils/src/gu_lock.hpp:45 #2 0x00007fa983aa8069 in galera::ServiceThd::thd_func (arg=0x55914fdf49b8) at galera/src/galera_service_thd.cpp:30 #3 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fa9714da700 (LWP 63829)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e614799 in safe_cond_wait (cond=0x7fa8ec1e5e98, mp=0x7fa8ec1e5de8, file=0x55914e77d7c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055914dd6a384 in inline_mysql_cond_wait (that=0x7fa8ec1e5e98, mutex=0x7fa8ec1e5de8, src_file=0x55914e77dae0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055914dd6d01c in Wsrep_thd_queue::pop_front (this=0x7fa8ec1e5d90) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055914dd6bdfc in wsrep_post_rollback_process (post_rollbacker=0x7fa8ec000b00, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:285 #5 0x000055914dd5c57f in start_wsrep_THD (arg=0x7fa9500154b0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fa939ffb700 (LWP 42576)): #0 0x00007fa98bba0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055914e155935 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055914e244911 in srv_master_sleep () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2311 #3 0x000055914e244a41 in srv_master_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2352 #4 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fa98bfaa880 (LWP 42205)): #0 0x00007fa989f4d20d in poll () from /lib64/libc.so.6 #1 0x000055914d9ebcea in handle_connections_sockets () at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:6417 #2 0x000055914d9eaf85 in mysqld_main (argc=162, argv=0x55914fc75d20) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:5853 #3 0x000055914d9df125 in main (argc=23, argv=0x7ffc25c501f8) at /home/stepan/galera/git/10.4/server/sql/main.cc:25 Thread 23 (Thread 0x7fa93affd700 (LWP 42574)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e154f4e in os_event::timed_wait (this=0x559150256f60, abstime=0x7fa93affcd60) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055914e155265 in os_event::wait_time_low (this=0x559150256f60, time_in_usec=10000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055914e155405 in os_event_wait_time_low (event=0x559150256f60, time_in_usec=10000000, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055914e38b31c in dict_stats_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/dict/dict0stats_bg.cc:460 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fa9327fc700 (LWP 42582)): #0 0x00007fa98bba0e3d in nanosleep () from /lib64/libpthread.so.0 #1 0x000055914e155935 in os_thread_sleep (tm=1000000) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0thread.cc:225 #2 0x000055914e2fc4ff in btr_defragment_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/btr/btr0defragment.cc:702 #3 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fa931ffb700 (LWP 42583)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e155633 in os_event::wait (this=0x55915004d030) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055914e1550b9 in os_event::wait_low (this=0x55915004d030, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055914e15542a in os_event_wait_low (event=0x55915004d030, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055914e30a6a2 in buf_resize_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0buf.cc:3099 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fa969ffa700 (LWP 42521)): #0 0x00007fa98b76a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055914e148950 in LinuxAIOHandler::collect (this=0x7fa969ff9b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055914e148e79 in LinuxAIOHandler::poll (this=0x7fa969ff9b00, m1=0x7fa969ff9ba8, m2=0x7fa969ff9bb0, request=0x7fa969ff9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055914e148fdc in os_aio_linux_handler (global_segment=5, m1=0x7fa969ff9ba8, m2=0x7fa969ff9bb0, request=0x7fa969ff9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055914e14db00 in os_aio_handler (segment=5, m1=0x7fa969ff9ba8, m2=0x7fa969ff9bb0, request=0x7fa969ff9be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055914e3a19b1 in fil_aio_wait (segment=5) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055914e246ed9 in io_handler_thread (arg=0x55914f3d0688 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fa971606700 (LWP 42585)): #0 0x00007fa989e9140a in sigwaitinfo () from /lib64/libc.so.6 #1 0x000055914d9df1a4 in my_sigwait (set=0x7fa971605cf0, sig=0x7fa971605cb4, code=0x7fa971605cb8) at /home/stepan/galera/git/10.4/server/include/my_pthread.h:196 #2 0x000055914d9e57e4 in signal_hand (arg=0x0) at /home/stepan/galera/git/10.4/server/sql/mysqld.cc:3233 #3 0x000055914e5a7261 in pfs_spawn_thread (arg=0x559150010900) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #4 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fa96a7fb700 (LWP 42520)): #0 0x00007fa98b76a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055914e148950 in LinuxAIOHandler::collect (this=0x7fa96a7fab00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055914e148e79 in LinuxAIOHandler::poll (this=0x7fa96a7fab00, m1=0x7fa96a7faba8, m2=0x7fa96a7fabb0, request=0x7fa96a7fabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055914e148fdc in os_aio_linux_handler (global_segment=4, m1=0x7fa96a7faba8, m2=0x7fa96a7fabb0, request=0x7fa96a7fabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055914e14db00 in os_aio_handler (segment=4, m1=0x7fa96a7faba8, m2=0x7fa96a7fabb0, request=0x7fa96a7fabe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055914e3a19b1 in fil_aio_wait (segment=4) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055914e246ed9 in io_handler_thread (arg=0x55914f3d0680 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fa9807c1700 (LWP 63828)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e614799 in safe_cond_wait (cond=0x7fa90808e3a8, mp=0x7fa90808e2f8, file=0x55914e77d7c0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055914dd6a384 in inline_mysql_cond_wait (that=0x7fa90808e3a8, mutex=0x7fa90808e2f8, src_file=0x55914e77dae0 "/home/stepan/galera/git/10.4/server/sql/wsrep_thd.h", src_line=69) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055914dd6d01c in Wsrep_thd_queue::pop_front (this=0x7fa90808e2a0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.h:69 #4 0x000055914dd6ba3c in wsrep_rollback_process (rollbacker=0x7fa908000ba0, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:143 #5 0x000055914dd5c57f in start_wsrep_THD (arg=0x7fa9501491d0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fa96affc700 (LWP 42519)): #0 0x00007fa98b76a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055914e148950 in LinuxAIOHandler::collect (this=0x7fa96affbb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055914e148e79 in LinuxAIOHandler::poll (this=0x7fa96affbb00, m1=0x7fa96affbba8, m2=0x7fa96affbbb0, request=0x7fa96affbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055914e148fdc in os_aio_linux_handler (global_segment=3, m1=0x7fa96affbba8, m2=0x7fa96affbbb0, request=0x7fa96affbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055914e14db00 in os_aio_handler (segment=3, m1=0x7fa96affbba8, m2=0x7fa96affbbb0, request=0x7fa96affbbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055914e3a19b1 in fil_aio_wait (segment=3) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055914e246ed9 in io_handler_thread (arg=0x55914f3d0678 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fa971570700 (LWP 42587)): #0 0x00007fa989f4d20d in poll () from /lib64/libc.so.6 #1 0x000055914e68040e in vio_io_wait (vio=0x5591503dd960, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:976 #2 0x000055914e67ea2e in vio_socket_io_wait (vio=0x5591503dd960, event=VIO_IO_EVENT_READ) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:118 #3 0x000055914e67ec14 in vio_read (vio=0x5591503dd960, buf=0x7fa9500016a0 "\a", size=4) at /home/stepan/galera/git/10.4/server/vio/viosocket.c:195 #4 0x000055914dfb43bc in my_real_read (net=0x7fa95000a338, complen=0x7fa97156f810, header=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:962 #5 0x000055914dfb4c43 in my_net_read_packet_reallen (net=0x7fa95000a338, read_from_server=1 '\001', reallen=0x7fa97156f878) at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1243 #6 0x000055914dfb4bf6 in my_net_read_packet (net=0x7fa95000a338, read_from_server=1 '\001') at /home/stepan/galera/git/10.4/server/sql/net_serv.cc:1227 #7 0x000055914db103a6 in do_command (thd=0x7fa95000a040) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1229 #8 0x000055914dc812c9 in do_handle_one_connection (connect=0x559150354560) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #9 0x000055914dc8100d in handle_one_connection (arg=0x559150354560) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #10 0x000055914e5a7261 in pfs_spawn_thread (arg=0x5591503dba00) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #11 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fa97148f700 (LWP 63833)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa983a84bdb in gu::Lock::wait (this=this@entry=0x7fa97148c6f0, cond=..., date=...) at galerautils/src/gu_lock.hpp:62 #2 0x00007fa983acf976 in galera::Monitor::wait (this=this@entry=0x55914fdf5240, gtid=..., wait_until=...) at galera/src/monitor.hpp:355 #3 0x00007fa983ac157f in galera::ReplicatorSMM::sync_wait (this=0x55914fdf42a0, upto=, tout=, gtid=0x7fa97148cb30) at galera/src/replicator_smm.cpp:1661 #4 0x00007fa983ae0643 in galera_sync_wait (wsrep=, upto=, tout=, gtid=) at galera/src/wsrep_provider.cpp:998 #5 0x000055914e6c8cb5 in wsrep::wsrep_provider_v26::causal_read (this=0x55914fd69e50, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:801 #6 0x000055914e6b41b5 in wsrep::server_state::causal_read (this=0x55914fdbe740, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/server_state.cpp:636 #7 0x000055914e6aeb0d in wsrep::client_state::sync_wait (this=0x7fa954026e60, timeout=-1) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/client_state.cpp:417 #8 0x000055914dd54d8a in wsrep_sync_wait (thd=0x7fa954021000, mask=1) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:1022 #9 0x000055914db17c54 in mysql_execute_command (thd=0x7fa954021000) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:3779 #10 0x000055914db26b95 in mysql_parse (thd=0x7fa954021000, rawbuf=0x7fa954033168 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7fa97148e0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:8120 #11 0x000055914db261c8 in wsrep_mysql_parse (thd=0x7fa954021000, rawbuf=0x7fa954033168 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", length=102, parser_state=0x7fa97148e0d0, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:7932 #12 0x000055914db12085 in dispatch_command (command=COM_QUERY, thd=0x7fa954021000, packet=0x7fa954029b21 "SELECT VARIABLE_VALUE = 'ON' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_ready'", packet_length=102, is_com_multi=false, is_next_command=false) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1790 #13 0x000055914db10a6d in do_command (thd=0x7fa954021000) at /home/stepan/galera/git/10.4/server/sql/sql_parse.cc:1356 #14 0x000055914dc812c9 in do_handle_one_connection (connect=0x55914fe0fc30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1399 #15 0x000055914dc8100d in handle_one_connection (arg=0x55914fe0fc30) at /home/stepan/galera/git/10.4/server/sql/sql_connect.cc:1302 #16 0x000055914e5a7261 in pfs_spawn_thread (arg=0x5591503dd470) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #17 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fa9317fa700 (LWP 42584)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e614799 in safe_cond_wait (cond=0x7fa9317f9c70, mp=0x7fa9317f9cb0, file=0x55914e8e18a0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1178) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:492 #2 0x000055914e06c513 in inline_mysql_cond_wait (that=0x7fa9317f9c70, mutex=0x7fa9317f9cb0, src_file=0x55914e8e33e0 "/home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc", src_line=294) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1178 #3 0x000055914e071132 in thd_destructor_proxy () at /home/stepan/galera/git/10.4/server/storage/innobase/handler/ha_innodb.cc:294 #4 0x000055914e5a7261 in pfs_spawn_thread (arg=0x5591502075b0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fa933fff700 (LWP 42579)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e155633 in os_event::wait (this=0x55915004cdb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055914e1550b9 in os_event::wait_low (this=0x55915004cdb0, reset_sig_count=166609) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055914e15542a in os_event_wait_low (event=0x55915004cdb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055914e2413a9 in srv_resume_thread (slot=0x55914f1927f0 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055914e244fad in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fa938ff9700 (LWP 42578)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e155633 in os_event::wait (this=0x55915004cd30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055914e1550b9 in os_event::wait_low (this=0x55915004cd30, reset_sig_count=178447) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055914e15542a in os_event_wait_low (event=0x55915004cd30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055914e2413a9 in srv_resume_thread (slot=0x55914f1927b8 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055914e244fad in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fa932ffd700 (LWP 42581)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e155633 in os_event::wait (this=0x55915004cf30) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055914e1550b9 in os_event::wait_low (this=0x55915004cf30, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055914e15542a in os_event_wait_low (event=0x55915004cf30, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055914e321bce in buf_dump_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0dump.cc:835 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fa968ff8700 (LWP 42572)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e154f4e in os_event::timed_wait (this=0x55915004ce30, abstime=0x7fa968ff7ba0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055914e155265 in os_event::wait_time_low (this=0x55915004ce30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055914e155405 in os_event_wait_time_low (event=0x55915004ce30, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055914e2435b2 in srv_error_monitor_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:1838 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fa9337fe700 (LWP 42580)): #0 0x00007fa98bb9d965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e155633 in os_event::wait (this=0x55915004ccb0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:159 #2 0x000055914e1550b9 in os_event::wait_low (this=0x55915004ccb0, reset_sig_count=220860) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:326 #3 0x000055914e15542a in os_event_wait_low (event=0x55915004ccb0, reset_sig_count=0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:515 #4 0x000055914e2413a9 in srv_resume_thread (slot=0x55914f192780 , sig_count=0, wait=true, timeout_usec=0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:903 #5 0x000055914e244fad in srv_worker_thread (arg=0x0) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0srv.cc:2485 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fa93bfff700 (LWP 42571)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e154f4e in os_event::timed_wait (this=0x559150224f40, abstime=0x7fa93bffed40) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055914e155265 in os_event::wait_time_low (this=0x559150224f40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055914e155405 in os_event_wait_time_low (event=0x559150224f40, time_in_usec=1000000, reset_sig_count=1) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055914e121e6a in lock_wait_timeout_thread () at /home/stepan/galera/git/10.4/server/storage/innobase/lock/lock0wait.cc:543 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fa9697f9700 (LWP 42522)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e154f4e in os_event::timed_wait (this=0x55915004cfb0, abstime=0x7fa9697f8af0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:276 #2 0x000055914e155265 in os_event::wait_time_low (this=0x55915004cfb0, time_in_usec=515000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:398 #3 0x000055914e155405 in os_event_wait_time_low (event=0x55915004cfb0, time_in_usec=515000, reset_sig_count=2) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0event.cc:498 #4 0x000055914e32ba47 in pc_sleep_if_needed (next_loop_time=1549452444869, sig_count=2, cur_time=1549452444354) at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:2592 #5 0x000055914e32c980 in buf_flush_page_cleaner_coordinator () at /home/stepan/galera/git/10.4/server/storage/innobase/buf/buf0flu.cc:3051 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fa970c05700 (LWP 42517)): #0 0x00007fa98b76a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055914e148950 in LinuxAIOHandler::collect (this=0x7fa970c04b00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055914e148e79 in LinuxAIOHandler::poll (this=0x7fa970c04b00, m1=0x7fa970c04ba8, m2=0x7fa970c04bb0, request=0x7fa970c04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055914e148fdc in os_aio_linux_handler (global_segment=1, m1=0x7fa970c04ba8, m2=0x7fa970c04bb0, request=0x7fa970c04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055914e14db00 in os_aio_handler (segment=1, m1=0x7fa970c04ba8, m2=0x7fa970c04bb0, request=0x7fa970c04be0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055914e3a19b1 in fil_aio_wait (segment=1) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055914e246ed9 in io_handler_thread (arg=0x55914f3d0668 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fa96b7fd700 (LWP 42518)): #0 0x00007fa98b76a644 in __io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x000055914e148950 in LinuxAIOHandler::collect (this=0x7fa96b7fcb00) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:1915 #2 0x000055914e148e79 in LinuxAIOHandler::poll (this=0x7fa96b7fcb00, m1=0x7fa96b7fcba8, m2=0x7fa96b7fcbb0, request=0x7fa96b7fcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2060 #3 0x000055914e148fdc in os_aio_linux_handler (global_segment=2, m1=0x7fa96b7fcba8, m2=0x7fa96b7fcbb0, request=0x7fa96b7fcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:2114 #4 0x000055914e14db00 in os_aio_handler (segment=2, m1=0x7fa96b7fcba8, m2=0x7fa96b7fcbb0, request=0x7fa96b7fcbe0) at /home/stepan/galera/git/10.4/server/storage/innobase/os/os0file.cc:5667 #5 0x000055914e3a19b1 in fil_aio_wait (segment=2) at /home/stepan/galera/git/10.4/server/storage/innobase/fil/fil0fil.cc:4402 #6 0x000055914e246ed9 in io_handler_thread (arg=0x55914f3d0670 ) at /home/stepan/galera/git/10.4/server/storage/innobase/srv/srv0start.cc:325 #7 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fa972857700 (LWP 42515)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e614a9b in safe_cond_timedwait (cond=0x55914fbe1ee0 , mp=0x55914fbe1e20 , abstime=0x7fa972856cb0, file=0x55914ea5a370 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055914e47e8cd in inline_mysql_cond_timedwait (that=0x55914fbe1ee0 , mutex=0x55914fbe1e20 , abstime=0x7fa972856cb0, src_file=0x55914ea5a3b8 "/home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c", src_line=116) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055914e47edf1 in my_service_thread_sleep (control=0x55914f196b00 , sleep_time=30000000000) at /home/stepan/galera/git/10.4/server/storage/maria/ma_servicethread.c:115 #4 0x000055914e47328a in ma_checkpoint_background (arg=0x1e) at /home/stepan/galera/git/10.4/server/storage/maria/ma_checkpoint.c:707 #5 0x000055914e5a7261 in pfs_spawn_thread (arg=0x55914ff47690) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #6 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fa9845f2700 (LWP 42221)): #0 0x00007fa98bb9dd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000055914e614a9b in safe_cond_timedwait (cond=0x55914fbf4680 , mp=0x55914fbf45c0 , abstime=0x7fa9845f1d70, file=0x55914ea9e5a0 "/home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h", line=1204) at /home/stepan/galera/git/10.4/server/mysys/thr_mutex.c:546 #2 0x000055914e615a55 in inline_mysql_cond_timedwait (that=0x55914fbf4680 , mutex=0x55914fbf45c0 , abstime=0x7fa9845f1d70, src_file=0x55914ea9e5e8 "/home/stepan/galera/git/10.4/server/mysys/thr_timer.c", src_line=292) at /home/stepan/galera/git/10.4/server/include/mysql/psi/mysql_thread.h:1204 #3 0x000055914e6165b8 in timer_handler (arg=0x0) at /home/stepan/galera/git/10.4/server/mysys/thr_timer.c:292 #4 0x000055914e5a7261 in pfs_spawn_thread (arg=0x55914fdbd7d0) at /home/stepan/galera/git/10.4/server/storage/perfschema/pfs.cc:1862 #5 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fa980776700 (LWP 63830)): #0 0x00007fa98bb9e9d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x000055914e60f16a in my_write_core (sig=6) at /home/stepan/galera/git/10.4/server/mysys/stacktrace.c:481 #2 0x000055914de4b7ff in handle_fatal_signal (sig=6) at /home/stepan/galera/git/10.4/server/sql/signal_handler.cc:305 #3 #4 0x00007fa989e90207 in raise () from /lib64/libc.so.6 #5 0x00007fa989e918f8 in abort () from /lib64/libc.so.6 #6 0x00007fa989e89026 in __assert_fail_base () from /lib64/libc.so.6 #7 0x00007fa989e890d2 in __assert_fail () from /lib64/libc.so.6 #8 0x00007fa983adbdc7 in galera::ReplicatorSMM::request_state_transfer (this=this@entry=0x55914fdf42a0, recv_ctx=recv_ctx@entry=0x7fa9807754d0, group_uuid=..., cc_seqno=cc_seqno@entry=70315, sst_req=0x0, sst_req_len=) at galera/src/replicator_str.cpp:945 #9 0x00007fa983acae78 in galera::ReplicatorSMM::process_conf_change (this=0x55914fdf42a0, recv_ctx=0x7fa9807754d0, cc=...) at galera/src/replicator_smm.cpp:2660 #10 0x00007fa983aaaec9 in galera::GcsActionSource::dispatch (this=this@entry=0x55914ff48090, recv_ctx=recv_ctx@entry=0x7fa9807754d0, act=..., exit_loop=@0x7fa98077529e: false) at galera/src/gcs_action_source.cpp:124 #11 0x00007fa983aab21d in galera::GcsActionSource::process (this=0x55914ff48090, recv_ctx=0x7fa9807754d0, exit_loop=@0x7fa98077529e: false) at galera/src/gcs_action_source.cpp:182 #12 0x00007fa983aca033 in galera::ReplicatorSMM::async_recv (this=0x55914fdf42a0, recv_ctx=0x7fa9807754d0) at galera/src/replicator_smm.cpp:383 #13 0x000055914e6c7ffe in wsrep::wsrep_provider_v26::run_applier (this=0x55914fd69e50, applier_ctx=0x7fa9807754d0) at /home/stepan/galera/git/10.4/server/wsrep-lib/src/wsrep_provider_v26.cpp:646 #14 0x000055914dd6a932 in wsrep_replication_process (thd=0x7fa8fc02e500, arg=0x0) at /home/stepan/galera/git/10.4/server/sql/wsrep_thd.cc:61 #15 0x000055914dd5c57f in start_wsrep_THD (arg=0x7fa95001f5f0) at /home/stepan/galera/git/10.4/server/sql/wsrep_mysqld.cc:2768 #16 0x00007fa98bb99dd5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fa989f57ead in clone () from /lib64/libc.so.6 Compressed file /home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/mysqld.2/data/core.42205 - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_cc_master-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.GCF-620 galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.GCF-561 galera_sr.GCF-1060 galera_sr.GCF-597 galera_sr.GCF-1043B galera_sr.GCF-572 galera_sr.galera_sr_blob galera_sr.GCF-1051 galera_sr.GCF-851 galera_sr.GCF-571 galera_sr.GCF-585 galera_sr.galera_sr_bf_abort galera_sr.GCF-845 galera_sr.GCF-1043A galera_sr.galera_sr_cc_master galera_sr.GCF-623 galera_sr.GCF-867 galera_sr.GCF-580 galera_sr.GCF-1008 galera_sr.galera-features#56 galera_sr.GCF-1018 galera_sr.mysql-wsrep-features#9 2019-02-06 13:27:09 20024 [Warning] WSREP: Failed to replicate rollback fragment for 33472: 3 2019-02-06 13:27:14 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.misc:2 2019-02-06 13:27:14 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 70315 last received: 70254 2019-02-06 13:27:14 20033 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) mysqld: galera/src/replicator_str.cpp:945: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed. Attempting backtrace. You can use the following information to find out galera_sr.galera_sr_cc_slave 'innodb' [ pass ] 17439 galera_sr.galera_sr_concurrent 'innodb' [ pass ] 2189 galera_sr.galera_sr_conflict 'innodb' [ pass ] 2122 galera_sr.galera_sr_conflict_on_commit 'innodb' [ pass ] 2143 galera_sr.galera_sr_conflict_on_commit2 'innodb' [ pass ] 3144 galera_sr.galera_sr_conflict_with_rollback_master 'innodb' [ pass ] 2131 galera_sr.galera_sr_ddl_master 'innodb' [ fail ] Test ended at 2019-02-06 13:28:06 CURRENT_TEST: galera_sr.galera_sr_ddl_master --- /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_ddl_master.result 2019-01-23 13:58:38.971427700 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/r/galera_sr_ddl_master.reject 2019-02-06 13:28:05.832265000 +0200 @@ -44,5 +44,5 @@ 1 SELECT COUNT(*) = 0 FROM mysql.wsrep_streaming_log; COUNT(*) = 0 -1 +0 DROP TABLE t1; mysqltest: Result content mismatch - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_ddl_master-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_ddl_master-innodb/' galera_sr.galera_sr_ddl_schema 'innodb' [ pass ] 2170 galera_sr.galera_sr_ddl_slave 'innodb' [ pass ] 2200 galera_sr.galera_sr_ddl_unrelated 'innodb' [ pass ] 2220 galera_sr.galera_sr_dupkey_error 'innodb' [ pass ] 2175 galera_sr.galera_sr_fk_conflict 'innodb' [ pass ] 6066 galera_sr.galera_sr_insert_select 'innodb' [ pass ] 65405 galera_sr.galera_sr_kill_connection 'innodb' [ pass ] 2165 galera_sr.galera_sr_kill_query 'innodb' [ pass ] 2611 galera_sr.galera_sr_load_data 'innodb' [ pass ] 47322 galera_sr.galera_sr_myisam 'innodb' [ pass ] 2082 galera_sr.galera_sr_parallel_apply 'innodb' [ pass ] 2155 galera_sr.galera_sr_rollback 'innodb' [ pass ] 2245 galera_sr.galera_sr_rollback_retry 'innodb' [ pass ] 2142 galera_sr.galera_sr_rollback_savepoint 'innodb' [ pass ] 2193 galera_sr.galera_sr_rollback_statement 'innodb' [ fail ] Test ended at 2019-02-06 13:30:49 CURRENT_TEST: galera_sr.galera_sr_rollback_statement mysqltest: At line 34: query 'INSERT INTO t2 SELECT * FROM t1' succeeded - should have failed with errno 1062... The result from queries just before the failure was: connection node_2; connection node_1; connection node_1; CREATE TABLE ten (f1 INTEGER); INSERT INTO ten VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10); CREATE TABLE t1 (f1 INTEGER PRIMARY KEY AUTO_INCREMENT, f2 CHAR(255)) ENGINE=InnoDB; CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 CHAR(255)) ENGINE=InnoDB; INSERT INTO t1 (f2) SELECT REPEAT('a', 255) FROM ten AS a1, ten AS a2, ten AS a3; ALTER TABLE t1 CHANGE f1 f1 INTEGER; ALTER TABLE t1 DROP PRIMARY KEY; INSERT INTO t1 VALUES (1, 'abc'); SET SESSION wsrep_trx_fragment_size = 1; INSERT INTO t2 SELECT * FROM t1; - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_rollback_statement-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_rollback_statement-innodb/' galera_sr.galera_sr_shutdown_master 'innodb' [ pass ] 20601 galera_sr.galera_sr_shutdown_slave 'innodb' [ pass ] 19660 galera_sr.galera_sr_table_contents 'innodb' [ fail ] Test ended at 2019-02-06 13:31:38 CURRENT_TEST: galera_sr.galera_sr_table_contents cp: cannot stat 'std_data/binlog-header.log': No such file or directory mysqltest: In included file "./suite/galera/include/galera_dump_sr_table.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/galera_sr_table_contents.test at line 21: At line 20: exec of 'cp std_data/binlog-header.log /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/sr_binlog.log' failed, error: 256, status: 1, errno: 11 Output from before failure: Warnings: Warning 1287 ' INTO FROM...' instead The result from queries just before the failure was: connection node_2; connection node_1; CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t1 VALUES (1),(2),(3); CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t2 VALUES (1),(2),(3); SET SESSION wsrep_trx_fragment_size = 1; SET AUTOCOMMIT=OFF; START TRANSACTION; Start of Simple Insert INSERT INTO t1 VALUES (4); Warnings: Warning 1287 ' INTO FROM...' instead - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_table_contents-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_table_contents-innodb/' galera_sr.galera_sr_transaction_replay 'innodb' [ pass ] 2254 galera_sr.galera_sr_unit_statements 'innodb' [ pass ] 2118 galera_sr.galera_sr_ws_size 'innodb' [ pass ] 2491 galera_sr.galera_sr_ws_size2 'innodb' [ pass ] 4196 galera_sr.galera_var_ignore_apply_errors_sr 'innodb' [ pass ] 2150 galera_sr.mysql-wsrep#215 'innodb' [ pass ] 2509 galera_sr.mysql-wsrep-features#138 'innodb' [ pass ] 2139 galera_sr.mysql-wsrep-features#14 'innodb' [ pass ] 2097 galera_sr.mysql-wsrep-features#148 'innodb' [ pass ] 2297 MTR's internal check of the test case 'galera_sr.mysql-wsrep-features#148' failed. This means that the test case does not preserve the state that existed before the test case was executed. Most likely the test case did not do a proper clean-up. It could also be caused by the previous test run by this thread, if the server wasn't restarted. This is the diff of the states of the servers before and after the test case was executed: mysqltest: Logging to '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16001 (socket /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: ... Done processing test commands. --- /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.result 2019-02-06 13:32:06.442265000 +0200 +++ /home/stepan/galera/git/10.4/server/mysql-test/var/tmp/check-mysqld_2.reject 2019-02-06 13:32:08.896265000 +0200 @@ -1099,4 +1099,4 @@ user_variables 1.0 DISABLED INFORMATION SCHEMA 100403.0 NULL NULL Sergey Vojtovich User-defined variables GPL OFF Gamma 1.0 partition 1.0 DISABLED STORAGE ENGINE 100403.0 NULL NULL Mikael Ronstrom, MySQL AB Partition Storage Engine Helper GPL OFF Stable 1.0 VARIABLE_NAME VARIABLE_VALUE -DEBUG_SYNC ON - current signal: '' +DEBUG_SYNC ON - current signal: 'signal.wsrep_apply_cb' mysqltest: Result length mismatch not ok galera_sr.mysql-wsrep-features#15 'innodb' [ pass ] 2101 galera_sr.mysql-wsrep-features#165 'innodb' [ fail ] Test ended at 2019-02-06 13:32:39 CURRENT_TEST: galera_sr.mysql-wsrep-features#165 mysqltest: In included file "/home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.inc": included from /home/stepan/galera/git/10.4/server/mysql-test/suite/galera_sr/t/mysql-wsrep-features#165.test at line 27: At line 101: query 'SELECT * FROM t1' failed: 1205: Lock wait timeout exceeded; try restarting transaction The result from queries just before the failure was: < snip > connection node_1; SET DEBUG_SYNC = 'now WAIT_FOR wait'; connection node_1; SET DEBUG_SYNC = 'now SIGNAL continue'; connection node_1c; UPDATE t1 SET f2 = 'x' WHERE f1 = 3; connection node_1a; COMMIT; connection node_1b; ERROR 40001: Deadlock found when trying to get lock; try restarting transaction connection node_1; SELECT * FROM t1; f1 f2 1 a 2 a 3 x 4 a 5 a connection node_2; SELECT * FROM t1; More results from queries before failure can be found in /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysql-wsrep-features#165.log - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#165-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.mysql-wsrep-features#165-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 2019-02-06 13:32:29 2 [Warning] WSREP: Event 3 Write_rows_v1 apply failed: 121, seqno 12 galera_sr.mysql-wsrep-features#22 'innodb' [ pass ] 2156 galera_sr.mysql-wsrep-features#27 'innodb' [ pass ] 4135 galera_sr.mysql-wsrep-features#35 'innodb' [ pass ] 2158 galera_sr.mysql-wsrep-features#93 'innodb' [ pass ] 2122 galera_sr.mysql-wsrep-features#96 'innodb' [ pass ] 2241 galera_sr.galera_sr_kill_all_norecovery 'innodb' [ pass ] 18544 galera_sr.galera_sr_kill_slave 'innodb' [ pass ] 20939 galera_sr.galera_sr_mysqldump_sst 'innodb' [ fail ] Found warnings/errors in server log file! Test ended at 2019-02-06 13:34:55 line 2019-02-06 13:34:50 14 [Warning] WSREP: no THD for trx: 51 2019-02-06 13:34:50 14 [Warning] WSREP: no THD for trx: 51 ^ Found warnings in /home/stepan/galera/git/10.4/server/mysql-test/var/log/mysqld.2.err ok - saving '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' to '/home/stepan/galera/git/10.4/server/mysql-test/var/log/galera_sr.galera_sr_mysqldump_sst-innodb/' galera_sr.galera_sr_small_gcache 'innodb' [ pass ] 63299 -------------------------------------------------------------------------- The servers were restarted 18 times Spent 1361.764 of 2087 seconds executing testcases mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/stepan/galera/git/10.4/server/mysql-test/var/log/warnings' for details. Check of testcase failed for: galera_sr.mysql-wsrep-features#148 Completed: Failed 9/83 tests, 89.16% were successful. Failing test(s): galera_sr.galera_sr_log_bin galera_sr.GCF-437 galera_sr.galera_sr_load_data_splitting galera_sr.galera_sr_cc_master galera_sr.galera_sr_ddl_master galera_sr.galera_sr_rollback_statement galera_sr.galera_sr_table_contents galera_sr.mysql-wsrep-features#165 galera_sr.galera_sr_mysqldump_sst The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html Errors/warnings were found in logfiles during server shutdown after running the following sequence(s) of tests: galera_sr.galera_sr_log_bin galera_sr.GCF-620 galera_sr.GCF-627 galera_sr.GCF-900 galera_sr.GCF-889 galera_sr.GCF-561 galera_sr.GCF-1060 galera_sr.GCF-597 galera_sr.GCF-1043B galera_sr.GCF-572 galera_sr.galera_sr_blob galera_sr.GCF-1051 galera_sr.GCF-851 galera_sr.GCF-571 galera_sr.GCF-585 galera_sr.galera_sr_bf_abort galera_sr.GCF-845 galera_sr.GCF-1043A galera_sr.galera_sr_cc_master galera_sr.GCF-623 galera_sr.GCF-867 galera_sr.GCF-580 galera_sr.GCF-1008 galera_sr.galera-features#56 galera_sr.GCF-1018 galera_sr.mysql-wsrep-features#9 galera_sr.mysql-wsrep-features#165 galera_sr.mysql-wsrep-features#15 mysql-test-run: *** ERROR: there were failing test cases