[MDEV-21768] `galera_3nodes.galera_gtid_2_cluster` fails Created: 2020-02-19  Updated: 2020-11-18  Resolved: 2020-11-18

Status: Closed
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.4
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Timofey Turenko Assignee: Jan Lindström (Inactive)
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-23775 Galera test failure on galera_3nodes.... Open

 Description   

galera_3nodes.galera_gtid_2_cluster 'innodb' w2 [ fail ]
Test ended at 2020-02-19 06:13:16

CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster

Failed to start mysqld.6
mysqltest failed but provided no output

  • saving '/var/tmp/mtr/2/log/galera_3nodes.galera_gtid_2_cluster-innodb/' to '/var/tmp/mtr/log/galera_3nodes.galera_gtid_2_cluster-innodb/'

Retrying test galera_3nodes.galera_gtid_2_cluster, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_gtid_2_cluster

2020-02-19 6:13:12 0 [Warning] WSREP: error while trying to listen 'tcp://0.0.0.0:16021?socket.non_blocking=1', asio error 'bind: Address already in use'
2020-02-19 6:13:12 0 [ERROR] WSREP: failed to open gcomm backend connection: 98: error while trying to listen 'tcp://0.0.0.0:16021?socket.non_blocking=1', asio error 'bind: Address already in use': 98 (Address already in use)
2020-02-19 6:13:12 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():222: Failed to open backend connection: -98 (Address already in use)
2020-02-19 6:13:12 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1614: Failed to open channel 'cluster2' at 'gcomm://127.0.0.1:16015': -98 (Address already in use)
2020-02-19 6:13:12 0 [ERROR] WSREP: gcs connect failed: Address already in use
2020-02-19 6:13:12 0 [ERROR] Plugin 'wsrep' init function returned error.
2020-02-19 6:13:12 0 [ERROR] Plugin 'wsrep' registration as a STORAGE ENGINE failed.
2020-02-19 6:13:12 0 [ERROR] Failed to initialize plugins.
2020-02-19 6:13:12 0 [ERROR] Aborting

galera_3nodes.galera_gtid_2_cluster 'innodb' w2 [ retry-fail ]
Test ended at 2020-02-19 06:13:26

CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster
CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster

Failed to start mysqld.5

  • skipping '/var/tmp/mtr/2/log/galera_3nodes.galera_gtid_2_cluster-innodb/'

Test galera_3nodes.galera_gtid_2_cluster has failed 2 times, no more retries!

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_gtid_2_cluster

WSREP_SST: [ERROR] rsync daemon port '16020' has been taken (20200219 06:13:19.128)
2020-02-19 6:13:19 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16020' --datadir '/var/tmp/mtr/2/mysqld.5/data/' --defaults-file '/var/tmp/mtr/2/my.cnf' --defaults-group-suffix '.5' --parent '16464' --binlog 'mysqld-bin' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.5 --defaults-file=/var/tmp/mtr/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2020-02-19 6:13:19 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16020' --datadir '/var/tmp/mtr/2/mysqld.5/data/' --defaults-file '/var/tmp/mtr/2/my.cnf' --defaults-group-suffix '.5' --parent '16464' --binlog 'mysqld-bin' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.5 --defaults-file=/var/tmp/mtr/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2020-02-19 6:13:19 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2020-02-19 6:13:19 2 [ERROR] WSREP: SST request callback failed. This is unrecoverable, restart required.
2020-02-19 6:13:18 0 [Warning] WSREP: error while trying to listen 'tcp://0.0.0.0:16021?socket.non_blocking=1', asio error 'bind: Address already in use'
2020-02-19 6:13:18 0 [ERROR] WSREP: failed to open gcomm backend connection: 98: error while trying to listen 'tcp://0.0.0.0:16021?socket.non_blocking=1', asio error 'bind: Address already in use': 98 (Address already in use)
2020-02-19 6:13:18 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():222: Failed to open backend connection: -98 (Address already in use)
2020-02-19 6:13:18 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1614: Failed to open channel 'cluster2' at 'gcomm://127.0.0.1:16015': -98 (Address already in use)
2020-02-19 6:13:18 0 [ERROR] WSREP: gcs connect failed: Address already in use
2020-02-19 6:13:18 0 [ERROR] Plugin 'wsrep' init function returned error.
2020-02-19 6:13:18 0 [ERROR] Plugin 'wsrep' registration as a STORAGE ENGINE failed.
2020-02-19 6:13:18 0 [ERROR] Failed to initialize plugins.
2020-02-19 6:13:18 0 [ERROR] Aborting

 Comments   
Comment by Stepan Patryshev (Inactive) [ 2020-11-18 ]

It failed also on 10.3 ES:

10.3.24-8-MariaDB-enterprise-debug 9848e25f11fc8213a2e70f66b253272a5e4d11f7

 
galera_3nodes.galera_gtid_2_cluster 'innodb' [ fail ]
 
        Test ended at 2020-11-10 22:04:16
 
 
 
CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster
 
--- /home/timofey_turenko_mariadb_com/MariaDBEnterprise/mysql-test/suite/galera_3nodes/r/galera_gtid_2_cluster.result	2020-11-10 20:04:29.000000000 +0100
 
+++ /home/timofey_turenko_mariadb_com/MariaDBEnterprise/mysql-test/suite/galera_3nodes/r/galera_gtid_2_cluster.reject	2020-11-10 22:04:15.865865937 +0100
 
@@ -1,27 +1,38 @@
 
+connection node_1;
 
 cluster 1 node 1
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connection node_2;
 
 cluster 1 node 2
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3;
 
+connection node_3;
 
 cluster 1 node 3
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_4, 127.0.0.1, root, , test, $NODE_MYPORT_4;
 
+connection node_4;
 
 cluster 2 node 1
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_5, 127.0.0.1, root, , test, $NODE_MYPORT_5;
 
+connection node_5;
 
 cluster 2 node 2
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_6, 127.0.0.1, root, , test, $NODE_MYPORT_6;
 
+connection node_6;
 
 cluster 2 node 3
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connection node_1;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_4, master_use_gtid=current_pos, ignore_server_ids=(12,13);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -31,6 +42,7 @@
 
 select @@gtid_slave_pos;
 
 @@gtid_slave_pos
 
 
+connection node_4;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_1, master_use_gtid=current_pos, ignore_server_ids=(22,23);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -41,15 +53,19 @@
 
 @@gtid_slave_pos
 
 
 cluster 1 node 1
 
+connection node_1;
 
 create table t1 (cluster_domain_id int ,node_server_id int, seq_no int);
 
 insert into t1 values (1, 11, 2);
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 1
 
+connection node_4;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2
 
@@ -62,9 +78,12 @@
 
 1	11	2
 
 2	21	1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 2
 
+connection node_2;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,2-21-1
 
@@ -73,9 +92,12 @@
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 3
 
+connection node_3;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
@@ -84,9 +106,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 2
 
+connection node_5;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
@@ -95,9 +120,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 3
 
+connection node_6;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
@@ -106,9 +134,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 1
 
+connection node_1;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
@@ -119,15 +150,21 @@
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
 cluster 2 node 1
 
+connection node_4;
 
 stop slave;
 
 reset slave;
 
 change master to master_use_gtid=no, ignore_server_ids=();
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
+connection node_2;
 
 reset master;
 
+connection node_3;
 
 reset master;
 
+connection node_5;
 
 reset master;
 
+connection node_6;
 
 reset master;
 
+connection node_1;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_6, master_use_gtid=current_pos, ignore_server_ids=(12,13);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -137,6 +174,7 @@
 
 select @@gtid_slave_pos;
 
 @@gtid_slave_pos
 
 
+connection node_4;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_3, master_use_gtid=current_pos, ignore_server_ids=(22,23);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -147,15 +185,19 @@
 
 @@gtid_slave_pos
 
 
 cluster 1 node 1
 
+connection node_1;
 
 create table t1 (cluster_domain_id int ,node_server_id int, seq_no int);
 
 insert into t1 values (1, 11, 2);
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 1
 
+connection node_4;
 
 insert into t1 values (2, 21, 1);
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
@@ -165,9 +207,12 @@
 
 1	11	2
 
 2	21	1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 2
 
+connection node_2;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,2-21-1
 
@@ -176,9 +221,12 @@
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 3
 
+connection node_3;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
@@ -187,9 +235,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 2
 
+connection node_5;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
@@ -198,9 +249,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 3
 
+connection node_6;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
@@ -209,9 +263,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 1
 
+connection node_1;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
@@ -221,11 +278,16 @@
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
 cluster 2 node 1
 
+connection node_4;
 
 stop slave;
 
 change master to master_use_gtid=no, ignore_server_ids=();
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
+connection node_2;
 
 reset master;
 
+connection node_3;
 
 reset master;
 
+connection node_5;
 
 reset master;
 
+connection node_6;
 
 reset master;
 
 
 
mysqltest: Result length mismatch
 
 
 
 - skipping '/var/tmp/mtr/log/galera_3nodes.galera_gtid_2_cluster-innodb/'
 
 
 
Retrying test galera_3nodes.galera_gtid_2_cluster, attempt(2/3)...
 
 
 
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_gtid_2_cluster
 
 
 
2020-11-10 22:04:10 14 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'could not find next log; the first event '.' at 4, the last event read from 'mysqld-bin.000003' at 2059, the last byte read from 'mysqld-bin.000003' at 2167.', Internal MariaDB error code: 1236

Comment by Stepan Patryshev (Inactive) [ 2020-11-18 ]

It duplicates MDEV-23775.

Generated at Thu Feb 08 09:09:38 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.