[MDEV-23775] Galera test failure on galera_3nodes.galera_gtid_2_cluster Created: 2020-09-21  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Galera, Tests
Affects Version/s: 10.3.24, 10.4.14, 10.2
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Jan Lindström (Inactive) Assignee: Seppo Jaakola
Resolution: Unresolved Votes: 0
Labels: Test_disabled, test_disabled_ES

Issue Links:
Blocks
blocks MDEV-19937 Galera test failures on 10.2/10.3 Closed
Duplicate
is duplicated by MDEV-21768 `galera_3nodes.galera_gtid_2_cluster`... Closed

 Description    

galera_3nodes.galera_gtid_2_cluster 'innodb' [ fail ]
 
        Test ended at 2020-09-21 13:45:54
 
 
 
CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster
 
mysqltest: In included file "./include/sync_with_master_gtid.inc": 
included from /home/jan/mysql/10.2-bugs/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.test at line 73:
 
At line 48: Failed to sync with master
 
 
 
The result from queries just before the failure was:
 
< snip >
 
include/wait_for_slave_to_start.inc
 
select @@gtid_binlog_state;
 
@@gtid_binlog_state
 
0-21-1
 
select @@gtid_slave_pos;
 
@@gtid_slave_pos
 
 
 
cluster 1 node 1
 
connection node_1;
 
create table t1 (cluster_domain_id int ,node_server_id int, seq_no int);
 
insert into t1 values (1, 11, 2);
 
select @@gtid_binlog_state;
 
@@gtid_binlog_state
 
0-21-1,1-11-2
 
#wait for sync  cluster 1 and 2
 
connection node_1;
 
include/save_master_gtid.inc
 
connection node_4;
 
include/sync_with_master_gtid.inc
 
Timeout in master_gtid_wait('0-21-1,1-11-2', 120), current slave GTID position is: 1-11-2.
 
 
 
More results from queries before failure can be found in /home/jan/mysql/10.2-bugs/mysql-test/var/log/galera_gtid_2_cluster.log
 
 
 
 - saving '/home/jan/mysql/10.2-bugs/mysql-test/var/log/galera_3nodes.galera_gtid_2_cluster-innodb/' to '/home/jan/mysql/10.2-bugs/mysql-test/var/log/galera_3nodes.galera_gtid_2_cluster-innodb/'
 
--------------------------------------------------------------------------



 Comments   
Comment by Jan Lindström (Inactive) [ 2020-09-21 ]

Note that this is not only failure on this test case.

Comment by Jan Lindström (Inactive) [ 2020-09-21 ] 

galera_3nodes.galera_gtid_2_cluster 'innodb' [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2020-09-21 13:34:56
 
line
 
2020-09-21 13:34:50 140589051225856 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'could not find next log; the first event '.' at 4, the last event read from 'mysqld-bin.000003' at 2059, the last byte read from 'mysqld-bin.000003' at 2167.', Internal MariaDB error code: 1236
 
^ Found warnings in /home/jan/mysql/10.2-bugs/mysql-test/var/log/mysqld.4.err

Comment by Jan Lindström (Inactive) [ 2020-09-21 ] 

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
 
worker[1] mysql-test-run: WARNING: Process [mysqld.2 - pid: 1799729, winpid: 1799729, exit: 256] died after mysql-test-run waited 31.9 seconds for /home/jan/mysql/10.2-bugs/mysql-test/var/run/mysqld.2.pid to be created.
 
galera_3nodes.galera_gtid_2_cluster 'innodb' [ fail ]
 
        Test ended at 2020-09-21 13:34:35
 
 
 
CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster
 
 
 
 
 
Failed to start mysqld.2
 
mysqltest failed but provided no output

Comment by Stepan Patryshev (Inactive) [ 2020-11-18 ]

It failed also on 10.3 ES:

10.3.24-8-MariaDB-enterprise-debug 9848e25f11fc8213a2e70f66b253272a5e4d11f7

 
galera_3nodes.galera_gtid_2_cluster 'innodb' [ fail ]
 
        Test ended at 2020-11-10 22:04:16
 
 
 
CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster
 
--- /home/timofey_turenko_mariadb_com/MariaDBEnterprise/mysql-test/suite/galera_3nodes/r/galera_gtid_2_cluster.result	2020-11-10 20:04:29.000000000 +0100
 
+++ /home/timofey_turenko_mariadb_com/MariaDBEnterprise/mysql-test/suite/galera_3nodes/r/galera_gtid_2_cluster.reject	2020-11-10 22:04:15.865865937 +0100
 
@@ -1,27 +1,38 @@
 
+connection node_1;
 
 cluster 1 node 1
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connection node_2;
 
 cluster 1 node 2
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3;
 
+connection node_3;
 
 cluster 1 node 3
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_4, 127.0.0.1, root, , test, $NODE_MYPORT_4;
 
+connection node_4;
 
 cluster 2 node 1
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_5, 127.0.0.1, root, , test, $NODE_MYPORT_5;
 
+connection node_5;
 
 cluster 2 node 2
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connect node_6, 127.0.0.1, root, , test, $NODE_MYPORT_6;
 
+connection node_6;
 
 cluster 2 node 3
 
 SHOW STATUS LIKE 'wsrep_cluster_size';
 
 Variable_name	Value
 
 wsrep_cluster_size	3
 
+connection node_1;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_4, master_use_gtid=current_pos, ignore_server_ids=(12,13);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -31,6 +42,7 @@
 
 select @@gtid_slave_pos;
 
 @@gtid_slave_pos
 
 
+connection node_4;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_1, master_use_gtid=current_pos, ignore_server_ids=(22,23);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -41,15 +53,19 @@
 
 @@gtid_slave_pos
 
 
 cluster 1 node 1
 
+connection node_1;
 
 create table t1 (cluster_domain_id int ,node_server_id int, seq_no int);
 
 insert into t1 values (1, 11, 2);
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 1
 
+connection node_4;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2
 
@@ -62,9 +78,12 @@
 
 1	11	2
 
 2	21	1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 2
 
+connection node_2;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,2-21-1
 
@@ -73,9 +92,12 @@
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 3
 
+connection node_3;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
@@ -84,9 +106,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 2
 
+connection node_5;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
@@ -95,9 +120,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 3
 
+connection node_6;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
@@ -106,9 +134,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 1
 
+connection node_1;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
@@ -119,15 +150,21 @@
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
 cluster 2 node 1
 
+connection node_4;
 
 stop slave;
 
 reset slave;
 
 change master to master_use_gtid=no, ignore_server_ids=();
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
+connection node_2;
 
 reset master;
 
+connection node_3;
 
 reset master;
 
+connection node_5;
 
 reset master;
 
+connection node_6;
 
 reset master;
 
+connection node_1;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_6, master_use_gtid=current_pos, ignore_server_ids=(12,13);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -137,6 +174,7 @@
 
 select @@gtid_slave_pos;
 
 @@gtid_slave_pos
 
 
+connection node_4;
 
 change master to master_host='127.0.0.1', master_user='root', master_port=NODE_MYPORT_3, master_use_gtid=current_pos, ignore_server_ids=(22,23);;
 
 start slave;
 
 include/wait_for_slave_to_start.inc
 
@@ -147,15 +185,19 @@
 
 @@gtid_slave_pos
 
 
 cluster 1 node 1
 
+connection node_1;
 
 create table t1 (cluster_domain_id int ,node_server_id int, seq_no int);
 
 insert into t1 values (1, 11, 2);
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 1
 
+connection node_4;
 
 insert into t1 values (2, 21, 1);
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
@@ -165,9 +207,12 @@
 
 1	11	2
 
 2	21	1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 2
 
+connection node_2;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,2-21-1
 
@@ -176,9 +221,12 @@
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 3
 
+connection node_3;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-11-2,1-12-3,2-21-1
 
@@ -187,9 +235,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
 #wait for sync  cluster 1 and 2
 
+connection node_1;
 
 include/save_master_gtid.inc
 
+connection node_4;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 2
 
+connection node_5;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1
 
@@ -198,9 +249,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 2 node 3
 
+connection node_6;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
@@ -209,9 +263,12 @@
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
 #wait for sync  cluster 2 and 1
 
+connection node_4;
 
 include/save_master_gtid.inc
 
+connection node_1;
 
 include/sync_with_master_gtid.inc
 
 cluster 1 node 1
 
+connection node_1;
 
 select @@gtid_binlog_state;
 
 @@gtid_binlog_state
 
 1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3
 
@@ -221,11 +278,16 @@
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
 cluster 2 node 1
 
+connection node_4;
 
 stop slave;
 
 change master to master_use_gtid=no, ignore_server_ids=();
 
 reset master;
 
 set global GTID_SLAVE_POS="";
 
+connection node_2;
 
 reset master;
 
+connection node_3;
 
 reset master;
 
+connection node_5;
 
 reset master;
 
+connection node_6;
 
 reset master;
 
 
 
mysqltest: Result length mismatch
 
 
 
 - skipping '/var/tmp/mtr/log/galera_3nodes.galera_gtid_2_cluster-innodb/'
 
 
 
Retrying test galera_3nodes.galera_gtid_2_cluster, attempt(2/3)...
 
 
 
***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_gtid_2_cluster
 
 
 
2020-11-10 22:04:10 14 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'could not find next log; the first event '.' at 4, the last event read from 'mysqld-bin.000003' at 2059, the last byte read from 'mysqld-bin.000003' at 2167.', Internal MariaDB error code: 1236

Comment by Stepan Patryshev (Inactive) [ 2020-11-18 ]

It failed also on 10.4 ES:

10.4.14-8-MariaDB-enterprise-debug 04eba9e5962d8c7ae9fdb39200e775a878f80655

 
CURRENT_TEST: galera_3nodes.galera_gtid_2_cluster
 
mysqltest: In included file "./include/sync_with_master_gtid.inc": 
included from /home/timofey_turenko_mariadb_com/MariaDBEnterprise/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.test at line 273:
 
At line 48: Failed to sync with master
 
 
 
The result from queries just before the failure was:
 
< snip >
 
#wait for sync  cluster 1 and 2
 
connection node_1;
 
include/save_master_gtid.inc
 
connection node_4;
 
include/sync_with_master_gtid.inc
 
cluster 2 node 2
 
connection node_5;
 
select @@gtid_binlog_state;
 
@@gtid_binlog_state
 
1-12-3,1-11-2,1-13-4,2-21-1
 
insert into t1 values (2, 22, 2);
 
select @@gtid_binlog_state;
 
@@gtid_binlog_state
 
1-12-3,1-11-2,1-13-4,2-21-1,2-22-2
 
#wait for sync  cluster 2 and 1
 
connection node_4;
 
include/save_master_gtid.inc
 
connection node_1;
 
include/sync_with_master_gtid.inc
 
Timeout in master_gtid_wait('1-13-4,2-22-3', 120), current slave GTID position is: 1-13-4,2-22-2.

Generated at Thu Feb 08 09:24:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.