[MDEV-31905] WSREP GTID MODE is inconsistent Created: 2023-08-11  Updated: 2024-01-22  Resolved: 2023-12-23

Status: Closed
Project: MariaDB Server
Component/s: Galera, Galera SST
Affects Version/s: 10.5.23, 10.6.14
Fix Version/s: 10.5.24, 10.6.17, 10.11.7, 11.0.5, 11.1.4, 11.2.3

Type: Bug Priority: Critical
Reporter: Edward Stoever Assignee: Julius Goryavsky
Resolution: Fixed Votes: 3
Labels: None

Attachments: PNG File after-restart-all-three-nodes-galera-new-cluster-was-on-m2.png     File n1.cnf     File n2.cnf     File n3.cnf    
Issue Links:
Relates
relates to MDEV-28015 Mariabackup | GTID value is missing, ... Closed
relates to MDEV-32740 wsrep_gtid_domain_id is disregarded o... Closed

 Description   

WSREP GTID MODE is inconsistent

When you create a Galera cluster with the following configuration according to documentation:

# NODE 1 (m1)
 
wsrep_gtid_mode=ON
 
wsrep_gtid_domain_id=1111
 
gtid_domain_id=2
 
server_id=10999
 
log_slave_updates=ON
 
log_bin = /var/log/mysql/mariadb-bin-log
 
 
 
# NODE 2 (m2)
 
wsrep_gtid_mode=ON
 
wsrep_gtid_domain_id=1111
 
gtid_domain_id=3
 
server_id=10999
 
log_slave_updates=ON
 
log_bin = /var/log/mysql/mariadb-bin-log
 
 
 
# NODE 3 (m3)
 
wsrep_gtid_mode=ON
 
wsrep_gtid_domain_id=1111
 
gtid_domain_id=4
 
server_id=10999
 
log_slave_updates=ON
 
log_bin = /var/log/mysql/mariadb-bin-log

Sometimes, the nodes will stay in sync. However, if the following is done:
1) transactions run on all three nodes
2) restart mariadb on any node
3) restart all three nodes, using galera_new_cluster on a different node from first startup

Then, GTIDs will become inconsistent both at the domain level and at the seq_no level.

This is not difficult to reproduce.

 Comments   
Comment by Edward Stoever [ 2023-08-23 ]

This was tested on 10.6.14 Enterprise.

Comment by Jan Lindström [ 2023-10-10 ]

Test case
1) Start the 3 node cluster
2) OLTP load started on all nodes
3) The second node is restarted
4) After 1 minute all nodes are stopped one by one
3) bootstrapped node3 and initialized remaining nodes

Note: reproduces with mariabackup not with rsync

Comment by Jan Lindström [ 2023-10-11 ]

edward Does customer use mariabacup as SST method?

With mariabackup I can reproduce it even at startup

connection node_1;
 
CREATE SCHEMA IF NOT EXISTS mysqlslap;
 
USE mysqlslap;
 
CREATE TABLE t1 (id bigint not null primary key auto_increment, name VARCHAR(64)) ENGINE=innodb;
 
# node_1
 
show variables like '%gtid_binlog_pos%';
 
Variable_name	Value
 
gtid_binlog_pos	1111-1-2
 
connection node_2;
 
# node_2
 
show variables like '%gtid_binlog_pos%';
 
Variable_name	Value
 
gtid_binlog_pos	0-1-2
 
connection node_3;
 
# node_3
 
show variables like '%gtid_binlog_pos%';
 
Variable_name	Value
 
gtid_binlog_pos	0-1-2

Comment by Richard Stracke [ 2023-12-12 ]

I checked the sourcecode and I shortened the relevant parts

 
 
    ******************************************
 
    uint  wsrep_gtid_domain_id=0;                   // Domain id on above structure
 
    
    void wsrep_init_globals()
 
          wsrep_init_gtid();
 
          
          wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
 
           stored_gtid.domain_id= wsrep_gtid_server.domain_id;     
               if (stored_gtid.server_id == 0)
 
                     rpl_gtid wsrep_last_gtid;
 
                     mysql_bin_log.lookup_domain_in_binlog_state(stored_gtid.domain_id,
 
                                                    &wsrep_last_gtid))
 
           wsrep_gtid_server.gtid(stored_gtid);
 
 
 
 
 
if (wsrep_new_cluster)
 
    new_gtid.domain_id= wsrep_gtid_domain_id;  #  wsrep_gtid_domain_id=0;       
        
            /* Try to search for domain_id and server_id combination in binlog if found continue from last seqno */
 
                wsrep_get_binlog_gtid_seqno(new_gtid);
 
    wsrep_gtid_server.gtid(new_gtid);
 
    
    *********************************************

So domain_id will be initialized with
wsrep_gtid_domain_id=0

wsrep_gtid_domain_id can be set from SST , otherwise it remain 0

wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
will set the grid from recovery process, otherwise it kept empty.

If new cluster the domain_id will be set to
new_gtid.domain_id= wsrep_gtid_domain_id;

IF the recovery process gives no result and new cluster flag is set,
wsrep_gtid_domain_id kept 0 and will be overwritten here

if (wsrep_new_cluster)
 
    new_gtid.domain_id= wsrep_gtid_domain_id;  #  wsrep_gtid_domain_id=0;

rsync copies maybe some more files for a successful recovery or whatever 

wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();

do .

So I suggest to init

uint wsrep_gtid_domain_id=0; // Domain id on above structure

to real wsrep_gtid_domain_id defined in config file
or sst_method = mariabackup have to copy the same files like rsync for a successul wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
seppo and janlindstrom

Can you check this, if this makes sense to you ?

Comment by Seppo Jaakola [ 2023-12-14 ]

Pull request to handle domain-id inconsistency due to mariabackup SST was submitted yesterday

Comment by Seppo Jaakola [ 2023-12-14 ]

PR is submitted here: https://github.com/MariaDB/server/pull/2932

Comment by Julius Goryavsky [ 2023-12-23 ]

Fix merged with head revision as https://github.com/MariaDB/server/commit/c89f769f2412202dd8d7e69b0942778c91602ec7

Generated at Thu Feb 08 10:27:21 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.