When you create a Galera cluster with the following configuration according to documentation:
# NODE 1 (m1)
wsrep_gtid_mode=ON
wsrep_gtid_domain_id=1111
gtid_domain_id=2
server_id=10999
log_slave_updates=ON
log_bin = /var/log/mysql/mariadb-bin-log
# NODE 2 (m2)
wsrep_gtid_mode=ON
wsrep_gtid_domain_id=1111
gtid_domain_id=3
server_id=10999
log_slave_updates=ON
log_bin = /var/log/mysql/mariadb-bin-log
# NODE 3 (m3)
wsrep_gtid_mode=ON
wsrep_gtid_domain_id=1111
gtid_domain_id=4
server_id=10999
log_slave_updates=ON
log_bin = /var/log/mysql/mariadb-bin-log
Sometimes, the nodes will stay in sync. However, if the following is done:
1) transactions run on all three nodes
2) restart mariadb on any node
3) restart all three nodes, using galera_new_cluster on a different node from first startup
Then, GTIDs will become inconsistent both at the domain level and at the seq_no level.
MDEV-28015Mariabackup | GTID value is missing, Galera Cluster , adding async slave to it
Closed
MDEV-34170table gtid_slave_pos entries never been deleted with wsrep_gtid_mode = 0
Closed
MDEV-34924gtid_slave_pos table rows newer been deleted on non replica nodes (wsrep_gtid_mode = 1)
Closed
MDEV-32740wsrep_gtid_domain_id is disregarded on any node other than bootstrapped node when the wsrep_gtid_mode is set to ON with wsrep_provider_version 26.4.14 (Galera 4)
Closed
MDEV-33610Circular replication breaks after upgrading
Test case
1) Start the 3 node cluster
2) OLTP load started on all nodes
3) The second node is restarted
4) After 1 minute all nodes are stopped one by one
3) bootstrapped node3 and initialized remaining nodes
Note: reproduces with mariabackup not with rsync
Jan Lindström
added a comment - - edited Test case
1) Start the 3 node cluster
2) OLTP load started on all nodes
3) The second node is restarted
4) After 1 minute all nodes are stopped one by one
3) bootstrapped node3 and initialized remaining nodes
Note: reproduces with mariabackup not with rsync
edward Does customer use mariabacup as SST method?
With mariabackup I can reproduce it even at startup
connection node_1;
CREATE SCHEMA IF NOT EXISTS mysqlslap;
USE mysqlslap;
CREATE TABLE t1 (id bigint not null primary key auto_increment, name VARCHAR(64)) ENGINE=innodb;
# node_1
show variables like '%gtid_binlog_pos%';
Variable_name Value
gtid_binlog_pos 1111-1-2
connection node_2;
# node_2
show variables like '%gtid_binlog_pos%';
Variable_name Value
gtid_binlog_pos 0-1-2
connection node_3;
# node_3
show variables like '%gtid_binlog_pos%';
Variable_name Value
gtid_binlog_pos 0-1-2
Jan Lindström
added a comment - edward Does customer use mariabacup as SST method?
With mariabackup I can reproduce it even at startup
connection node_1;
CREATE SCHEMA IF NOT EXISTS mysqlslap;
USE mysqlslap;
CREATE TABLE t1 (id bigint not null primary key auto_increment, name VARCHAR(64)) ENGINE=innodb;
# node_1
show variables like '%gtid_binlog_pos%';
Variable_name Value
gtid_binlog_pos 1111-1-2
connection node_2;
# node_2
show variables like '%gtid_binlog_pos%';
Variable_name Value
gtid_binlog_pos 0-1-2
connection node_3;
# node_3
show variables like '%gtid_binlog_pos%';
Variable_name Value
gtid_binlog_pos 0-1-2
rsync copies maybe some more files for a successful recovery or whatever
wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
do .
So I suggest to init
uint wsrep_gtid_domain_id=0; // Domain id on above structure
to real wsrep_gtid_domain_id defined in config file
or sst_method = mariabackup have to copy the same files like rsync for a successul wsrep_get_SE_checkpoint<wsrep_server_gtid_t>(); seppo and janlindstrom
Can you check this, if this makes sense to you ?
Richard Stracke
added a comment - - edited I checked the sourcecode and I shortened the relevant parts
******************************************
uint wsrep_gtid_domain_id= 0 ; // Domain id on above structure
void wsrep_init_globals()
wsrep_init_gtid();
wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
stored_gtid.domain_id= wsrep_gtid_server.domain_id;
if (stored_gtid.server_id == 0 )
rpl_gtid wsrep_last_gtid;
mysql_bin_log.lookup_domain_in_binlog_state(stored_gtid.domain_id,
&wsrep_last_gtid))
wsrep_gtid_server.gtid(stored_gtid);
if (wsrep_new_cluster)
new_gtid.domain_id= wsrep_gtid_domain_id; # wsrep_gtid_domain_id= 0 ;
/* Try to search for domain_id and server_id combination in binlog if found continue from last seqno */
wsrep_get_binlog_gtid_seqno(new_gtid);
wsrep_gtid_server.gtid(new_gtid);
*********************************************
So domain_id will be initialized with
wsrep_gtid_domain_id=0
wsrep_gtid_domain_id can be set from SST , otherwise it remain 0
wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
will set the grid from recovery process, otherwise it kept empty.
If new cluster the domain_id will be set to
new_gtid.domain_id= wsrep_gtid_domain_id;
IF the recovery process gives no result and new cluster flag is set,
wsrep_gtid_domain_id kept 0 and will be overwritten here
if (wsrep_new_cluster)
new_gtid.domain_id= wsrep_gtid_domain_id; # wsrep_gtid_domain_id= 0 ;
rsync copies maybe some more files for a successful recovery or whatever
wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
do .
So I suggest to init
uint wsrep_gtid_domain_id=0; // Domain id on above structure
to real wsrep_gtid_domain_id defined in config file
or sst_method = mariabackup have to copy the same files like rsync for a successul wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
seppo and janlindstrom
Can you check this, if this makes sense to you ?
Bug
Critical
This was tested on 10.6.14 Enterprise.