Details
Description
I know that changing wsrep_gtid_domain_id in the configuration file on all nodes requires a full cluster stop and restart to pick up the change as only the first node will actually use the config file value, while all further nodes receive the value to use from their donor during IST/SST.
But when e.g. changing the relevant configuration settings from
wsrep-gtid-mode=ON
|
wsrep_gtid_domain_id=100
|
gtid_domain_id=...different value per node....
|
to
wsrep_gtid_domain_id=200
|
then stopping all nodes, staring node-1 with galera_new_cluster, and node-2 with systemctl start mariadb, all is fine so far, both nodes show wsrep_gtid_domain_id=200 in SHOW VARIABLES LIKE 'wsrep_gtid_domain_id'. But when starting a 3rd or further nodes they all usually still show 100 instead of 200.
Looking at the error logs I see node-3 usually use node-2 as donor, not node-1.
When I force the node started with galera_new_cluster as the default donor with wsrep_sst_donor=node-1 I get the correct new 200 value on all nodes though.
So somehow nodes seem to remember the previous value and pass that on to joiners instead of the value they received from their own donor, or read from their configuration file.
Attachments
Issue Links
- relates to
-
MDEV-25115 Changes to wsrep_gtid_domain_id in my.cnf are ignored on node restart
-
- Closed
-
-
-
- Closed
-
Activity
I still have my original VM test setup for this somewhere, "just" need to figure out where ...
hholzgra The wsrep_gtid_domain_id is only changed by SST when the secondary node is used as a donor node. Can you confirm if node3 has used SST or IST to join the cluster.
How to reproduce:
start cluster of at least three nodes with
wsrep_gtid_domain_id=100
Shut down all nodes, node 1 last. Change configuration to
wsrep_gtid_domain_id=200
Start node 1 with galera-new-cluster, then start up the remaining nodes with systemctl start mariabd
Run
mysql -e "show variables like 'wsrep_gtid_domain_id'"
on all nodes, see that first and second node show the correct value 200, but later nodes that join using a different donor than node 1 show the old value 100
When enforcing an SST on one of the later nodes (node 3 and later) it will show the correct value 200 regardless of the actual donor node picked, but when then restarting the node after SST startup has completed and a different node than the first one is picked for SST (eg. by enforcing that with wsrep_sst_donor=node-2) it will flip back to the old value 100 again.
When enforcing an SST on all but the first node by purging the data directory the correct value 200 is picked up by all nodes, and persists over later restarts, too.
So we should either clearly document the procedure necessary to change wsrep_gtid_domain_id, or – preferred – figure out and fix the behavior so that changing the domain id is possible without enforced SST on all but the first node.
julien.fritsch jplindst The problem has been reproduced. Even after SST, the wsrep code always chooses the old wsrep_gtid_domain_id if the donor is a non-bootstrap node when the joiner is restarted.
Please check the --gtid-domain-id value on IST log
xtrabackup IST info
When Node1 becomes the donor (--gtid-domain-id is 200)
2022-11-26 5:27:30 1 [Note] WSREP: Server status change synced -> donor
|
2022-11-26 5:27:30 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2022-11-26 5:27:30 0 [Note] WSREP: Donor monitor thread started to monitor
|
2022-11-26 5:27:30 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.100.30:4444/xtrabackup_sst//1' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid 'ee9f290d-6d41-11ed-a940-ffb1d812fa61:61' --gtid-domain-id 200 --bypass --mysqld-args --wsrep-new-cluster --wsrep_start_position=ee9f290d-6d41-11ed-a940-ffb1d812fa61:8,100-101-3'
|
2022-11-26 5:27:30 1 [Note] WSREP: sst_donor_thread signaled with 0
|
2022-11-26 5:27:30 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.100.30:4568 sending 62-63, preload starts from 63
|
2022-11-26 5:27:30 0 [Note] WSREP: IST sender 62 -> 63
|
When Node2 becomes the donor (--gtid-domain-id is 100)
2022-11-26 5:04:02 1 [Note] WSREP: Server status change synced -> donor
|
2022-11-26 5:04:02 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2022-11-26 5:04:02 0 [Note] WSREP: Donor monitor thread started to monitor
|
2022-11-26 5:04:02 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '192.168.100.30:4444/xtrabackup_sst//1' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 0 --datadir '/var/lib/mysql/' --gtid 'ee9f290d-6d41-11ed-a940-ffb1d812fa61:55' --gtid-domain-id 100 --bypass --mysqld-args --wsrep_start_position=ee9f290d-6d41-11ed-a940-ffb1d812fa61:7,100-101-3'
|
2022-11-26 5:04:02 1 [Note] WSREP: sst_donor_thread signaled with 0
|
2022-11-26 5:04:02 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.100.30:4568 sending 56-57, preload starts from 57
|
2022-11-26 5:04:02 0 [Note] WSREP: IST sender 56 -> 57
|
rsync IST info
When Node1 becomes the donor
2022-11-26 5:36:23 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.100.30:4444/rsync_sst' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 1 --datadir '/var/lib/mysql/' --gtid 'ee9f290d-6d41-11ed-a940-ffb1d812fa61:69' --gtid-domain-id 200 --bypass --mysqld-args --wsrep-new-cluster --wsrep_start_position=ee9f290d-6d41-11ed-a940-ffb1d812fa61:66,200-101-0'
|
2022-11-26 5:36:23 1 [Note] WSREP: sst_donor_thread signaled with 0
|
2022-11-26 5:36:23 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.100.30:4568 sending 70-71, preload starts from 71
|
2022-11-26 5:36:23 0 [Note] WSREP: IST sender 70 -> 71
|
When Node2 becomes the donor.
2022-11-26 5:35:18 1 [Note] WSREP: Server status change synced -> donor
|
2022-11-26 5:35:18 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
2022-11-26 5:35:18 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.100.30:4444/rsync_sst' --local-port 3306 --socket '/run/mysqld/mysqld.sock' --progress 1 --datadir '/var/lib/mysql/' --gtid 'ee9f290d-6d41-11ed-a940-ffb1d812fa61:63' --gtid-domain-id 100 --bypass --mysqld-args --wsrep_start_position=ee9f290d-6d41-11ed-a940-ffb1d812fa61:64,100-101-3'
|
2022-11-26 5:35:18 0 [Note] WSREP: Donor monitor thread started to monitor
|
2022-11-26 5:35:18 1 [Note] WSREP: sst_donor_thread signaled with 0
|
2022-11-26 5:35:18 0 [Note] WSREP: async IST sender starting to serve tcp://192.168.100.30:4568 sending 64-69, preload starts from 69
|
2022-11-26 5:35:18 0 [Note] WSREP: IST sender 64 -> 69
|
This issue causes the GTID sequence to be incorrectly generated on the cluster nodes. If we generate any transaction on node3 then gtid_binlog_pos on node3 and node1 will use new wsrep_domain_id but replicated event on node2 will select old wsrep_gtid_domain_id.
Node3
MariaDB [(none)]> select @@gtid_binlog_pos,@@gtid_current_pos,@@wsrep_gtid_domain_id; |
+-------------------+--------------------+------------------------+ |
| @@gtid_binlog_pos | @@gtid_current_pos | @@wsrep_gtid_domain_id |
|
+-------------------+--------------------+------------------------+ |
| 200-101-2 | 200-101-2 | 100 |
|
+-------------------+--------------------+------------------------+ |
1 row in set (0.000 sec) |
|
|
MariaDB [(none)]>
|
Node1
MariaDB [(none)]> select @@gtid_binlog_pos,@@gtid_current_pos,@@wsrep_gtid_domain_id; |
+-------------------+--------------------+------------------------+ |
| @@gtid_binlog_pos | @@gtid_current_pos | @@wsrep_gtid_domain_id |
|
+-------------------+--------------------+------------------------+ |
| 200-101-2 | 200-101-2 | 200 |
|
+-------------------+--------------------+------------------------+ |
1 row in set (0.000 sec) |
|
|
MariaDB [(none)]>
|
Node2
MariaDB [(none)]> select @@gtid_binlog_pos,@@gtid_current_pos,@@wsrep_gtid_domain_id; |
+-------------------+--------------------+------------------------+ |
| @@gtid_binlog_pos | @@gtid_current_pos | @@wsrep_gtid_domain_id |
|
+-------------------+--------------------+------------------------+ |
| 100-101-5 | 100-101-5 | 200 |
|
+-------------------+--------------------+------------------------+ |
1 row in set (0.001 sec) |
|
|
MariaDB [(none)]>
|
{code:sql}
|
|
|
Similarly if we create any transaction on node2 then gtid_binlog_pos on node3 and node2 will use old wsrep_domain_id but replicated transaction on node1 will select new wsrep_gtid_domain_id. |
*Node2*
|
{code:sql}
|
MariaDB [(none)]> create database db1; |
Query OK, 1 row affected (0.006 sec)
|
|
|
MariaDB [(none)]> select @@gtid_binlog_pos,@@gtid_current_pos,@@wsrep_gtid_domain_id; |
+-------------------+--------------------+------------------------+ |
| @@gtid_binlog_pos | @@gtid_current_pos | @@wsrep_gtid_domain_id |
|
+-------------------+--------------------+------------------------+ |
| 100-101-3 | 100-101-3 | 200 |
|
+-------------------+--------------------+------------------------+ |
1 row in set (0.000 sec) |
|
|
MariaDB [(none)]>
|
Node3
MariaDB [(none)]> select @@gtid_binlog_pos,@@gtid_current_pos,@@wsrep_gtid_domain_id; |
+-------------------+--------------------+------------------------+ |
| @@gtid_binlog_pos | @@gtid_current_pos | @@wsrep_gtid_domain_id |
|
+-------------------+--------------------+------------------------+ |
| 100-101-3 | 100-101-3 | 100 |
|
+-------------------+--------------------+------------------------+ |
1 row in set (0.000 sec) |
|
|
MariaDB [(none)]>
|
Node 1
MariaDB [(none)]> select @@gtid_binlog_pos,@@gtid_current_pos,@@wsrep_gtid_domain_id; |
+-------------------+--------------------+------------------------+ |
| @@gtid_binlog_pos | @@gtid_current_pos | @@wsrep_gtid_domain_id |
|
+-------------------+--------------------+------------------------+ |
| 200-101-1 | 200-101-1 | 200 |
|
+-------------------+--------------------+------------------------+ |
1 row in set (0.000 sec) |
|
|
MariaDB [(none)]>
|
ramesh Based on Hartmut comments, could you please try to reproduce?