With wsrep_gtid_mode=ON, a node can adopt a GTID server_id that no cluster member is using, causing nodes to stamp different GTIDs for the same writeset

Description

With wsrep_gtid_mode=ON, the cluster-wide GTID server_id is established by the bootstrap node and thereafter read from the wsrep XID persisted in the InnoDB rollback segment (wsrep_init_gtid(), sql/wsrep_mysqld.cc). A non-bootstrap node deliberately ignores its own configured server_id and uses the recovered one — this is intentional and covered by galera_gtid_server_id.test.

The problem is that the recovered value is only ever a historical record of what this datadir was stamping when it was last written. Nothing validates it against the identity the cluster is currently using. As a result a node can adopt a server_id that no member is using any more, and different nodes then stamp different GTIDs for the same writeset — which breaks the invariant that a GTID identifies the same transaction on every node.

Observed behaviour

Two-node cluster, both nodes configured with the same server_id, both stamping the same seqno but different server_id:

node 1 (writer):   gtid_binlog_state | 7-99621-118

node 2 (applier):  gtid_binlog_state | 7-60621-118

60621 was a server_id the cluster had used previously. Node 2's recovered position confirms where it came from:

WSREP: Recovered position: e0e61b35-8bf2-11f1-b976-766050a0191d:110,7-60621-85

An asynchronous replica attached to this cluster cannot be repositioned between the two nodes, since the same transaction has two different GTIDs depending on which node it reads from.

Reproduction

This needs no SST and no timing window — a plain rejoin is enough:

Bootstrap a two-node cluster with wsrep_gtid_mode=ON and server_id=A on both nodes. Run some writes so both nodes persist a checkpoint carrying A.

Stop both nodes. Change server_id to B in the configuration on both.

Bootstrap node 1 (--wsrep-new-cluster). It takes the configured B, as intended.

Start node 2 normally. It recovers A from its own checkpoint and logs Ignoring server id B for non bootstrap node, using A.

Write on node 1 and compare gtid_binlog_state on both nodes: node 1 stamps domain-B-seqno, node 2 stamps domain-A-seqno for the same writeset.

Node 2 has adopted an identity that no member of the cluster is using.

The same outcome is reachable through SST, by a different route: for rsync and mariabackup the joiner's identity comes from the donor's copied InnoDB pages (wsrep_sst_received() only rewrites the checkpoint when !wsrep_before_SE(), i.e. for mysqldump), so it inherits whatever the donor had persisted at snapshot time. The rpl-GTID part of the checkpoint is refreshed only by a commit, a primary view change, or logical SST — a bootstrap writes it in memory only — so a donor that has just been bootstrapped with a new server_id and has not yet committed anything will hand out the previous value. Note also that recovery selects the XID with the highest cluster seqno across all 128 rollback segments, which is another way for an older record to win.

Incidentally, this also means the effective identity depends on the SST method: mysqldump SST gives the joiner its own configured value, rsync/mariabackup give it the donor's persisted one.

Impact

• Divergent GTIDs for the same transaction across nodes; an async replica cannot switch between nodes.
• Not correctable through configuration — check_server_id() blocks runtime changes under wsrep_gtid_mode, and a non-bootstrap restart re-reads the stale value, so recovery requires bootstrapping and re-provisioning the other nodes.
• Hard to diagnose: the effective identity is not observable in 10.6. The wsrep_se_checkpoint status variable, which prints the persisted XID including the GTID identity, is present in current main (13.1) but not in 10.6.25 — we have not checked which release first carried it. Without it the value can only be inferred from the middle field of gtid_binlog_state, or from a --wsrep-recover run.

Suggested direction

The underlying issue is that the GTID identity is cluster state but is persisted only node-locally, and a node trusts its local copy unconditionally. Some options, in increasing order of scope:

Minimum: on a non-bootstrap start, when the recovered server_id differs from the configured one, make that visible enough to act on - the existing message is logged at note level (WSREP_INFO), so it is easy to miss and is not collected by MTR's error-log check; raising it to warning would make it visible, and most affected tests already carry a matching mtr.add_suppression call. Exposing the effective identity as a status variable would also help, (backporting wsrep_se_checkpoint would be enough). This does not fix the divergence but makes it detectable.

Workaround

Where all nodes agree on the same effective server_id and only the configuration disagrees, setting server_id in the configuration on every node to the effective value (a.k.a "change every node's config to match a value nobody chose") and performing a rolling restart resolves the visible symptoms: @@server_id then matches what the cluster stamps, and gtid_current_pos tracks the wsrep domain again. No bootstrap or SST is needed. server_id cannot be changed at runtime under wsrep_gtid_mode (check_server_id()), so this requires a configuration change and restart. Confirmed effective on a production three-node cluster.

Note that this makes the effective value a property of the cluster rather than of each node: any node added later must be configured with the same value, or it will inherit the cluster's value and disagree with its own configuration again.

Where nodes have actually diverged and are stamping different server_ids, this does not help — the GTID streams have already split. That case requires bootstrapping the node with the intended identity and re-provisioning the others from a fresh datadir.

Narrow fix: persist the identity when it is adopted at bootstrap, so a donor can never hand out a value it is no longer using. Note that rewriting the checkpoint with an unchanged cluster seqno trips ut_ad(xid_seqno > wsrep_seqno) in trx_rseg_update_wsrep_checkpoint() on debug builds, so this needs care about where it is done. This closes the SST route but not the rejoin route above.

Complete fix: give the GTID server_id an explicit channel so a joining or rejoining node learns the cluster's current identity from the group rather than from its own disk. wsrep_gtid_domain_id already works this way — it is transferred over the SST channel and applied in wsrep_sst.cc — and wsrep_schema already persists cluster state such as the view, so the mechanism exists. A node whose recovered identity disagrees with the cluster's could then adopt the cluster value (or refuse to start) rather than silently diverge.

Related

The same latch has a second, separately visible consequence: because binlog GTIDs whose server_id differs from @@server_id are filtered out of gtid_current_pos (rpl_slave_state::iterate()), a non-bootstrap node omits the cluster's own GTID domain from gtid_current_pos, which MaxScale and master_use_gtid=current_pos consume. This is visible in the committed galera_gtid_server_id.result, where node 2 shows gtid_binlog_pos 0-12-1,1-11-2 but gtid_current_pos 0-12-1. Happy to file that separately if preferred.

Narrow fix: persist the identity when it is adopted at bootstrap, so a donor can never hand out a value it is no longer using. Note that rewriting the checkpoint with an unchanged cluster seqno trips ut_ad(xid_seqno > wsrep_seqno) in trx_rseg_update_wsrep_checkpoint() on debug builds, so this needs care about where it is done. This closes the SST route but not the rejoin route above.

Complete fix: give the GTID server_id an explicit channel so a joining or rejoining node learns the cluster's current identity from the group rather than from its own disk. wsrep_gtid_domain_id already works this way — it is transferred over the SST channel and applied in wsrep_sst.cc — and wsrep_schema already persists cluster state such as the view, so the mechanism exists. A node whose recovered identity disagrees with the cluster's could then adopt the cluster value (or refuse to start) rather than silently diverge.

Related

The same latch has a second, separately visible consequence: because binlog GTIDs whose server_id differs from @@server_id are filtered out of gtid_current_pos (rpl_slave_state::iterate()), a non-bootstrap node omits the cluster's own GTID domain from gtid_current_pos, which MaxScale and master_use_gtid=current_pos consume. This is visible in the committed galera_gtid_server_id.result, where node 2 shows gtid_binlog_pos 0-12-1,1-11-2 but gtid_current_pos 0-12-1. Happy to file that separately if preferred.

Narrow fix: persist the identity when it is adopted at bootstrap, so a donor can never hand out a value it is no longer using. Note that rewriting the checkpoint with an unchanged cluster seqno trips ut_ad(xid_seqno > wsrep_seqno) in trx_rseg_update_wsrep_checkpoint() on debug builds, so this needs care about where it is done. This closes the SST route but not the rejoin route above.

Complete fix: give the GTID server_id an explicit channel so a joining or rejoining node learns the cluster's current identity from the group rather than from its own disk. wsrep_gtid_domain_id already works this way — it is transferred over the SST channel and applied in wsrep_sst.cc — and wsrep_schema already persists cluster state such as the view, so the mechanism exists. A node whose recovered identity disagrees with the cluster's could then adopt the cluster value (or refuse to start) rather than silently diverge.

Related

The same latch has a second, separately visible consequence: because binlog GTIDs whose server_id differs from @@server_id are filtered out of gtid_current_pos (rpl_slave_state::iterate()), a non-bootstrap node omits the cluster's own GTID domain from gtid_current_pos, which MaxScale and master_use_gtid=current_pos consume. This is visible in the committed galera_gtid_server_id.result, where node 2 shows gtid_binlog_pos 0-12-1,1-11-2 but gtid_current_pos 0-12-1. Happy to file that separately if preferred.