Switchover freezes the primary for switchover_timeout and then fails when the target has log_slave_updates=ON but its gtid_binlog_pos is behind its gtid_current_pos

Problem

A switchover to a replica that has log_slave_updates=ON but a stale gtid_binlog_pos can never succeed. The monitor makes the primary read-only, waits the full switchover_timeout for a catch-up that cannot happen while the primary is frozen, then undoes the switchover. When the switchover is retried automatically (for example by the Kubernetes Operator during a rolling update), writes are blocked for about 90 s in every 92 s until a single transaction happens to commit on the primary between attempts.

This was seen in a production-like environment and reproduced in a lab: 4 of 4 runs with no write traffic stalled.

How the target gets a stale gtid_binlog_pos

1. Server A is the primary and writes its own GTIDs to its binlog (0-10-N).
2. A switchover makes server B the primary. A now replicates with log_slave_updates=OFF, so replicated events (0-11-M) are applied but not written to A's binlog. A's gtid_binlog_pos stays at 0-10-N, while gtid_current_pos / gtid_slave_pos follow the primary.
3. A restarts with log_slave_updates=ON (a config change). Its gtid_binlog_pos is still 0-10-N until it logs its first replicated event.
4. With no writes on the primary, a switchover to A is requested.

Why the catch-up can't finish

• MariaDBServer::catchup_to_master uses gtid_binlog_pos when log_bin && log_slave_updates (mariadbserver.cc:531-547 in 25.10.3), and compares it to the demotion target's gtid_binlog_pos (cluster_manipulation.cc:950).
• The target's binlog pos (0-10-N) is behind the primary's (0-11-M) in domain 0. It can only advance when a new event replicates and is logged, and the demotion has just frozen the primary (read_only=1, FTWRL). The wait always times out.
• The target's gtid_current_pos is already equal to the primary's, so the data is fully caught up; only the binlog is missing those events.
• A target that never logged anything (empty gtid_binlog_pos) passes, because events_ahead(..., MISSING_DOMAIN_IGNORE) skips domains missing from the target (gtid.cc:190-240). So in practice only a former primary is affected. (Read in code; not tested as a target.)
• Demotion already treats gtid_binlog_pos as untrustworthy when log_slave_updates is off (mariadbserver.cc:1120-1124, with the TODO "reduce dependency on gtid_binlog_pos"). The promotion target's binlog pos is trusted as soon as the setting is on, even when it is visibly behind the server's own gtid_current_pos.

Lab evidence

Setup: 3 servers, MariaDB Enterprise Server 11.8.8-5, MaxScale 25.10.4, GTID replication, semi-sync, switchover_timeout default (90s). Topology was managed by MariaDB Enterprise Operator 26.6.7. The switchover failure was also reproduced with a direct REST switchover while the Operator was paused.

GTIDs captured during the catch-up wait (server-0 = target, server-1 = primary being demoted):

server-0  log_slave_updates=1  gtid_binlog_pos=0-10-38  gtid_current_pos=0-11-41  gtid_slave_pos=0-11-41  read_only=1

server-1  log_slave_updates=0  gtid_binlog_pos=0-11-41  gtid_current_pos=0-11-41                          read_only=1

server-2  log_slave_updates=1  gtid_binlog_pos=(empty)  gtid_current_pos=0-11-41  gtid_slave_pos=0-11-41  read_only=1

MaxScale log for each failed attempt (92 s apart):

error  : [mariadbmon] Replica catchup timed out on replica 'server-0'.

notice : [mariadbmon] Attempting to undo changes to 'server-1'.

notice : [mariadbmon] 'server-1' restored to original status.

error  : [mariadbmon] Switchover server-1 -> server-0 failed.

notice : [mariadbmon] Disabling automatic cluster operations for 5 monitor ticks.

Other gaps captured in separate runs: 0-10-602 vs 0-11-603, 0-10-399 vs 0-11-400, 0-10-794 vs 0-11-795. A gap of one transaction is enough.

What ended the stall in the lab:

• Any single commit on the primary between attempts (an application INSERT through MaxScale). The next attempt succeeded.
• As a side effect, a runtime config change (PATCH of the monitor) with config_sync_cluster set, because the config sync INSERT into mysql.maxscale_config commits on the primary.
• switchover-force: it waits the full timeout, then promotes anyway.

Expected

The switchover should either succeed (the target has every transaction, per gtid_current_pos) or be rejected before the primary is frozen, with a message naming the cause.

Possible fix directions (your call)

• Before demotion, detect a target whose gtid_binlog_pos is behind its own gtid_current_pos. Then either reject it with a clear reason, or treat it like log_slave_updates=OFF (compare gtid_current_pos and log the existing "replication will break for lagging replicas" warning).
• During catch-up, stop early when the target's gtid_current_pos has reached the primary and the binlog pos is not moving, instead of holding the primary frozen for the full timeout.