STOP SLAVE stalls for several seconds on a slave whose master is unreachable, despite rpl_semi_sync_slave_kill_conn_timeout=1

Summary

On a semi-sync slave whose master has become unreachable (node stopped, or network blocked), STOP SLAVE takes about 3 to 10 seconds instead of returning promptly, even though the relay log is fully applied. After the wait the slave logs "cannot connect to master to kill slave io_thread's connection". rpl_semi_sync_slave_kill_conn_timeout=1 shortens that last step to 1 s, but not the wait before it. In all 106 stalls where the slave's error log could be read, the SQL thread had stopped at the same master binlog position and GTID that the IO thread had read up to, so nothing was left to apply.

Related: MXS-5688 (MaxScale added a workaround, check_repl_on_stop_slave_timeout, and could not reproduce the stall on its side).

Environment

• Enterprise Server 11.8.8-5 (not yet tried on Community), 1 master + 2 slaves, GTID replication over TLS
• Semi-sync AFTER_SYNC, master timeout effectively infinite, rpl_semi_sync_slave_kill_conn_timeout=1, slave_net_timeout=10
• MaxScale 25.10.4 mariadbmon issues the STOP SLAVE during failover, with max_statement_time=6 and backend_timeout=7s
• Kubernetes (kind) with Calico; zone loss simulated by stopping nodes or by blocking new connections between zones

Steps to reproduce

1. Run the topology above with steady writes on the master.
2. Make the master unreachable from a slave: stop its node, or block new connections between zones while existing ones stay open.
3. On the slave, run STOP SLAVE and time it. (In the lab, MaxScale runs it as part of an automatic failover.)

Expected

STOP SLAVE returns promptly when there is nothing left to apply, because the slave no longer needs the master.

Actual

• Master's node stopped (no network answer at all): STOP SLAVE took 3.25 s to 8.39 s over the 5 attempts listed in stalls.md, in 3 runs.
• Network blocked between zones (existing connections open, new ones blocked): 2.94 s to 8.73 s over 8 attempts in 7 runs.
• Hard network cut (existing connections dropped too): 9.49 s in 1 run.
• Relay log state: in every stall where both lines could be read, SQL thread exit position and IO thread exit position were equal (same binlog file, same position, same GTID).
• In one stall the IO thread logged "Slave I/O thread killed while connecting to master", so it was in a connection attempt to the unreachable master when it was stopped. This suggests STOP SLAVE waits for that attempt, but the logs do not prove it.
• When the wait exceeds MaxScale's limit, the failover step fails (error 2026, connection timed out) and is retried later, or completes only partly (the promoted slave is left without its peer attached).

  2026-10-30 18:21:38 16 [Note] Error reading relay log event: slave SQL thread was killed

  2026-10-30 18:21:38 16 [Note] Slave SQL thread exiting, replication stopped in log 'core-db-bin.000001' at position 805582; GTID position '0-10-2473'

  2026-10-30 18:21:46 15 [Note] Slave I/O thread killed during or after a reconnect done to recover from failed read

  2026-10-30 18:21:46 15 [Note] Slave I/O thread exiting, read up to log 'core-db-bin.000001', position 805582; GTID position 0-10-2473

  2026-10-30 18:21:47 15 [Note] cannot connect to master to kill slave io_thread's connection

  STOP SLAVE was issued at 18:21:38; the IO thread exited 7.6 s later and the "cannot connect" line came 1 s after that.

Evidence

• core-db-1-mariadb-error-log-2026-10-30T182100-182330Z.txt: slave error log around two stalls (18:21:38 and 18:22:49). A restart of mariadbd by the orchestrator at 18:21:59 is marked as omitted.
• maxscale-log-2026-10-30T182100-182330Z.txt: MaxScale log for the same window, showing the STOP SLAVE attempts and their outcome.
• stalls.md: table of 14 stalls from lab runs (timing, fault type, relay log state), and how the relay log state was decided.

Not captured

• SHOW SLAVE STATUS, SHOW FULL PROCESSLIST and performance_schema.metadata_locks at the moment of STOP SLAVE. Relay log state comes from the positions in the error log at thread exit.
• What the IO thread was waiting on during the stall.

Not tested

• Community Server
• Other 11.x versions
• Bare metal or VMs outside Kubernetes

Impact

Automatic failover after a zone loss can fail or complete only partly, depending on how long STOP SLAVE takes relative to the monitor's time limit.