# STOP SLAVE stalls: relay log state and timing

Built 2026-11-02 from lab logs only (local kind cluster, Enterprise Server 11.8.8-5, MaxScale 25.10.4). No SHOW SLAVE STATUS was captured at the moment of STOP SLAVE, so "relay log state" below comes from the candidate's own error log (details under "How relay log state was decided").

## Result

Every stall with a readable log had the relay log fully applied. The candidate's SQL thread stopped at the same master binlog position (and GTID) that the IO thread had read up to. So nothing was left to apply, and the replica still waited several seconds before STOP SLAVE returned. This meets the bar the Server team set ("a situation where it should not", meaning no unprocessed relay log).

Across the Oct 21 to Oct 29 lab runs with MaxScale's "Failed to stop replica connection" error, 106 of 113 stalls show equal SQL and IO positions, 0 show a difference, and 7 could not be read (IO thread exit line not in the window). Plus the 5 no-operator stalls below, all equal.

## How relay log state was decided

For each STOP SLAVE on a candidate replica, the server error log has these lines in order:

1. `Error reading relay log event: slave SQL thread was killed` (STOP SLAVE starts, SQL thread stops).
2. `Slave SQL thread exiting, replication stopped in log 'F' at position P; GTID position 'G'` (P is the master binlog position the SQL thread had applied).
3. `Slave I/O thread killed during or after a reconnect done to recover from failed read` (only in some runs).
4. `Slave I/O thread exiting, read up to log 'F', position P2; GTID position G2` (P2 is the position the IO thread had read up to).
5. `cannot connect to master to kill slave io_thread's connection` (about 1 s after line 4 when `rpl_semi_sync_slave_kill_conn_timeout=1`).

If F and P match F and P2, the SQL thread had applied everything the IO thread had received. That is what every stall shows. Limit: this is the position at thread exit, not a polled SHOW SLAVE STATUS. A rerun that polls SHOW SLAVE STATUS every 200 ms on the candidate would confirm it directly (not run).

## Stalls named for the report

Times are UTC from the candidate's error log. "STOP SLAVE start" is the SQL thread kill line. "IO exit" is how long after that the IO thread exited. "Kill msg" is when the `cannot connect to master to kill` line was logged. Fault: Node kill = the primary's node was stopped (SIGKILL on the kind worker container). Cut-3 = NetworkPolicy blocking new connections between zones, then STOP/START SLAVE on the replicas. Cut-2 = hard cut that also drops existing connections.

Settings key: B/C = defaults for `rpl_semi_sync_slave_kill_conn_timeout`, MaxScale `backend_timeout=3s`. I1/I2 = kill_conn 1 s / 2 s, `backend_timeout=3s`. M and P = kill_conn 1 s, `backend_timeout=7s`, `check_repl_on_stop_slave_timeout=true`.

| # | Run (relative to work-products/) | Settings | Fault | Candidate | STOP SLAVE start | IO exit | Kill msg | Relay log at exit | MaxScale result |
|---|---|---|---|---|---|---|---|---|---|
| 1 | local-zone-loss-lab/runs-v267-matrix-005/s2-B-cut3-2 | B | Cut-3 | core-db-0 | 2026-10-21 17:11:40.357 | +2.94 s | +2.94 s | applied (132186 0-11-2947 both) | STOP SLAVE finished, no error |
| 2 | local-zone-loss-lab/runs-v267-matrix-006/s2-B-cut3-3 | B | Cut-3 | core-db-1 | 2026-10-21 17:33:35.702 | +3.22 s | +3.22 s | applied (952371 0-10-2515 both) | Failed to stop replica connection |
| 3 | same run, second attempt | B | Cut-3 | core-db-1 | 2026-10-21 17:34:11.416 | +8.73 s | +8.73 s | applied (same) | Failed to stop replica connection |
| 4 | local-zone-loss-lab/runs-v267-matrix-040/s1-I1-cut3-1 | I1 | Cut-3 | core-db-1 | 2026-10-27 16:19:57.051 | +4.87 s | +4.87 s | applied (991356 0-10-2514 both) | Failed to stop replica connection |
| 5 | local-zone-loss-lab/runs-v267-matrix-049/s2-I2-cut3-2 | I2 | Cut-3 | core-db-1 | 2026-10-27 19:45:01.672 | +4.77 s | +4.77 s | applied (992417 0-10-2517 both) | Failed to stop replica connection |
| 6 | local-zone-loss-lab/runs-v267-matrix-044/s2-I1-cut3-2 | I1 | Cut-3 | core-db-1 | 2026-10-27 17:41:55.800 | +7.83 s | +7.83 s | applied (993460 0-10-2520 both) | Failed to stop replica connection |
| 7 | local-zone-loss-lab/runs-v267-matrix-054/s2-M-cut3-2 | M, operator running | Cut-3 | core-db-1 | 2026-10-28 16:25:16.795 | +7.62 s | +7.62 s | applied (71063 0-10-3387 both) | Failed to stop replica connection (limit 7 s) |
| 8 | local-zone-loss-lab/runs-v267-matrix-070/s1-P-cut3-3 | P, operator running | Cut-3 | core-db-0 | 2026-10-29 00:22:24.346 | +7.45 s | +7.45 s | applied (1033558 0-11-2650 both) | Failed to stop replica connection |
| 9 | local-zone-loss-lab-noop/runs/no-operator/fixture-003/s3b1-M-node-loss-1 | M, plain (operator scaled to zero) | Node kill | core-db-1 | 2026-10-30 18:21:38.647 | +7.63 s | +8.63 s | applied (805582 0-10-2473 both) | Failed to stop replica connection; failover failed, retried about a minute later and succeeded |
| 10 | local-zone-loss-lab-noop/runs/no-operator/fixture-003/s3b1-M-node-loss-2 | M, plain | Node kill | core-db-0 | 2026-10-30 18:29:36.253 | +3.25 s | +4.25 s | applied (273906 0-11-3347 both) | STOP SLAVE finished, failover performed |
| 11 | local-zone-loss-lab-noop/runs/no-operator/fixture-003/s3b1-M-node-loss-3 | M, plain | Node kill | core-db-1 | 2026-10-30 18:41:06.800 | +7.58 s | +8.59 s | applied (565145 0-10-4274 both) | Failed to stop replica connection |
| 12 | same run, second attempt | M, plain | Node kill | core-db-1 | 2026-10-30 18:42:03.651 | +7.63 s | +7.63 s | applied (same) | Failed to stop replica connection |
| 13 | same run, third attempt | M, plain | Node kill | core-db-2 | 2026-10-30 18:43:21.065 | +8.39 s | +9.39 s | applied (same) | Failed to stop replica connection |
| 14 | local-zone-loss-lab-noop/runs/no-operator/fixture-011/s1-M-cut2-2 | M, plain | Cut-2 | core-db-2 | 2026-10-30 22:03:21.941 | +9.49 s | +10.49 s | applied (805596 0-10-2473 both) | Failed to stop replica connection; promoted core-db-1 left with no replica attached |

In row 9 the SQL thread stopped at 18:21:38.647 and the IO thread logged "killed during or after a reconnect done to recover from failed read" at 18:21:46.280, so the IO thread was busy with a reconnect attempt to the unreachable primary for about 7.6 s. The `cannot connect to master to kill` line came 1 s later, which matches `rpl_semi_sync_slave_kill_conn_timeout=1`. So the 1 s setting shortens the kill step, but the 7 s to 9 s wait before it is a separate wait and is not covered by that setting. What the IO thread was waiting on is not established from these logs.

Primary state when the stall happened:
- Node kill (rows 9 to 13): the primary's node was gone, so no connection could be made.
- Cut-3 (rows 1 to 8): the primary was running but new connections from the replica's zone were blocked by a NetworkPolicy.
- Cut-2 (row 14): hard network cut (existing connections dropped, new ones dropped).

## Not found

- No failed STOP SLAVE was found inside the fault window of run s2-M-cut3-3 (same set as row 7).
- Per-run log files carry lines from the previous run on the same cluster. Rows above are limited to each run's fault-to-end window from `markers.json`.