Details
-
Bug
-
Status: Open (View Workflow)
-
Critical
-
Resolution: Unresolved
-
10.6(EOL), 10.11, 11.4, 10.6.28, 10.11.18
-
Can result in hang or crash
Description
Summary
`master_retry_count=0` is documented as "never stop attempting to reconnect", but the
slave I/O thread gives up after a single reconnect attempt and exits **without logging any
error**. `Last_IO_Errno` stays `0` and `Last_IO_Error` stays empty, so the failure is
invisible to any monitoring that watches for replication errors.
Documented behaviour
https://mariadb.com/docs/server/server-management/starting-and-stopping-mariadb/mariadbd-options
states for `--master-retry-count`: A value of 0 means the replica will not stop attempting to reconnect.
Actual behaviour
Zero is exempted in one of the two retry paths but not the other.
- 1. `connect_to_master()` — honours the documented semantics.** 10.11 `sql/slave.cc:8000`:
if ((++err_count == master_retry_count) && master_retry_count ) |
The trailing `&& master_retry_count` exempts zero, so connection attempts within a single
reconnect are indeed unlimited. This is the behaviour the documentation describes.
- 2. `try_to_reconnect()` — does not.** 10.11 `sql/slave.cc:4944-4948`:
if ((*retry_count)++) |
{
|
if (*retry_count > master_retry_count) |
return 1; // Don't retry forever |
slave_sleep(thd, mi->connect_retry, io_slave_killed, mi);
|
}
|
No zero-exemption. With `master_retry_count = 0`:
- 1st call — `(*retry_count)++` evaluates to `0` → falsy → block skipped entirely, no
give-up check, no sleep. Reconnect proceeds. Counter becomes 1. - 2nd call — evaluates to `1` → truthy → enters block with counter now `2` →
`2 > 0` → `return 1`.
All four call sites (`sql/slave.cc:5178, 5197, 5225, 5283`) respond to `return 1` with
`goto err`, whose only output is:
[Note] Slave I/O thread exiting, read up to log '<file>', position <pos>; GTID position <gtid>
|
No error, no warning. Notably *not* the `Slave I/O thread killed during or after
reconnect` message, even with `log_warnings=2`.
The counter is reset only after an event is successfully read — `sql/slave.cc:5289`:
retry_count=0; // ok event, reset retry counter |
not after a successful reconnect. So the effective behaviour of `master_retry_count=0` is
*exactly one reconnect permitted between successful event reads*, rather than unlimited.
Why it is rarely noticed, and why it is serious when hit
An ordinary network blip is absorbed inside `connect_to_master()`'s own loop; an event
eventually arrives and resets the counter. The bug only bites when a reconnect succeeds
but no event arrives before the next failure — e.g. a path that completes a TCP handshake
but delivers nothing, or simply an idle master. Then `read_event()` waits out
`slave_net_timeout`, fails a second time, and the thread exits silently and permanently.
Because `Last_IO_Errno` is `0`, the usual monitoring signals show nothing wrong. The SQL
thread stays `Yes` and drains the relay log, so `SHOW SLAVE STATUS` looks healthy apart
from `Slave_IO_Running: No` and `Seconds_Behind_Master: NULL`.
Reproduction
Deterministic, using an idle master so the reset at `slave.cc:5289` never fires.
# Slave configuration
|
# master-retry-count=0
|
# slave-net-timeout=10 (shortened to speed the test up)
|
|
|
# Route the slave to the master through a proxy we can break at will
|
socat TCP-LISTEN:3307,fork,reuseaddr TCP:<master-host>:3306 &
|
-- On the slave
|
CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=3307, |
MASTER_USER='<user>', MASTER_PASSWORD='<pw>', |
MASTER_USE_GTID=slave_pos;
|
START SLAVE;
|
-- confirm Slave_IO_Running: Yes |
Keep the master *completely idle* (no writes), then:
1. Kill and restart the `socat` proxy → read failure #1. The slave reconnects
successfully; `retry_count` is now 1 and, with no events flowing, never resets.
2. Kill and restart the proxy again → read failure #2.
Expected: the I/O thread keeps reconnecting indefinitely, per the documentation.
Actual: the I/O thread exits. Error log contains only `Slave I/O thread exiting`;
`SHOW SLAVE STATUS` reports `Slave_IO_Running: No`, `Last_IO_Errno: 0`,
`Last_IO_Error:` empty. Replication never resumes without manual `START SLAVE`.
Setting `master-retry-count=100000` and repeating the same sequence recovers normally,
isolating the variable.
Suggested fix
Apply the same zero-exemption `connect_to_master()` already uses
Remove the extraneous retry_count check in try_to_reconnect() by backporting the first commit from PR 3764 (MDEV-35304) to 10.11–11.8.
The refactor in `main` removes the defect as a side effect: `try_to_reconnect()` no longer
takes a retry counter and contains no give-up guard, and the surviving limit in
`connect_to_master()` carries the zero-exemption:
if ((++(mi->connects_tried) == mi->retry_count) && mi->retry_count) |
Related
- MySQL Bug #68203, "slave does not retry connect" — https://bugs.mysql.com/bug.php?id=68203
Describes the same defect in the same function; MariaDB appears to have inherited it. MDEV-36002: also about --master-retry-count=0 not working, but about a different cause in connect_to_master().
Its patch has a reüsable test snippet that this can base on.
It did not encounter this issue, likely because the test worked by closing an already established connection, whereas this issue occurs with a connection that’s never established.
Field-observed impact (context, not required for the report)
Found on a production MariaDB 10.11.18 replica whose link to its master was disrupted by an
IP address conflict on the LAN. Over ~6 weeks the I/O thread died silently ~14 times
without any server restart. The longest single outage ran roughly 10 hours before anyone
noticed, because no error was ever recorded and the SQL thread continued to report healthy.
Attachments
Issue Links
- relates to
-
MDEV-36002 --master-retry-count=0 can still stop
-
- Closed
-
- split from
-
MDEV-35304 More Details for Ongoing IO Thread Re-connection Attempts
-
- Closed
-