Semi-sync master hangs when a replica reconnects while its old connection is dead: Ack_receiver::remove_slave() never wakes the ack receiver

Summary

When a semi-sync replica disappears without closing its TCP connection (host powered off / network cut, so no FIN or RST reaches the primary) and later reconnects from a different IP address, the primary kills the old ("zombie") binlog dump thread for that server_id.

That thread then blocks in Ack_receiver::remove_slave(), waiting for the ack receiver thread to acknowledge the removal. But the ack receiver thread is blocked in poll(..., -1) on the replica sockets and is never woken: remove_slave() does not call signal_listener(), unlike add_slave() and stop(). It only wakes when the kernel finally errors the dead socket, which takes up to ~15 minutes with the default net.ipv4.tcp_retries2 = 15.

Until then:

• the old dump thread stays in "Waiting to finalize termination";
• the new dump thread stays in "starting" (in kill_zombie_dump_threads());
• in our observation, new client logins also hang: mariadb -e "SELECT 1" from localhost times out after 30 s.

If the replica reconnects from the same IP, the problem does not occur: the replica's kernel answers the primary's retransmissions with RST, the socket errors, and poll() returns.

How to reproduce

It reproduces reliably. On the replica's host, silently drop all its traffic in both directions, so the existing connection is not reset:

iptables -t raw -I PREROUTING 1 -s <replica_ip> -j DROP

iptables -t raw -I PREROUTING 1 -d <replica_ip> -j DROP

1. On the primary, open a session beforehand and keep it logged in, polling SHOW FULL PROCESSLIST every 2 s. New logins will hang later.
2. Write once on the primary, e.g. CREATE DATABASE x; DROP DATABASE x;. It waits 10 s, then logs "Timeout waiting for reply of binlog" and "Semi-sync replication switched OFF". (This step is optional; the result was the same with semi-sync still ON.)
3. Wait a few minutes. The primary still shows the old Binlog Dump thread as healthy ("Master has sent all binlog to slave; waiting for more updates").
4. Start a fresh replica process with a new IP address: same server_id, same GTID position. In our case the replica container was deleted and recreated, and got a new pod IP. It connects to the primary.
5. Keep the DROP rules for the old IP in place.

Observed

Primary error log at the moment the new replica connects:

[Note] Stop semi-sync binlog_dump to slave (server_id: 11)

No "Start binlog_dump to slave_server(11) ..." follows, and nothing more is logged until the server is restarted.

Processlist, from the session opened before the test:

| 4420 | repl | <old_ip>:54998 |      | Busy        | 732 | Waiting to finalize termination | NULL |

| 5575 | repl | <new_ip>:34186 | NULL | Binlog Dump |   0 | starting                        | NULL |

It stays like this, and every new login (mariadb -uroot -e "SELECT 1" over the local socket) times out; our client-side timeout was 30 s.

With the old IP reachable again (the replica returning from the same IP), the same sequence completes within the same second: "Stop semi-sync binlog_dump", then "Start binlog_dump", then "Start semi-sync binlog_dump", then "Semi-sync replication switched ON".

Analysis (12.3.3 source)

mysql_binlog_send() sets stage_waiting_to_finalize_termination and calls repl_semisync_master.dump_end(thd), which calls ack_receiver.remove_slave(thd).

Ack_receiver::remove_slave() (sql/semisync_master_ack_receiver.cc):

  if (slaves_changed)

  {

    m_slaves_changed= true;

    mysql_cond_broadcast(&m_cond);

    ...

    if (m_status != ST_DOWN)

      mysql_cond_wait(&m_cond_reply, &m_mutex);

  }

Ack_receiver::run() only signals m_cond_reply after it re-enters its loop. But while slaves exist, it is blocked in:

  bool listen_on_sockets()

  {

    return poll(m_fds.data(), m_fds.size(), -1);   // no timeout

  }

m_cond does not wake a thread that is blocked in poll().

• add_slave() handles this with signal_listener(); // Inform listener that there are new slaves (the signal pipe is part of the polled fds), and so does stop().
• remove_slave() doesn't. So with a dead peer (no data, and no HUP or ERR until TCP gives up), remove_slave() waits until the kernel errors the socket.

Secondary: kill_zombie_dump_threads() is meant to wait at most SECONDS_TO_WAIT_FOR_DUMP_THREAD_KILL, but its loop

for (uint i= 10 * SECONDS_TO_WAIT_FOR_DUMP_THREAD_KILL; --i > 0 && !thd->killed; i++)

decrements and increments i in each iteration, so it never counts down.

The code is unchanged on main.

Suggested fix

• Call signal_listener() in Ack_receiver::remove_slave() after setting m_slaves_changed, as add_slave() does.
• Optionally, give listen_on_sockets() a finite timeout as a safety net.
• Fix the loop counter in kill_zombie_dump_threads().

Workarounds evaluated

• A lower net.ipv4.tcp_retries2 (e.g. 5, so a dead connection errors within ~12 s) bounds the hang. It is not always settable: Kubernetes, for example, treats it as an unsafe sysctl.
• net_write_timeout has no effect (tested with 15).
• TCP keepalive doesn't apply: the heartbeats keep the connection in retransmission.
• Disabling semi-sync avoids the code path, at the cost of semi-sync durability.

Possibly related: MySQL bug #90857 (dump thread stuck in "Waiting to finalize termination", new dump threads stuck in "starting"; closed as "can't repeat").