Semi-sync ACKs through readwritesplit are counted as pending requests and leak memory after a master status change

Summary

Since the fix for MXS-5910, a replica with semi-sync replication can replicate through a readwritesplit listener. MaxScale routes each semi-sync ACK from the replica to the primary as a request that expects a reply. The primary never replies to an ACK, so the pending request count of the session grows without limit. If the primary is not a valid routing target for a moment, readwritesplit puts all later packets of the session into its query queue, and the queue never empties. The MaxScale process then grows until it is OOM-killed. In our environment, it grew by about 200 to 250 MB per minute.

Setup

• MaxScale 25.01.5 with readwritesplit and mariadbmon.
• MariaDB 10.6 primary and replicas, semi-sync on for the primary and the replicas.
• An external replica with semi-sync on, which replicates from the readwritesplit listener.

Observed behavior

The numbers in this section come from the environment where we saw the problem. They are examples only. The ACK rate, the growth rate and the time until the failure depend on the write load of the primary and on how often the primary status changes.

• When the replication session starts, active_operations of the primary grows by the ACK rate (about 700 per second here) and never goes down. On a MaxScale without a replication session, the value stays at 0 to 2.
• Some hours later, active_operations stops at a high value (for example 43,038,904). In the same minute, the RSS of the process starts to grow linearly (about 230 MB per minute here).
• /v1/maxscale/memory reported about 170 MB while the RSS was 21.5 GB.
• At that minute, the monitor changed the primary to Slave and back to Master within about 5 ms:

  notice : Server changed state: server1[...]: new_slave. [Master, Running] -> [Slave, Running]

  notice : Server changed state: server1[...]: new_master. [Slave, Running] -> [Master, Running]

  In our case, this status change was a side effect of our setup: two MaxScale instances with cooperative monitoring locks, and a runtime update of the monitor on both at the same time. It is not a required part of the problem. Any other event that removes the Master status of the primary for a moment can start the queue in the same way (see "Trigger").

• When the process dies, the replica reconnects to the other MaxScale, and the cycle starts again there.

Analysis

— added as a comment —

Trigger

Any moment in which the primary does not have the Master status can start the queue, if an ACK arrives in that moment. In our case, it was a monitor restart after a runtime change of the monitor, with cooperative monitoring locks. The passive MaxScale did not see the master lock for one tick and removed the Master status for about 5 ms. A primary in maintenance or another short status change can probably start the queue in the same way. Because the pending replies never complete, the session cannot recover from such a moment.

Expected behavior

Packets that a client sends after COM_BINLOG_DUMP do not produce a reply, so MaxScale must not count them as pending requests. A replication session must not depend on the routing plan after the binlog dump starts.

Possible fix

• Treat packets of a session in replication mode as NO_RESPONSE, in the client protocol and in readwritesplit.
• In readwritesplit, route all packets of a replicating session directly to the backend that received COM_BINLOG_DUMP, without a routing plan or the query queue.
• Optionally, include the router session queues in the memory statistics.

Workaround

• Replicate through a readconnroute service. readconnroute sends each packet directly to its backend, with no reply tracking and no queue.
• Or disable semi-sync on the replica that connects through MaxScale.

Related

• MXS-5910: the fix that made semi-sync ACKs routable during replication.
• MXS-4792: MaxScale no longer sends COM_QUIT on replication connections.
• MXS-2694: readwritesplit treats COM_BINLOG_DUMP as an endless reply.