DBRM controller keeps a dead worker socket after a worker restart; next write halts 300 s then sets read-only

Summary

The DBRM controller never probes or re-establishes its connections to the workers after startup. When a worker restarts, the controller keeps a dead socket, finds out on its next command, sets halting and waits up to 300 s for a RESUME that only dbrmctl can send. At 300 s it sets the whole cluster read-only. The socket was already repaired by the failed read, so an immediate retry would have succeeded; no code path tries it. Separately, a read error leaves later workers' replies unread and a read error in the confirm phase is reported to the client as success.

Impact

• Reproduced on our 3-node test cluster (23.02.24): restarting one replica's worker during a cpimport load, followed by dbrmctl reload, put DBRM read-only in 5 of 7 runs. The controller logged "Network error reading from node N" followed 299-300 s later by "A node is unresponsive for cmd = N, no reconfigure in at least 300 seconds. Setting read-only mode."
• Every writer in the cluster stalls for the 300 s. The failing load gets ERR_READONLY, its bulk rollback also needs BRM writes and fails, so the table lock leaks. A hit in the commit phase loses the whole load.
• Seen likely, not confirmed, on a customer cluster (Sep-Oct 2026): two read-only events preceded by the same "unresponsive ... no reconfigure" message, on a command that is typically the first write after a worker bounce. Worker restart history was consistent but not proven for each event.
• dbrmctl reload does not release a halt already in progress (see below). Running it after the worker returns is a race against the next command.

Where it happens (tag 23.02.24, 4e11b0ed9)

All in versioning/BRM/masterdbrmnode.cpp unless noted.

• No liveness check: connectToWorkers() (:168-209) runs only at startup and in doReload. isConnected() is local socket state (utils/messageqcpp/messagequeue.h:270-273). A write to a peer-closed socket succeeds locally.
• gatherResponses (:816-904): the read throws, MessageQueueClient::read closes the socket (messagequeue.cpp:280-290), then :854-857 sets halting = true and returns ERR_OK with the failed worker left in iSlave.
• CHECK_ERROR1 sees halting, calls undo() (:965-998; its write to the failed worker lazily reconnects, messagequeue.cpp:295-310), unlocks and jumps to retrycmd.
• retrycmd (:534-556): while (halting && ++haltloops < 300) sleep(1) (:537-538), then at the limit logs the message above and sets readOnly = true; halting = false (:542-549). Every other write thread parks in the same loop. Only doHalt, doResume (:1155-1160) and this timeout write halting.
• doReload (:1273-1326) rebuilds the sockets and sets readOnly = false (:1319) but never clears halting; its comment says the caller must bracket it with halt/resume. The parked thread still times out and overwrites the clear with readOnly = true. doReload also swaps slaves without taking slaveLock.
• doResume calls unlock() unconditionally (:1159). When halting was set by gatherResponses, slaveLock is not held, so a bare resume unlocks an unowned mutex.
• Nobody sends HALT/RESUME/RELOAD except dbrmctl (versioning/BRM/dbrmctl.cpp:85/93/101). A grep of cmapi/ at the tag finds no call; cmapi/mcs_node_control/models/dbrm_socket.py:7-14 only knows readonly/readwrite/status-style commands. The "reconfigure" the message waits for was the legacy ProcMgr halt, reload, resume sequence; it does not exist in a CMAPI cluster.

Two further defects in the same path (code read, not reproduced; effects are inferred):

• Stale replies: distribute() writes to all workers, but gatherResponses returns at the failed one without reading workers after it, and undo() only reaches workers up to iSlave (:977-990). Their unread replies stay in the socket buffers, so after a RESUME or readwrite without reload each later read on those sockets is one message behind. Identical one-byte replies hide this; the first reply carrying data (commands 28/42/44) then mismatches and trips the "image inconsistency" path. Only doReload recreates the sockets. Our test failures always hit the last worker, so this was not exercised.
• Confirm phase: confirm() (:1009-1022) uses gatherResponses, which on a dead worker sets halting and returns; the caller (:743-762) checks only for an exception, so the client is told SUCCESS while halting stays set and the next write parks for 300 s.

Why MCOL-3917 does not settle this

MCOL-3917 was closed Won't Fix with the answer "restart the full system", on the basis that services are expected to be restarted together. That assumption no longer holds. CMAPI itself restarts single workers: put_config stops and starts the local node's services on every config broadcast, a node add/remove or failover restarts workers on the surviving nodes, and systemd restarts a worker after an OOM kill. In a CMAPI cluster nothing sends the halt/reload/resume the controller is waiting for, and a full controller restart is not part of any of those flows. The controller therefore has to survive a worker reconnect on its own. The first retry after the reconnect works today; the 300 s wait is only policy inherited from ProcMgr (comment at masterdbrmnode.cpp:840-844).

How to reproduce

On a 3-node cluster with CMAPI:

1. Start a cpimport (or a loop of DML) against the cluster.
2. While it runs, restart one replica's worker: systemctl restart mcs-workernode@2.
3. The controller log shows "Client read close socket ... Remote is closed", then "Network error reading from node N ... Will see if retry is possible".
4. Writes stall. About 300 s later: "no reconfigure in at least 300 seconds. Setting read-only mode." The load fails with ERR_READONLY and leaks its table lock.
5. dbrmctl reload run a few seconds after the worker returns does not shorten the wait once a command has already hit the dead socket. Across 7 runs with a dbrmctl reload after the worker returned, 5 went read-only and 2 completed normally. A further run without the reload went read-only as expected.
   An idle restart followed by any write takes the same path (the next command finds the dead socket).

Suggested fix

1. In retrycmd, when halting was set by gatherResponses (not doHalt), reconnect all workers every second or so and retry once they are connected, within the existing 300 s budget, then read-only as today. Prerequisite: make the retried command idempotent (the in-place rewrite in BEGIN_VB_COPY must not apply twice).
2. Make RELOAD clear halting after reconnecting, and take slaveLock while swapping slaves.
3. After any gather failure, reset every worker socket (or drain replies from workers after iSlave) before retrying.
4. Treat a read error in the confirm phase as a command failure, or retry CONFIRM only.
5. Make doResume unlock only a lock it holds.

Related

• MCOL-5508 (configurable unresponsive timeout, stable-23.10): makes the 300 s wait tunable. It does not remove the wait, and the controller still never retries.
• MCOL-3917 (Won't Fix): "services must survive workernode/controllernode restarts"; see the section above for why it should be revisited.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.