messageqcpp clients (DBRM client, DEC, WEClients) have no read timeout; one hung or vanished peer stalls callers forever

Summary

The messageqcpp clients used between ColumnStore services (the DBRM client in mariadbd, DistributedEngineComm (DEC) in ExeMgr, WEClients) wait for replies with no read timeout. If a peer hangs while its sockets stay open, or its host vanishes without a FIN/RST, the waiting caller blocks forever. TCP keepalive would only catch a dead host or a partition; a hung but alive peer needs an application-level read deadline.

Impact

Two forms of the same gap, both on 23.02.24.

1. One frozen PrimProc stalls catalog queries on every node. Seen in the logs of a customer cluster (Sep-Oct 2026): stack dumps from the two surviving nodes show catalog queries (plan-time column lookups and two information_schema queries) blocked in DistributedEngineComm::read, waiting on the PM of the DBRM master node, which had stopped answering. We did not capture that node, so why it stopped is unknown. That those DEC reads caused the cluster-wide stall is likely, not proven on the customer side.

We reproduced the pattern on our 3-node test cluster. With one PrimProc stopped (SIGSTOP), six clients on another node (catalog queries, a SELECT, a full count, an INSERT, a cpimport) all hung, as did a catalog query and a SELECT on the third node. CMAPI kept getting heartbeats, so no failover happened. After SIGCONT everything finished with correct results, about 10 minutes later. Responses are not lost; only a timeout is missing. If the PM never returns, the cluster stays stuck.

2. After failover of a hung primary, mariadbd ColumnStore queries hang until SIGKILL. Reproduced once on our test cluster; not seen in the customer's logs (their stacks show form 1 only), though the trigger class is the same. A query had sent a request to the old controller just before it was hard-rebooted, leaving a half-open socket. That one reader blocked forever holding the DBRM mutex, and every other ColumnStore query in that mariadbd queued behind it. KILL had no effect, a CMAPI cluster stop/start does not restart mariadbd, and systemctl restart mariadb sat in stop until the 30 minute timeout. SIGKILL of mariadbd on each survivor cleared it.

Where it happens (tag 23.02.24, 4e11b0ed9)

• utils/messageqcpp/messagequeue.h:198: read(const timespec* timeout = 0, ...) defaults to no timeout. In inetstreamsocket.cpp a null timeout becomes msecs = -1; readToMagic only polls when msecs >= 0 (line 309), so it goes straight to a blocking ::read() (line 364) and retries on EINTR (lines 366-372). The timed overloads exist but these callers don't use them.
• versioning/BRM/dbrm.cpp:934: DBRM::send_recv takes the per-object mutex, then lines 953-954 do msgClient->write(in); out = msgClient->read(); with no timeout. The comment at line 956 ("If we add a timeout to the read() call, uncomment this clause") shows the gap is known. The error path after line 960 reconnects fine if the socket ever errors; only detection is missing.
• mariadbd has one process-global SessionManager holding one DBRM (calpontsystemcatalog.cpp:182, sessionmanager.h:209). checkSysCatVer makes a controller RPC per referenced column on the plan path (lookupOID at calpontsystemcatalog.cpp:495-499, colType at 1028-1032). One lost reply freezes all ColumnStore planning in that mariadbd. makeCalpontSystemCatalog also calls sysCatVerID under map_mutex (lines 1871, 1972), so new sessions join the pile-up.
• dbcon/joblist/distributedenginecomm.cpp:388: the per-PM Listen thread loops on an untimed client->read(0, NULL, &stats). DistributedEngineComm::read (line 490, pop at 510) is an untimed queue pop, as are read_all and read_some. DEC already pushes an empty ByteStream to every session queue on EOF or exception (lines 400-436); a silent peer never triggers it.
• writeengine/client/we_clients.cpp:438-462: WEClients::read is the same untimed queue pop. ddlproc/ddlprocessor.cpp:420-450 commitTransaction waits for PMCount replies and bails only on an empty ByteStream, which needs EOF.
• No SO_KEEPALIVE, TCP_KEEPIDLE, TCP_USER_TIMEOUT or SO_RCVTIMEO is set outside storage-manager. Socket setup sets only buffer sizes, TCP_NODELAY and SO_REUSEADDR (inetstreamsocket.cpp:179-201). Same on 23.10.
• KILL can't break form 2: planning has one KILL_QUERY check (ha_mcs_impl.cpp:4735), before the per-column RPCs, and the waiters sit in a blocking read() and pthread_mutex_lock. The server-side KILL and shutdown behaviour is our reading of MariaDB semantics, not checked in server source.

How to reproduce

Form 1 (our 3-node test cluster): kill -STOP PrimProc on one node, then from another node run a catalog query (select * from information_schema.columnstore_tables), a SELECT on a ColumnStore table, and a cpimport. All hang with no error and no failover. kill -CONT the PrimProc and they complete. Stacks show DistributedEngineComm::read, ClientRotator::read and, for new connections, connect() stuck in SYN-SENT once the frozen PM's listen backlog fills.

Form 2: reproduced once, not on demand. Hang the primary, fail over, hard-reboot the hung node while a controller request from mariadbd is in flight. Stack of the stuck mariadbd: one thread in InetStreamSocket::readToMagic <- DBRM::send_recv <- DBRM::sysCatVerID, others in pthread_mutex_lock inside send_recv and on map_mutex. Next time, capture ss -tnpie and /proc/<pid>/fd before restarting.

Suggested fix

• Use bounded reads on the request/response paths: pass a timeout to the existing read(timespec*) in DBRM::send_recv, and on timeout drop the client and return ERR_NETWORK. Pick a deadline longer than the slowest legitimate controller operation, or make it per command.
• DEC: an inactivity deadline in Listen while work is outstanding (or PM heartbeats), feeding the existing empty-ByteStream fan-out; a timed pop in DEC::read and WEClients::read; a connect timeout in Setup.
• Add SO_KEEPALIVE (idle/interval/count) and TCP_USER_TIMEOUT for host loss. Cheap, but not sufficient: a hung PrimProc on a live kernel still ACKs probes.
• Fix the unlocked fSessionManager.reset() in checkSysCatVer (calpontsystemcatalog.cpp:6286-6290) in the same change. A read timeout makes that error path reachable, and it would destroy a DBRM and mutex that other threads are still using.
• Reduce the blast radius: don't hold DBRM::mutex across the network wait, don't call sysCatVerID under map_mutex, avoid one controller RPC per column, and make the wait interruptible so KILL and shutdown can abort it.

Related

• MCOL-6312: a TupleBPS stall after lost connectivity, same no-timeout family; the engineer could not reproduce it. This ticket gives a deterministic trigger and names the shared cause.
• MCOL-6603: mentions in one line that DEC::read has no timeout. This ticket covers the whole messageqcpp client family, including the DBRM client.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.