CMAPI cluster stop wedges ~6 min on the primary, fails with 422 and leaves the primary on a reduced node list

Summary

mcs cluster stop is implemented as a config broadcast (maintenance flag plus revision bump) sent to all nodes at once, with no ordering. Replicas stop their workers and failover heartbeats within seconds, while the primary does a graceful DMLProc stop that can take up to 300 s. During that window the primary's failover monitor sees both peers silent and queues a node deactivation, and stopping the monitor deadlocks against the failover agent's begin-transaction retry until the 400 s request timeout. The stop then fails with HTTP 422 and the queued failover commits, removing the replicas from the primary's config.

Impact

Reproduced twice on our 3-node test cluster (23.02.24):

• A cluster restart with an open DML transaction.
• A plain mcs cluster stop with the controller only in read-only mode (no open transaction). It took about 6 minutes, ended in CMAPI 422 "Broadcasting config to nodes failed ... config put failed by Timeout", and left the primary on a 2-node config revision while the replicas stayed on the previous revision with 3 nodes.
  Recovery needed the replicas' config copied back to all nodes by hand and a CMAPI restart. The MCOL-6093 fix content is already in 23.02.24 (via the CMAPI sync) and this still reproduced.

Where it happens (tag 23.02.24, 4e11b0ed9)

1. cmapi/cmapi_server/handlers/cluster.py:51-72 toggle_cluster_state(): sets the maintenance flag, bumps the revision and calls broadcast_new_config(). There is no separate stop RPC; success means every node answered the config PUT.
2. cmapi/cmapi_server/helpers.py broadcast_new_config() PUTs to all nodes concurrently, each with timeout=LONG_REQUEST_TIMEOUT (:404-406; 400 s, constants.py:125). Any failure raises, giving the 422.
3. cmapi/cmapi_server/controllers/endpoints.py:526-538 put_config: publish('failover', False) runs only after stop_node() returns. Replicas reach it in seconds. The primary reaches it after the graceful DMLProc stop, which waits up to DMLPROC_SHUTDOWN_TIMEOUT = 300 s (constants.py:124) when the controller is read-only or the rollback cannot complete. Meanwhile neither failover/node_monitor.py nor failover_agent.py consults the maintenance flag, so a planned stop looks like a 2-node failure and deactivateNodes is queued.
4. Deadlock: publish('failover', False) reaches NodeMonitor.stop() (failover/node_monitor.py:54-60) and then AgentComm.die() (failover/agent_comm.py:102-104), which does self._thread.join() with no timeout. That thread sits in FailoverAgent.startTransaction(), which loops while not got_txn (failover_agent.py:121) with no check of _die. Begin fails with 422 "active operation" because the cluster-stop transaction is still open, and that transaction closes only when the primary's own PUT answers, which is waiting on the join. Only the 400 s aiohttp timeout breaks the cycle.
5. After the timeout the agent gets its transaction and runs the queued FailoverAgent.deactivateNodes() (failover_agent.py:44-66) with no re-check of liveness or maintenance. The commit writes the smaller node list and broadcasts it only to the nodes it still considers alive, which is how the primary ends up with a different revision.
   Whether rollback of the failed transaction reverted the replicas before the primary's commit is not settled from code alone.

How to reproduce

1. 3-node cluster with failover enabled (default). Put the controller in read-only mode (for example after a worker restart that left DBRM read-only), or leave a DML transaction open.
2. Run mcs cluster stop on any node.
3. Replicas log their NodeMonitor and heartbeat stopping within seconds. The primary logs "Stopping Failover monitor thread." only after the DMLProc graceful-stop wait, then blocks about 400 s from the start of the PUT.
4. The command ends with 422 "Broadcasting config to nodes failed ... config put failed by Timeout". The primary's Columnstore.xml has a higher revision and fewer nodes than the replicas' copies.

Suggested fix

• Order the shutdown: publish failover=False (and stop the heartbeat responder) on every node before any stop_node(), then stop the primary's DMLProc, then the replicas.
• Make the failover agent honour the maintenance flag, and drop queued events when the monitor is stopped. Re-validate liveness and maintenance inside deactivateNodes().
• Give FailoverAgent.startTransaction() a deadline and a _die check, and give AgentComm.die() a bounded join, so NodeMonitor.stop() cannot deadlock against an open cluster transaction.
• Keep a failed stop broadcast from leaving the failover thread free to commit a membership change.

Related

• MCOL-6093 (multiple PUT /config at shutdown): the fix content is in 23.02.24 and this still reproduced, so this is a separate cause (unordered stop plus the monitor/agent deadlock).
• MCOL-6097: related CMAPI shutdown/failover behaviour; this ticket covers the stop ordering and the stale queued failover.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.