CMAPI failover evicts a node without fencing it, and put_config ignores ConfigRevision, giving two DBRM masters

Summary

When CMAPI's failover agent deactivates a node that missed heartbeats, it removes the node from the config and pushes the new config only to the survivors. The evicted node is never told, stopped or fenced: it keeps its old Columnstore.xml, its controllernode and any running load. The new primary starts a second controllernode over the same shared dbroot. In addition, put_config never compares ConfigRevision, so a cluster operation run from the stale node bumps its old revision to the same number as the survivors' and overwrites their config with its old topology.

Impact

• Reproduced on our 3-node test cluster (23.02.24, shared storage, auto_failover on) in about 10 minutes by freezing CMAPI on the primary for roughly 45 s with SIGSTOP while its ColumnStore services kept running. After SIGCONT, mcs cluster status from the old primary shows two nodes with dbrm_mode: master.
• Split read view: a row inserted through the new primary was invisible from the old one. An INSERT on the old primary failed with MCS-2045, and its controller tried to drive remote workers (undo: could not contact worker). The client stayed in Update with no error returned.
• Revision collision: mcs cluster stop run from the old primary bumped its stale revision to the survivors' revision number and overwrote their config; the pushed stop killed the legitimate master's controllernode. The old primary's own failover agent then evicted the other two nodes (mirror-image failover), leaving a 1-node cluster owning all dbroots and the others orphaned with services down.
• Seen on a customer cluster (Sep-Oct 2026): two nodes reported dbrm_mode: master against the same shared dbroot for about an hour, followed by a same-number config overwrite when a cluster-wide operation ran from the stale node; later, repeated failovers of live nodes within a day. The evicted node there was thrashing (memory pressure), which is exactly the kind of node that misses heartbeats while its services keep running.
• No data loss on our light test write load, but two controllers can save BRM/EM files under the same systemFiles/dbrm.

Where it happens (tag 23.02.24, 4e11b0ed9)

• cmapi/failover/agent_comm.py:232-236: the deactivated nodes go to startTransaction(extra_nodes=..., remove_nodes=...). cmapi/cmapi_server/helpers.py:67+ (start_transaction) builds the targets as active nodes plus extras minus remove_nodes.
• cmapi/cmapi_server/failover_agent.py:151-156 (commitTransaction): update_revision_and_manager(), then broadcast_new_config(nodes=nodes) with the node list that excludes the evicted node. No begin, config, stop or fence is ever sent to it.
• cmapi/cmapi_server/helpers.py:485+ (update_revision_and_manager) sets the revision to the local value plus one. Two nodes with diverged configs therefore mint the same number.
• cmapi/cmapi_server/controllers/endpoints.py:361+ (put_config) requires only an open transaction from the expected manager address. There is no ConfigRevision comparison, so an older or equal revision replaces a newer config. The only versioning (StatefulVersionModel term/seq) covers the in-memory stateful flags, not Columnstore.xml, and does nothing for DBRM.
• cmapi/failover/node_monitor.py:125-127: on the survivors the monitor stops heartbeating when len(desiredNodes) < 3, and the evicted node is no longer in their desired or inactive lists, so it can never be re-activated. On the evicted node the local XML still shows 3 desired nodes with itself as primary, and the survivors still answer heartbeats once it recovers, so it sees nothing to act on.
• cmapi/cmapi_server/helpers.py:738-743 (get_current_config_file, run at CMAPI start): adopts a config only when its revision is strictly greater than the highest seen, so two different files with the same number are never reconciled.

How to reproduce

On a 3-node cluster on shared storage with auto_failover enabled and failover monitoring active on all nodes (the log shows "Failover support is active"):

1. Record ConfigRevision, the Columnstore.xml checksum and mcs cluster status on every node.
2. On the primary, freeze CMAPI only: kill -STOP <cmapi pid>. Leave controllernode, workernode and PrimProc running.
3. After 45-60 s (the default sampling window is 30 s, FailoverSamplingIntervalSeconds), a survivor logs FA.deactivateNodes(): deactivating nodes: [...] and commits revision+1 with itself as primary.
4. kill -CONT the frozen CMAPI. The old primary's XML is unchanged, and mcs-controllernode is active on both it and the new primary. mcs cluster status on the old primary shows two dbrm_mode: master.
5. Compare a row count or syscat query from the old primary and a survivor; insert through one side and look from the other.
6. Run mcs cluster stop from the old primary. It publishes the same revision number as the survivors' and overwrites their config; the stop kills the legitimate controller.

Suggested fix

1. Fence the evicted node: best-effort stop of its services (or a config push to it) when it is removed, and refuse to start a second controllernode while the old one still answers.
2. Make a node self-demote when it learns a peer holds a newer revision, and stop acting as primary or DBRM master when its own revision is behind.
3. put_config should reject a revision that is not newer than the local one unless forced, and the revision should carry a term/epoch (the existing StatefulVersionModel) so same-number forks are detectable.
4. Reconcile configs at CMAPI start and in the monitor by comparing against all configured nodes, not only the node's own desired list.

Related

• Epic MCOL-6008 (HA improvements): this belongs under it.
• MCOL-4540: earlier failover/HA item (Confirmed, Icebox); neither fencing nor revision checks were added.
• MCOL-6006 ("Disable failover when shared storage not detected"): adds the shared-storage gate and the term/seq model for the stateful flag only; it does not cover Columnstore.xml or DBRM fencing.
• MCOL-6194: related HA work; does not add fencing of an evicted node or a revision guard (not re-read in detail for this draft).
• MCOL-6093 and MCOL-5508 reduce duplicate broadcasts and make the unresponsive window configurable. A longer window makes eviction less likely but does not change what happens once a node is evicted.
• CMAPI at 23.02.24 is byte-identical to 23.02.23, and no commit on develop-23.02 or stable-23.10 since then adds fencing or a revision guard.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.