Free-list order diverges across nodes after a snapshot reload, putting DBRM read-only

Summary

The extent map free list is a flat slot array with no ordering invariant, and allocation is first-fit by slot index. A live delete puts the freed range in the highest empty slot, but loading a snapshot ignores the saved free list and rebuilds it in ascending LBID order. After any mid-space extent delete, a worker that loads a fresh snapshot therefore holds the same extent map as its peers but a different free-list order. The next extent allocation returns different LBIDs from different workers, the controller sees mismatching replies, retries three times and sets DBRM read-only.

Impact

• Reproduced every time on our 3-node test cluster (23.02.24): delete some extents (for example drop a table created before later ones), restart only the primary's worker, then run CREATE TABLE. The controller logs "image inconsistency detected at node N" three times within about 60 ms, then "Setting read-only mode".
• Also reproduced via cpimport and DML stripe allocation (command 44) after a CMAPI cluster start.
• Seen likely, not confirmed, on a customer cluster (Sep-Oct 2026): repeated "image inconsistency detected ... command 44" read-only events. That cluster had the ingredients (failed loads rolled back, restarted workers, failovers), but the free-list state was never captured, so this is a signature match only.
• dbrmctl reload clears read-only but does not fix the cause; the next allocation trips it again. Extent maps compare equal (editem -i), so nothing points at the free list.

Where it happens (tag 23.02.24, 4e11b0ed9)

• Allocation: versioning/BRM/extentmap.cpp:3880-3905 (getLBIDsFromFreeList) walks slots 0..capacity and takes the front of the first slot with size >= request (:3887-3890). The LBID handed out depends on slot order.
• Free: extentmap.cpp:4574-4578 (deleteExtent) sets freeFLIndex on every empty slot, so it ends as the highest-index empty slot. With no adjacent free range the freed extent goes there (:4667-4669), after the big tail range.
• Load: extentmap.cpp:1603-1738 (loadVersion4or5) reads the saved size but not the saved entries, zeroes the array and seeds one range (:1631-1633), then calls reserveLBIDRange for each extent in ascending LBID order (:1718-1721). Result: holes first in LBID order, tail last. save() (:1979-1992) writes the live array, which load never reads. The fixFL argument of load() is unused.
• Compare: versioning/BRM/masterdbrmnode.cpp:914-958 (compareResponses) byte-compares worker replies. Only commands whose reply carries LBIDs expose the difference: 28 (dict store extent), 36/42 (column extent), 44 (stripe extents). The caller retries the same command (:656-709) and sets readOnly = true when retry > 1 (:700-707).

Mechanism: journal replay re-executes the same commands, so a node's order equals canonical(order at its last load) plus the operations since. A snapshot truncates the journal (slavecomm.cpp:1959-2015), so a node loading it gets canonical order while nodes that lived through the delete keep the post-delete order.

Ways one node ends up reloading while others do not:

• Master-only worker restart: ExecStopPost runs mcs-savebrm.py on the primary only (cmapi/engine_files/mcs-workernode.service:10), which takes a fresh snapshot and truncates the journal.
• Replica restart after the primary snapshotted post-delete: mcs-loadbrm.py pulls the snapshot from the primary.
• CMAPI cluster start or any config broadcast: nodes are stopped and started concurrently (helpers.py:449, put_config in endpoints.py) with no barrier making them load the same snapshot (inferred from code; the ordering is a race).
• Callers that free extents: drop/truncate (deleteOID), partition drop, cpimport bulk rollback, empty-extent cleanup, dbroot removal. Only a delete with no free neighbour changes slot order.

How to reproduce

On a 3-node cluster:

1. Create several tables so extents are allocated at increasing LBIDs, then drop one that is not the last.
2. Restart only the primary's worker (systemctl restart mcs-workernode@1). It saves a snapshot on stop and loads it canonical on start.
3. Run CREATE TABLE (command 42) or a cpimport (command 44).
4. Expect three "image inconsistency detected at node N" log lines from the controller, then read-only. editem -l on each node shows the two slot orders (one node lists the hole before the tail, the others the tail before the hole).

Suggested fix

1. Make allocation order-independent: choose the lowest-start range that fits (or best fit with lowest-start tie-break) in getLBIDsFromFreeList. One function, same O(capacity) scan. All nodes must upgrade together, since mixed versions would diverge.
2. Add a free-list checksum to the controller's connect/reload handshake (or to editem -i) so a mismatch is logged and reported instead of surfacing as read-only.
3. Alternative, riskier: canonicalise the list after every delete batch to exactly the layout reserveLBIDRange produces, or persist and load the saved list.

Related

No existing MCOL ticket covers this. MCOL-6496 and MCOL-6434 concern shared-memory/BRM corruption, a different mechanism: here the images are intact and only the free-list slot order differs.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.