DBRM worker and controller start on an empty or failed BRM image when mcs-loadbrm fails

Summary

When mcs-loadbrm fails, systemd still starts mcs-workernode@N, then the controller, PrimProc and the rest of the stack. Nothing in the unit files, CMAPI or the C++ DBRM code checks that the extent map (EM) was actually loaded, so the node runs on an empty image. The worker's own C++ snapshot has no empty-image guard either, so an empty image can later be written over the good save files.

Impact

• Customer cluster (a customer site, Sep-Oct 2026): during a failover, a replica's mcs-loadbrm failed and its DBRM worker started in the same second. The load failure itself was loud (a short journal); the start-anyway behaviour is what let the node join with a bad image.
• Our 3-node test cluster (23.02.24): reproduced on demand on a replica, and once by accident on the primary. After a failed load the worker started 1 ms later, the controller connected to all workers about 5 s later, PrimProc followed, and the cluster looked healthy (dbrmctl status OK, master/readwrite).
• On a replica, the first write that depends on the image makes the workers disagree and puts the whole cluster in read-only mode. Catalog lookups on that node already say the table does not exist.
• On the primary, with no replica to disagree, the first extent allocation is confirmed from LBID 0, on top of live data, with no error (the allocation path is verified; the end-to-end corruption is inferred from it).

Where it happens (tag 23.02.24, 4e11b0ed9)

systemd ordering only, no dependency.

• oam/install_scripts/mcs-workernode.service.in:3: After=network.target mcs-loadbrm.service, no Requires=, BindsTo= or ExecStartPre=.
• oam/install_scripts/mcs-loadbrm.service.in:7,12: Type=oneshot, and WantedBy=mcs-workernode@1/@2 is its only link to the worker. That acts as a Wants= from the worker's side, and a failed Wants= dependency does not fail the wanting job, so systemctl start mcs-workernode@N succeeds anyway.
• mcs-controllernode.service.in, PrimProc and the WriteEngine/DMLProc/DDLProc units chain on After= only.
• CMAPI treats loadbrm as systemd's business (cmapi/cmapi_server/constants.py:92-93); it starts the services in order and sees systemctl's success.

The C++ side accepts an empty image.

• An absent EM segment is created as "a copy of nothing" with one free range starting at LBID 0: versioning/BRM/extentmap.cpp:2349-2357.
• ExtentMap::getLBIDsFromFreeList returns the first range that fits (extentmap.cpp:3880-3896), so on an empty node the first new extent gets LBID 0.
• masterdbrmnode.cpp, workernode.cpp and controllernode.cpp have no EM size, generation or checksum check when a worker connects. CMAPI /node/status reports master/slave and read-only state only, not image content.

Unguarded C++ snapshot.

• versioning/BRM/slavecomm.cpp:1965-2005: every snapshotInterval journal entries (default 100000) or on a takeSnapshot request, the first worker writes the in-memory image to BRM_saves_A or BRM_saves_B and repoints BRM_saves_current. There is no size check, and the file is not written under a temporary name and renamed.
• Only the Python mcs-savebrm.py has a guard: em_is_empty refuses an EM under 1000 bytes (oam/install_scripts/mcs-savebrm.py.in:213-227, used at :371-373 and :381-383). That guard is why a clean stop did not overwrite the good save in our test. The periodic C++ snapshot has no equivalent; likely it can overwrite the save, but we did not run an empty node long enough to hit the interval.

What the first write does (inferred from the sites above).

• With a replica on the real image: the empty node answers CREATE_STRIPE_COLUMN_EXTENTS with LBID 0, the replicas with their real next-free LBID. compareResponses (masterdbrmnode.cpp:917-962) sees the mismatch, retries three times, goes read-only and undoes the change. The cluster is protected only because the reply happens to carry image-derived data.
• With no replica to disagree (1-node cluster, or the other workers also empty or disconnected): the allocation is confirmed and journaled, and a later snapshot persists it.
• BRM lookups are served from the node's own shared memory, so reads on the empty node see no extents without any write happening.

How to reproduce

On a 3-node cluster, make mcs-loadbrm exit non-zero on a replica (we denied it a directory it needs to create), then start mcs-workernode@1 there. systemd logs Failed to start loadbrm and Starting mcs-workernode about 1 ms later. Start the controller and PrimProc: all come up, the controller connects to every worker, and the node's shared-memory free list reads 0 67108864 (one range from LBID 0). The next write that allocates an extent puts DBRM in read-only mode.

Suggested fix

• Make the worker unit depend on the load: Requires= (or BindsTo=) plus After= on mcs-loadbrm.service, or an ExecStartPre= that checks a success marker written by loadbrm. CMAPI should fail the node start, and not promote it, when the load failed.
• Have the worker refuse to serve an empty EM when BRM_saves_current names a non-empty save. Exchange an image checksum, size or generation when a worker connects and before DBRM goes read-write, so divergence is rejected at connect time rather than found at the first allocation.
• Port the em_is_empty guard into the C++ snapshot (slavecomm.cpp:1965-2005) and write the save files to a temporary name, then rename.

Related

• MCOL-5709 (Fixed in 23.02.9 / 23.10.2): mcs-savebrm.py skips saving an empty image. That covers the Python savebrm side only. It does nothing on the load side (nothing stops the worker starting after a failed load), nothing at worker connect (no image check), and nothing for the C++ snapshot in slavecomm.cpp, which has no guard at all. MCOL-5405 is closed as a duplicate of MCOL-5709.
• Sibling tickets: the pull-integrity ticket (a cut or stale transfer is one way a load fails) and the load_brm-reports-success ticket (a load can also succeed on a bad image, which a start-gate on load success would not catch).

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.