load_brm returns success after failed journal replay or stale shared memory left by a failed savebrm

Summary

load_brm exits 0 and prints "Successfully replayed N BRM transactions" even when every replayed journal command failed. The commonest trigger we found: when mcs-savebrm.py exits 1 at worker stop, systemd skips the following clearShm step, so the next load_brm runs against the previous worker's shared memory, fails on each EM-mutating command, and still reports success. The worker then runs on an image that differs from the other workers.

Impact

• Customer cluster (a customer site, Sep-Oct 2026): not seen directly. A live replica worker restart followed by a "successful" reload and an image-inconsistency error a few minutes later fits this mechanism, but we could not confirm it without that node's savebrm and load_brm output.
• Our 3-node test cluster (23.02.24): reproduced. A replica with a near-empty EM (812 bytes) made savebrm exit 1; the next load printed File exists 1030 times, replayed 1032 records, and returned 0. The identical journal replayed cleanly with no File exists in two other runs, so the failure depends on the shared-memory state, not on the files.
• The divergence is only caught at the next write, when workers answer an extent allocation differently and DBRM goes read-only ("same extents, different LBIDs" is the observed shape; the mechanism for it is inferred).

Where it happens (tag 23.02.24, 4e11b0ed9)

Why clearShm is skipped

• oam/install_scripts/mcs-workernode.service.in:16-17: ExecStopPost=mcs-savebrm.py then ExecStopPost=... clearShm, neither prefixed with -. Per systemd.service(5), if an ExecStopPost line fails the remaining lines of that type do not run.
• oam/install_scripts/mcs-savebrm.py.in:371-373 (primary) and :381-383 (replica) exit 1 when save_brm fails or em_is_empty is true (EM under 1000 bytes, :213-227). On a replica this only decides whether clearShm runs; the guard exists to protect the primary's saves.
• :365-367: if clear_shmem_locks() is None is dead code, since call_executable_with_params returns True/False, never None; a failed mcs-shmem-locks is ignored.
• Likely (not run): a missing EM file makes get_file_size return None (:203-211), and None < 1000 raises TypeError in em_is_empty, which also exits non-zero.

Why load_brm says success

• SlaveDBRMNode mutators catch every exception, print e.what() to stderr and return -1, for example versioning/BRM/slavedbrmnode.cpp:320-334 (setLocalHWM).
• In standalone (load_brm) mode the caller drops that code: SlaveComm::do_setLocalHWM (versioning/BRM/slavecomm.cpp:942-975) only sends the reply when not standalone.
• SlaveComm::replayJournal (slavecomm.cpp:2178-2194) only aborts if an exception escapes processCommand, which the catches above prevent. Otherwise it calls confirmChanges() and ret++ unconditionally.
• versioning/BRM/load_brm.cpp:96-101 fails only when err < 0; :125-126 then prints "Successfully replayed N". mcs-loadbrm.py sees exit 0 and carries on.

The EEXIST site is not confirmed. Best candidate (inferred): each EM write goes through createDataArea (versioning/BRM/brmshmimpl.cpp, near :1348-1374), which calls shared_memory_object::remove with the result ignored and then create_only. If the remove fails (for example a segment owned by another user in sticky /dev/shm), the create throws "File exists". Checking needs ls -ln /dev/shm before the load and strace -f -e trace=shm_unlink,openat load_brm ....

How to reproduce

On a 3-node cluster running 23.02.24:

1. On a replica, make the saved EM smaller than 1000 bytes (we used a near-empty image), then stop its worker. mcs-savebrm.py logs "EM file is none or its size N is less than 1000 bytes" and exits 1; clearShm does not run.
2. Start the worker again, which runs mcs-loadbrm against the leftover shared memory.
3. load_brm prints File exists once per EM-mutating journal record and then "Successfully replayed N BRM transactions", exit 0.
   We also saw a truncated journal replay partway and return 0 on the test cluster; we have not traced that code path here, so treat that part as observed but unexplained.

Suggested fix

• Make clearShm run regardless of savebrm's result: prefix the savebrm ExecStopPost with -, or run clearShm as an ExecStartPre of the load. Then a failed save cannot leave stale shared memory for the next load.
• Make load_brm fail when any replayed command fails: count non-zero results from the SlaveDBRMNode mutators in standalone mode and return non-zero.
• Treat a short or truncated journal as a load failure rather than a partial replay.
• Fix the dead clear_shmem_locks() is None check and handle a missing EM file in em_is_empty.
• Pair with the sibling ticket so that a failed load stops the worker from starting.

Related

• MCOL-5338 (journal file missing; fixed): that covers a missing journal file. This ticket is about a journal that exists but replays against stale shared memory, or is short, and still reports success.
• MCOL-5105 (systemd timeouts): related only through the same unit files. It does not address ExecStopPost ordering or load_brm's exit code.
• Sibling new tickets: DBRM starts on a failed load, and the loadbrm pull integrity ticket.

Source

Found while investigating a customer issue on 23.02.24. Reproduction steps and logs from our test cluster can be shared as needed.