Lone worker restart leaves its node on a stale BRM copy (MCS-2031/MCS-2039)

Problem

When mcs-workernode@N is restarted on its own (by hand, Restart=on-failure after a crash or an OOM kill, or by a dependency), its ExecStopPost always runs clearShm. mcs-loadbrm then loads BRM into new shared memory segments under the same names.

PrimProc, WriteEngineServer and DMLProc on that node keep running. They still map the old, unlinked segments and never see the reload. Their view of the extent map is frozen at the moment of the restart, until they are restarted themselves.

Queries on data that existed before the restart keep working, so nothing looks wrong at first. Any extent allocated afterwards (a new table, or a load that adds extents) fails on that node with one of:

• MCS-2031: Blocks are missing: the stale metadata area names a data area that no longer exists (brmshmimpl.cpp refreshLocked(): "... cannot be opened ... has to be cleared").
• MCS-2039: Data file does not exist: the frozen extent map doesn't have the new LBID and resolves it to the nearest lower extent of another OID. PrimProc logs "Invalid Range from HWM for lbid ..." and "CAL0053 could not open file for OID ..." (the other OID's file, on another dbroot).

Which of the two you get depends on whether the stale metadata's current slot is one the reader had already mapped.

There is no automatic recovery. You have to restart the readers on that node, or do a full cluster stop/start.

Reproduce (3-node cluster, stock 23.02.24 BRM code)

1. On a replica: systemctl restart mcs-workernode@2. It loads cleanly. dbrmctl status is OK on all nodes.
2. On that replica, every MCS-shm-* mapping of PrimProc, WriteEngineServer (and mariadbd) is now "(deleted)", and the inodes differ from /dev/shm.
3. Create a new table and cpimport 36,000 rows into it, so there are new extents on every dbroot. The load succeeds.
4. Every query on the new table fails (ERROR 1815 ... error in PrimitiveServer, then MCS-2039). Queries on old tables are still correct, even after calflushcache().

Open question for engineering

In step 4 the HWM check caught the bad lookup. Can a stale lookup land inside a real range instead (for example, LBIDs freed by a DROP and then reused) and read the wrong file without any error? Not tested.

Suggested fix (branch ready, after-tests passed on a CI build)

Decide at the worker's stop, where clearShm already runs, based on who still maps the segments. The default stays "clear".

• New mcs-workernode-stop-shm.sh (ExecStopPost, after mcs-savebrm.py). It keeps the segments while PrimProc, WriteEngineServer, DMLProc or DDLProc is alive, so load_brm loads through the existing metadata area and the readers follow it the MCOL-6496 way. It clears as before when none of them is alive (a cluster stop: CMAPI stops them first), when one of them runs a replaced binary (an upgrade, where the layout may have changed), or when MCS_WORKER_STOP_CLEARSHM=always is set in systemd.env. It logs the decision either way. mariadbd isn't counted because it outlives a cluster stop. The cmapi container unit gets the same change.
• loadState() empties the copy locks together with the load. They aren't saved and the journal replay rebuilds them. On kept segments the replay would otherwise meet locks that are already held, wait 60 s per range in beginVBCopy, and force-release them.
• A failed load discards its copies (EM, index, free list, VSS, VBBM, copy locks) instead of publishing half-loaded ones to the readers.
• mcs-loadbrm.py: if load_brm says the shared memory has to be cleared, it clears and loads once more. If the load doesn't complete, it clears on exit, so the worker starts on no BRM (as before) rather than on the pre-restart image. Both clears log the readers still running, at ERROR, because those readers have to be restarted.

It's safe to keep the segments after a worker that was killed mid-update: the next writer recovers the robust update mutex (EOWNERDEAD) and drops the unpublished copy. No BRM code takes the old RWLock locks since MCOL-6496.

Tested (CI build with the fix, 3 nodes)

• Lone worker restart on a replica under query load: segments kept, readers follow. New table + load + UPDATE are exact. EM and free list are equal on all nodes. 0 failed queries.
• SIGKILL of the replica worker during a cpimport: segments kept, worker restarted, data exact.
• Full cluster stop/start: cleared. PrimProc + worker stopped together: kept (WES alive). PrimProc + WES + worker stopped together: cleared.
• Replaced PrimProc binary: cleared. MCS_WORKER_STOP_CLEARSHM=always: cleared.
• Not exercised: the EOWNERDEAD recovery itself (the kill never landed inside the update-mutex section). That part is checked in the code only.
• Python unit tests for the new loadbrm functions (Python 3.6). Drone doesn't run them.

Not covered

• cmapi/engine_files/mcs-loadbrm.py (container) has neither the retry nor the clear on failure.