Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
23.02.24
-
None
-
None
Description
Summary: restarting a single mcs-workernode (without a full cluster stop/start) leaves that node's PrimProc, WriteEngineServer, DMLProc and DDLProc reading a frozen copy of BRM. The worker's stop always runs clearShm, and mcs-loadbrm then loads BRM into new segments under the same names. The running processes keep the unlinked segments mapped and never see the reload. Nothing shows until a query needs an extent allocated after the restart. Then it fails with MCS-2031: Blocks are missing (the stale metadata names a data area that is gone) or MCS-2039: Data file does not exist (the stale extent map resolves the new LBID to the nearest older extent of another OID). First reported as a side finding on MCOL-6607 (comment of 2026-10-15). Affects builds with the MCOL-6496 shared-memory scheme.
Change (in progress):
- mcs-workernode-stop-shm.sh (ExecStopPost, after mcs-savebrm.py): keep the segments while one of those processes is alive, so load_brm loads through them and they follow. Clear as before when none is alive (cluster stop: CMAPI stops them first), when one runs a replaced binary (upgrade), or with MCS_WORKER_STOP_CLEARSHM=always in systemd.env. Logs the decision. mariadbd isn't counted: it outlives a cluster stop. Also in the cmapi container unit.
- loadState(): empty the copy locks with the load. They aren't saved; the journal replay rebuilds them. On kept segments the replay would otherwise meet locks already held, wait 60 s per range in beginVBCopy and force-release them.
- A failed load discards its copies (EM, index, free list, VSS, VBBM, copy locks) instead of publishing half-loaded ones.
- mcs-loadbrm.py: if load_brm says the shared memory has to be cleared, clear and load once more. If the load doesn't complete, clear on exit, so the worker starts on no BRM (as before) rather than on the image from before the restart, and name the readers that still need a restart at ERROR.
Not covered:
- mariadbd stays on the old segments after a cluster stop/start (information_schema tables and partition UDFs read a frozen extent map). Needs a reader-side generation check; separate issue.
- A reader alive at a cluster stop keeps the segments through stop, upgrade and start; data areas carry no layout version.
- cmapi/engine_files/mcs-loadbrm.py (container) has neither the retry nor the clear on failure.
Status: branch mcs2031-worker-stop-shm (72e1c26d5, committed under MCOL-6607 until this ticket existed). Not yet proven.
Attachments
Issue Links
- relates to
-
MCOL-6607 DBRM controller keeps a dead worker socket after a worker restart; next write halts 300 s then sets read-only
-
- Open
-