Details
-
Bug
-
Status: Open (View Workflow)
-
Critical
-
Resolution: Unresolved
-
23.02.24
-
None
-
None
Description
Summary: if any BRM client running as root (for example sudo editem -i) touches a node's BRM while that node's shared memory is empty (worker stopped, clearShm done, before mcs-loadbrm runs), the next load_brm on that node silently drops journal records. It reports success ("Successfully replayed N BRM transactions", exit 0), but the loaded extent map has lower HWMs and older CP values than the primary's. The replica then starts with a BRM that disagrees with the primary, and nothing says so.
Mechanism (23.02.24, MCOL-6496 shared-memory scheme, versioning/BRM/brmshmimpl.cpp):
- A reader that finds no segment bootstraps one: openOrCreateMetadataArea() plus an empty data area. Run as root, it leaves root-owned objects in /dev/shm (here MCS-shm-00020001, 00020003, 00060001, 00060003: extent map and free list).
- load_brm runs as mysql. When a write needs a new data area, createDataArea() calls shared_memory_object::remove(name) and ignores the result. /dev/shm is sticky (1777), so mysql can't unlink a root-owned object; the remove fails silently and the following create_only throws EEXIST ("File exists").
- On 23.02.24 that error is printed and replay carries on, so the update that hit it is lost while the transaction count still includes it.
Reproduced with a snapshot and journal saved from a 3-node test cluster (44,119 extents, 37 journal records), each run in an isolated /dev/shm (unshare -m + tmpfs) so no live node is touched:
| Run (stock 23.02.24 binaries) | Result |
|---|---|
| Clean shm, load_brm as mysql | exit 0, 37 replayed, extent map A |
| editem -i as mysql first, then load_brm | exit 0, 37 replayed, extent map A |
| editem -i as root first, then load_brm | 13 x "File exists", exit 0, "Successfully replayed 37 BRM transactions", extent map B |
Map B against A: four objects (one table's OIDs 3611-3614) have lower HWMs (80 vs 84, 160 vs 168, 20 vs 21) and older min/max/sequence values. The free list was the same.
With a strict journal replay (the MCOL-6608 fix), the same input fails loudly instead: "File exists / Journal record 3 failed to apply. Journal replay was incomplete." exit 1, and the worker doesn't start until the shared memory is cleared. That's how we found it: a routine sudo editem -i health check on a replica whose BRM pull had just failed.
Who can hit it: the services run as mysql, so it takes a root BRM client on a node in the window between clearShm and load_brm. That window is longest exactly when someone is diagnosing a node whose worker didn't come back, which is when people run tools like editem by hand as root. dbrmctl status did not create segments in our tests; editem -i did.
Suggested fix
- createDataArea(): check the result of remove(). If the name still exists, either open the existing (0666) object and resize it, or fail loudly. Never continue past EEXIST.
- Replay: any record that fails to apply must fail the load (MCOL-6608).
- Readers shouldn't bootstrap data areas, or should create them owned by the service user.
- mcs-loadbrm could run clearShm right before load_brm.
Workaround: don't run BRM tools as root on a node whose worker is down; if you have, run clearShm on that node before starting its worker.
