Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
23.02.24, 23.02.25
-
None
-
None
Description
Problem
Rolling back a write transaction crashes DBRMWorkerNode (SIGSEGV) when the VBBM (or VSS) on that node has never been published.
VBBM::discardUpdate() (versioning/BRM/vbbm.cpp:254) drops the copy and points back at the published data area:
fPVBBMImpl->discardUpdate();
|
fPVBBMImpl->refresh();
|
setPointers(fPVBBMImpl->get()); // get() is null if nothing was ever published |
setPointers(nullptr) then reads the header through the null pointer (vbbm.cpp:137). VSS::discardUpdate() (vss.cpp:217) is the same. It is a signal, so the attempt() wrapper in SlaveDBRMNode::undoChanges() does not catch it.
Backtrace (from a unit test, see Reproduce):
VBBM::setPointers(header=0x0)
|
SlaveDBRMNode::undoChanges() lambda (slavedbrmnode.cpp:932)
|
SlaveComm::applyJournalRecord()
|
SlaveComm::replayJournal()
|
Why "never published" happens
An empty data area is published by lock(READ) and by a save (initDataAreaIfNeeded() / ensureDataArea()), but not by lock(WRITE) (vbbm.cpp:320-334). So if the first access to a node's VBBM is a write that is then rolled back, the rollback crashes.
When it can happen
- A worker starts without load_brm having run on that node. For example, a replica whose mcs-loadbrm failed but whose worker started anyway (systemd before MCOL-6606, or the container dispatcher). Replicas never save, and VBBM is read only for versioned blocks and rollbacks, so it can stay unpublished for a long time.
- Then any write that takes the VBBM write lock and fails on that node (e.g. deleteOID on its empty extent map returns -1), or any undo() the master sends while that write is open, kills the worker.
Not reachable on a fresh install (the first system-catalog extents force a save, which publishes) or after a failed load_brm (it releases, which publishes).
Reproduce
Found by a unit test: replay a journal containing one failing command (e.g. deleteOID of an OID with no extents, which returns -1) onto empty, fresh BRM shared memory, with no loadState() first. The replay rolls the failed command back and the process segfaults. (load_brm itself is safe: it runs loadState() before replaying.)
Suggested fix
In VBBM::discardUpdate() and VSS::discardUpdate(): call setPointers() only when get() is non-null; otherwise clear the members, as release(READ) does. VBBM::initDataAreaIfNeeded()'s catch calls discardUpdate() in the same state and is covered by the same change.
The same code is on stable-23.10.
Attachments
Issue Links
- relates to
-
MCOL-6496 Change the way of working with shared memory
-
- Closed
-