Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6682

DBRM worker segfaults rolling back a VBBM/VSS that was never published

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 23.02.24, 23.02.25
    • None
    • workernode
    • None

    Description

      Problem

      Rolling back a write transaction crashes DBRMWorkerNode (SIGSEGV) when the VBBM (or VSS) on that node has never been published.

      VBBM::discardUpdate() (versioning/BRM/vbbm.cpp:254) drops the copy and points back at the published data area:

      fPVBBMImpl->discardUpdate();
      fPVBBMImpl->refresh();
      setPointers(fPVBBMImpl->get());   // get() is null if nothing was ever published

      setPointers(nullptr) then reads the header through the null pointer (vbbm.cpp:137). VSS::discardUpdate() (vss.cpp:217) is the same. It is a signal, so the attempt() wrapper in SlaveDBRMNode::undoChanges() does not catch it.

      Backtrace (from a unit test, see Reproduce):

      VBBM::setPointers(header=0x0)
      SlaveDBRMNode::undoChanges() lambda  (slavedbrmnode.cpp:932)
      SlaveComm::applyJournalRecord()
      SlaveComm::replayJournal()

      Why "never published" happens

      An empty data area is published by lock(READ) and by a save (initDataAreaIfNeeded() / ensureDataArea()), but not by lock(WRITE) (vbbm.cpp:320-334). So if the first access to a node's VBBM is a write that is then rolled back, the rollback crashes.

      When it can happen

      • A worker starts without load_brm having run on that node. For example, a replica whose mcs-loadbrm failed but whose worker started anyway (systemd before MCOL-6606, or the container dispatcher). Replicas never save, and VBBM is read only for versioned blocks and rollbacks, so it can stay unpublished for a long time.
      • Then any write that takes the VBBM write lock and fails on that node (e.g. deleteOID on its empty extent map returns -1), or any undo() the master sends while that write is open, kills the worker.

      Not reachable on a fresh install (the first system-catalog extents force a save, which publishes) or after a failed load_brm (it releases, which publishes).

      Reproduce

      Found by a unit test: replay a journal containing one failing command (e.g. deleteOID of an OID with no extents, which returns -1) onto empty, fresh BRM shared memory, with no loadState() first. The replay rolls the failed command back and the process segfaults. (load_brm itself is safe: it runs loadState() before replaying.)

      Suggested fix

      In VBBM::discardUpdate() and VSS::discardUpdate(): call setPointers() only when get() is non-null; otherwise clear the members, as release(READ) does. VBBM::initDataAreaIfNeeded()'s catch calls discardUpdate() in the same state and is covered by the same change.

      The same code is on stable-23.10.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              kyle.hutchinson Kyle Hutchinson
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.