Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6434

Corruption of BRM data after a fatal unrecoverable signal

    XMLWordPrintable

Details

    • Bug
    • Status: Confirmed (View Workflow)
    • Major
    • Resolution: Unresolved
    • 23.10.8
    • None
    • None
    • None
    • 2026-8

    Description

      On June 3, 2026, the Linux OOM (Out of Memory) killer terminated PrimProc processes on mcs2 and mcs3 due to excessive memory consumption (~150 GB RSS each on 156 GB nodes with no swap). This triggered a cascade of failures that ultimately led to corruption of the BRM (Block Resolution Manager) extent map. The corrupted BRM was persisted to disk when save_brm ran after the workernode's shared memory was destroyed, producing VBBM and VSS files that were 99.97% smaller than normal (84 bytes and 8 bytes vs. hundreds of kilobytes). Every subsequent restart loaded this corrupted BRM, causing continuous assertion failures, DMLProc crash loops, and inability to perform data operations.

      It is necessary to investigate in which situations the problem may occur, and how to avoid it:

      1. make the write to shmem atomic?
      2. check the consistency of data during loading?
      3. anything else?

      Attachments

        Activity

          People

            sergey.zefirov Sergey Zefirov
            alexey.antipovsky Aleksei Antipovskii
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 15d
                15d
                Remaining:
                Remaining Estimate - 15d
                15d
                Logged:
                Time Spent - Not Specified
                Not Specified

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.