Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6434

Corruption of BRM data after a fatal unrecoverable signal

    XMLWordPrintable

Details

    • Bug
    • Status: In Progress (View Workflow)
    • Major
    • Resolution: Unresolved
    • 23.10.8
    • None
    • None
    • None
    • 2026-8, 2026-9, 2026-10

    Description

      On June 3, 2026, the Linux OOM (Out of Memory) killer terminated PrimProc processes on mcs2 and mcs3 due to excessive memory consumption (~150 GB RSS each on 156 GB nodes with no swap). This triggered a cascade of failures that ultimately led to corruption of the BRM (Block Resolution Manager) extent map. The corrupted BRM was persisted to disk when save_brm ran after the workernode's shared memory was destroyed, producing VBBM and VSS files that were 99.97% smaller than normal (84 bytes and 8 bytes vs. hundreds of kilobytes). Every subsequent restart loaded this corrupted BRM, causing continuous assertion failures, DMLProc crash loops, and inability to perform data operations.

      It is necessary to investigate in which situations the problem may occur, and how to avoid it:

      1. make the write to shmem atomic?
      2. check the consistency of data during loading?
      3. anything else?

      Attachments

        Activity

          People

            alexey.antipovsky Aleksei Antipovskii
            alexey.antipovsky Aleksei Antipovskii
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 30d Original Estimate - 30d
                30d
                Remaining:
                Time Spent - 22d Remaining Estimate - 13d
                13d
                Logged:
                Time Spent - 22d Remaining Estimate - 13d
                22d

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.