Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6434

Corruption of BRM data after a fatal unrecoverable signal

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 23.10.8
    • 23.02.24, 25.10.7
    • None
    • None
    • 2026-8, 2026-9, 2026-10, 2026-11, 2026-12

    Description

      On June 3, 2026, the Linux OOM (Out of Memory) killer terminated PrimProc processes on mcs2 and mcs3 due to excessive memory consumption (~150 GB RSS each on 156 GB nodes with no swap). This triggered a cascade of failures that ultimately led to corruption of the BRM (Block Resolution Manager) extent map. The corrupted BRM was persisted to disk when save_brm ran after the workernode's shared memory was destroyed, producing VBBM and VSS files that were 99.97% smaller than normal (84 bytes and 8 bytes vs. hundreds of kilobytes). Every subsequent restart loaded this corrupted BRM, causing continuous assertion failures, DMLProc crash loops, and inability to perform data operations.

      It is necessary to investigate in which situations the problem may occur, and how to avoid it:

      1. make the write to shmem atomic?
      2. check the consistency of data during loading?
      3. anything else?

      Attachments

        Issue Links

          Activity

            People

              alexey.antipovsky Aleksei Antipovskii
              alexey.antipovsky Aleksei Antipovskii
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 30d Original Estimate - 30d
                  30d
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 35d
                  35d

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.