Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6674

23.02: backport MCOL-5950 (bounded disk-aggregation finalize) and fix a use-after-free in RowAggStorage::dump()

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 23.02.24
    • None
    • PrimProc
    • None

    Description

      Summary: on 23.02, the final merge of disk-based aggregation (RowAggStorage::finalize) loads every spilled generation back into memory and frees nothing until it finishes. A daily aggregation that spills many generations has a large memory spike at the end, and the merge time grows with the square of the generation count. On a shared/NFS spill directory this can run for a long time with no log output. MCOL-5950 fixed the older-generation side in 23.10.4 and 25.10; 23.02 doesn't have it.

      Change (in progress):

      1. Backport of MCOL-5950 (bd0f59910, #3525): walk the older generation in block order (getRowForFinalization) and free each block with dropRGData() once merged. 23.02 lacks MCOL-5852, so rowGidRidToIdx()/rowIdxToGidRid() are defined locally in rowstorage.cpp.
      2. Fix on top of upstream, external keys: when the key and value rowgroups differ, the upstream loop calls dropRGData() twice on the key storage and never on the value storage, so the older generation's value blocks are never freed. This needs forward-porting to 23.10/25.10.
      3. Fix for a crash the backport introduces: dump() sizes its free-memory reserve with fRowGroupOut->getSizeWithStrings(fMaxRows), which reads the string store of whatever RGData fRowGroupOut is attached to. After dropRGData() that is freed memory. Reproduced on a 3-node test cluster (daily aggregation with disk aggregation spilling): PrimProc SIGSEGV on all 3 nodes in StringStore::getSize <- RowGroup::getSizeWithStrings <- RowAggStorage::dump <- RowAggregation::addRowGroup. Fixed by sizing with getDataSize(fMaxRows) in both dump() and getAproxRGSize().

      stable-23.10 has the same getSizeWithStrings line and the same dropRGData() pattern. Not checked yet whether it crashes there.

      Not covered (follow-up): the newer generation is still probed through getRow() -> loadRG() with no eviction, and the memory manager is non-strict with disk aggregation on, so finalize is still not bounded by the configured limit. The cost stays O(generations^2).

      Status: branch mcol-5950-backport (a6b3701f7, 976599646). Compile-checked only. A rerun of the test that crashed is pending.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              kyle.hutchinson Kyle Hutchinson
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.