Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
23.02.24
-
None
-
None
Description
Summary: on 23.02, the final merge of disk-based aggregation (RowAggStorage::finalize) loads every spilled generation back into memory and frees nothing until it finishes. A daily aggregation that spills many generations has a large memory spike at the end, and the merge time grows with the square of the generation count. On a shared/NFS spill directory this can run for a long time with no log output. MCOL-5950 fixed the older-generation side in 23.10.4 and 25.10; 23.02 doesn't have it.
Change (in progress):
- Backport of
MCOL-5950(bd0f59910, #3525): walk the older generation in block order (getRowForFinalization) and free each block with dropRGData() once merged. 23.02 lacksMCOL-5852, so rowGidRidToIdx()/rowIdxToGidRid() are defined locally in rowstorage.cpp. - Fix on top of upstream, external keys: when the key and value rowgroups differ, the upstream loop calls dropRGData() twice on the key storage and never on the value storage, so the older generation's value blocks are never freed. This needs forward-porting to 23.10/25.10.
- Fix for a crash the backport introduces: dump() sizes its free-memory reserve with fRowGroupOut->getSizeWithStrings(fMaxRows), which reads the string store of whatever RGData fRowGroupOut is attached to. After dropRGData() that is freed memory. Reproduced on a 3-node test cluster (daily aggregation with disk aggregation spilling): PrimProc SIGSEGV on all 3 nodes in StringStore::getSize <- RowGroup::getSizeWithStrings <- RowAggStorage::dump <- RowAggregation::addRowGroup. Fixed by sizing with getDataSize(fMaxRows) in both dump() and getAproxRGSize().
stable-23.10 has the same getSizeWithStrings line and the same dropRGData() pattern. Not checked yet whether it crashes there.
Not covered (follow-up): the newer generation is still probed through getRow() -> loadRG() with no eviction, and the memory manager is non-strict with disk aggregation on, so finalize is still not bounded by the configured limit. The cost stays O(generations^2).
Status: branch mcol-5950-backport (a6b3701f7, 976599646). Compile-checked only. A rerun of the test that crashed is pending.