Details
-
Bug
-
Status: Confirmed (View Workflow)
-
Major
-
Resolution: Unresolved
-
23.10.8
-
None
-
None
-
None
-
2026-8
Description
On June 3, 2026, the Linux OOM (Out of Memory) killer terminated PrimProc processes on mcs2 and mcs3 due to excessive memory consumption (~150 GB RSS each on 156 GB nodes with no swap). This triggered a cascade of failures that ultimately led to corruption of the BRM (Block Resolution Manager) extent map. The corrupted BRM was persisted to disk when save_brm ran after the workernode's shared memory was destroyed, producing VBBM and VSS files that were 99.97% smaller than normal (84 bytes and 8 bytes vs. hundreds of kilobytes). Every subsequent restart loaded this corrupted BRM, causing continuous assertion failures, DMLProc crash loops, and inability to perform data operations.
It is necessary to investigate in which situations the problem may occur, and how to avoid it:
- make the write to shmem atomic?
- check the consistency of data during loading?
- anything else?