Repeated automatic syscat upgrade leaks extents for `calpontsys.syscolumn.charsetnum` (OID 1043), causing 50-second retries and slow or failing ColumnStore queries

MariaDB ColumnStore 25.10.4
MariaDB Server 10.6.25-21
RHEL 8, x86_64
Three-node ColumnStore deployment using StorageManager-backed shared object storage

Description:

After an Extent Map recovery, simple reads and writes against ColumnStore tables became extremely slow. A SELECT from a one-row table took approximately 100 seconds, while INSERT followed by commit took more than six minutes. Direct queries against selected columns of calpontsys.systable and calpontsys.syscolumn remained fast.

The delay was caused by an inconsistent extent layout in calpontsys.syscolumn. Normal syscolumn OIDs had one effective extent, while charsetnum, OID 1043, had nine unique extents in one segment file.

The OID 1043 layout was:

FBO 0, HWM 0, unavailable
FBO 4096, HWM 0, unavailable
FBO 8192, HWM 0, unavailable
FBO 12288, HWM 0, unavailable
FBO 16384, HWM 0, unavailable
FBO 20480, HWM 0, unavailable
FBO 24576, HWM 0, unavailable
FBO 28672, HWM 0, unavailable
FBO 32768, HWM 3, available

The corresponding StorageManager object was only 1,048,576 bytes. With 8 KiB blocks, FBO 4096 represents an offset of 32 MiB, already beyond the physical object. Therefore only FBO 0 was physically valid.

A query that explicitly read OID 1043 failed:

SELECT count(charsetnum) FROM calpontsys.syscolumn;

PrimProc logged invalid HWM ranges and early EOF errors for OID 1043.

The performance delay comes from TupleBPS::goodExtentCount(). TupleBPS requires all columns participating in a scan to have the same number of extents. On mismatch, TupleBPS::run() retries 1,000 times with a 50,000 microsecond sleep and reloads extent lists after each sleep. This creates a minimum delay of approximately 50 seconds for each affected internal syscat query.

The internal columnRIDs() query took 50.286 seconds. Multiple metadata queries before user-query execution produced an observed delay of approximately 100 seconds.

Measured before repair:

INSERT into one-row test table including commit: 6 min 41.593 sec
SELECT from one-row test table: 1 min 40.507 sec
SELECT count from calpontsys.systable: 0.012 sec

Likely creation mechanism:

CMAPI runs dbbuilder on the primary after starting DDLProc. OID 1043 is an automatic syscat upgrade candidate.

dbbuilder decides whether to upgrade OID 1043 only by checking whether its default physical segment file exists. It does not verify Extent Map state.

WriteEngine allocates the new extent in BRM before creating and initializing the physical file. If file creation, file open, StorageManager access, initialization, filling, CP update, or HWM update fails, the allocated extent is not explicitly removed by the dbbuilder upgrade path.

If the physical file is still reported as absent during the next startup, dbbuilder attempts the upgrade again and allocates another extent. This can produce the observed FBO sequence 0, 4096, 8192, and so on.

The exact historical StorageManager error that caused the failed attempts is not available because dbbuilder.log is overwritten on every run. The non-atomic allocation and missing compensation are confirmed by source inspection; the specific initial I/O failure remains a hypothesis.

Relevant source locations:

CMAPI startup invocation:
cmapi_server/process_dispatchers/systemd.py, lines 140-150

OID 1043 automatic upgrade selection:
tools/dbbuilder/dbbuilder.cpp, lines 169-210

OID 1043 fill using OID 1024 as reference:
tools/dbbuilder/systemcatalog.cpp, lines 620-663

Physical-file-only existence check:
writeengine/shared/we_fileop.cpp, lines 2248-2267

BRM allocation before physical file creation:
writeengine/shared/we_fileop.cpp, lines 209-260
writeengine/shared/we_fileop.cpp, lines 783-886

Extent-count retry:
dbcon/joblist/tuple-bps.cpp, lines 1224-1234
dbcon/joblist/tuple-bps.cpp, lines 1404-1439

Recovery:

A complete BRM backup was captured and verified using SHA-256. It contained EM, journal, VSS, and VBBM. The journal replayed 489 BRM operations identically on all three workers.

OID 1043 was repaired by retaining the first extent and removing all later extents. The final layout was:

OID 1043
one extent
FBO 0
DBRoot 1
partition 0
segment 0
HWM 3
status available

Measured after repair:

Read syscolumn.charsetnum: 0.02 sec
Read one-row user table: 0.45 sec
Insert one row: 0.45 sec
Read inserted row: 0.20 sec

The cluster returned to read-write mode. The previous catalog delays and OID 1043 early EOF errors disappeared.

Proposed fix:

Do not simply create the physical file before allocating the extent. File initialization requires allocation results such as startLbid and allocSize. If the file is created first and allocation then fails, an orphan file remains. The next dbbuilder run would see the file and incorrectly consider the upgrade complete even though no EM mapping exists.

The operation should instead be transactional or compensating:

1. Retain the exact identity of the newly allocated extent.
2. On any subsequent error, delete exactly that extent.
3. Create and initialize a staging object before publishing the final file.
4. Mark the extent available only after file publication, CP update, and HWM update succeed.
5. Validate both the physical file and Extent Map before deciding that the upgrade is required or complete.
6. Detect and repair stale extent-without-file and file-without-extent states.
7. Rotate or preserve dbbuilder logs instead of overwriting the only failure record.
8. Wait for StorageManager ownership and metadata visibility before running automatic syscat upgrade.

Acceptance criteria:

1. A failure after extent allocation leaves no stale extent.
2. A failure after staging-file creation leaves no final file and no stale extent.
3. Repeated startup after a failed attempt does not increase the extent count.
4. A stale extent without a file is repaired or reported without allocating another extent.
5. A file without an EM mapping is reported as inconsistent and is not treated as a completed upgrade.
6. Repeated successful starts leave OID 1043 with exactly one readable extent.
7. columnRIDs() completes without a 50-second retry.
8. SELECT count(charsetnum) FROM calpontsys.syscolumn succeeds.
9. Simple ColumnStore reads and writes complete without catalog delays.
10. Tests cover both local storage and StorageManager-backed object storage.