Details
-
Bug
-
Status: In Review (View Workflow)
-
Major
-
Resolution: Unresolved
-
13.1
Description
Description
While A/B benchmarking MDEV-38975 in preview-13.1 (4411ba183b5 = branch minus all MDEV-38975 commits, vs 91f37a32f3e = branch tip) I found a severe regression in a SHOW FULL COLUMNS loop workload (ORM/admin-tool startup pattern: SHOW FULL COLUMNS for each of 73 tables per iteration, N client threads), and it worsens with concurrency:
| threads | baseline QPS | new QPS | delta |
|---|---|---|---|
| 16 | 297.14 | 191.52 | -36% |
| 64 | 360.10 | 180.21 | -50% |
| 128 | 328.79 | 132.32 | -60% |
(2x Xeon E5-2699A v4, 256GB RAM; server started with --max-heap-table-size=16G --tmp-table-size=16G.)
Reproduced locally at 16 threads (AMD 7840HS 8c/16t, same server options): 334.20 -> 212.89 QPS (-36%), and profiled both sides with perf record -F 997 -g --call-graph fp on mariadbd (RelWithDebInfo, -fno-omit-frame-pointer).
Observe, that the new side burns less CPU per iteration than the baseline (19.9 ms vs 23.6 ms on-CPU) yet is 36% slower — the losses are off-CPU. The perf attribution of the new side (see new-show_columns_loop-children.txt):
heap_write 16.22% incl.
|
hp_get_new_block -> my_malloc -> sysmalloc_mmap 9.1%
|
free_tmp_table -> hp_free -> hp_free_level -> __munmap 12.8%
|
do_anonymous_page (fault-in + page zeroing) 14.6%
|
rwsem_down_write_slowpath (mmap_lock) 6.3%
|
rwsem_down_read_slowpath (mmap_lock) 2.4%
|
smp_call_function_many_cond / flush_tlb_mm_range (TLB shootdown IPIs)
|
None of this exists in the baseline profile (base-show_columns_loop-children.txt), which shows the Aria path writing through the shared page cache (maria_create, write_block_record, shmem_file_write_iter).
Mechanism
1. Every SHOW FULL COLUMNS materializes an I_S temporary table containing blob columns. Before MDEV-38975 these were created as Aria from the start (HA_NO_BLOBS); now they legitimately stay in HEAP.
2. heap_prepare_hp_create_info() (ha_heap.cc:879) computes max_records = tmp_memory_table_size / mem_per_row.
3. init_block() (hp_create.c:382) sizes the first block as MY_MAX(min_records, max_records/heap_allocation_parts) * recbuffer, rounded up to a power of 2 and capped at INT_MAX32.
4. With tmp_table_size=16G this is a multi-hundred-MB allocation per statement, which glibc serves via mmap and the table drop immediately calls munmap.
5. Per statement the kernel pays: page faults + page zeroing on first touch, page-table teardown and TLB-shootdown IPIs on unmap, and — critically — the process-wide mmap_lock write lock. mmap_lock serialization is what makes the regression grow with thread count (-36/-50/-60%).
The sizing heuristic is the problem, not the BLOB support
Note, that at default-sized tmp_table_size=16M the regression is not seen at all — it inverts into a 1.55x win for the new code: new 594.43 QPS vs baseline 382.70 QPS (the baseline is nearly insensitive to the setting: 334.20 @ 16G -> 382.70 @ 16M). The first-block sizing predates MDEV-38975; it was simply never exercised by a create-and-drop-per-statement workload before, because blob-bearing I_S temp tables could not be HEAP. Large tmp_table_size is common production tuning, so any tuned server with a SHOW/I_S-heavy workload (ORMs, admin tools, monitoring) hits this.
Proposed fix directions
1. Do not derive the first block allocation from max_records (i.e. from tmp_table_size): start from a small initial block (or min_records-based) and grow geometrically toward max_records/heap_allocation_parts.
2. Alternatively cap the initial alloc_size at a few MB regardless of max_records.
3. Orthogonally, per-statement internal temp tables could reuse/pool blocks instead of malloc/free per statement, but 1-2 alone removes the mmap churn.
Attachments
- base-show_columns_loop-children.txt / new-show_columns_loop-children.txt — perf inclusive call trees, both sides
- base-show_columns_loop-flat-n.txt / new-show_columns_loop-flat-n.txt — flat tables with absolute sample counts (997 Hz; samples ~ ms on-CPU)
- base-show_columns_loop-flame.html / new-show_columns_loop-flame.html — flamegraphs
- base-show_columns_loop.log / new-show_columns_loop.log — QPS logs of the recorded 120s runs
Attachments
Issue Links
- is caused by
-
MDEV-38975 BLOBs in MEMORY (HEAP)
-
- In Testing
-