Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40447

HEAP initial block allocation is sized from max_records, causing per-statement giant mmap/munmap and mmap_lock contention with large tmp_table_size

    XMLWordPrintable

Details

    Description

      Description

      While A/B benchmarking MDEV-38975 in preview-13.1 (4411ba183b5 = branch minus all MDEV-38975 commits, vs 91f37a32f3e = branch tip) I found a severe regression in a SHOW FULL COLUMNS loop workload (ORM/admin-tool startup pattern: SHOW FULL COLUMNS for each of 73 tables per iteration, N client threads), and it worsens with concurrency:

      threads baseline QPS new QPS delta
      16 297.14 191.52 -36%
      64 360.10 180.21 -50%
      128 328.79 132.32 -60%

      (2x Xeon E5-2699A v4, 256GB RAM; server started with --max-heap-table-size=16G --tmp-table-size=16G.)

      Reproduced locally at 16 threads (AMD 7840HS 8c/16t, same server options): 334.20 -> 212.89 QPS (-36%), and profiled both sides with perf record -F 997 -g --call-graph fp on mariadbd (RelWithDebInfo, -fno-omit-frame-pointer).

      Observe, that the new side burns less CPU per iteration than the baseline (19.9 ms vs 23.6 ms on-CPU) yet is 36% slower — the losses are off-CPU. The perf attribution of the new side (see new-show_columns_loop-children.txt):

      heap_write                         16.22% incl.
        hp_get_new_block -> my_malloc -> sysmalloc_mmap    9.1%
      free_tmp_table -> hp_free -> hp_free_level -> __munmap   12.8%
      do_anonymous_page (fault-in + page zeroing)         14.6%
      rwsem_down_write_slowpath (mmap_lock)                6.3%
      rwsem_down_read_slowpath (mmap_lock)                 2.4%
      smp_call_function_many_cond / flush_tlb_mm_range (TLB shootdown IPIs)
      

      None of this exists in the baseline profile (base-show_columns_loop-children.txt), which shows the Aria path writing through the shared page cache (maria_create, write_block_record, shmem_file_write_iter).

      Mechanism

      1. Every SHOW FULL COLUMNS materializes an I_S temporary table containing blob columns. Before MDEV-38975 these were created as Aria from the start (HA_NO_BLOBS); now they legitimately stay in HEAP.
      2. heap_prepare_hp_create_info() (ha_heap.cc:879) computes max_records = tmp_memory_table_size / mem_per_row.
      3. init_block() (hp_create.c:382) sizes the first block as MY_MAX(min_records, max_records/heap_allocation_parts) * recbuffer, rounded up to a power of 2 and capped at INT_MAX32.
      4. With tmp_table_size=16G this is a multi-hundred-MB allocation per statement, which glibc serves via mmap and the table drop immediately calls munmap.
      5. Per statement the kernel pays: page faults + page zeroing on first touch, page-table teardown and TLB-shootdown IPIs on unmap, and — critically — the process-wide mmap_lock write lock. mmap_lock serialization is what makes the regression grow with thread count (-36/-50/-60%).

      The sizing heuristic is the problem, not the BLOB support

      Note, that at default-sized tmp_table_size=16M the regression is not seen at all — it inverts into a 1.55x win for the new code: new 594.43 QPS vs baseline 382.70 QPS (the baseline is nearly insensitive to the setting: 334.20 @ 16G -> 382.70 @ 16M). The first-block sizing predates MDEV-38975; it was simply never exercised by a create-and-drop-per-statement workload before, because blob-bearing I_S temp tables could not be HEAP. Large tmp_table_size is common production tuning, so any tuned server with a SHOW/I_S-heavy workload (ORMs, admin tools, monitoring) hits this.

      Proposed fix directions

      1. Do not derive the first block allocation from max_records (i.e. from tmp_table_size): start from a small initial block (or min_records-based) and grow geometrically toward max_records/heap_allocation_parts.
      2. Alternatively cap the initial alloc_size at a few MB regardless of max_records.
      3. Orthogonally, per-statement internal temp tables could reuse/pool blocks instead of malloc/free per statement, but 1-2 alone removes the mmap churn.

      Attachments

      • base-show_columns_loop-children.txt / new-show_columns_loop-children.txt — perf inclusive call trees, both sides
      • base-show_columns_loop-flat-n.txt / new-show_columns_loop-flat-n.txt — flat tables with absolute sample counts (997 Hz; samples ~ ms on-CPU)
      • base-show_columns_loop-flame.html / new-show_columns_loop-flame.html — flamegraphs
      • base-show_columns_loop.log / new-show_columns_loop.log — QPS logs of the recorded 120s runs

      Attachments

        Issue Links

          Activity

            People

              monty Michael Widenius
              arcivanov Arcadiy Ivanov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.