[Draft] corrupted double-linked list in mem_heap_create_block_func

2026-08-20  8:05:34 18 [Note] InnoDB: Cannot close file ./mysql/innodb_table_stats.ibd because of pending fsync

corrupted double-linked list

260820  8:06:04 [ERROR] /server/sql/mysqld got signal 6 ;

#10 0x00007f11e4eadc4e in malloc () from /lib/x86_64-linux-gnu/libc.so.6

#11 0x00005607de3055f7 in ut_allocator<unsigned char, true>::allocate (this=<optimized out>, n_elements=8096, autoevent_idx=38, set_to_zero=false, throw_on_error=false) at /server/storage/innobase/include/ut0new.h:376

#12 mem_heap_create_block_func (heap=heap@entry=0x7711c0000c68, n=<optimized out>, type=0) at /server/storage/innobase/mem/mem0mem.cc:276

#13 0x00005607de3057a2 in mem_heap_add_block (heap=0x7711c0000c68, n=n@entry=520) at /server/storage/innobase/mem/mem0mem.cc:362

#14 0x00005607de4083a2 in mem_heap_alloc (n=520, heap=<optimized out>) at /server/storage/innobase/include/mem0mem.inl:179

#15 mtr_buf_t::add_block (this=0x7f11e0db5998) at /server/storage/innobase/include/dyn0buf.h:323

#16 mtr_buf_t::push<unsigned char*> (this=0x7f11e0db5998, size=33) at /server/storage/innobase/include/dyn0buf.h:234

#17 mtr_buf_t::push (this=0x7f11e0db5998, ptr=0x77116cef1fb8 "./partition_db/thash_8_8#P#p8.ibd", len=33) at /server/storage/innobase/include/dyn0buf.h:250

#18 mtr_t::log_file_op (this=this@entry=0x7f11e0db5868, type=type@entry=FILE_MODIFY, space_id=<optimized out>, path=path@entry=0x77116cef1fb8 "./partition_db/thash_8_8#P#p8.ibd", new_path=new_path@entry=0x0) at /server/storage/innobase/fil/fil0fil.cc:1616

#19 0x00005607ddadf740 in fil_names_clear (lsn=lsn@entry=31307280) at /server/storage/innobase/fil/fil0fil.cc:3193

#20 0x00005607ddad7c12 in log_checkpoint_low (oldest_lsn=oldest_lsn@entry=31307280, end_lsn=end_lsn@entry=49357253) at /server/storage/innobase/buf/buf0flu.cc:2239

#21 0x00005607de3d3cd8 in log_checkpoint () at /server/storage/innobase/buf/buf0flu.cc:2314

#22 buf_flush_page_cleaner () at /server/storage/innobase/buf/buf0flu.cc:2875

## Verdict

The crash signature (glibc `corrupted double-linked list`, signal 6, in the

background `buf_flush_page_cleaner` thread while walking

`fil_system.named_spaces` during a checkpoint) is the same failure class as

**MDEV-37755** ("Mysqld crash in fil_names_clear" — a stale/dangling

`fil_space_t` left reachable from `fil_system.named_spaces` after the

tablespace it belongs to has been freed). That ticket is *Closed/Fixed*, and

the fix (GitHub commit `7301fbac576b2d8630cdaea029e6a682c25d8412`,

"`fil_space_t::drop()` doesn't remove space from `fil_system.named_spaces`")

**is present** in the exact revision that crashed

(`2c0f583a065bdfcaf20b451922fb6caeadd72aa4`) — confirmed both by

`git merge-base --is-ancestor` and by direct inspection of the checked-out

source at that commit (the `named_spaces.remove(*space)` call added by the fix

is there, storage/innobase/fil/fil0fil.cc:1733).

So **this specific trial is not a recurrence of MDEV-37755 as filed** — it is

either a distinct, still-open variant of the same class of race (a tablespace

being freed while the checkpoint thread iterates `named_spaces`), most likely

exposed by a code path the MDEV-37755 fix didn't cover, or a much narrower

residual window in the fixed code. I could not pin down the exact defect or

produce a standalone reproducer within the time/resource budget available

here (see "Reproduction attempts" below); this write-up hands off the

evidence chain and a concrete, actionable lead for whoever picks it up next.

## Evidence chain

### 1. The crash itself (`trial.log`, `vardir96/s1/mysql.err`)

```

corrupted double-linked list

260820  8:06:04 [ERROR] .../sql/mysqld got signal 6 ;

```

Coredump backtrace (`vardir96/threads_s1_core.169482_1124106240.txt`):

```

#10 malloc () from libc.so.6

#11 ut_allocator<unsigned char, true>::allocate (...) at .../ut0new.h:376

#12 mem_heap_create_block_func (...) at .../mem0mem.cc:276

#13 mem_heap_add_block (...) at .../mem0mem.cc:362

#14 mem_heap_alloc (...) at .../mem0mem.inl:179

#15 mtr_buf_t::add_block (...) at .../dyn0buf.h:323

#16 mtr_buf_t::push<unsigned char*> (...) at .../dyn0buf.h:234

#17 mtr_buf_t::push (..., path=0x77116cef1fb8 "./partition_db/thash_8_8#P#p8.ibd", ...)

       at .../dyn0buf.h:250

#18 mtr_t::log_file_op (..., type=FILE_MODIFY, ...) at fil0fil.cc:1616

#19 fil_names_clear (lsn=31307280) at fil0fil.cc:3193

#20 log_checkpoint_low (...) at buf0flu.cc:2239

#21 log_checkpoint () at buf0flu.cc:2314

#22 buf_flush_page_cleaner () at buf0flu.cc:2875

```

This is the classic "detected late" signature: the actual out-of-bounds

write/use-after-free happens somewhere else; glibc's malloc only notices the

corrupted heap metadata later, on an unrelated allocation — here, the

`mem_heap` block that `fil_names_clear()` allocates while writing a

`FILE_MODIFY` record for `./partition_db/thash_8_8#P#p8.ibd` during a routine

checkpoint. **The named table/partition (`thash_8_8`, partition `p8`) is the

one whose file-name string ends up in the corrupted heap block** — i.e. the

checkpoint was in the middle of processing exactly the table that had just

been rebuilt (see below), which is what points at a `named_spaces`

lifetime bug rather than something unrelated like page-compression buffer

sizing.

### 2. What was happening to `thash_8_8` right before the crash (`vardir96/s1/mysql.log`)

The general query log shows connection 16 (`WRK-1`) issuing:

```

260820  8:05:50   ... (last timestamp before the ALTER)

    16 Query  /* WRK-1 QNO 89 */  ALTER TABLE `thash_8_8` ORDER BY `col_int_nokey` /* Transformed by DisableOptimizations */

```

`thash_8_8` is an InnoDB table created with `PARTITIONS 98` (98 hash

partitions, see `conf/yy/partition_by_hash.yy`). `ALTER TABLE ... ORDER BY`

on InnoDB always uses `ALGORITHM=COPY`: it builds a whole new table (98 new

partition tablespaces), copies every row, then drops the 98 old partition

tablespaces as part of the same statement's commit. **Connection 16 never

issues another query after this line** — the ALTER was still in flight (columns

in-progress dropping ~98 tablespaces back-to-back) when the server crashed at

8:06:04, roughly 12-14 seconds later. That is a much bigger, tighter burst of

tablespace-drop operations than a single `DROP TABLE` would produce, and it

lines up with the checkpoint thread choking on `thash_8_8`'s own files.

The trial also had `SET GLOBAL INNODB_COMPRESSION_ALGORITHM=...` firing

throughout via `conf/yy/innodb_compression_algorithms.yy`, and

`innodb_compression_default=ON` was set for the whole trial, so every InnoDB

table (including `thash_8_8`) was page-compressed with whatever algorithm was

globally active at write time. This does not appear to be the trigger (see

"Ruled out" below) but it does mean every one of the 98 dropped/recreated

tablespaces was going through the page-compression write path as well, adding

extra background I/O/flush activity concurrent with the ALTER.

## Ruled out

Two candidate known bugs were checked and **both excluded** — not by

guesswork, but by confirming their fixes are already present in the code that

actually crashed:

* **MDEV-37755** (`fil_space_t::drop()` doesn't remove space from

  `fil_system.named_spaces`) — same crashing thread, same function

  (`fil_names_clear` via `log_checkpoint`), same underlying defect class

  (freed tablespace still reachable from `named_spaces`). Fix commit

  `7301fbac576b2d8630cdaea029e6a682c25d8412` is an ancestor of

  `2c0f583a065bdfcaf20b451922fb6caeadd72aa4` (`git merge-base

  --is-ancestor` confirms it), and the fixed code

  (`fil0fil.cc:1725-1733`, the `named_spaces.remove(*space)` re-check

  after waiting on pending references) is present verbatim in the crashing

  revision. **This trial cannot be a plain recurrence of the bug as filed and

  fixed** — see "Recommendation" for why it may still be a related, uncovered

  variant.

* **MDEV-39532** (ASAN heap-buffer-overflow in `lzo1x_1_15_compress` /

  `fil_page_compress_low`, still *Open*) — same background thread

  (`buf_flush_page_cleaner`), and the trial's grammar (

  `conf/yy/innodb_compression_algorithms.yy`) does drive

  `SET GLOBAL INNODB_COMPRESSION_ALGORITHM=lzo` at runtime even though the

  server was started with `lzma`, so this looked like a strong candidate.

  Ruled out **both by code inspection and empirically**:

  * Code: `buf_tmp_reserve_compression_buf()` in `storage/innobase/buf/buf0flu.cc`

    already sizes the scratch buffer to `LZO1X_1_15_MEM_COMPRESS` (not just

    `srv_page_size`) whenever the LZO provider is loaded — identical at

    `2c0f583a065b...`, its parent `34b9dc4f...`, and the unrelated ASAN build

    `b26a` (revision `2b127d5fb919f91bccf2402c222baeb66a73349e`, 2026-08-09)

    used below. This is exactly the fix MDEV-39532 needs; it's already there.

  * Empirical: `lzo_repro_driver.sh` (this directory) drives a `PAGE_COMPRESSED=1`

    InnoDB table through `provider_lzo`/`innodb_compression_algorithm=lzo`

    under the ASAN build `b26a`, forces ~2400 buffer-pool flushes via a

    shrunk buffer pool, and shuts down cleanly with **no ASAN report**.

## Reproduction attempts (all negative)

| # | Grammar | Build | Seed | Duration | Result |

|---|---|---|---|---|---|

| 1 | full original mix (5 grammars) | bb-main-atomic-rel (Release-like) | 1787213082 (original) | 180s | `STATUS_OK`, clean |

| 2 | full original mix (5 grammars) | bb-main-atomic-rel (Release-like) | 1787213082 (rerun, thread interleaving differs) | 180s | `STATUS_OK`, clean |

| 3 | full original mix (5 grammars) | b26a (ASAN) | 1787213083 | 180s (much lower effective query count: ASAN + `max_statement_time=20` combine to time out most multi-row INSERTs) | `STATUS_OK`, clean |

| 4 | narrowed: `partition_by_hash.yy` + `innodb_compression_algorithms.yy` only, `gendata=simple` | bb-main-atomic-rel (Release-like) | 1787213200 | 240s | `STATUS_OK`, clean |

Total: **4 full-duration trials, ~0 hits**, roughly 20 minutes of cumulative

RQG wall-clock. This is a small sample for a race this narrow (the original

CI run needed ~82s of an 8-thread, higher-throughput Jenkins host to hit it

once), so a clean run here is weak evidence of absence, not evidence the bug

is gone. Scripts for attempts 1/2 and attempt 4 are `repro_original.sh` and

`repro_narrowed.sh` in this directory (`VARDIR=... PORT=... SEED=... bash

repro_original.sh`); attempt 3 is the same as `repro_original.sh` with

`BASEDIR=/data/bld/b26a`.

## Is this specific to the `bb-main-atomic` branch?

Almost certainly not — best guess is that **main reproduces this too**,

based on code identity rather than an actual hit on main (none was obtained,

see above):

* The crashing commit's branch ("bb-main-atomic") diverges from mainline

  `main` at `b8dbc264f03ad4ebb483f6380715d81b67cc2a42` (2026-08-03). From

  there, the atomic branch carries only 4 commits of its own (the "Atomic

  CREATE OR REPLACE TABLE" work, MDEV-25292 + MDEV-40776), while `main` has

  191 commits on top of the same base by 2026-08-19.

* Those 4 atomic-branch commits touch almost nothing outside `mysql-test/`;

  the only non-test change is a one-line whitespace fix to an error string in

  `handler0alter.cc`.

* `storage/innobase/fil/fil0fil.cc`, `storage/innobase/buf/buf0flu.cc`, and

  `storage/innobase/handler/handler0alter.cc` — everything this crash's call

  stack runs through — are **byte-identical** between the divergence point,

  the crashing commit, and current `main` (fetched fresh, HEAD `bab03b0f`,

  2026-08-19): zero commits touched any of them on either side of the split.

So the atomic-DDL work neither introduces nor fixes anything in the

tablespace-lifecycle/checkpoint code this bug lives in; it's plain inherited

InnoDB code. Worth confirming with an actual `main` build if one becomes

available, but there is no code-level reason to expect it to be

atomic-branch-specific.

## Recommendation for whoever continues this

1. **File a new MDEV** referencing MDEV-37755 as related-but-distinct,

   including this signature and the evidence above. Do not assume it's a

   duplicate — the fix commit is confirmably present.

2. The most promising lead is the **volume/burst of tablespace drops**:

   MDEV-37755's own reproduction (per its JIRA comments) appears to have been

   single/few-tablespace scenarios; this trial's trigger candidate is an

   `ALTER TABLE ... ORDER BY` on a **98-partition** table, i.e. ~98

   tablespace drops fired back-to-back at ALTER commit time, racing a

   concurrent checkpoint. A targeted reproducer should drive repeated

   full-rebuild ALTERs (`ORDER BY`, or any other `ALGORITHM=COPY` ALTER) on

   wide-partition InnoDB tables with a background thread continuously forcing

   checkpoints (e.g. tight `innodb_flush_log_at_trx_commit`/small log file,

   or explicit periodic `FLUSH TABLES` / small `innodb_log_file_size`), and

   should not need `full_text_search`/`all_selects`/`cor.yy` at all — those

   appear to be noise, not part of the trigger.

3. Given how rare this race is even under conditions close to the original,

   **`rr` chaos-mode recording** (as MariaDB's own QA used for MDEV-37755) is

   almost certainly a better investment than more blind RQG stress runs,

   especially in a disk-constrained sandbox like this one.

4. If a hit is ever obtained, get the `gdb` backtrace of *all* threads at

   crash time (not just the one that aborted) — the actual corrupting write

   is presumably in a DDL/purge thread that had already returned by the time

   the corruption was noticed, so a single-thread backtrace from the crash

   point will not show it; an `rr` replay or a watchpoint on the freed

   `fil_space_t`'s `named_spaces` list-node pointers (as MDEV-37755's

   diagnosis did) is the way to actually catch the write.