Reproducible `InnoDB: Assertion failure ... trx0rseg.cc line 370, !UT_LIST_GET_LEN(undo_list)` on every clean `systemctl stop mariadb`, no open transactions

1. 1. Environment

1. 1. Summary

The server reliably crashes with `SIGABRT` on *every* clean shutdown (`systemctl stop mariadb`), immediately after `Buffer pool(s) dump completed`, with an InnoDB assertion failure in `trx0rseg.cc` line 370 (`!UT_LIST_GET_LEN(undo_list)`). This happens even though `SHOW ENGINE INNODB STATUS` immediately before the stop shows no open transactions, no prepared XA transactions, and a purge history list length of 0.

The crash is *not* related to `innodb_fast_shutdown` — it was reproduced identically with both `innodb_fast_shutdown = 0` and `innodb_fast_shutdown = 1` (the default).

The crash does not appear to cause table-level data corruption: `mariadb-check --all-databases --check` reports every table across all databases as `OK` after multiple crash/restart cycles.

1. 1. Steps to Reproduce

1. Start MariaDB normally (`systemctl start mariadb`).
2. Let the server run under normal application load for a period (in our case, a general-purpose multi-app database server — Nextcloud, phpBB3, piwigo, phpMyAdmin, and others — with no unusual transaction patterns).
3. Stop the server cleanly: `systemctl stop mariadb`.
4. Observe the crash in `/var/log/mysql/mysql_error.log` and `journalctl -u mariadb`.
5. `systemctl start mariadb` succeeds afterward without issue — the crash is reliably reproducible on the next stop as well.

This has now been reproduced identically *4 times in a row* across a ~2.5 hour window, with different process IDs and different ASLR base addresses each time (ruling out a one-off memory issue tied to a specific process instance).

1. 1. Expected Behavior

Clean, silent shutdown with exit code 0, as is normal for InnoDB when there are no open transactions and the purge history list is empty.

1. 1. Actual Behavior

`systemd` reports:
```
mariadb.service: Main process exited, code=killed, status=6/ABRT
mariadb.service: Failed with result 'signal'.
```

1. 1. Relevant Log Excerpt (`/var/log/mysql/mysql_error.log`)

```
2026-09-11 10:13:04 0 [Note] /usr/sbin/mariadbd (unknown): Normal heruntergefahren
2026-09-11 10:13:04 0 [Note] InnoDB: FTS optimize thread exiting.
2026-09-11 10:13:04 0 [Note] InnoDB: Starting shutdown...
2026-09-11 10:13:04 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2026-09-11 10:13:04 0 [Note] InnoDB: Buffer pool(s) dump completed at 260911 10:13:04
2026-09-11 10:13:04 0x7f94d3c4bfc0 InnoDB: Assertion failure in file ./storage/innobase/trx/trx0rseg.cc line 370
InnoDB: Failing assertion: !UT_LIST_GET_LEN(undo_list)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mariadbd startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/docs/server/server-usage/storage-engines/innodb/innodb-troubleshooting/innodb-recovery-modes
InnoDB: about forcing recovery.
260911 10:13:04 [ERROR] /usr/sbin/mariadbd got signal 6 ;
```

1. 1. Full Backtrace

```
Thread pointer: 0x0
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mariadbd(my_print_stacktrace+0x2e) [0x55b29cf94f6e]
/usr/sbin/mariadbd(handle_fatal_signal+0x269) [0x55b29cb05079]
/lib/x86_64-linux-gnu/libc.so.6(+0x3fdf0) [0x7f94d304bdf0]
/lib/x86_64-linux-gnu/libc.so.6(+0x9495c) [0x7f94d30a095c]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x7f94d304bcc2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x22) [0x7f94d30344ac]
/usr/sbin/mariadbd(+0x6f3684) [0x55b29c73c684]
/usr/sbin/mariadbd(+0x6f137c) [0x55b29c73a37c]
/usr/sbin/mariadbd(+0xe3b3dc) [0x55b29ce843dc]
/usr/sbin/mariadbd(+0xe26f6b) [0x55b29ce6ff6b]
/usr/sbin/mariadbd(+0xd5731b) [0x55b29cda031b]
/usr/sbin/mariadbd(_Z22ha_finalize_handlertonPv+0x3a) [0x55b29cb0834a]
/usr/sbin/mariadbd(+0x873279) [0x55b29c8bc279]
/usr/sbin/mariadbd(+0x8739d6) [0x55b29c8bc9d6]
/usr/sbin/mariadbd(_Z15plugin_shutdownv+0xa6) [0x55b29c8be0d6]
/usr/sbin/mariadbd(+0x75a960) [0x55b29c7a3960]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0xe90) [0x55b29c7adde0]
/lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7f94d3035ca8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f94d3035d65]
/usr/sbin/mariadbd(_start+0x21) [0x55b29c7a1f81]
```

(Note: symbol addresses vary between crash instances due to ASLR, but the crash signature — `ha_finalize_handlerton` → `plugin_shutdown` → `abort()` via the InnoDB assertion — is identical across all 4 reproductions.)

1. 1. Server State Immediately Before Shutdown (`SHOW ENGINE INNODB STATUS`)

```
TRANSACTIONS
------------
Trx id counter 330653926
Purge done for trx's n:o < 330653926 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION (0x7ffa203c4b80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
```

`XA RECOVER;` returned an empty set (no prepared XA transactions).

1. 1. Additional Data Point (possibly related, possibly a separate issue)

On one occasion, after explicitly setting `SET GLOBAL innodb_fast_shutdown = 0;` before stopping, the shutdown log additionally showed InnoDB system tablespace autoshrink-related activity before hitting the same assertion:

```
2026-09-11 9:50:46 0 [Note] InnoDB: :autoshrink freed the segment 10226 in page 876
...
2026-09-11 9:50:46 0 [ERROR] InnoDB: File './ibdata1' is corrupted
2026-09-11 9:50:46 0 [ERROR] InnoDB: :autoshrink failed to free the segment 8306 in page 913
2026-09-11 9:50:46 0 [Note] InnoDB: Starting shutdown...
```

This is notable because `innodb_data_file_path` on this server does *not* have the `:autoshrink` attribute set (confirmed via `SHOW VARIABLES LIKE 'innodb_data_file_path';` → `ibdata1:12M:autoextend`). It's unclear whether this is the same underlying bug surfacing differently under a slow shutdown, or an unrelated second issue. `mariadb-check --all-databases --check` found no corrupted tables after this event either.

Also relevant: on every startup, the following warning appears, suggesting the server has been unable to complete a requested undo tablespace count change for some time:
```
[Warning] InnoDB: Cannot change innodb_undo_tablespaces=3 because previous shutdown was not with innodb_fast_shutdown=0
```

1. 1. Impact / Data Safety

• No data loss or table corruption has been observed (`mariadb-check --all-databases --check`: all tables `OK` across all databases, checked multiple times after crashes).
• The crash occurs after `InnoDB: Buffer pool(s) dump completed`, i.e. after data has already been durably flushed, and appears limited to internal rollback-segment bookkeeping during plugin finalization.
• `systemd` restarts the service automatically and it comes back up cleanly every time.
• Nonetheless, a `SIGABRT` + core dump on every shutdown is concerning for production use and should not be considered "safe to ignore" long-term.

1. 1. What We've Already Ruled Out

• *Galera / wsrep:* not active (`wsrep_on` unset, `60-galera.cnf` entirely commented out).
• *Encryption plugin:* `SHOW PLUGINS` shows only built-in plugins (`Library = NULL` for all rows) — no dynamically loaded encryption plugin is in use.
• *Open/lingering transactions:* none (see `SHOW ENGINE INNODB STATUS` and `XA RECOVER` above).
• *`innodb_fast_shutdown` value:* reproduced identically with both `0` and `1`.
• *Available package updates:* none (`apt-get -s upgrade | grep -i maria` returns nothing; 11.4.13 is current in the Debian repository as of 2026-09-11).

1. 1. Effective Configuration (`mariadbd --print-defaults`)

```
--socket=/var/run/mysqld/mysqld.sock --pid-file=/run/mysqld/mysqld.pid --basedir=/usr --bind-address=127.0.0.1 --max_allowed_packet=256M --expire_logs_days=10 --character-set-server=utf8mb4 --character-set-collations=utf8mb4=uca1400_ai_ci --socket=/var/run/mysqld/mysqld.sock --bind-address=127.0.0.1 --binlog_format=ROW --binlog_expire_logs_seconds=86400 --character-set-server=utf8mb4 --character-set-collations=utf8mb4=uca1400_ai_ci --datadir=/var/lib/mysql --default_storage_engine=InnoDB --general_log=0 --general_log_file=/var/log/mysql/mysql.log --innodb_buffer_pool_size=20G --innodb_buffer_pool_instances=8 --innodb_io_capacity=1000 --innodb_io_capacity_max=2000 --innodb_read_only_compressed=OFF --innodb_log_buffer_size=32M --innodb_log_file_size=512M --join_buffer_size=2M --key_buffer_size=512M --lc_messages_dir=/usr/share/mariadb --lc_messages=de_DE --log_bin=/var/log/mysql/mariadb-bin --log_bin_index=/var/log/mysql/mariadb-bin.index --log_bin_trust_function_creators=true --log_error=/var/log/mysql/mysql_error.log --log_warnings=2 --long_query_time=1 --slow_query_log=1 --slow_query_log_file=/var/log/mysql/mysql-slow.log --log_slow_verbosity=query_plan --max_connections=150 --max_heap_table_size=64M --max_allowed_packet=512M --myisam_sort_buffer_size=512M --port=3306 --pid-file=/var/run/mysqld/mysqld.pid --query_cache_limit=0 --query_cache_size=0 --read_buffer_size=2M --read_rnd_buffer_size=2M --skip-name-resolve --sort_buffer_size=2M --table_open_cache=400 --table_definition_cache=800 --thread_cache_size=50 --tmp_table_size=32M --tmpdir=/tmp --transaction_isolation=READ-COMMITTED --user=mysql --wait_timeout=600
```

Note: `-max_allowed_packet` and `bind-address`/`socket`/`character-set*` appear twice due to option files being merged from multiple `conf.d`/`mariadb.conf.d` files (the last occurrence of each wins and is what's actually active — confirmed via `SHOW VARIABLES` at runtime).

Also note `--innodb_buffer_pool_instances=8` is present in the config but is a no-op on this version: the server logs `[Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files.` on every startup — kept here only for completeness/transparency, not believed to be related to the crash.

1. 1. Attachments

No `gdb`-resolved core dump backtrace is attached (raw addresses only, see above) — `systemd-coredump` was not installed on this system at the time of the crashes and the default `LimitCORE=0` prevented a core file from being written despite the "Writing a core file..." log line. The raw addresses/offsets are consistent across all 4 reproductions and may still allow matching against the `mariadbd` binary if needed.