Purge coordinator loops forever in purge_sys_t::close_and_reopen() and leaks THD mem_root (22+ GB in < 2 min) for tables with an indexed virtual column

Symptom

On an otherwise idle Galera node (no client traffic, it only applies write sets), mariadbd anonymous memory grew from about 4 GB to 26 GB within two minutes and never went back down. The process reached the systemd MemoryHigh limit and then thrashed in reclaim (PSI memory "full" ~52 %, 56 % sys CPU, swap full). After that it stopped answering SQL. systemctl stop hung for over 10 minutes, so it had to be SIGKILLed. After restart it rejoined via IST.

We had seen the same pattern about 50 times between August and October 2026 on an earlier 13.0.x cluster (same Galera version). One of those episodes ended in an InnoDB semaphore-wait abort.

Evidence

performance_schema, taken at 25 GB RSS. All the memory is in the main_mem_root of one InnoDB thread-pool thread:

| event_name                     | cur_mb     | high_mb    | current_count_used |

| memory/sql/thd::main_mem_root  | 22079.5509 | 22079.9763 |              13467 |

| memory/innodb/buf_buf_pool     | 16384.0000 | 16384.0000 |                  1 |

| thread_id | name                          | event_name                    | cur_mb     |

|    333889 | thread/innodb/ib_tpool_worker | memory/sql/thd::main_mem_root | 22098.8554 |

In /proc/<pid>/smaps the growth is in the main [heap] (10 GB → 16 GB in 30 s). glibc arenas are normal (~64 MB each).

Stacks of that thread (eu-stack, several samples over about 30 minutes). It is always in the purge table open/close cycle:

#0 open_table(THD*, TABLE_LIST*, Open_table_context*)

#1 open_purge_table(THD*, char const*, unsigned long, char const*, unsigned long, MDL_ticket*)

#2 trx_purge_table_open(unsigned long, MDL_context*)

#3 purge_coordinator_callback(void*) [clone .cold]

---

#0 lf_hash_delete / MDL_context::release_lock(enum_mdl_duration, MDL_ticket*)

#4 trx_purge_close_tables(THD*, purge_table*, bool)

#5 purge_coordinator_callback(void*) [clone .cold]

---

#3 bool dict_table_t::parse_name<false>(...)

#4 trx_purge_table_open(unsigned long, MDL_context*)

#5 purge_coordinator_callback(void*) [clone .cold]

gdb (with dbgsym), full frame list of the same thread:

my_mb_wc_filename

strconvert

filename_to_tablename

dict_table_t::parse_name<true>

trx_purge_table_acquire

trx_purge_table_open

purge_sys_t::close_and_reopen

trx_purge_attach_undo_recs

trx_purge

purge_coordinator_state::do_purge

purge_coordinator_callback

tpool::task_group::execute

Analysis (12.3.3 source)

MDEV-39261 added the following. When open_purge_table() returns NULL for a table with an indexed virtual column, trx_purge_table_open() returns table == -1 (must_wait()). Then purge_sys_t::close_and_reopen() does goto retry, with no retry limit and no back-off other than VALGRIND_YIELD.

Each attempt calls open_purge_table(), which allocates a TABLE_LIST (plus whatever open_table() allocates) on thd->mem_root of the purge coordinator THD:

TABLE *open_purge_table(THD *thd, ...)

{

  Open_table_context ot_ctx(thd, MYSQL_OPEN_HAS_MDL_LOCK);

  TABLE_LIST *tl= thd->alloc<TABLE_LIST>(1);     // <-- every retry

  ...

  bool error= open_table(thd, tl, &ot_ctx);

  DBUG_RETURN(error ? NULL : tl->table);

}

trx_purge_close_tables() calls close_thread_tables() and releases the MDL tickets. Nothing calls free_root() (or otherwise resets thd->mem_root) inside the loop. While open_table() keeps failing, the coordinator therefore spins at full speed and leaks on every iteration. That matches both the 22 GB in thd::main_mem_root and the high sys CPU. The loop also never ends, which is why shutdown hangs: purge never finishes its batch.

Only tables with an index on a VIRTUAL column take this path (the mdl && result.table->has_virtual_index() branch). Purge also visits INSERTed records, to reset DB_TRX_ID. So a table that only receives INSERTs is enough to enter this branch on every purge batch.

In our cluster the only such tables were INSERT-only logging tables with

`x_id` varchar(50) GENERATED ALWAYS AS (json_value(`request_content`,'$.subscription.id')) VIRTUAL,

KEY `x_id` (`x_id`)

(about 1 insert per minute, 1.09 M rows, 12.8 GB, ROW_FORMAT=DYNAMIC, longtext JSON column with a CHECK (json_valid(...))).

We could not determine why open_table() failed persistently on that node. There was no FLUSH TABLES, BACKUP STAGE, ALTER or RENAME on the affected tables at the time (Com_flush=0, Com_backup=0). The only DDL in the cluster at the time was an application job that does CREATE/TRUNCATE/DROP of unrelated tables every 15 minutes, applied through Galera TOI. table_open_cache=4000, table_definition_cache=2048, Opened_table_definitions ~190, so no cache pressure. Once the loop has started, it never seems to recover.

Workaround

We converted the generated columns from VIRTUAL to STORED, so the tables no longer have a virtual index and purge no longer goes through open_purge_table(). A direct MODIFY ... STORED gives ERROR 1907 This is not yet supported for generated columns. We had to copy into a new table with DROP KEY, DROP COLUMN, ADD COLUMN ... STORED, ADD KEY and swap the tables with RENAME TABLE.

Suggested fix

• Free or reset the coordinator's thd->mem_root on each close_and_reopen() iteration (e.g. free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC)) after close_thread_tables()), or allocate the TABLE_LIST on the stack or in a per-attempt mem_root in open_purge_table().
• Bound or back off the retry (sleep, or wait on the TDC flush condition instead of spinning), and let shutdown interrupt the loop.
• Report why open_table() failed (ot_ctx action / error code). That would help identify the persistent failure condition.

Available on request

Full eu-stack dumps of all threads (two samples), smaps mappings, performance_schema memory tables, SHOW ENGINE INNODB STATUS and processlist at the time of the spike, gdb frame list. We kept the process running for diagnostics before killing it, but no core dump (26 GB, mostly swapped).