MariaDB Hangs Due to Stuck Queries, Refuses to Restart

Summary
MariaDB occasionally enters an unrecoverable state where multiple queries get stuck indefinitely and the mysqld process becomes unresponsive. Attempting to restart the process either fails or results in a hang during shutdown, requiring forceful intervention (e.g., kill -9).

Steps to Reproduce
Unfortunately, this issue is intermittent and non-deterministic. However, it has occurred multiple times across different workloads.

Observed Behavior

• Queries remain stuck indefinitely with no progress, running for hours until they reach Killed state and does not clean up the Killed state queries.
• Running KILL ID from inside the MySQL client does not work for the stuck query.
• Process does not respond to mysqladmin shutdown or systemctl stop mariadb
• Requires `kill -9` to terminate.
• No significant errors reported in the MariaDB error log around the time of the hang.

Environment Details

• MariaDB runs inside LXD Containers
• Storage: ZFS on pd-ssd, no high IO noticed
• OS: Ubuntu Noble
• Kernel: 6.8.0-1029-gcp #31-Ubuntu SMP

You may notice that the InnoDB buffer pool is low, and that's intended as we have a daemon that automatically raise innodb buffer pool limit based on certain metrics and cache performance. We also confirmed that this issue is happening with InnoDB buffer limit set from 16M limit to 2G limit and with InnoDB buffer usage didn't hit the limit as well.