Startup hangs in init_slave(): slave deadlock-handler thread never signals start; server accepts connections but never services them, ignores SIGTERM

BELOW IS A AI GENERATE REPORT. The hanging Maria definitely happened, as for the accuracy of the summary below.... that's down the new robot overlords.

During a fleet restart to apply configuration changes, 4 out of 4 mariadbd instances (two instances on each of two identically-configured hosts) hung permanently during startup. The same binary/config combination started normally on 6 subsequent boots, so this appears to be a startup race whose window was widened by an unusually heavy first boot (details below).

State of a hung instance

• Error log shows normal startup through Buffer pool(s) load completed and Server socket created on IP ..., then nothing further — the ready for connections line is never printed.
• TCP/socket connections are accepted at the OS level but never serviced: SELECT 1 over the unix socket blocks indefinitely.
• systemctl stop / SIGTERM has no effect: no Starting shutdown is ever logged; the process must be SIGKILLed.
• Process is idle (~0% CPU, all threads in futex/cond waits).
• On some (not all) hung instances the error log contains, during startup: [Warning] InnoDB: A long wait (229 seconds) was observed for dict_sys.latch

Stack analysis (gdb, with mariadb-server-core-dbgsym)

The main thread is blocked inside init_slave(), waiting for the slave background (deadlock handler) thread to signal it has started:

Thread 1 (LWP 51299) "mariadbd":

#5 inline_mysql_cond_wait (src_line=571, src_file="./sql/slave.cc", ...)

#6 start_slave_deadlock_handler_thread () at sql/slave.cc:571

#7 init_slave () at sql/slave.cc:675

#8 mysqld_main (...) at sql/mysqld.cc:6059

Crucially, the thread being waited for is NOT present in the process: the full thread apply all bt dump contains 22 threads — timer, checkpoint worker, io_uring, buf_flush, mem_pressure, 12 fil_crypt threads (idle on fil_crypt_threads_cond / fil_crypt_alloc_iops), handle_manager, binlog_background_thread, one tpool worker, and main — and no slave background/deadlock-handler thread. It either failed to spawn or exited before signalling, leaving main in an infinite cond_wait.

Because mysqld_main never proceeds past init_slave, sd_notify readiness is never sent (systemd unit hangs in activating/deactivating), connection servicing never starts (hence accepted-but-dead connections), and the shutdown path is unreachable.

Full thread apply all bt dumps for all four hung instances, error logs, configs, journal extracts and package lists are attached (one bundle per host).

What was special about the triggering boot

All four instances hung on the first boot after a config change that made startup unusually heavy, all at once:

• innodb_buffer_pool_size raised 128M → 24G (with buffer pool dump/load on)
• innodb_log_file_size raised → 16G (redo resized on this boot)
• innodb_encrypt_tables=ON via file_key_management, 12 encryption threads, ~20,000 InnoDB tablespaces per instance to scan for rotation
• table_definition_cache=2097152, table_open_cache=1048576
• slave_parallel_threads=40
• Each instance is a GTID multi-source replica (one named channel) that was mid-stream; slave threads auto-start at boot (no skip_slave_start)

Six later boots — including one with the identical full configuration — started normally, so none of these settings is individually necessary or sufficient; the combination appears to widen a race in the init_slave thread-start handshake.

Steps attempted to reproduce

• Removing slave_parallel_threads only: starts fine.
• Removing table caches only: starts fine.
• Full original configuration re-applied on one instance: started fine.
• No deterministic repro found; original incident was 4/4 simultaneously.

Suggested areas to inspect

The handshake in start_slave_deadlock_handler_thread (sql/slave.cc:571, 10.11.18) appears to have no timeout and no handling for the background thread exiting before setting its running flag — whatever the underlying cause of the thread failing to come up, the result is an unkillable, half-started server rather than a startup error.