There's also a possibility that it's related to MDEV-34134.

It looks like this is caused by MDEV-33799 (fixed in 10.11.8), but presents as a hang instead of a segfault.

With more investigation (I didn't see the flame graph earlier), I think I see what is going on. When MDEV-31413 added the replication of mysql.gtid_slave_pos, it seems there is no cleanup of records (on a regular slave, the records are automatically cleaned up as they are no longer needed). This is problematic, because at recovery time, the entire mysql.gtid_slave_pos table is read into memory (during the load_gtid_state_cb), along with additional malloc calls for each row during mysql_manager_submit (which builds a linked list with individual mallocs per each row in mysql.gtid_slave_pos). On the regular slave, this is not a problem as gtid_slave_pos won't be too big. However, on the node which replicates gtid_slave_pos, I imagine this can take up a significant amount of memory. On my laptop, I can generate 12MB worth of records in just 5 minutes (just in the size of the mysql.gtid_slave_pos alone, i.e. not counting the additional metadata/linked list).

So I would expect that during the hang, the server runs out of resources and leads to an effective halt of the system. To confirm, do we have any memory profiling information about the hang? E.g. memory/swap usage?

We are currently blocked going to production because of this bug. This has been assigned to the original reporter for feedback. What kind of feedback is needed? Is it some kind of tests that need to be done or some additional details need to be provided? We can chip in. Anything that will help to progress on this issue would be helpful.

Thanks ryudidoo for the report. Moving this out of needs feedback, as Richard has reproduced the issue and confirmed my theory (and filed MDEV-34924 accordingly). Assigning to sysprg of the Galera team for next steps.