We encountered a problem with one node hanging in a Galera cluster. At the time one connection was issuing:

SET GLOBAL server_audit_events='CONNECT,QUERY_DDL,QUERY_DCL',server_audit_output_type=syslog,server_audit_excl_users='XXXXXXXXXX',server_audit_logging=1,binlog_row_image='minimal',expire_logs_days=1;

Other connections seemed to be internal healthcheck / logging utilities running SHOW STATUS LIKE 'wsrep_local_state' or dumping wsrep status variables via SHOW GLOBAL STATUS LIKE 'wsrep%'.

Attached is thread apply all bt output indicating the mutex contention. It looks like all threads were waiting on the server_audit mutex.

We are writing to only a single node. The other nodes in the cluster seemed to not be affected at the time.

Encountered on MariaDB 10.1.18 / Galera 25.3.17.