Hi,

Our DB is a 3 nodes Galera cluster and all traffic are mainly handled by one DB node only.

We experienced DB hang today. During real time monitoring, found DML were just querying or killed. We cannot find any reason of the blocking. NO Galera flow control , no row blocking. Show engine innodb status has the transaction but no useful information. You can see below example thread 4932094.

ProcessList :

show engine innodb status:

---TRANSACTION 1251810249, ACTIVE 1027 sec inserting
mysql tables in use 1, locked 1
1 lock struct(s), heap size 1128, 0 row lock(s)
MariaDB thread id 4932094, OS thread handle 140036119787264, query id 1453748629 172.24.150.132 besdb_user Update
INSERT INTO tb_bes_op_status_upload_hist (ti_upload_toll_domain_id, bi_bes_env_id, i_op_status, vc_create_user_id, vc_update_user_id) VALUES ('9', '3', '0', 'DASTSC1', 'DASTSC1')
TABLE LOCK table `bes_core`.`tb_bes_op_status_upload_hist` trx id 1251810249 lock mode IX

Metadata lock:
------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------+

------------------------------------------------------------------------------------------------------------+

Attached below for reference
mariadb-error.log - DB error log
list_mariadb_session_tdffsbe_tdb04.log - process list every 1 min (only 100 each time)
show-engine-innodb-status.txt - output of "show engine innodb status"
metadata-lock-info.txt - metadata lock info
mariadb.cnf - DB option file

Regards,
William Wong