Secondary monitor cannot get locks after Primary monitor power failed

either the power failed or the primary MaxScale node was disconnected from the network. The locks on backend servers are still exist. MaxScale secondary node cannot acquire locks and cannot perform cluster operation, such as automatic failover or manual switch over.

• Error when try to switch over:
  Cannot perform cluster operation because this MaxScale does not have exclusive locks on a majority of servers. Run "SELECT IS_USED_LOCK('maxscale_mariadbmonitor');" on the servers to find out which connection id has a lock.

• SELECT IS_USED_LOCK() at backend servers:
  • At master mariadb backend
    MariaDB [(none)]> SELECT IS_USED_LOCK('maxscale_mariadbmonitor'), IS_USED_LOCK('maxscale_mariadbmonitor_master')\G
    IS_USED_LOCK('maxscale_mariadbmonitor'): 26061
    IS_USED_LOCK('maxscale_mariadbmonitor_master'): 26061
  • At one of slave mariadb backend server
    MariaDB [(none)]> SELECT IS_USED_LOCK('maxscale_mariadbmonitor'), IS_USED_LOCK('maxscale_mariadbmonitor_master')\G
    IS_USED_LOCK('maxscale_mariadbmonitor'): 26061
    IS_USED_LOCK('maxscale_mariadbmonitor_master'): NULL

This does not happen when the primary MaxScale is graceful shutdown.

I think set a small wait_timeout in monitor session should help remove the stale backend locks.