InnoDB lock wait timeout watchdog

As transactions which are executed by appliers have passed certification in the cluster, they must be applied and committed successfully. These transactions are called brute force (BF). Normally, when BF transaction is requesting conflicting lock that is held by local transaction, local transaction is aborted and rolled back. However, occasionally BF aborting local transactions may not work perfectly due to race conditions or unforeseen behavior of the lock manager, which may cause appliers to wait locks indefinitely. There has been bug reports containing error log message "WSREP: BF lock wait long for trx:" (.e.g. MDEV-20082, MDEV-28070, MDEV-15822, MDEV-17809 etc). These are caused by bug or BF abort did not happen or was unsuccessful, when the applier originally faced a lock conflict with the lock holder. Reproducing these problems have proved to be very hard and in many cases reason for this long lock wait was not found.

In order to resolve indefinite applier waits, a short applier lock wait timeout is introduced. However instead of giving up with lock wait, a background thread is used to retry BF abort on behalf of the applier which is waiting for the lock.

A variable to control the applier lock wait timeout is innodb_wsrep_applier_lock_wait_timeout with default value of five seconds. If the value is zero, the background BF aborting is disabled.

The value of innodb_wsrep_applier_lock_wait_timeout is set to zero in Galera suite MTR test configuration to avoid non-deterministic behavior.