The function DeadlockChecker::check_and_resolve() is being executed while continuously holding lock_sys.mutex. Moving the deadlock detection to lock_wait() (which was refactored in MDEV-24671) will allow the check to release lock_sys.mutex between search iterations.

It may also be possible to avoid holding lock_sys.mutex altogether, and instead only use lock_sys.wait_mutex and various trx_t::mutex.