The InnoDB deadlock checker is holding lock_sys.mutex for a long time. After
MDEV-24671 and MDEV-24731, we could almost perform the check by using a combination of lock_sys.wait_mutex and individual trx_t::mutex to protect the reads of trx->lock.wait_lock (the field may change from a non-null pointer to another non-null pointer without holding lock_sys.wait_mutex).
We can do even better: use Brent’s cycle detection algorithm on trx->lock.wait_trx.. (a new field protected by lock_sys.wait_mutex), and then release lock_sys.wait_mutex, acquire lock_sys.mutex and reacquire lock_sys.wait_mutex to dump information on all transactions that participate in the the deadlock, as well as to choose the victim transaction to be the one with the smallest weight. No dynamic memory allocation is needed, except possibly for some character strings that are part of the output.
There may be at most 8×innodb_page_size active transactions (524,288 when using innodb_page_size=64k). It is probably easiest to treat a waits-for path that exceeds 255 steps as a deadlock. In the old implementation, the cut-off was 200.
The old deadlock reporter only displayed information about 2 transactions, even though a deadlock may involve multiple transactions.