Unnecessary index latch contention in BTR_MODIFY_LEAF operations

There could be a better way to reduce index tree contention than what was implemented in MDEV-38814: Implement an optimistic lock escalation path.

That is, when during a page split or merge we need to retracting from the leaf towards the root, we could employ try_lock operations for acquiring non-leaf page latches. Only if a page latch needs to be waited for, we would backtrack, acquiring the page latches in the correct order with respect to WL#6326.

A lock order inversion can only lead to a deadlock when the locks are being waited for. That is what we would avoid.

I don't know how feasible this is to implement in different operations, but I think that the mtr_t::m_memo data structure should contain most of the needed information.

Backtracking (releasing and reacquiring page latches) should be trivial, as long as we have not modified any index pages that we have acquired. If we have modified pages, we must discard any log records from the mini-transaction log and restore the original contents before releasing the page latch, like fsp_system_tablespace_truncate() (MDEV-14795) does in some error scenarios. Perhaps we could adjust the code so that tentative modifications will be written to mtr_t::m_log and applied from there to the pages, once all pages have successfully been latched and the operation is know to succeed.