The function btr_cur_search_to_nth_level_func() may be invoked during rollback with the flag BTR_RTREE_UNDO_INS applied on top of BTR_MODIFY_TREE. In the BTR_MODIFY_TREE mode, we are not supposed to release any upper-level page latches. But we are doing that. The following would avoid that:

diff --git a/storage/innobase/include/btr0btr.h b/storage/innobase/include/btr0btr.h

index 29ece955702..4563da31cd2 100644

--- a/storage/innobase/include/btr0btr.h

+++ b/storage/innobase/include/btr0btr.h

@@ -160,6 +160,8 @@ record is in spatial index */

 					 | BTR_ALREADY_S_LATCHED	\

 					 | BTR_LATCH_FOR_INSERT		\

 					 | BTR_LATCH_FOR_DELETE		\

+					 | BTR_RTREE_UNDO_INS		\

+					 | BTR_RTREE_DELETE_MARK	\

 					 | BTR_MODIFY_EXTERNAL)))

Also, some function parameters or local variables could be optimized. For example, the function rtr_pcur_open_low() is always being invoked with level=0.

Note: This does not fix the race condition that could explain MDEV-15284, MDEV-16269. Those failures are still repeatable after this cleanup.