I would like to see this repeated with ./mtr --rr using the patch from MDEV-22179.
With a rr replay of the server that was killed right before the failed recovery, we should be able to figure out the cause.

I believe that this bug should be possible also in 10.5. The code was heavily refactored related to MDEV-12353, and the MLOG_CHECKPOINT record was replaced by FILE_CHECKPOINT, but the logic should be similar ever since 10.2.2. Only in MDEV-14425 (which missed 10.5) we would eliminate this logic and drastically reduce the amount of log written by a checkpoint.

I believe that this bug should also be repeatable with RQG. It could help to use innodb_encrypt_tables, innodb_page_size=64k and innodb_checksum_algorithm=crc32, to make page flushing as slow as possible. (The default 10.5 innodb_checksum_algorithm=full_crc32 would only compute one checksum per page.) We should only need lots of writes, and frequent checkpoints. Checkpoints can be triggered in debug builds with the following:

SET GLOBAL innodb_log_checkpoint_now=ON;

We might also want to set --debug-dbug=d,ib_log_checkpoint_avoid to forbid checkpoints being from being triggered by background activity. Repeating this bug might require that there be long enough pauses between checkpoints.

I suspected that this bug could share a root cause with MDEV-22027, but that bug turned out to be a 10.5-specific regression.

Before killing the server:
=============
Thread A tries to do log_checkpoint() and does fil_names_clear() It does log_write_up_to().
Another thread B does the similar operation like fil_names_clear() and it also does log_write_up_to()
Another thread C does the similar operation like fil_names_clear() and finishes log_write_up_to(). Writes the checkpoint info in log file header.

After that, Thread A and Thread B return from log_checkpoint() without writing checkpoint info because it has log_sys.n_pending_checkpoint_writes value as 0.

During recovery:
=============
First scan: By using end_lsn, InnoDB does parse the redo log from end_lsn (thread C's one) and finds the FILE_CHECKPOINT lsn for it.
(mlog_checkpoint_lsn is 110052239)
Second scan: It starts from checkpoint_lsn, parse the redo log and find the FILE_CHECKPOINT lsn of thread As. (110051441).

The recovery has the following check in recv_sys_t::parse()

          if (lsn == checkpoint_lsn)

          {

            ut_ad(mlog_checkpoint_lsn <= recovered_lsn);

            if (mlog_checkpoint_lsn)

              continue;

            mlog_checkpoint_lsn= recovered_lsn;

            l+= 8;

            recovered_offset= l - buf;

            return true;

          }

For the above code, recovered_lsn value is 110051441. It leads to debug assert. IMO, we should remove the assert and it is wrong.

Thank you. The debug assertion is indeed bogus and should be removed.