It indeed seems to be the case that the use of a ‘sub-minitransaction’ ibuf_mtr in fsp_fill_free_list() is ‘overtaking’ the write of FSP_SIZE.

I believe that the reason for using a separate ibuf_mtr in fsp_fill_free_list() is to avoid trouble with any caller that could access further pages while the change buffer bitmap page latch is being held.

I see two alternative fixes:

1. Before invoking fsp_fill_free_list() in fsp_header_init(), commit and restart the mini-transaction. To prevent page eviction, buf_block_buf_fix_inc() could be invoked before the mtr_t::commit().
2. Remove the ibuf_mtr and use the passed mtr instead. Ensure that no caller is accessing any further pages before invoking mtr_t::commit().

The transitive closure of all callers of fsp_fill_free_list() in that compilation unit is as follows:

• fsp_alloc_free_extent()
• fsp_alloc_free_page()
• fseg_fill_free_list()
• fseg_alloc_free_extent()
• fseg_alloc_free_page_low()
• fsp_alloc_seg_inode_page()
• fsp_alloc_seg_inode()
• fseg_create_general()
• fseg_create()
• fseg_alloc_free_page_general()
• fsp_header_init()

None of the functions are invoking mtr_t::commit(). The last 4 functions are global, and hence we must conduct a transitive closure of them in all files until the mtr_t::commit() call. Here is the closure of fseg_create_general(), fseg_create(), fseg_alloc_free_page_general() but not fsp_header_init():

• fseg_create_general(): trx_undo_create(), trx_undo_assign_undo() looks OK
• fseg_create(): btr_create() is allocating a further page in the same mini-transaction. Not OK!
• fseg_create(): buf_dblwr_create() is allocating multiple pages. Possibly OK, because this only happens during bootstrap.
• fseg_create(): dict_hdr_create() is allocating multiple pages. Possibly OK, because this only happens during bootstrap.
• fseg_create(): trx_rseg_header_create() is looking up the TRX_SYS page after allocating a page. Might be problematic, although it only occurs during bootstrap.
• fseg_create(): trx_sysf_create() might be OK, because it only happens during bootstrap.
• fseg_alloc_free_page_general(): btr_page_alloc_low(), btr_page_alloc() get the mtr from their callers
• fseg_alloc_free_page_general(): trx_undo_add_page() gets the mtr from its callers, and its callers trx_undo_report_row_operation() and trx_undo_report_rename() do not seem to be accessing any further pages before invoking mtr_t::commit().
• btr_page_alloc(): btr_root_raise_and_insert() can allocate further pages in btr_page_split_and_insert(): Not OK!
• btr_page_alloc(): There are also other callers, which I did not investigate.

If we let the fsp_alloc_free_extent() call in fseg_alloc_free_extent() continue to use a sub-minitransaction, the transitive closure of affected callers would be limited to fsp_header_init(), which is listed below:

• fsp_header_init(): Most callers are immediately invoking mtr_t::commit().
• fsp_header_init(): srv_undo_tablespaces_init() is using a single mini-transaction for creating all undo tablespaces. Should be OK, because it happens at bootstrap.
• fsp_header_init(): (10.2 and 10.3 only) srv_undo_tablespaces_init() is initializing the undo tablespaces in the recovery of undo tablespace truncation. Probably OK, because it happens at startup.
• fsp_header_init(): innobase_start_or_create_for_mysql() is invoking btr_create() on the ibuf->index in the same mini-transaction. Probably OK, because it happens at bootstrap.

We should also check the transitive closure of callers of the following functions, which are modifying FSP_SIZE. No redo log for referring to newly added pages must be written ahead of the FSP_SIZE modification:

• fsp_header_inc_size(): OK; only called by innobase_start_or_create_for_mysql(), which immediately calls mtr_t::commit()
• fsp_try_extend_data_file_with_pages()
• fsp_try_extend_data_file(): fsp_fill_free_list() is using the problematic sub-minitransaction as noted above. And this can occur also as part of fseg_alloc_free_extent() when a new change buffer bitmap page is created.

I do not yet see how a deadlock involving the change buffer could be possible, as implied by the comment:

			/* Initialize the ibuf bitmap page in a separate

			mini-transaction because it is low in the latching

			order, and we must be able to release its latch

			before returning from the fsp routine */

Any page allocation operation should be protected by an exclusive fil_space_t::latch. Other threads should not be able to access any of the newly allocated pages (including the change buffer bitmap page) while our mini-transaction is holding the tablespace latch. The only exception that I can think of would be read-ahead. In both buf_read_ahead_random() and buf_read_ahead_linear() we are reading fil_space_t::size without holding fil_space_t::latch, which should be protecting it. If we introduce the proper protection to fil_space_t::size, we should be able to eliminate the sub-minitransaction. If the tablespace is being extended while a read-ahead operation is executing, it would either block until the extension has been completed, or it would limit the read-ahead to pages that were not extended yet.

Because introducing extra synchronization is expensive, we might rather replace the problematic use of fil_space_t::size with fil_space_t::free_limit. It should not make sense to read-ahead any unallocated pages in any case.

I prototyped 3 different solutions on 10.1 so far:

1. Cache FSP_FREE_LIMIT in fil_space_t::free_limit, and protect access to it with fil_space_t::latch when determining the bounds for read-ahead and flushing. Problem: FSP_FREE_LIMIT can actually be larger than FSP_SIZE!
2. Access fil_space_t::size protected by fil_space_t::latch when determining the bounds for read-ahead and neighbour flushing. Problem: Some operations will hang, because the latch acquisition would be violating the latching order.
3. Copy fil_space_t::size to a new field fil_space_t::committed_size when mtr_commit() is releasing an exclusive fil_space_t::latch. There are some crashes, which I have not debugged yet.

Once I have a workable solution for preventing read-ahead or flushing from accessing a newly extended change buffer bitmap page, I can remove the ibuf_mtr and use the single mini-transaction object in fsp_fill_free_list(), to fix the ordering problem that is breaking the recovery.

I pushed a fix to bb-10.1-marko and merged it up to bb-10.5-MDEV-23190, with some conflict resolution in each of 10.2, 10.3, 10.4, 10.5. mleich, please test.

I pushed this to 10.1 and merged up to 10.5. Between each major version, some additional conflict resolution was needed.