Thanks mleich!

Technically reproducible on 10.2 with explicit minimum (1M) innodb-buffer-pool-chunk-size, MDEV-25342 just pushed the default value for this lower.

$ podman run  -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 quay.io/mariadb-foundation/mariadb-devel:10.2 --innodb-buffer-pool-size=256M --innodb-buffer-pool-chunk-size=1M --innodb-page-size=64k

...

Version: '10.2.42-MariaDB-1:10.2.42+maria~bionic' as '10.2.42-MariaDB-80d33261ea1a731f228b712b0e763740f87b89be'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: Resizing buffer pool from 268435456 to 8388608 (unit=1048576).

2022-01-11  5:25:55 140319263479552 [Note] InnoDB: 

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: Disabling adaptive hash index.

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: disabled adaptive hash index.

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: Withdrawing blocks to be shrunken.

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: buffer pool 0 : start to withdraw the last 3968 blocks.

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: buffer pool 0 : withdrawing blocks. (3840/3968)

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: buffer pool 0 : withdrew 3799 blocks from free list. Tried to relocate 41 pages (3840/3968).

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: buffer pool 0 : withdrawing blocks. (3840/3968)

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: buffer pool 0 : withdrew 0 blocks from free list. Tried to relocate 0 pages (3840/3968).

2022-01-11  5:25:55 140318846473984 [Note] InnoDB: buffer pool 0 : withdrawing blocks. (3840/3968)

2022-01-11  5:28:00 140318846473984 [Note] InnoDB: buffer pool 0 : withdrawing blocks. (3840/3968)

2022-01-11  5:28:00 140318846473984 [Note] InnoDB: buffer pool 0 : withdrew 0 blocks from free list. Tried to relocate 0 pages (3840/3968).

2022-01-11  5:28:00 140318846473984 [Note] InnoDB: buffer pool 0 : withdrawing blocks. (3840/3968)

2022-01-11  5:28:00 140318846473984 [Note] InnoDB: buffer pool 0 : withdrew 0 blocks from free list. Tried to relocate 0 pages (3840/3968).

2022-01-11  5:28:00 140318846473984 [Note] InnoDB: buffer pool 0 : will retry to withdraw later.

2022-01-11  5:28:00 140318846473984 [Note] InnoDB: Will retry to withdraw 10 seconds later.

I don't quite get the assertion in the timeout thread, but if the above endless loop is prevented it becomes resolved.

I analyzed the hang yesterday. It was caused by buf_pool.mutex being continuously held by buf_LRU_free_page(), for buf_page_init_for_read() while buf_pool_t::resize() in another thread was desperately trying to complete. We had innodb_buffer_pool_size=8M and innodb_page_size=64k during that hang.

What the the page read was issued for should not matter. I think that it was purge or rollback.

From marko: My preferred fix would be to avoid the hang when trying to allocate a page while resize is running. I did not investigate how easy that would be to do.

I think that the test failure demonstrates two problems. The main one is that the minimum buffer pool size is not being enforced consistently. I hope that you can fix that, danblack.

To me, the more interesting one is that the server is hanging. I will try to determine the root cause of that, and see what it would take to fix that.

I had a look at the code. The function buf_LRU_get_free_block() already works in a rather reasonable fashion. It is releasing the buf_pool.mutex between iterations. The infamous message InnoDB: Difficult to find free blocks in the buffer pool is being suppressed when buffer pool resizing is active.

Because buf_LRU_get_free_block() is not waiting for buffer pool resizing to finish and because buf_pool_t::resize() is only polling the status every 10 seconds, the infinite loop could be unavoidable. In MDEV-27461.txt we have the following:

• Thread 13 is attempting to allocate a buffer page for reading a page when opening a table, causing the infinite wait in buf_LRU_get_free_block().
• Thread 11 (finishing ROLLBACK) is attempting to allocate a buffer page for reading a page.

I do not think that either thread should be holding many buffer pool page latches. I will have to debug the trace again to see if anything else could be improved, apart from the trivial fix to enforce a reasonable minimum buffer pool size (measured in pages of innodb_page_size, not bytes).

I analyzed a new trace of this. In it, we have 229 pages in both buf_pool.LRU and buf_pool.flush_list, and at least 2 of those pages are really waiting to be written out to a data file. (After MDEV-25113, buf_pool.flush_list may contain clean pages.)

The page cleaner is in an untimed wait that was introduced in MDEV-24278. The buffer pool resizer should probably wake it up. Both buf_flush_async_lsn and buf_flush_sync_lsn are 0. During the execution of the entire trace, the page cleaner was woken up only once:

Thread 2 hit Breakpoint 3, buf_flush_page_cleaner () at /data/Server/bb-10.8-MDEV-14425D/storage/innobase/buf/buf0flu.cc:2235

2235	    set_timespec(abstime, 1);

I believe that a potential hang is possible in 10.5 or later even with larger buffer pool sizes, due to MDEV-24278.

For 10.5 and later, I pushed a fix to ensure that the page cleaner will start writing out pages so that some space will become available in the buffer pool. According to mleich, it helps to reduce the problem, but it will not prevent hangs when larger values of innodb_max_dirty_pages_pct are in use.

With that fix (fixing a minor regression caused by MDEV-24278), the problem is no longer repeatable with the default innodb_max_dirty_pages_pct=90. Therefore, I do not think that this bug should block MDEV-25342 anymore.

Root cause:
I was wondering why buf_flush_LRU_list_batch() fails to write out any dirty pages. The reason is that the condition UT_LIST_GET_LEN(buf_pool.LRU) > BUF_LRU_MIN_LEN does not hold, because there are fewer than 256 pages in the buf_pool.LRU list. This would seem to be the hard minimum size of the buffer pool in pages. 256*16384=4MiB, and 256*65536=16MiB. The buffer pool is smaller than the latter.

I do not know why the minimum sizes at startup are steeper than the hard limit (5MiB for up to innodb_page_size=16384, and 24MiB for larger page sizes), but it indeed would be consistent to enforce the same limit at startup and in SET GLOBAL. danblack, can you please fix the SET GLOBAL?

Consistent enforcement of the minimum value in SET GLOBAL innobd_buffer_pool_size will be covered by MDEV-27467.
The scope of this ticket was reduced to only fix the recent regression that was caused by MDEV-24278 in MariaDB Server 10.5.9: the page cleaner would fail to wake up while the buffer pool is being shrunk.