One thing that is somewhat unclear if this hang could possibly have been fixed by MDEV-35409 or MDEV-34265. The 10.6.18 release is also affected by MDEV-34166, but I assume that it was already ruled out.

marko I checked this scenario in 10.6 latest. So I think it is not fixed by any other MDEV yet.

commit 15139c88a8f14c535f6888892f3f6dc3765bc765 (HEAD -> 10.6, origin/10.6)

Merge: e95a8f84de7 3a4c0295ae0

Author: Julius Goryavsky <julius.goryavsky@mariadb.com>

Date:   Wed Mar 5 01:54:40 2025 +0100

    Merge branch '10.5' into '10.6'

There are several open bugs about the fact that InnoDB is holding dict_sys.latch while waiting for something in the buffer pool. MDEV-33594 was encountered by the same customer earlier, and some more bugs are linked in MDEV-34988.

Created a patch for fixing the issue along with some diagnostics log. Pull Request

I am confident that debarun’s fix is going to improve things in this area, but it would be even better if it could be validated. We were never able to reproduce the original hang so far. I assume that it could require a buffer pool size that is small compared to the working set, maybe also a virtual machine with few vCPUs and lots of concurrent connections.

Here is some suggestion for attempting to repeat the issue.

1. Large enough buffer pool so that it takes 15 minutes+ time to flush 80% of the buffer pool.

innodb_max_io_capacity = 4000, innodb_page_size=16k, innodb_buffer_pool_size= 72G

2. Avoid page flushing based on dirty page.

innodb_max_dirty_pages_pct= 99.9

3. Have some write-load and set innodb_redo_log_size such that buffer pool dirty page percentage is very high ( at least > 80%) when checkpoint age is high and close to the of the total redo log margin.

4. Some concurrent read-load might help pushing a dirty page to LRU tail before it is actually flushed.

I have managed to get 10.6.18 to hang with a hammerdb workload, it hasn't crashed but it is stuck at 0 TPM. Just before it dropped to 0 it gave the message "2025-03-25 15:38:36 0 [Warning] InnoDB: Could not free any blocks in the buffer pool! 4737408 blocks are in use and 0 free. Consider increasing innodb_buffer_pool_size". This was after 1 hr of running a workload.
To do I this increased the buffer pool size to 72GB and I decreased the redo log size to 512MB, I also doubled the number of threads to 200 and used the "use all warehouses option" and used the parameters above. Any query to any of the tables then hangs e.g. "select * from warehouse".
Top functions in this state are:
25.16% mariadbd [.] buf_flush_list_holding_mutex(unsigned long, unsigned long)
3.50% mariadbd [.] buf_flush_page_cleaner() [clone .cold]
3.09% mariadbd [.] fil_flush_file_spaces() [clone .part.0]
1.57% mariadbd [.] buf_flush_list(unsigned long, unsigned long)
The database would not shut down and needed the process to be killed, but recovered on startup.
Will attempt to repeat to ensure it is reproducible and then try with the pull request.

I can confirm that the behaviour is reproducible, after running the workload for up to an hour it prints the message "Could not free any blocks in the buffer pool! 4737408 blocks are in use and 0 free. Consider increasing innodb_buffer_pool_size" and the transaction rate drops to 0, the database hangs and will not shutdown with the same functions active.

Hi Steve Shaw

That is great news!. Can you please capture the following information for me to analyze and check if this is same issue ?

1. Server error log
2. SHOW ENGINE INNODB STATUS;
3. Stack trace for all the threads by attaching gdb

Steve Shaw can you please run 10.6.21 with and without the fix and verify that the fix solves this issue?

Here are some possible work around that would make this bug less likely to happen for 10.6.18+ versions. The tuning could adversely affect performance.

1. Increase redo log size. The actual value needs to be derived by observing the check point age calculated as
[Innodb_lsn_current - Innodb_lsn_last_checkpoint] in bytes. It is advised to keep the checkpoint age < 70% of the total redo size.

MariaDB [(none)]> show status like "%lsn%";

+----------------------------+---------+

| Variable_name              | Value   |

+----------------------------+---------+

| Innodb_lsn_current         | 3913386 |

| Innodb_lsn_flushed         | 3913386 |

| Innodb_lsn_last_checkpoint | 3913370 |

+----------------------------+---------+

2. Increase innodb_io_capacity (more than 1/2 of innodb_max_io_capacity) to allow more aggressive flushing.
https://mariadb.com/kb/en/innodb-system-variables/#innodb_io_capacity

3. Reduce the value of innodb_max_dirty_pages_pct to allow some early flushing from the buffer pool.
https://mariadb.com/kb/en/innodb-system-variables/#innodb_max_dirty_pages_pct

4. Increase LRU scan depth to allow for more free page availability, It could affect overall performance.
https://mariadb.com/kb/en/innodb-system-variables/#innodb_lru_scan_depth

Yes I will, I will run on the unmodified 10.6.21 first to try and reproduce on this version.
For the earlier questions the error log only showed
"Could not free any blocks in the buffer pool! 4737408 blocks are in use and 0 free. Consider increasing innodb_buffer_pool_size"
before hanging and no other messages.
I ran SHOW ENGINE INNODB STATUS, however did not save the output, I will do this for 10.6.21.

Thanks Steve Shaw. Could you please also try to capture the stack trace when it hangs for 10.6.21 run ? It would be of great help in analysis.

debarun the same test has run against MariaDB 10.6.21-MariaDB source revision 066e8d6aeabc13242193780341e0f845528105de and the issue has not been encountered after running for 3 hours, 3X longer than when it is encountered on 10.6.18. There is also no messages on "Could not free any blocks". It does not appear to be encountered on 10.6.21 (but does not guarantee it).
Can the pull request be applied to 10.6.18 to see if it fixes the known affected version?

Steve Shaw Possibly some other change have made the behaviour different in 10.6.21 but as you mentioned it doesn't guarantee that the issue is not there. Yes, I think if the issue is repeatable in 10.6.18, it makes good sense to run it with the pull request changes. btw, the stack trace (and other details) information would be still useful for me even without the patch in 10.6.18 (in case it doesn't repeat with the pull-request patch).

I also ran against 10.6.19 and did not see the issue or the message. I re-ran again vs 10.6.18 and this gave the message "2025-03-27 13:24:22 0 [Warning] InnoDB: Could not free any blocks in the buffer pool! 4737408 blocks are in use and 0 free. Consider increasing innodb_buffer_pool_size." yet did not hang in the subsequent hour as previously, yet stopping and restarting the test with the sessions reconnecting produced a hang immediately. So, so far "Could not free any blocks in the buffer pool!" and the hang state has only been reproduced every time after receiving the message on 10.6.18 and no other version.

MDEV-34265 must be included in ES 10.6.19-15, as it is fixed in CS 10.6.19.
Thanks

pandi.gurusamy right, customer encountered the current bug on 10.6.19 and that stack trace also had confirmed earlier that it is not related to MDEV-34265. My analysis in this bug is mainly based on customer stack trace and this bug is expected to fix a symptom that should result in the exact symptom customer is facing. However, we could not repeat this issue in black-box testing and Steve Shawis kindly helping with it.

Steve Shaw I see. In 10.6.18, you could be hitting MDEV-34265. For current customer case (CS-211883), however, we are sure that he is hitting a different issue which could be explained by the current bug MDEV-36226.

To know for sure whether you are hitting MDEV-34265, I would need to check the complete stack trace in your case when it hangs. Could you please attach gdb to the hanging mariadbd and get the stack trace for all threads ?
Also, I recommend to move to 10.6.19 for future testing of this bug to ensure that we are not blocked by MDEV-34265.

axel reported in MDEV-35155 observing a performance improvement. This is most likely thanks to debarun’s fix.