If I make dict_check_tablespaces_and_store_max_id() open all files when encryption_key_id_exists(FIL_DEFAULT_ENCRYPTION_KEY) (or !ibuf.empty), then all encryption tests except encryption.innodb-remove-encryption will pass. That still leaves some non-encryption test failures. The trickiest is the following:

CURRENT_TEST: sys_vars.innodb_buffer_pool_dump_abort_loads

mysqltest: At line 143: Can't initialize replace from 'replace_result $fulldump Previously_dumped'

This would be fixed by making buf_load() invoke another SYS_TABLES scan and open all data files for those missing tablespaces that are mentioned in the file identified by innodb_buffer_pool_filename.

Before I implemented the tweak to buf_load(), I had tweaked the following tests:

encryption.innodb-remove-encryption

innodb.row_format_redundant

innodb.table_flags

innodb_zip.restart

It turns out that my changes to these tests can be omitted, because the ib_buffer_pool file would ensure that the tablespaces of interest be loaded.

If the ib_buffer_pool file were removed, these tests would fail. To make these failures deterministic, I would have created all tables with STATS_PERSISTENT=0 and run a slow shutdown (innodb_fast_shutdown=0) so that no background tasks would cause the tables to be loaded and data files to be opened.

I can imagine that opening the files can be slow also in other environments than Microsoft Windows, such as when the data directory resides on network attached storage or NFS.

origin/10.6-MDEV-32027 bddd1da9e939835dbbf61ea3c90638789db86733 2023-11-13T16:27:37+02:00
behaved well in RQG testing.

Even on Windows it is not slow. I could not get it as slow, as the original reporter, but 85 000 files add 6-15 seconds startup (Windows seems to caches some metadata, and it is faster second time around). However, delay scales linearly with number of files, so startup with 850000 files can take 2.5 minutes on my machine. The reporter has something that is factors of magnitude slower, and for him it is really much too much. We do not know what kind of storage reporter had, but it would be interesting to know . kennyliao, can you comment on that?