While testing MDEV-23855, it became clear that the fil_system.mutex is a contention point, causing the page cleaner thread to spend 5% of its time on spinning on a mutex. The mutex would be acquired and released several times per page flush operation, mostly by the thread that initiates the write, but also by the I/O completion callback.

In fil_space_t, we should use a single atomic counter with 3 flags, instead of a mix of flags and least 3 counters, some of which are protected by fil_system.mutex. That should reduce the use of fil_system.mutex to the rare cases where files need to be created, opened, or closed.

On Microsoft Windows, a concurrent synchronous write and FlushFileBuffers() on the same file handle will kill performance. Some special tricks will be needed on that platform. This is especially prominent with the synchronous writes for the doublewrite buffer in buf_dblwr_t::flush_buffered_writes().