When using simulated AIO, the IO (pread/pwrite), and the completion function are executed
in the same io callback. This might lead to deadlock, when completion function executes other AIOs - an example is the completion of the large doublewrite, which then issues multiple page writes.
The reason for the deadlock is that current implementation does not release the io control block aiocb soon enough, but only after the user callback, which, if we're unlucky, might itself require a free aiocb. This does not happen in native AIO implementations where control blocks are released as soon as OS indicates that write is finished.
MDEV-24313 workarounds the problem by having at least 2 IO write threads, but the solution is hacky, and only relies on the fact that at any point of time there will be only one IO callback that submits more AIOs, which is still true in 10.5 .
The correct solution, which is more in line with native AIO, is to split pread/pwrite from completion code, i.e submit task to threadpool after pwrite completion, instead of executing it right after pwrite.