danblack should this be deployed to all builder?
What would a good value be?
I found https://blog.pythian.com/importance-oracle-database-related-kernel-parameters-aio-max-nr-bonus-track/ but I am not sure that I understand all the implication of such a change.

FYI:

faust@hz-bbw1 ~ main

❯ docker run -it quay.io/mariadb-foundation/bb-worker:debian10 bash -c "sysctl -n fs.aio-max-nr"

65536

Also, why this default value is not sufficient anymore?

Well this number is global for the machine and not per container/exe instance.

> should this be deployed to all builder?

Seems resonable. There's higher requirements depending on how many mtr tests are concurrent.

> What would a good value be?

Double the current would be my starting point. I haven't seen a downside in raising this parameter. If the kernel can handle the number of jobs it can use a bit more kernel memory to keep track of AIO requests.

Monitor fs.aio-nr (https://docs.kernel.org/admin-guide/sysctl/fs.html#aio-nr-aio-max-nr) as the number being used.

Ok thanks danblack!
vladbogo before I make the change, please can you also take a look?

My understanding is that we are hitting this because our BB load balancing is now working better (more parallel build/test on strong machines).

MR ready for review: https://gitlab.com/mariadb/sysadmin/-/merge_requests/35

The change has been deployed on all BB docker workers.

Summary of discussion with marko, danblack and vladbogo

Marko:
In 10.5+ the code would look like this:

  int max_read_events= int(n_reader_threads * OS_AIO_N_PENDING_IOS_PER_THREAD);

  int max_write_events= int(n_writer_threads * OS_AIO_N_PENDING_IOS_PER_THREAD);

  int events= max_read_events + max_write_events;

  /** Do the Linux AIO dance (this will try to create a new

  io context with changed max_events ,etc*/

  int ret= srv_thread_pool->reconfigure_aio(srv_use_native_aio, events);

That would be (innodb_read_io_threads+innodb_write_io_threads)×256, or (4+4)×256=2048 per server instance by default.
Each mtr can start at least up to 2×--parallel servers (if we consider replication tests), maybe 3× or 4× if we consider galera_3nodes.
So, if we can have 2 ./mtr --parallel=60 running concurrently on the same kernel, the worst case should be 2×4×60×2048 = 60×16384 = 983040.

Vlad:
Max number of builders for each machine is defined in https://github.com/MariaDB/buildbot/blob/dev/locks.py#L14-L44
but each builder might use a different number for --parallel
but the container still starts, but waits for others to finish before actually doing something

--parallel is usualy 14 (but MSAN or Valgrind have much more).

Faustin:
Value given by n×4×p×2048 where n is the number of concurrent builds and p the value of --parallel.
Suggestion is the following then --parallel 30, n= see https://github.com/MariaDB/buildbot/blob/dev/locks.py.
Keep the actual value for small builders and only modify strong builders (n>=5).

Deployed with https://gitlab.com/mariadb/sysadmin/-/commit/0fb9deffe.