Faustin Lammler added a comment - 2022-03-22 15:24 What would be needed to enable it (just install liburing?)

Daniel Black added a comment - 2022-03-22 22:53 Nope, seccomp filter to allow the syscalls ( https://github.com/moby/moby/issues/39396 ) And maybe raising the MEMLOCK Limit. This will be indicated by an innodb error message in the logs if there's a newer kernel liburing version (2+, not 0.7 anyway). This is only needed on kernels < 5.12 ( https://github.com/axboe/liburing/issues/246 ). I think 1M limit if the default is lower is sufficient.

Faustin Lammler added a comment - 2022-03-23 09:08 - edited Ok, I see. I am wondering if we are not hitting the limits of using containers in BB and if this is not a job for libvirt runners ( vladbogo can I have you opinion on this?). Anyway, from what I understand of https://github.com/moby/moby/issues/39396 , we could enable uring directly in the docker daemon by modifying https://gitlab.com/mariadb/sysadmin/-/blob/master/ansible/roles/bb_worker_docker/templates/docker_override.conf.j2 danblack do you feel to propose something directly there? I am really not sure about how the default seccomp filter should be modified. Then I am happy to deploy/test on some runners. Also I can make sure of deploying this only on <5.12 kernels (should be doable with jinja2 and ansible facter).

Vlad Bogolin added a comment - 2022-03-23 09:31 Hi. Does this happen on other hosts? If yes, then it might be indeed a limitation. Otherwise, I would vote to apply the solution from the mentioned issue only for this host but maybe allowing all syscalls (as proposed in the issue) is too extreme

Daniel Black added a comment - 2022-03-24 00:19 From https://github.com/moby/moby/commit/f4d41f1dfa52caa8f12b070315e230e7eded5f4a it looks like docker has these enabled by default. Are we actually running a version that is covered by the tags at the top of that commit?

Faustin Lammler added a comment - 2022-03-24 09:54 I have made an upgrade round on our runners: faust@serv ~/MariaDB/ansible main .venv ❯ ansible bb_workers_docker -a "docker --version" bg-bbw1-x64 | CHANGED | rc=0 >> Docker version 20.10.12, build e91ed57 bg-bbw3-x64 | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 bg-bbw4-x64 | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 bg-bbw2-x64 | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 ci-bbw1-arm | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 db-p9-bbw1 | CHANGED | rc=0 >> Docker version v20.10.13, build a224086 ci-bbw4-arm | CHANGED | rc=0 >> Docker version 20.10.12, build e91ed57 ci-bbw3-arm | CHANGED | rc=0 >> Docker version 20.10.12, build e91ed57 ci-bbw2-arm | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 us-intel-bbw1-x64 | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 fi-bbw-p9rhel7 | CHANGED | rc=0 >> Docker version 18.03.1-ce, build ccde200 fi-bbw-p9rhel8 | CHANGED | rc=0 >> Docker version v20.10.12, build e91ed57 hz-bbw2 | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 hz-bbw1 | CHANGED | rc=0 >> Docker version 20.10.14, build a224086 ibm-s390x-sles15 | CHANGED | rc=0 >> Docker version 20.10.9-ce, build 79ea9d308018 ibm-s390x-ubuntu20.04 | CHANGED | rc=0 >> Docker version 20.10.12, build e91ed57 ibm-s390x-rhel8 | CHANGED | rc=0 >> Docker version 20.10.12, build e91ed57 If you danblack can confirm that for those runners uring is enabled, it would be good because maybe something else is needed

Daniel Black added a comment - 2022-03-24 23:54 bg-bbw2-docker-fedora-35 https://ci.mariadb.org/23511/logs/aarch64-fedora-35/mysqld.1.err.1 2022-03-23 11:17:57 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd 2022-03-23 11:17:57 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF https://docs.docker.com/engine/reference/commandline/dockerd/#default-ulimit-settings (json file example on this page)

Faustin Lammler added a comment - 2022-04-11 15:52 danblack , vladbogo do we have a way to query the BB DB or API (or something else) in order to quickly and programmatically spot this kind of problems?

Daniel Black added a comment - 2022-04-11 22:41 Not really. It was quite painful to go through and find this one warning (I'm not sure I found any successes) Best I can think of at the moment is to take a test from https://github.com/axboe/liburing/tree/master/test , suggestion read-write.c, compile against the worker liburing (so don't pull the latest feature tests) into the bb workers, and run it on some pre-step in worker jobs. Like MDBF-386 it can be a worker information finding stage. I don't think properties are a cross reference searchable item but it should be possible to extend it that way.

Faustin Lammler added a comment - 2022-05-23 16:15 FYI (and as already discussed) liburing-devel is not available in rhel9, see also: https://github.com/MariaDB/mariadb.org-tools/pull/160

Daniel Black added a comment - 2022-09-23 06:52 Please update the memlock limits. Spent a long time working out why MDEV-29610 isn't affecting bb.org builders. And we're delivering significant functionality (10.6+ liburing), that is complicated, and untested, to delivering this packaged to users. Just 1M of locked memory as the default memlock limit per container. This is needed in all hosts running a kernel < 5.12.

Faustin Lammler added a comment - 2022-09-23 13:34 Assigned to vladbogo since it's better to be managed at BB level. Re-assign to me if we want to manage this at docker daemon level.

Daniel Black added a comment - 2022-09-23 22:05 Thanks vladbogo , already showing useful results in MDEV-29610 .