[MDEV-26569] 10.6 mariadbd error: io_uring_queue_init() failed with ENOSYS, and then asserts Created: 2021-09-08  Updated: 2022-10-22  Resolved: 2022-07-29

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Otto Kekäläinen Assignee: Daniel Black
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-24883 add io_uring support for tpool Closed

 Description   

While preparing 10.6.4 for official Debian upload I noticed that the builds fail when mariadb-test-run tries to start mariadbd on Launchpad with the following error on multiple platforms:

Logging: /<<PKGBUILDDIR>>/mysql-test/mariadb-test-run.pl  --force --testcase-timeout=120 --suite-timeout=540 --retry=3 --parallel=4 --skip-rpl --suite=main --skip-test-list=unstable-tests
 
vardir: /<<PKGBUILDDIR>>/builddir/mysql-test/var
 
Removing old var directory...
 
Creating var directory '/<<PKGBUILDDIR>>/builddir/mysql-test/var'...
 
Checking supported features...
 
MariaDB Version 10.6.4-MariaDB-1~ubuntu21.10.1~1631027360.b24724e0e83.debia
 
 - SSL connections supported
 
Using suites: main
 
Collecting tests...
 
Installing system database...
 
mysql-test-run: *** ERROR: Error executing mariadbd --bootstrap
 
Could not install system database from /<<PKGBUILDDIR>>/builddir/mysql-test/var/log/bootstrap.sql
 
The /<<PKGBUILDDIR>>/builddir/mysql-test/var/log/bootstrap.log file contains:
 
/<<PKGBUILDDIR>>/builddir/sql/mariadbd --no-defaults --disable-getopt-prefix-matching --bootstrap --basedir=/<<PKGBUILDDIR>> --datadir=/<<PKGBUILDDIR>>/builddir/mysql-test/var/install.db --plugin-dir=/<<PKGBUILDDIR>>/builddir/mysql-test/var/plugins --default-storage-engine=myisam --loose-skip-plugin-feedback --loose-skip-plugin-innodb --loose-skip-plugin-innodb-buffer-page-lru --loose-skip-plugin-innodb-buffer-pool-stats --loose-skip-plugin-innodb-cmp --loose-skip-plugin-innodb-cmp-per-index --loose-skip-plugin-innodb-cmp-per-index-reset --loose-skip-plugin-innodb-cmp-reset --loose-skip-plugin-innodb-cmpmem --loose-skip-plugin-innodb-cmpmem-reset --loose-skip-plugin-innodb-ft-being-deleted --loose-skip-plugin-innodb-ft-config --loose-skip-plugin-innodb-ft-default-stopword --loose-skip-plugin-innodb-ft-deleted --loose-skip-plugin-innodb-ft-index-cache --loose-skip-plugin-innodb-ft-index-table --loose-skip-plugin-innodb-sys-columns --loose-skip-plugin-innodb-sys-fields --loose-skip-plugin-innodb-sys-foreign --loose-skip-plugin-innodb-sys-foreign-cols --loose-skip-plugin-innodb-sys-indexes --loose-skip-plugin-innodb-sys-tables --loose-skip-plugin-innodb-sys-tablespaces --loose-skip-plugin-innodb-sys-tablestats --loose-skip-plugin-innodb-sys-virtual --loose-skip-plugin-innodb-tablespaces-encryption --loose-skip-plugin-partition --loose-skip-plugin-sequence --loose-skip-plugin-thread-pool-groups --loose-skip-plugin-thread-pool-queues --loose-skip-plugin-thread-pool-stats --loose-skip-plugin-thread-pool-waits --loose-skip-plugin-unix-socket --loose-skip-plugin-user-variables --loose-innodb --loose-innodb-log-file-size=10M --disable-sync-frm --tmpdir=/<<PKGBUILDDIR>>/builddir/mysql-test/var/tmp/ --core-file --console --character-set-server=latin1 --lc-messages-dir=/<<PKGBUILDDIR>>/builddir/sql/share/ --character-sets-dir=/<<PKGBUILDDIR>>/sql/share/charsets
 
2021-09-07 17:12:28 0 [Note] /<<PKGBUILDDIR>>/builddir/sql/mariadbd (server 10.6.4-MariaDB-1~ubuntu21.10.1~1631027360.b24724e0e83.debia) starting as process 16598 ...
 
2021-09-07 17:12:28 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32186)
 
2021-09-07 17:12:28 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
 
2021-09-07 17:12:29 0 [Note] Plugin 'partition' is disabled.
 
2021-09-07 17:12:29 0 [Note] Plugin 'SEQUENCE' is disabled.
 
2021-09-07 17:12:29 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
 
2021-09-07 17:12:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2021-09-07 17:12:29 0 [Note] InnoDB: Number of pools: 1
 
2021-09-07 17:12:29 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
2021-09-07 17:12:29 0 [Note] mariadbd: O_TMPFILE is not supported on /<<PKGBUILDDIR>>/builddir/mysql-test/var/tmp/ (disabling future attempts)
 
2021-09-07 17:12:29 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOSYS: try uprading the kernel
 
210907 17:12:29 [ERROR] mysqld got signal 6 ;

Full log:

The only platform that worked was armhf.

 Comments   
Comment by Marko Mäkelä [ 2021-09-08 ]

I checked one of the linked logs. It starts with the following:

https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+build/22048215
 
RUN: /usr/share/launchpad-buildd/bin/builder-prep 
Kernel version: Linux bos02-s390x-002 4.15.0-154-generic #161-Ubuntu SMP Fri Jul 30 12:59:37 UTC 2021 s390x

I found a claim that io_uring first appeared in the Linux 5.1 kernel. I would not expect it to be backported to older kernels, such as the 4.15 based kernel here. One of the first mainstream distributions whose kernel supports io_uring (Ubuntu 20.04) is claimed to have shipped with a 5.4 kernel. As far as I understood, it did not include liburing. Here you seem to have a ‘too new’ userspace running on an old kernel.

If it is not possible to upgrade the affected systems to a newer kernel, you have two options:

  • cmake -DCMAKE_DISABLE_FIND_PACKAGE_URING=1 to remove the dependency at build time
  • innodb_use_native_aio=OFF to disable the interface at runtime; it might be a little challenging to pass this to the bootstrap
Comment by Otto Kekäläinen [ 2021-09-08 ]

Thanks for a quick reply. I guess the Launchpad builders run Ubuntu Bionic and buildd in containers, thus inheriting the kernel 4.15: https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=linux-image-generic

I asked for access to Ubuntu 20.04 Launchpad build hosts in https://answers.launchpad.net/launchpad/+question/698668

Comment by Tuukka Pasanen [ 2021-09-12 ]

This solved or need more work to stop requiring uring?

Comment by Otto Kekäläinen [ 2021-09-12 ]

There is nothing to be fixed in MariaDB for this – it is correct for MariaDB builds to assume that on new Debian/Ubuntu releases a kernel 5.4+ would be available.

Tracking Launchpad.net builder host updates in https://bugs.launchpad.net/launchpad/+bug/1943292

Comment by Tuukka Pasanen [ 2021-11-04 ]

I'm closing this as there is nothing to-do

Comment by Daniel Black [ 2022-02-09 ]

I'm going to open to fix the assertion aspect of this, and perhaps fall back to innodb_use_native_aio=0.

I plan on using 22.04 as the basis for 10.8+ container images once Ubuntu-22.04 released. Those are going to operate on older kernels, potentially even ones that don't support io_uring.

An assertion looks really sloppy, and we could just fall back to innodb_use_native_aio=0 like we did MDEV-26674 (though I'd want to rely on ENOSYS rather than explicit kernel versions again).

Comment by Daniel Black [ 2022-02-10 ]

MariaDB does appear to be forcing AIO itself off with a simulated ENOSYS.

$ mariadblocal --innodb_use_native_aio=ON
 
2022-02-10 20:58:23 0 [Note] sql/mysqld (server 10.6.7-MariaDB) starting as process 251713 ...
 
2022-02-10 20:58:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2022-02-10 20:58:23 0 [Note] InnoDB: Number of pools: 1
 
2022-02-10 20:58:23 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
2022-02-10 20:58:23 0 [Warning] mysqld: io_uring_queue_init() failed with ENOSYS: try uprading the kernel
 
2022-02-10 20:58:23 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF

I don't know how the 2021-12-15 builds are asserting https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all

Comment by Daniel Black [ 2022-02-15 ]

Waiting to see if 10.6.7 suffers same problem on https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all

Comment by Otto Kekäläinen [ 2022-02-17 ]

I uploaded 10.6.7 to https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all and unfortunately it still fails to build.

However on a closer look the error is related to OpenSSL 3.0 entering repos:
noformat
– Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "3.0.1")
CMake Error at cmake/ssl.cmake:157 (MESSAGE):
Cannot find appropriate system libraries for SSL. Use WITH_SSL=bundled to
enable SSL support
Call Stack (most recent call first):
CMakeLists.txt:382 (MYSQL_CHECK_SSL)
noformat

This was visible both in latest 10.6.7 and for the 10.6.6 build https://launchpadlibrarian.net/585123790/buildlog_ubuntu-jammy-amd64.mariadb-10.6_1%3A10.6.6-1~ubuntu22.04.1~1644378769.2423f1afc81.debian.latest_BUILDING.txt.gz

Comment by Otto Kekäläinen [ 2022-02-24 ]

Launchpad builders are now fixed in https://bugs.launchpad.net/launchpad-buildd/+bug/1943292 and builds at https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.6/+builds?build_text=&build_state=all work again

Generated at Thu Feb 08 09:46:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.