Both crashes in the install test are the same failure. io_uring_get_sqe() returns NULL and MariaDB writes through it. Running the server with a liburing library older than 2.3 reproduces both crashes exactly, and the production VM that crashed has liburing2 2.1.
From liburing 2.3 on, the inline helper io_uring_get_sqe() reads two queue-size fields that libraries older than 2.3 leave at zero. When the server runs with such an old library, every call reports the submission queue as full and returns NULL. MariaDB's io_uring code doesn't check for NULL and writes the request into address 0. The install test hit this twice: first in mariadb-install-db, then in InnoDB's shutdown after the first crash had left ibdata1 empty.
The MariaDB-server RPM only asks for the LIBURING_2.1 symbol version, so zypper accepts an old liburing2 and never upgrades it. SUSE shipped liburing2 2.1 for SLES 15 until May 2025, and the production VM image still has liburing2-2.1-150400.2.4.
liburing2-2.1-150400.2.4; the dev VM has 2.9| VM | Role | Kernel | liburing2 | Reading |
|---|---|---|---|---|
| bb-hz-bbw5-sles-1507-amd64 | Production. Ran build 76750 and crashed | 6.4.0-150700.51 (2025-04-30) | 2.1-150400.2.4 | older than 2.3 the version that reproduces both crashes |
| bb-ro-apexis-bbw03-x64-sles-1507-amd64 | Dev. Checked on 2026-10-29 | 6.4.0-150700.53.73 | 2.9-150700.10.3.1 | 2.3 or newer same package as the container, where nothing crashed |
The production VM that crashed has liburing2-2.1-150400.2.4, a package from the SLE 15 SP4 code stream (release 150400). That is the old-library case reproduced in the container. Its image was never updated to the 2.9 package that the dev VM has.
Each libvirt host keeps its own base image (<VM name>.qcow2), and dev and production use different hosts, which is how the two drifted apart. The dev VM also runs a later maintenance kernel, while bbw5 still has the April 2025 release kernel. Builds on the dev VM should not hit this crash.
mariadb-install-db --rpm --user=mysql. The plugin packages (rocksdb, duckdb, connect, oqgraph, videx…) and galera were installed after it, so none of them was present yet.Thread pointer: 0x0). It is the first asynchronous page write that fails, so page 0 of ibdata1 is never written. The Aria system tables, which don't go through this code, were already on disk.systemctl start mariadb finds a header page of zeros: InnoDB: Header page consists of zero bytes in datafile: ./ibdata1, then Plugin initialization aborted with error Data structure corruption. This error is a result of crash 1; on its own it only stops InnoDB from starting.InnoDB: Starting shutdown... The shutdown tears down the io_uring queue and hits the same NULL. With a working liburing, the same corrupt datadir makes the server exit cleanly with rc=1.struct io_uringliburing commit f5cd4eb2 (August 2022, released in 2.3) added cached copies of the queue mask and size to the public struct, in bytes that used to be padding. The inline helpers in the 2.3+ headers read those copies. Older libraries keep the library version liburing.so.2 and clear the padding in io_uring_queue_mmap(), so the cached size reads as 0.
| Offset | liburing ≤ 2.2 (e.g. SUSE 2.1) | liburing ≥ 2.3 (MariaDB built against 2.9) |
|---|---|---|
| 0x38 | sqes | sqes |
| 0x44 | sqe_tail | sqe_tail |
| 0x48 | ring_sz | ring_sz |
| 0x50 | ring_ptr | ring_ptr |
| 0x58 | pad[0] = 0 | ring_mask |
| 0x5c | pad[1] = 0 | ring_entries |
| 0x60 | pad[2], pad[3] | pad[0], pad[1] |
liburing 2.9 src/include/liburing.h, compiled into mariadbd
IOURINGINLINE struct io_uring_sqe *_io_uring_get_sqe(struct io_uring *ring) { ... head = *sq->khead; /* ring_entries is 0 with a pre-2.3 library */ if (next - head <= sq->ring_entries) { sqe = &sq->sqes[(sq->sqe_tail & sq->ring_mask) << shift]; ... return sqe; } return NULL; }
MariaDB tpool/aio_liburing.cc at e9ff0bf7
// line 99, ~aio_uring() -> crash 2 io_uring_sqe *sqe= io_uring_get_sqe(&uring_); io_uring_prep_nop(sqe); // line 125, submit_io() -> crash 1 io_uring_sqe *sqe= io_uring_get_sqe(&uring_); if (cb->m_opcode == aio_opcode::AIO_PREAD) io_uring_prep_readv(sqe, ...); else io_uring_prep_writev(sqe, ...);
Disassembly of the stripped 12.3.4 mariadbd (build ID 5e20b928…). uring_ sits at offset 8 inside aio_uring, so +0x60/+0x64 are the struct's 0x58/0x5c.
submit_io(), write request: crash 1 address +0x119a5e0 119a589: cmp 0x64(%r14),%r9d ; next - head vs ring_entries (0) 119a58d: ja 119a640 ; queue reported full 119a640: xor %eax,%eax ; sqe = NULL 119a642: jmp 119a5d0 ; prepare the request anyway 119a5e0: movb $0x2,(%rax) ; sqe->opcode = IORING_OP_WRITEV at address 0 ~aio_uring(): crash 2 address +0x119aa90 (compiler-isolated NULL path) 119aa90: movb $0x0,0x0 ; sqe->opcode = IORING_OP_NOP at address 0 119aa98: ud2
The RPM dependencies generated for MariaDB-server only record the newest liburing symbol version the binary calls:
rpm -q --requires MariaDB-server | grep uring
liburing.so.2()(64bit)
liburing.so.2(LIBURING_2.0)(64bit)
liburing.so.2(LIBURING_2.1)(64bit)
rpm -q --changelog liburing2 (SLE 15 SP7, 2.9-150700.10.3.1)
* Tue May 13 2025 gabriel.bertazi@suse.com
- Update to liburing 2.9 (PED-12844)
...
* Wed Oct 13 2021 ddiss@suse.com
- update to 2.1 (bsc#1193522)
A system with liburing2 2.1 satisfies every requirement, so the install succeeds and the server crashes on its first asynchronous write. Nothing in liburing's library version or symbol versions marks the 2.3 change.
Base image registry.suse.com/bci/bci-base:15.7, MariaDB-server from build 76750, --security-opt seccomp=unconfined so InnoDB can use io_uring, host kernel 7.0.
| Setup | liburing | InnoDB I/O | Result |
|---|---|---|---|
| Datadir on overlayfs | 2.9 | io_uring, buffered log | 21 / 21 ok bootstrap runs |
| Datadir on an ext4 volume | 2.9 | io_uring, O_DIRECT log, same as the VM | 31 / 31 ok bootstrap runs |
ext4, page 0 of ibdata1 zeroed | 2.9 | same | clean abort same InnoDB errors as the VM, rc=1, no crash |
| ext4, fresh datadir | 2.1 | same | SIGSEGV bootstrap at +0x119a5e0, ibdata1 header left as zeros |
| ext4, datadir left by the previous row | 2.1 | same | SIGSEGV server start at +0x119aa90 after "Data structure corruption", rc=139 |
| Frame | VM, crash 1 | Container, liburing 2.1 | VM, crash 2 | Container, liburing 2.1 |
|---|---|---|---|---|
| crash site | +0x119a5e0 | +0x119a5e0 | +0x119aa90 | +0x119aa90 |
| caller | +0xfdb8b2 | +0xfdb8b2 | +0xfdb004 | +0xfdb004 |
| +0x112da43 | +0x112da43 | +0x1067ac6 | +0x1067ac6 | |
| +0x10f4268 | +0x10f4268 | +0xf71eb5 | +0xf71eb5 | |
| +0x10f5f3c | +0x10f5f3c | ha_initialize_handlerton+0x87 | ha_initialize_handlerton+0x87 | |
| +0x7c9c5e | +0x7c9c5e | plugin_init+0x9da | plugin_init+0x9da | |
| thread entry | +0x10f6b58 | +0x10f6b58 | mysqld_main+0x4a2 | mysqld_main+0x4a2 |
Minimal reproducer, inside a SLES 15.7 container with MariaDB-server from build 76750
zypper -n in gcc make curl -sL https://github.com/axboe/liburing/archive/refs/tags/liburing-2.1.tar.gz | tar xz cd liburing-liburing-2.1 && ./configure && make -C src ln -s liburing.so.2.1.0 src/liburing.so.2 LD_LIBRARY_PATH=$PWD/src mariadb-install-db --rpm --user=mysql # crash 1 LD_LIBRARY_PATH=$PWD/src /usr/sbin/mariadbd --user=mysql # crash 2
The test itself copied the liburing-2.1 source from a local clone of the upstream repository instead of downloading the tarball; the build steps are the same.
| Suspect | Why it isn't the cause |
|---|---|
| SELinux | The log collection reports "SELinux config not present". SLES 15 uses AppArmor, and a denial returns a permission error, not SIGSEGV. |
| Other MariaDB packages | Crash 1 happened before any plugin package was installed. Crash 2 reproduces with MariaDB-server alone. |
| Kernel (VM 6.4.0-150700.51) | Not needed for the crash. Both crashes reproduce on a 7.0 host kernel with liburing 2.1 and don't happen there with 2.9. The VM that crashed has liburing 2.1. |
| Container seccomp / filesystem | io_uring was active in every run. The ext4 runs used the same O_DIRECT log path as the VM's xfs. |
| "Data structure corruption" | A result of crash 1 leaving ibdata1 empty. With a working liburing it ends in a clean abort. |
liburing2 to 2.1 for SLE 15. SUSE changelog, bsc#1193522.so.2. In the thread, the Debian maintainer proposes raising the minimum liburing version in distro packaging instead. axboe/liburingio_uring_get_sqe() "breaks seamless updates of liburing.so". axboe/liburingliburing2 to 2.9 (PED-12844). Systems that never pulled this update still have 2.1. SUSE changelogbb-hz-bbw5-sles-1507-amd64, whose kernel is the April 2025 SP7 build. buildbot logrpm -q liburing2 on bb-ro-apexis-bbw03-x64-sles-1507-amd64 returns 2.9-150700.10.3.1. Its kernel is 6.4.0-150700.53.73, newer than bbw5's 6.4.0-150700.51, so the two hosts' images differ. worker consolerpm -q liburing2 on bb-hz-bbw5-sles-1507-amd64, the VM that crashed, returns 2.1-150400.2.4. worker consolebbw5 has liburing2-2.1-150400.2.4. The dev VM has 2.9.
Owner: CI
Update liburing2 in the bbw5 SLES 15.7 image, or refresh the whole image to match the dev VM. Its kernel is also still the April 2025 release.
Owner: CI
RPMs built against liburing 2.3+ headers should require liburing2 >= 2.3, so an old system fails at install time instead of crashing at runtime.
Owner: MariaDB packaging
submit_io() and ~aio_uring() in tpool/aio_liburing.cc should handle a NULL from io_uring_get_sqe() instead of writing through it.
Owner: MariaDB server
Unrelated, found in the same log: scripts/rpm-install.sh writes wsrep_provider=/usr/lib*/galera*/libgalera_smm.so literally into galera.cnf, because shells don't expand wildcards in a variable assignment. It's harmless while wsrep is off.