[MDEV-28751] mariadb-install-db fails writing ibdata1 on overlayfs - Jira

Details

Type: Bug
Status: Open (View Workflow)
Priority: Major
Resolution: Unresolved
Affects Version/s: 10.6.8
Fix Version/s: 10.6
Component/s: Storage Engine - InnoDB
Labels:
- crash
- foundation
- innodb
- not-10.5
Environment:
OS: Debian GNU/Linux 11 (bullseye)
Software platform: debian-live-11.3.0-amd64-standard.iso
Filesystem: overlay
QEMU: qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 2
Download: mariadb-10.6.8-linux-systemd-x86_64.tar.gz

Description

Following INSTALL-BINARY instructions,
scripts/mysql_install_db --user=mysql
failed:

$ sudo scripts/mysql_install_db --user=mysql

Installing MariaDB/MySQL system tables in './data' ...

2022-06-06  4:17:03 0 [Warning] InnoDB: Retry attempts for writing partial data failed.

2022-06-06  4:17:03 0 [ERROR] InnoDB: Write to file ./ibdata1 failed at offset 0, 1048576 bytes should have been written, only 0 were written. Operating system error number 22. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.

2022-06-06  4:17:03 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'

2022-06-06  4:17:03 0 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space

2022-06-06  4:17:03 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.

2022-06-06  4:17:04 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-06-06  4:17:04 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

2022-06-06  4:17:04 0 [ERROR] Unknown/unsupported storage engine: InnoDB

2022-06-06  4:17:04 0 [ERROR] Aborting

Installation of system tables failed!

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

strace.log.gz
2022-06-06 07:14
13 kB
Daniel Lewart

Issue Links

relates to

MDEV-27593 Crashing on I/O error is unhelpful

Closed

MDEV-28100 reiserfs O_DIRECT unsupported: Assertion `p == pages.end() || p->first > page_id' failed in recv_sys_t::apply

Stalled

Activity

Ascending order - Click to sort in descending order

View 5 older comments

Faustin Lammler added a comment - 2022-06-07 07:50

Thanks lewart!
danblack FYI the initial report is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009290.

Faustin Lammler added a comment - 2022-06-07 07:50 Thanks lewart ! danblack FYI the initial report is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009290 .

Daniel Lewart added a comment - 2022-06-11 09:27

MDEV-28100.c failed to write to an ext4 filesystem.
Am I doing something wrong?

$ df -hT /
Filesystem Type Size Used Avail Use% Mounted on
/dev/vda1 ext4 4.0G 1.9G 1.9G 51% /

$ uname -a
Linux debian 5.18.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.2-1 (2022-06-06) x86_64 GNU/Linux

$ gcc -Wall MDEV-28100.c

$ strace ./a.out foo
execve("./a.out", ["./a.out", "foo"], 0x7fffa9e8d228 /* 22 vars */) = 0
brk(NULL) = 0x55be57cb2000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14247, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 14247, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f20d0b91000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@9\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\200\0\300\4\0\0\0\1\0\0\0\0\0\0\0", 32, 848) = 32
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\263\353\274.\303\361o\256\340\2\364\377wD'#"..., 68, 880) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1900992, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f20d0b8f000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1934632, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f20d09b6000
mmap(0x7f20d09d8000, 1409024, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f20d09d8000
mmap(0x7f20d0b30000, 327680, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17a000) = 0x7f20d0b30000
mmap(0x7f20d0b80000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c9000) = 0x7f20d0b80000
mmap(0x7f20d0b86000, 34088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f20d0b86000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f20d09b4000
arch_prctl(ARCH_SET_FS, 0x7f20d0b90580) = 0
mprotect(0x7f20d0b80000, 16384, PROT_READ) = 0
mprotect(0x55be56c16000, 4096, PROT_READ) = 0
mprotect(0x7f20d0bc4000, 8192, PROT_READ) = 0
munmap(0x7f20d0b91000, 14247) = 0
openat(AT_FDCWD, "foo", O_RDWR|O_CREAT|O_TRUNC, 0740) = 3
close(3) = 0
openat(AT_FDCWD, "foo", O_RDWR|O_DIRECT|O_CLOEXEC) = 3
pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 0) = -1 EINVAL (Invalid argument)
close(3) = 0
exit_group(0) = ?
+++ exited with 0 +++

$ ls -l foo
~~rwxr~~---- 1 user user 0 Jun 11 04:19 foo

Daniel Lewart added a comment - 2022-06-11 09:27 MDEV-28100 .c failed to write to an ext4 filesystem. Am I doing something wrong? $ df -hT / Filesystem Type Size Used Avail Use% Mounted on /dev/vda1 ext4 4.0G 1.9G 1.9G 51% / $ uname -a Linux debian 5.18.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.2-1 (2022-06-06) x86_64 GNU/Linux $ gcc -Wall MDEV-28100 .c $ strace ./a.out foo execve("./a.out", ["./a.out", "foo"] , 0x7fffa9e8d228 /* 22 vars */) = 0 brk(NULL) = 0x55be57cb2000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14247, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 14247, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f20d0b91000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@9\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\200\0\300\4\0\0\0\1\0\0\0\0\0\0\0", 32, 848) = 32 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\263\353\274.\303\361o\256\340\2\364\377wD'#"..., 68, 880) = 68 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1900992, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f20d0b8f000 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 mmap(NULL, 1934632, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f20d09b6000 mmap(0x7f20d09d8000, 1409024, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f20d09d8000 mmap(0x7f20d0b30000, 327680, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17a000) = 0x7f20d0b30000 mmap(0x7f20d0b80000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c9000) = 0x7f20d0b80000 mmap(0x7f20d0b86000, 34088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f20d0b86000 close(3) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f20d09b4000 arch_prctl(ARCH_SET_FS, 0x7f20d0b90580) = 0 mprotect(0x7f20d0b80000, 16384, PROT_READ) = 0 mprotect(0x55be56c16000, 4096, PROT_READ) = 0 mprotect(0x7f20d0bc4000, 8192, PROT_READ) = 0 munmap(0x7f20d0b91000, 14247) = 0 openat(AT_FDCWD, "foo", O_RDWR|O_CREAT|O_TRUNC, 0740) = 3 close(3) = 0 openat(AT_FDCWD, "foo", O_RDWR|O_DIRECT|O_CLOEXEC) = 3 pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 0) = -1 EINVAL (Invalid argument) close(3) = 0 exit_group(0) = ? +++ exited with 0 +++ $ ls -l foo rwxr ---- 1 user user 0 Jun 11 04:19 foo

Arnaud R added a comment - 2022-08-01 12:16 - edited

Hello! Just for confirmation, we are seeing the same issue with the Kali Linux live image. Reported at https://bugs.kali.org/view.php?id=7823.

The command systemctl start mariadb fails, and below are the logs from the journal:

Aug 01 11:59:18 kali systemd[1]: Starting MariaDB 10.6.8 database server...

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] /usr/sbin/mariadbd (server 10.6.8-MariaDB-1) starting as process 3957 ...

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Number of pools: 1

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Using liburing

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Completed initialization of buffer pool

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: Plugin initialization aborted with error I/O error

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Starting shutdown...

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/

Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation

Aug 01 11:59:18 kali mariadbd[3957]: 220801 11:59:18 [ERROR] mysqld got signal 6 ;

Aug 01 11:59:18 kali mariadbd[3957]: This could be because you hit a bug. It is also possible that this binary

Aug 01 11:59:18 kali mariadbd[3957]: or one of the libraries it was linked against is corrupt, improperly built,

Aug 01 11:59:18 kali mariadbd[3957]: or misconfigured. This error can also be caused by malfunctioning hardware.

Aug 01 11:59:18 kali mariadbd[3957]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs

Aug 01 11:59:18 kali mariadbd[3957]: We will try our best to scrape up some info that will hopefully help

Aug 01 11:59:18 kali mariadbd[3957]: diagnose the problem, but since we have already crashed,

Aug 01 11:59:18 kali mariadbd[3957]: something is definitely wrong and this may fail.

Aug 01 11:59:18 kali mariadbd[3957]: Server version: 10.6.8-MariaDB-1

Aug 01 11:59:18 kali mariadbd[3957]: key_buffer_size=134217728

Aug 01 11:59:18 kali mariadbd[3957]: read_buffer_size=131072

Aug 01 11:59:18 kali mariadbd[3957]: max_used_connections=0

Aug 01 11:59:18 kali mariadbd[3957]: max_threads=153

Aug 01 11:59:18 kali mariadbd[3957]: thread_count=0

Aug 01 11:59:18 kali mariadbd[3957]: It is possible that mysqld could use up to

Aug 01 11:59:18 kali mariadbd[3957]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467959 K  bytes of memory

Aug 01 11:59:18 kali mariadbd[3957]: Hope that's ok; if not, decrease some variables in the equation.

Aug 01 11:59:18 kali mariadbd[3957]: Thread pointer: 0x0

Aug 01 11:59:18 kali mariadbd[3957]: Attempting backtrace. You can use the following information to find out

Aug 01 11:59:18 kali mariadbd[3957]: where mysqld died. If you see no messages after this, something went

Aug 01 11:59:18 kali mariadbd[3957]: terribly wrong...

Aug 01 11:59:18 kali mariadbd[3957]: stack_bottom = 0x0 thread_stack 0x49000

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(my_print_stacktrace)[0x564ff3aab3ce]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(handle_fatal_signal)[0x564ff35f6f78]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(__restore_rt)[0x7f9bf4f50200]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(gsignal)[0x7f9bf4a388c1]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(abort)[0x7f9bf4a22546]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x564ff3293b0d]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(std::unique_lock<std::mutex>::unlock())[0x564ff39116a5]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a17ab8]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a17b21]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a1a1d0]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a1b2f1]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x564ff398010f]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x564ff38c18d5]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(ha_initialize_handlerton(st_plugin_int*))[0x564ff35fa0ee]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(sys_var_pluginvar::sys_var_pluginvar(sys_var_chain*, char const*, st_plugin_int*, st_mysql_sys_var*, char const*))[0x564ff33f0ac6]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(plugin_init(int*, char**, int))[0x564ff33f1a88]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(unireg_abort)[0x564ff331baca]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(mysqld_main(int, char**))[0x564ff3321664]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(__libc_start_main)[0x7f9bf4a2381d]

Aug 01 11:59:18 kali mariadbd[3957]: ??:0(_start)[0x564ff3316bfa]

Aug 01 11:59:18 kali mariadbd[3957]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

Aug 01 11:59:18 kali mariadbd[3957]: information that should help you find out what is causing the crash.

Aug 01 11:59:18 kali mariadbd[3957]: Writing a core file...

Aug 01 11:59:18 kali mariadbd[3957]: Working directory at /var/lib/mysql

Aug 01 11:59:18 kali mariadbd[3957]: Resource Limits:

Aug 01 11:59:18 kali mariadbd[3957]: Limit                     Soft Limit           Hard Limit           Units

Aug 01 11:59:18 kali mariadbd[3957]: Max cpu time              unlimited            unlimited            seconds

Aug 01 11:59:18 kali mariadbd[3957]: Max file size             unlimited            unlimited            bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max data size             unlimited            unlimited            bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max stack size            8388608              unlimited            bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max core file size        0                    unlimited            bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max resident set          unlimited            unlimited            bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max processes             15385                15385                processes

Aug 01 11:59:18 kali mariadbd[3957]: Max open files            32768                32768                files

Aug 01 11:59:18 kali mariadbd[3957]: Max locked memory         524288               524288               bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max address space         unlimited            unlimited            bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max file locks            unlimited            unlimited            locks

Aug 01 11:59:18 kali mariadbd[3957]: Max pending signals       15385                15385                signals

Aug 01 11:59:18 kali mariadbd[3957]: Max msgqueue size         819200               819200               bytes

Aug 01 11:59:18 kali mariadbd[3957]: Max nice priority         0                    0

Aug 01 11:59:18 kali mariadbd[3957]: Max realtime priority     0                    0

Aug 01 11:59:18 kali mariadbd[3957]: Max realtime timeout      unlimited            unlimited            us

Aug 01 11:59:18 kali mariadbd[3957]: Core pattern: core

Aug 01 11:59:18 kali systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT

Aug 01 11:59:18 kali systemd[1]: mariadb.service: Failed with result 'signal'.

Aug 01 11:59:18 kali systemd[1]: Failed to start MariaDB 10.6.8 database server.

Aug 01 11:59:18 kali polkitd(authority=local)[1182]: Unregistered Authentication Agent for unix-process:3919:47143 (system bus name :1.56, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

Aug 01 11:59:23 kali systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.

Aug 01 11:59:23 kali systemd[1]: Stopped MariaDB 10.6.8 database server.

The issue can be worked around by adding this config snippet:

└─$ cat /etc/mysql/mariadb.conf.d/90-live.cnf

[mysqld]

innodb_flush_method = fsync

In case it's of any interest, the kernel version:

└─$ uname -a

Linux kali 5.18.0-kali5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.5-1kali6 (2022-07-07) x86_64 GNU/Linux

Arnaud R added a comment - 2022-08-01 12:16 - edited Hello! Just for confirmation, we are seeing the same issue with the Kali Linux live image. Reported at https://bugs.kali.org/view.php?id=7823 . The command systemctl start mariadb fails, and below are the logs from the journal: Aug 01 11:59:18 kali systemd[1]: Starting MariaDB 10.6.8 database server... Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] /usr/sbin/mariadbd (server 10.6.8-MariaDB-1) starting as process 3957 ... Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Number of pools: 1 Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Using liburing Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Completed initialization of buffer pool Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: Plugin initialization aborted with error I/O error Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Starting shutdown... Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: Operating system error number 22 in a file operation. Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/ Aug 01 11:59:18 kali mariadbd[3957]: 2022-08-01 11:59:18 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation Aug 01 11:59:18 kali mariadbd[3957]: 220801 11:59:18 [ERROR] mysqld got signal 6 ; Aug 01 11:59:18 kali mariadbd[3957]: This could be because you hit a bug. It is also possible that this binary Aug 01 11:59:18 kali mariadbd[3957]: or one of the libraries it was linked against is corrupt, improperly built, Aug 01 11:59:18 kali mariadbd[3957]: or misconfigured. This error can also be caused by malfunctioning hardware. Aug 01 11:59:18 kali mariadbd[3957]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Aug 01 11:59:18 kali mariadbd[3957]: We will try our best to scrape up some info that will hopefully help Aug 01 11:59:18 kali mariadbd[3957]: diagnose the problem, but since we have already crashed, Aug 01 11:59:18 kali mariadbd[3957]: something is definitely wrong and this may fail. Aug 01 11:59:18 kali mariadbd[3957]: Server version: 10.6.8-MariaDB-1 Aug 01 11:59:18 kali mariadbd[3957]: key_buffer_size=134217728 Aug 01 11:59:18 kali mariadbd[3957]: read_buffer_size=131072 Aug 01 11:59:18 kali mariadbd[3957]: max_used_connections=0 Aug 01 11:59:18 kali mariadbd[3957]: max_threads=153 Aug 01 11:59:18 kali mariadbd[3957]: thread_count=0 Aug 01 11:59:18 kali mariadbd[3957]: It is possible that mysqld could use up to Aug 01 11:59:18 kali mariadbd[3957]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467959 K bytes of memory Aug 01 11:59:18 kali mariadbd[3957]: Hope that's ok; if not, decrease some variables in the equation. Aug 01 11:59:18 kali mariadbd[3957]: Thread pointer: 0x0 Aug 01 11:59:18 kali mariadbd[3957]: Attempting backtrace. You can use the following information to find out Aug 01 11:59:18 kali mariadbd[3957]: where mysqld died. If you see no messages after this, something went Aug 01 11:59:18 kali mariadbd[3957]: terribly wrong... Aug 01 11:59:18 kali mariadbd[3957]: stack_bottom = 0x0 thread_stack 0x49000 Aug 01 11:59:18 kali mariadbd[3957]: ??:0(my_print_stacktrace)[0x564ff3aab3ce] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(handle_fatal_signal)[0x564ff35f6f78] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(__restore_rt)[0x7f9bf4f50200] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(gsignal)[0x7f9bf4a388c1] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(abort)[0x7f9bf4a22546] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x564ff3293b0d] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(std::unique_lock<std::mutex>::unlock())[0x564ff39116a5] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a17ab8] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a17b21] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a1a1d0] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x564ff3a1b2f1] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x564ff398010f] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x564ff38c18d5] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(ha_initialize_handlerton(st_plugin_int*))[0x564ff35fa0ee] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(sys_var_pluginvar::sys_var_pluginvar(sys_var_chain*, char const*, st_plugin_int*, st_mysql_sys_var*, char const*))[0x564ff33f0ac6] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(plugin_init(int*, char**, int))[0x564ff33f1a88] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(unireg_abort)[0x564ff331baca] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(mysqld_main(int, char**))[0x564ff3321664] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(__libc_start_main)[0x7f9bf4a2381d] Aug 01 11:59:18 kali mariadbd[3957]: ??:0(_start)[0x564ff3316bfa] Aug 01 11:59:18 kali mariadbd[3957]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains Aug 01 11:59:18 kali mariadbd[3957]: information that should help you find out what is causing the crash. Aug 01 11:59:18 kali mariadbd[3957]: Writing a core file... Aug 01 11:59:18 kali mariadbd[3957]: Working directory at /var/lib/mysql Aug 01 11:59:18 kali mariadbd[3957]: Resource Limits: Aug 01 11:59:18 kali mariadbd[3957]: Limit Soft Limit Hard Limit Units Aug 01 11:59:18 kali mariadbd[3957]: Max cpu time unlimited unlimited seconds Aug 01 11:59:18 kali mariadbd[3957]: Max file size unlimited unlimited bytes Aug 01 11:59:18 kali mariadbd[3957]: Max data size unlimited unlimited bytes Aug 01 11:59:18 kali mariadbd[3957]: Max stack size 8388608 unlimited bytes Aug 01 11:59:18 kali mariadbd[3957]: Max core file size 0 unlimited bytes Aug 01 11:59:18 kali mariadbd[3957]: Max resident set unlimited unlimited bytes Aug 01 11:59:18 kali mariadbd[3957]: Max processes 15385 15385 processes Aug 01 11:59:18 kali mariadbd[3957]: Max open files 32768 32768 files Aug 01 11:59:18 kali mariadbd[3957]: Max locked memory 524288 524288 bytes Aug 01 11:59:18 kali mariadbd[3957]: Max address space unlimited unlimited bytes Aug 01 11:59:18 kali mariadbd[3957]: Max file locks unlimited unlimited locks Aug 01 11:59:18 kali mariadbd[3957]: Max pending signals 15385 15385 signals Aug 01 11:59:18 kali mariadbd[3957]: Max msgqueue size 819200 819200 bytes Aug 01 11:59:18 kali mariadbd[3957]: Max nice priority 0 0 Aug 01 11:59:18 kali mariadbd[3957]: Max realtime priority 0 0 Aug 01 11:59:18 kali mariadbd[3957]: Max realtime timeout unlimited unlimited us Aug 01 11:59:18 kali mariadbd[3957]: Core pattern: core Aug 01 11:59:18 kali systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT Aug 01 11:59:18 kali systemd[1]: mariadb.service: Failed with result 'signal'. Aug 01 11:59:18 kali systemd[1]: Failed to start MariaDB 10.6.8 database server. Aug 01 11:59:18 kali polkitd(authority=local)[1182]: Unregistered Authentication Agent for unix-process:3919:47143 (system bus name :1.56, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus) Aug 01 11:59:23 kali systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1. Aug 01 11:59:23 kali systemd[1]: Stopped MariaDB 10.6.8 database server. The issue can be worked around by adding this config snippet: └─$ cat /etc/mysql/mariadb.conf.d/90-live.cnf [mysqld] innodb_flush_method = fsync In case it's of any interest, the kernel version: └─$ uname -a Linux kali 5.18.0-kali5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.5-1kali6 (2022-07-07) x86_64 GNU/Linux

Otto Kekäläinen added a comment - 2023-01-22 00:05

I would personally also be keen to figure out how to make the MariaDB datadir be able to run off overlayfs or something similar so I can implement my long time idea to have shadow upgrades where the old mariadbd and old data is kept running and intact, and every new version mariadbd would first run off overlayfs with real data to simulate the upgrade before it is done for real.

Otto Kekäläinen added a comment - 2023-01-22 00:05 I would personally also be keen to figure out how to make the MariaDB datadir be able to run off overlayfs or something similar so I can implement my long time idea to have shadow upgrades where the old mariadbd and old data is kept running and intact, and every new version mariadbd would first run off overlayfs with real data to simulate the upgrade before it is done for real.

Christian Hesse added a comment - 2025-02-06 07:57

Just tested, looks like this is no longer an issue - at least with versions 11.4.4 and 11.6.2.

Christian Hesse added a comment - 2025-02-06 07:57 Just tested, looks like this is no longer an issue - at least with versions 11.4.4 and 11.6.2.

MariaDB Server

mariadb-install-db fails writing ibdata1 on overlayfs

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration