[MDEV-29617] MariaDB server crashes on startup after 10.9.3 on ARM Created: 2022-09-23  Updated: 2023-10-23  Resolved: 2023-10-23

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.9.3
Fix Version/s: 10.9.8, 10.10.6, 10.11.5, 11.0.3, 11.1.2

Type: Bug Priority: Major
Reporter: Piotr Witosławski Assignee: Daniel Black
Resolution: Duplicate Votes: 0
Labels: None
Environment:

4.14.180-3-ARCH, armv7


Attachments: File mariadbd.strace    
Issue Links:
Duplicate
duplicates MDEV-31642 Upgrade from 10.7 or earlier may cras... Closed
is duplicated by MDEV-29618 MariaDB server crashes on startup aft... Closed
Relates
relates to MDEV-29551 InnoDB: Plugin initialization aborted... Closed

 Description   

After the update to 10.9.3, server doesn't start anymore.

220923 12:27:28 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
2022-09-23 12:27:28 0 [Note] /usr/bin/mariadbd (server 10.9.3-MariaDB) starting as process 20919 ...
2022-09-23 12:27:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2022-09-23 12:27:28 0 [Note] InnoDB: Number of transaction pools: 1
2022-09-23 12:27:28 0 [Note] InnoDB: Using generic crc32 instructions
2022-09-23 12:27:28 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2022-09-23 12:27:28 0 [Note] InnoDB: Completed initialization of buffer pool
2022-09-23 12:27:28 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2022-09-23 12:27:28 0 [Warning] InnoDB: Retry attempts for reading partial data failed.
2022-09-23 12:27:28 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.
2022-09-23 12:27:28 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2022-09-23 12:27:28 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2022-09-23 12:27:28 0 [ERROR] InnoDB: File (unknown): 'read' returned OS error 222. Cannot continue operation
220923 12:27:28 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.9.3-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 466613 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
??:0(my_print_stacktrace)[0x10881cc]
??:0(handle_fatal_signal)[0xb70b54]
??:0(__default_sa_restorer)[0xb6705d10]
??:0(pthread_key_delete)[0xb674e310]
??:0(raise)[0xb6704708]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    0                    bytes
Max resident set          unlimited            unlimited            bytes
Max processes             14864                14864                processes
Max open files            32190                32190                files
Max locked memory         8388608              8388608              bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       14864                14864                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
 
Kernel version: Linux version 4.14.180-3-ARCH (builduser@leming) (gcc version 10.2.0 (GCC)) #1 SMP PREEMPT Sat Jun 5 22:29:47 UTC 2021
 
220923 12:27:28 mysqld_safe mysqld from pid file /var/lib/mysql/alarm.pid ended

I'm attaching strace file as well.
Filesystem used: f2fs
Tried innodb_flush_method = fsync and innodb_use_native_aio = 0. Didn't help.



 Comments   
Comment by Alice Sherepa [ 2022-09-23 ]

it seems to be similar to MDEV-29551

Comment by Marko Mäkelä [ 2022-09-23 ]

alice, this is not similar to MDEV-29551, because according to witos, he overrode the default parameter innodb_flush_method=O_DIRECT.

In mariadbd.strace, I see the following:

mariadb-10.9.3

21004 pread64(11, "PHYS\0\0\0\2\0\0\0\1~&\374\0MariaDB 10.5.5\0\0"..., 4096, 0) = 4096
21004 pread64(11, 0x217a000, 512, 71620096) = -1 EINVAL (Invalid argument)

Would setting innodb_log_file_buffering=ON (MDEV-28766) allow the upgrade from MariaDB 10.5? If yes, I would like to know the physical block size of the underlying storage. What would /sys/block/*/queue/physical_block_size show for the device?

I see that according to stat(2), the size of the ib_logfile0 is a multiple of 4096 bytes.

Comment by Piotr Witosławski [ 2022-09-23 ]

innodb_log_file_buffering=ON

This setting helped. Thank you!

cat /sys/block/mmcblk0/queue/physical_block_size
512

Comment by Marko Mäkelä [ 2022-09-23 ]

witos, thank you.

danblack had an idea to enable the O_DIRECT already during open() and not by fcntl(). Hopefully that would fail on any file system that does not support O_DIRECT, and as a simple fallback, we could retry the open() without O_DIRECT.

Comment by Daniel Black [ 2023-09-19 ]

I think this has been resolved in MDEV-31642

Generated at Thu Feb 08 10:09:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.