Details
-
Bug
-
Status: Open (View Workflow)
-
Minor
-
Resolution: Unresolved
-
10.3.21, 10.6.14
-
None
-
Centos 7
Description
Hey there,
there is a reproduceable crash of MariaDB 10.3.x when running mariabackup on it. Reporting as advised in the resulting Error Log:
[00] 2020-03-30 12:39:07 Connecting to MySQL server host: localhost, user: root, password: set, port: 3306, socket: /var/lib/mysql/mysql.sock
|
[00] 2020-03-30 12:39:07 Using server version 10.3.21-MariaDB
|
mariabackup based on MariaDB server 10.3.21-MariaDB Linux (x86_64)
|
[00] 2020-03-30 12:39:07 uses posix_fadvise().
|
[00] 2020-03-30 12:39:07 cd to /var/lib/mysql/
|
[00] 2020-03-30 12:39:07 open files limit requested 0, set to 1024
|
[00] 2020-03-30 12:39:07 mariabackup: using the following InnoDB configuration:
|
[00] 2020-03-30 12:39:07 innodb_data_home_dir =
|
[00] 2020-03-30 12:39:07 innodb_data_file_path = ibdata1:12M:autoextend
|
[00] 2020-03-30 12:39:07 innodb_log_group_home_dir = ./
|
[00] 2020-03-30 12:39:07 InnoDB: Using Linux native AIO
|
2020-03-30 12:39:07 0 [Note] InnoDB: Number of pools: 1
|
[00] 2020-03-30 12:39:07 mariabackup: Generating a list of tablespaces
|
2020-03-30 12:39:07 0 [Warning] InnoDB: Allocated tablespace ID 2325 for sf_XXXXXX_relaunch/ezinfocollection_attribute, old maximum was 0
|
2020-03-30 12:39:07 0 [ERROR] InnoDB: Operating system error number 24 in a file operation.
|
2020-03-30 12:39:07 0 [ERROR] InnoDB: Error number 24 means 'Too many open files'
|
2020-03-30 12:39:07 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
|
2020-03-30 12:39:07 0 [ERROR] InnoDB: File ./XXXXXX_magento2/salesrule.ibd: 'open' returned OS error 224.
|
2020-03-30 12:39:07 0x7ff942b7d8c0 InnoDB: Assertion failure in file /root/rpmbuild/BUILD/mariadb-10.3.21/storage/innobase/fil/fil0fil.cc line 677
|
InnoDB: Failing assertion: success
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
200330 12:39:07 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.3.21-MariaDB
|
key_buffer_size=0
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=1
|
thread_count=0
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5600 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
mariabackup(my_print_stacktrace+0x3d)[0x5570572ef63d]
|
mariabackup(handle_fatal_signal+0x327)[0x557056e2ce17]
|
/lib64/libpthread.so.0(+0xf5f0)[0x7ff9427685f0]
|
/lib64/libc.so.6(gsignal+0x37)[0x7ff940824337]
|
/lib64/libc.so.6(abort+0x148)[0x7ff940825a28]
|
mariabackup(+0x4e4616)[0x557056aec616]
|
mariabackup(+0xaa5b99)[0x5570570adb99]
|
mariabackup(+0xaa5c1f)[0x5570570adc1f]
|
mariabackup(+0x507b7e)[0x557056b0fb7e]
|
mariabackup(+0x5071bd)[0x557056b0f1bd]
|
mariabackup(+0x5076ad)[0x557056b0f6ad]
|
mariabackup(+0x50d0b6)[0x557056b150b6]
|
mariabackup(main+0x194)[0x557056aee5f4]
|
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ff940810505]
|
mariabackup(+0x50073d)[0x557056b0873d]
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Max. Open Files should not be the Source I guess as it's raised to the limit already:
# cat /var/run/mariadb/mariadb.pid
|
2805
|
# cat /proc/2805/limits
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 62987 62987 processes
|
Max open files 1048576 1048576 files
|
Max locked memory unlimited unlimited bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 514290 514290 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Hope someone here can give me a helping hand on this! 
thx, bye from Austria
Andreas
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Open
-
Activity
I'm a bit puzzled by this as "Max Open Files" is set to 1048576 on this Server - so should be more than high enough?! But you are right - setting [mariabackup] open_files_limit=65535 did solve the Problem! Thank you! 
As marko pointed out many times, InnoDB is overly eager to abort on system errors. Ideally, InnoDB shouldn't crash on this. If it really has to abort, it would be better to go down in a more graceful fashion.
Futureweb, could you try to diagnose this with strace? Or construct a repeatable test case?
@Marko - problem was solved setting [mariabackup] open_files_limit=65535
Even though Open File Limit was set to 1048576 on Centos itself it seems it was not detected correctly until open_files_limit was added to the MariaDB config.
This is conceptually similar to MDEV-13542, but should be much easier to fix than that one.
In fact, there is no reason why mariabackup --backup is using the InnoDB file handling wrapper functions. It could use the mysys functions.
I do not think that fixing this (removing the use of InnoDB file API wrappers in mariabackup --backup) requires any deep InnoDB knowledge.
The error message says it's 'Too many open files', could you please try one of suggested here solutions https://mariadb.com/kb/en/mariabackup-overview/#too-many-open-files