[MDEV-21722] Mariabackup - Preparing incremental backup crash with mysqld got signal 6 Created: 2020-02-13  Updated: 2021-09-30  Resolved: 2020-03-12

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.4.10
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Wilson Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: crash, need_feedback
Environment:

Debian 10 = OS

Issue Links:
Relates
relates to MDEV-20755 InnoDB: Database page corruption on d... Closed

 Description   

InnoDB: Failing assertion: free + TRX_UNDO_LOG_XA_HDR_SIZE < srv_page_size - 100
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200213 10:23:21 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.10-MariaDB-1:10.4.10+maria~buster
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5917 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
mariabackup(my_print_stacktrace+0x2e)[0x563c2ed0971e]
mariabackup(handle_fatal_signal+0x54d)[0x563c2e864aad]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f708c43b730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f708b8837bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f708b86e535]
mariabackup(+0x5b2371)[0x563c2e4e1371]
mariabackup(+0x5b1f98)[0x563c2e4e0f98]
mariabackup(+0xc21ef0)[0x563c2eb50ef0]
mariabackup(+0xb5c9b5)[0x563c2ea8b9b5]
mariabackup(+0xb5d59b)[0x563c2ea8c59b]
mariabackup(+0x595449)[0x563c2e4c4449]
mariabackup(+0x578a62)[0x563c2e4a7a62]
mariabackup(+0xab70dc)[0x563c2e9e60dc]
mariabackup(+0xbf3c20)[0x563c2eb22c20]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f708c430fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f708b9454cf]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /path/to/backup
Resource Limits:
Fatal signal 11 while backtracing
Segmentation fault

 Comments   
Comment by Marko Mäkelä [ 2020-02-13 ]

This looks like undo log header page corruption.
We would need a reproducible test case for this.

Please note that a mariabackup --prepare step will be needed before the server can be started up. The incremental backup writes some additional .delta files that are ignored by the normal InnoDB crash recovery.

Comment by Wilson [ 2020-02-13 ]

Hi Marko Makela i encounter this issue when i prepare my incremental backup to my full backup, yes the mariadb service was stop.. Before i didn't encounter this issue..

Just a quick question because i'm a newbie on mariadb, is there a way of preparing multiple incremental backup on my fullbackup in just one session??

Comment by Somanos Sar [ 2020-06-03 ]

Hi all,
I've just tumbled upon the same issue.
10.4.8-MariaDB debian-linux-gnu (x86_64)
...
[00] 2020-06-03 17:51:49 mariabackup: Generating a list of tablespaces
2020-06-03 17:51:49 0 [Warning] InnoDB: Allocated tablespace ID 972953 for 377eafb4377eafb6/FTS_00000000000ed89d_00000000001ca9e9_INDEX_6, old maximum was 0
2020-06-03 17:51:49 0 [ERROR] InnoDB: Operating system error number 24 in a file operation.
2020-06-03 17:51:49 0 [ERROR] InnoDB: Error number 24 means 'Too many open files'
2020-06-03 17:51:49 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
2020-06-03 17:51:49 0 [ERROR] InnoDB: File ./a_1222c6951222c696/_src_temp_media.ibd: 'open' returned OS error 224.
2020-06-03 17:51:49 0x7ff2c93797c0 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.8/storage/innobase/fil/fil0fil.cc line 576

...
Is there any updates ?

My instance contains large amount of schemas...

Thanks

Comment by Somanos Sar [ 2020-06-05 ]

In case some others are facing the same issue, I found some hints from this link
https://mariadb.com/kb/en/mariabackup-overview/#too-many-open-files and end up with this conf, which works for me. Btw, 65535 wasn't enough for me... needed to make 10 fold
[mariabackup]
open_files_limit=655350
use-memory=1048576
max_connections=1

Comment by Marko Mäkelä [ 2020-09-11 ]

The problem should be caused by a DB_ROLL_PTR system column pointing to something that is no longer the start of a valid undo log record. In other words, the clustered index leaf page and the undo log pages of the transaction management are not corresponding to the same logical point of time.

MDEV-20755 might explain this. If the table was being rebuilt by ALTER TABLE using the default setting innodb_log_optimize_ddl=ON, then incremental backup or restore could make an incorrect copy of the data file.

I am not entirely convinced of this hypothesis, and I did not analyze what exactly is broken in the incremental backup or restore.

Generated at Thu Feb 08 09:09:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.