[MDEV-23454] Incremental Backup Preparation Mariabackup --prepare crashed [ERROR] mysqld got signal 6 ; Created: 2020-08-12  Updated: 2022-04-22  Resolved: 2022-04-22

Status: Closed
Project: MariaDB Server
Component/s: mariabackup
Affects Version/s: 10.2.27, 10.4.10, 10.4.13
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Wilson Echavez Assignee: Allen Lee (Inactive)
Resolution: Incomplete Votes: 0
Labels: crash
Environment:

Debian Buster


Attachments: Text File mysqld_bt_all_threads.txt     Text File mysqld_full_bt_all_threads.txt    

 Description   

We have big database we are backing up we do full backup then full backup prepare and then incremental backup it worked fine by the time we are doing incremental backup prepare we experience this issue/error for the past 3 weeks. We`ve done upgrading versions from version 10.4.10 to 10.4.13 still the same issue/error we encounter.

2020-08-11 17:01:20 0x7f4b5ef8e700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.13/storage/innobase/log/log0recv.cc line 1520
InnoDB: Failing assertion: Unable to render embedded object: File (page ) not found.!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200811 17:01:20 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.13-MariaDB-1:10.4.13+maria~buster
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5932 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
mariabackup(my_print_stacktrace+0x2e)[0x559862857bfe]
mariabackup(handle_fatal_signal+0x54d)[0x5598623b48ed]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f4b781b9730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f4b776017bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f4b775ec535]
mariabackup(+0x5c340f)[0x55986203540f]
mariabackup(+0x5a5cc1)[0x559862017cc1]
mariabackup(+0xb6d6ed)[0x5598625df6ed]
mariabackup(+0x5a629f)[0x55986201829f]
mariabackup(+0x588fae)[0x559861ffafae]
mariabackup(+0xac319c)[0x55986253519c]
mariabackup(+0xc03600)[0x559862675600]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f4b781aefa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f4b776c34cf]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /BackupDirectory
Resource Limits:
Fatal signal 11 while backtracing



 Comments   
Comment by Daniel Black [ 2020-08-12 ]

Thanks for the bug report. Though a quick search shows no similar fixes in 10.4.14 are you able to try it. It was released yesterday.

Before you rerun mariabackup can you install the debug package to make the backtrace more readable to developers

install debug package for mariadb-backp

 apt-get install mariadb-backup-dbgsym

If you have a core file from the above crash can you install gdb and attach the output file from:

gdb for getting comprensive backtrace

gdb --batch --eval-command="thread apply all bt"/usr/bin/mariabackup /path/to/core.file  > mysqld_bt_all_threads.txt

Comment by Wilson Echavez [ 2020-08-14 ]

Hello Daniel Black we upgrade our mariadb version to latest mariadb 10.4.14 and run our incremental prepare again and same issue we encountered i already put options on my mariabackup command --core-file but i can`t find from our directories the core file.

2020-08-14 04:59:32 0x7f9a36d22700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.14/storage/innobase/log/log0recv.cc line 1521
InnoDB: Failing assertion: Unable to render embedded object: File (page ) not found.!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200814 4:59:32 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.14-MariaDB-1:10.4.14+maria~buster
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5933 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
mariabackup(my_print_stacktrace+0x2e)[0x55609825634e]
mariabackup(handle_fatal_signal+0x54d)[0x556097db3dfd]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f9abfa1b730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f9abee637bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f9abee4e535]
mariabackup(+0x5cdcfc)[0x556097a2fcfc]
mariabackup(+0x5aff21)[0x556097a11f21]
mariabackup(+0xb7d3bd)[0x556097fdf3bd]
mariabackup(+0x5b04ea)[0x556097a124ea]
mariabackup(+0x59349e)[0x5560979f549e]
mariabackup(+0xad0c8c)[0x556097f32c8c]
mariabackup(+0xc11a10)[0x556098073a10]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f9abfa10fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f9abef254cf]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /Backup/Directory
Resource Limits:
Fatal signal 11 while backtracing

Additional Information we have done our Full Backup on Mariadb version 10.4.10 and it was successful, we prepare the Full Backup on Mariadb version 10.4.13 and it was successful, and then we have done the Incremental Backup on Mariadb version 10.4.13 and it was successful, and then we have done the Incremental Backup Prepare on two version Mariadb version 10.4.13 and Mariadb version 10.4.14 and on those two version this error/issue we encounter.

Comment by Wilson Echavez [ 2020-08-19 ]

Hello we already upgraded our Mariadb Version up to the latest Version Mariadb 10.4.14 we do Full Backup and it was a success and then we prepare the Full Backup and it was a success and then we do our first Incremental Backup based from the full backup and it was a success and then we prepare the first incremental backup to the full backup and it was a success and then we do the second incremental backup based from the first incremental backup and it was a success and then when we prepare the second incremental backup to the full backup and we encounter error/issue again.

200819 11:55:34 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.14-MariaDB-1:10.4.14+maria~buster
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5933 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
mariabackup(my_print_stacktrace+0x2e)[0x55755256834e]
mariabackup(handle_fatal_signal+0x54d)[0x5575520c5dfd]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fc1c0602730]
mariabackup(+0x5b498f)[0x557551d2898f]
mariabackup(+0xb973a0)[0x55755230b3a0]
mariabackup(+0xb7c66d)[0x5575522f066d]
mariabackup(+0xb7d3bd)[0x5575522f13bd]
mariabackup(+0x5b04ea)[0x557551d244ea]
mariabackup(+0x59349e)[0x557551d0749e]
mariabackup(+0xad0c8c)[0x557552244c8c]
mariabackup(+0xc11a10)[0x557552385a10]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fc1c05f7fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc1bfb0c4cf]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /BackupDirectory
Resource Limits:

I don`t have a core file from mariabackup to be able to backtrace but i have a backtrace from the mysqld process which i will attach here maybe it can help..
mysqld_bt_all_threads.txt mysqld_full_bt_all_threads.txt

Generated at Thu Feb 08 09:22:32 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.