Debian Buster

We perform again the second incremental backup prepare since we have a backup copy of the incremental backup that we had and same thing happen it crashed..

200820 13:33:33 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.14-MariaDB-1:10.4.14+maria~buster
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5933 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
mariabackup(my_print_stacktrace+0x2e)[0x558cd931634e]
mariabackup(handle_fatal_signal+0x54d)[0x558cd8e73dfd]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f4d9f6a4730]
mariabackup(+0x5b4cf7)[0x558cd8ad6cf7]
mariabackup(+0xb97f1d)[0x558cd90b9f1d]
mariabackup(+0xb986cc)[0x558cd90ba6cc]
mariabackup(+0xb7c96f)[0x558cd909e96f]
mariabackup(+0xb7d3bd)[0x558cd909f3bd]
mariabackup(+0x5b04ea)[0x558cd8ad24ea]
mariabackup(+0x59349e)[0x558cd8ab549e]
mariabackup(+0xad0c8c)[0x558cd8ff2c8c]
mariabackup(+0xc11a10)[0x558cd9133a10]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f4d9f699fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f4d9ebae4cf]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /BackupDirectory mariabackup_bt_all_threads.txt mysqld_bt_all_threads.txt mysqld_full_bt_all_threads.txt
Resource Limits:

We perform an incremental backup and was based from our full backup.. Incremental Backup was successful and completed but by the time we prepare the incremental backup we encounter issue again.

2020-08-21 12:28:22 0x7f194640e700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.14/storage/innobase/log/log0recv.cc line 1521
InnoDB: Failing assertion: Unable to render embedded object: File (page ) not found.!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
200821 12:28:22 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.14-MariaDB-1:10.4.14+maria~buster
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5933 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
mariabackup(my_print_stacktrace+0x2e)[0x559c7eee434e]
mariabackup(handle_fatal_signal+0x54d)[0x559c7ea41dfd]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f1959717730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f1958b5f7bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f1958b4a535]
mariabackup(+0x5cdcfc)[0x559c7e6bdcfc]
mariabackup(+0x5aff21)[0x559c7e69ff21]
mariabackup(+0xb7d3bd)[0x559c7ec6d3bd]
mariabackup(+0x5b04ea)[0x559c7e6a04ea]
mariabackup(+0x59349e)[0x559c7e68349e]
mariabackup(+0xad0c8c)[0x559c7ebc0c8c]
mariabackup(+0xc11a10)[0x559c7ed01a10]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f195970cfa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f1958c214cf]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /BackupDirectory mariabackup_bt_all_threads08212020.txt mysqld_bt_all_threads08212020.txt mysqld_full_bt_all_threads08212020.txt
Resource Limits:
Fatal signal 11 while backtracing

Steps till crash tested on 10.5.15 attached steps_crash

I can repeat this with the following change to an existing test:

diff --git a/mysql-test/suite/mariabackup/incremental_backup.test b/mysql-test/suite/mariabackup/incremental_backup.test

index 88e277fd95a..eb3f769f9a2 100644

--- a/mysql-test/suite/mariabackup/incremental_backup.test

+++ b/mysql-test/suite/mariabackup/incremental_backup.test

@@ -1,5 +1,5 @@

 --source include/have_aria.inc

---source include/innodb_page_size.inc

+--source include/innodb_page_size_small.inc

 # see suite.pm "check for exact values, in case the default changes to be small everywhere"

 if (`select @@max_binlog_stmt_cache_size = 4294963200 and @@innodb_page_size = 65536`) {

@@ -12,7 +12,7 @@ let basedir=$MYSQLTEST_VARDIR/tmp/backup;

 let incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1;

 CREATE TABLE t_aria(i INT) ENGINE ARIA;

-CREATE TABLE t(i INT PRIMARY KEY) ENGINE INNODB;

+CREATE TABLE t(i INT PRIMARY KEY) ENGINE INNODB ROW_FORMAT=COMPRESSED;

 BEGIN;

 INSERT INTO t VALUES(2);

 connect (con1,localhost,root,,);

ulimit -c unlimited

./mtr mariabackup.incremental_backup,16k

so that the page size 16384 bytes would be used, the test will cause a crash:

[00] 2022-04-19 21:04:29 incremental backup from 47858 is enabled.

[00] 2022-04-19 21:04:29 cd to /dev/shm/10.5m/mysql-test/var/tmp/backup/

[00] 2022-04-19 21:04:29 open files limit requested 0, set to 1024

[00] 2022-04-19 21:04:29 This target seems to be already prepared.

[00] 2022-04-19 21:04:29 mariabackup: using the following InnoDB configuration for recovery:

[00] 2022-04-19 21:04:29 innodb_data_home_dir = .

[00] 2022-04-19 21:04:29 innodb_data_file_path = ibdata1:12M:autoextend

[00] 2022-04-19 21:04:29 innodb_log_group_home_dir = /dev/shm/10.5m/mysql-test/var/tmp/backup_inc1/

[00] 2022-04-19 21:04:29 InnoDB: Using Linux native AIO

[00] 2022-04-19 21:04:29 mariabackup: Generating a list of tablespaces

220419 21:04:29 [ERROR] mysqld got signal 11 ;

…

#4  0x000055de1ffbd9ab in buf_is_zeroes (buf={data_ = 0x55de21c98000 "{\343n\351", size_ = 8192}) at /mariadb/10.5m/storage/innobase/buf/buf0buf.cc:708

#5  0x000055de2011503d in page_zip_verify_checksum (data=data@entry=0x55de21c98000 "{\343n\351", size=size@entry=8192) at /mariadb/10.5m/storage/innobase/page/page0zip.cc:4659

#6  0x000055de1ffbf494 in buf_page_is_corrupted (check_lsn=check_lsn@entry=false, read_buf=0x55de21c98000 "{\343n\351", fsp_flags=41) at /mariadb/10.5m/storage/innobase/buf/buf0buf.cc:817

#7  0x000055de2002d9dd in Datafile::validate_first_page (this=this@entry=0x55de21c28240, flush_lsn=flush_lsn@entry=0x7fff5f4a1258) at /mariadb/10.5m/storage/innobase/fsp/fsp0file.cc:573

#8  0x000055de1fa080ed in xb_load_single_table_tablespace (dirname=<optimized out>, filname=0x7fff5f4a2290 "t.ibd", is_remote=<optimized out>, skip_node_page0=<optimized out>) at /mariadb/10.5m/extra/mariabackup/xtrabackup.cc:3327

#9  0x000055de1fa07190 in enumerate_ibd_files (callback=callback@entry=0x55de1fa07fb3 <xb_load_single_table_tablespace(char const*, char const*, bool, bool)>) at /mariadb/10.5m/extra/mariabackup/xtrabackup.cc:3700

#10 0x000055de1fa073ca in xb_load_tablespaces () at /mariadb/10.5m/extra/mariabackup/xtrabackup.cc:3876

#11 0x000055de1fa0d9c9 in xtrabackup_prepare_func (argv=argv@entry=0x55de21be8178) at /mariadb/10.5m/extra/mariabackup/xtrabackup.cc:5930

#12 0x000055de1fa0e4e5 in main_low (argv=0x55de21be8178) at /mariadb/10.5m/extra/mariabackup/xtrabackup.cc:6930

#13 0x000055de1fa0e6c7 in main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.5m/extra/mariabackup/xtrabackup.cc:6726

It crashes even if I add KEY_BLOCK_SIZE=16 so that each page would be 16384 bytes.

marko, I have analyzed the test case and filed new MDEV-28473, as it does not relate to this case because: 1) the crash in this case happens in 10.4.14, but 82d59945 was committed in 10.5.12; 2) for this case the assertion fails in recv_parse_or_apply_log_rec_body() in 10.4.14, but the modified mtr test case fails with SIGSEGV in buf_is_zeroes().

Customer complains that for 10.6, when using table with ROW_FORMAT=COMPRESSED, the prepare of incremental backup crashes: mysqld got signal 11.
I have tested:
10.3 (does not crash)
10.4 (does not crash)
10.5 (crashes)
10.6 (crashes)
I will attach images to show the results.

Apparently when using 10.6.8-4-MariaDB-enterprise the crash does not occur. I will attach an image. Thank you.

The issue the customer suffered from was solved in MDEV-28473. We failed to reproduce this certain case, so I close it until we have the ability to get more details from our users.