[MDEV-26983] Server crash in buf_is_zeroes or assertion failure in xb_delta_open_matching_space upon incremental backup prepare with compressed partitioned table Created: 2021-11-05  Updated: 2022-09-14  Resolved: 2022-09-14

Status: Closed
Project: MariaDB Server
Component/s: mariabackup, Partitioning, Storage Engine - InnoDB
Affects Version/s: 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.4.27, 10.5.18, 10.6.11, 10.7.7, 10.8.6, 10.9.4, 10.10.2

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-18589 Assertion `fil_space_t::physical_size... Closed
Problem/Incident
is caused by MDEV-18493 Remove page_size_t Closed
Relates
relates to MDEV-28473 field_ref_zero is not initialized in ... Closed

 Description   

Note: the test case is made universal (10.4-10.7) for convenience. For the branch-specific regression test suite, at least the conditional logic of setting the variable shouldn't be needed

--source include/have_innodb.inc
--source include/have_partition.inc
 
# For 10.6+
if (`select count(*) from information_schema.system_variables where variable_name = 'innodb_read_only_compressed'`)
{
  SET @read_only_compressed.save= @@innodb_read_only_compressed;
  SET GLOBAL innodb_read_only_compressed= OFF;
}
 
CREATE TABLE t (pk INT PRIMARY KEY) ENGINE=InnoDB ROW_FORMAT=COMPRESSED;
 
--exec $XTRABACKUP --backup --target-dir=$MYSQL_TMP_DIR/backup_0 --protocol=tcp --port=$MASTER_MYPORT --user=root
 
ALTER TABLE t PARTITION BY KEY(pk);
 
--exec $XTRABACKUP --backup --target-dir=$MYSQL_TMP_DIR/backup_1 --incremental-basedir=$MYSQL_TMP_DIR/backup_0 --protocol=tcp --port=$MASTER_MYPORT --user=root > $MYSQL_TMP_DIR/backup_1.log 2>&1
--exec $XTRABACKUP --prepare --target-dir=$MYSQL_TMP_DIR/backup_0 --user=root > $MYSQL_TMP_DIR/backup_prepare_0.log 2>&1
--exec $XTRABACKUP --prepare --target-dir=$MYSQL_TMP_DIR/backup_0 --incremental-dir=$MYSQL_TMP_DIR/backup_1 --user=root >  $MYSQL_TMP_DIR/backup_prepare_1.log
--cat_file $MYSQL_TMP_DIR/backup_prepare_1.log
 
# Cleanup
DROP TABLE t;
# For 10.6+
if (`select @read_only_compressed.save is not null`)
{
  SET GLOBAL innodb_read_only_compressed= @read_only_compressed.save;
}

On 10.5-10.7 it crashes with sigsegv in buf_is_zeroes:

bb-10.5-release 8635be6a2

#2  <signal handler called>
#3  __memcmp_avx2_movbe () at ../sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S:70
#4  0x000055a4aecbb83f in buf_is_zeroes (buf=...) at /data/src/bb-10.5-release/storage/innobase/buf/buf0buf.cc:708
#5  0x000055a4aeee3f84 in page_zip_verify_checksum (data=0x55a4b0b64000 "{\343n\351", size=8192) at /data/src/bb-10.5-release/storage/innobase/page/page0zip.cc:4648
#6  0x000055a4aecbbb52 in buf_page_is_corrupted (check_lsn=false, read_buf=0x55a4b0b64000 "{\343n\351", fsp_flags=41) at /data/src/bb-10.5-release/storage/innobase/buf/buf0buf.cc:820
#7  0x000055a4aed6a049 in Datafile::validate_first_page (this=0x55a4b0b47640, flush_lsn=0x7ffce6ff5f08) at /data/src/bb-10.5-release/storage/innobase/fsp/fsp0file.cc:573
#8  0x000055a4ae367a61 in xb_load_single_table_tablespace (dirname=0x7ffce6ff5fc0 "test", filname=0x7ffce6ff6fa0 "t.ibd", is_remote=false, skip_node_page0=false) at /data/src/bb-10.5-release/extra/mariabackup/xtrabackup.cc:3313
#9  0x000055a4ae36884e in enumerate_ibd_files (callback=0x55a4ae3677cc <xb_load_single_table_tablespace(char const*, char const*, bool, bool)>) at /data/src/bb-10.5-release/extra/mariabackup/xtrabackup.cc:3686
#10 0x000055a4ae368ed0 in xb_load_tablespaces () at /data/src/bb-10.5-release/extra/mariabackup/xtrabackup.cc:3862
#11 0x000055a4ae370319 in xtrabackup_prepare_func (argv=0x55a4b0aff388) at /data/src/bb-10.5-release/extra/mariabackup/xtrabackup.cc:5925
#12 0x000055a4ae372efb in main_low (argv=0x55a4b0aff388) at /data/src/bb-10.5-release/extra/mariabackup/xtrabackup.cc:6883
#13 0x000055a4ae37272b in main (argc=5, argv=0x7ffce6ff8a18) at /data/src/bb-10.5-release/extra/mariabackup/xtrabackup.cc:6681

On 10.4 it fails on an assertion:

bb-10.4-release a19ab67318

mariabackup: /data/src/bb-10.4-release/extra/mariabackup/xtrabackup.cc:5305: pfs_os_file_t xb_delta_open_matching_space(const char*, const char*, const xb_delta_info_t&, char*, size_t, bool*): Assertion `fil_space_t::physical_size(flags) == info.page_size' failed.
211106  1:18:03 [ERROR] mysqld got signal 6 ;
 
sigaction.c:0(__restore_rt)[0x7f8a17d9f3c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f8a177e918b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f8a177c8859]
/lib/x86_64-linux-gnu/libc.so.6(+0x25729)[0x7f8a177c8729]
/lib/x86_64-linux-gnu/libc.so.6(+0x36f36)[0x7f8a177d9f36]
mariabackup/xtrabackup.cc:5307(xb_delta_open_matching_space(char const*, char const*, xb_delta_info_t const&, char*, unsigned long, bool*))[0x5624aceb6bcf]
mariabackup/xtrabackup.cc:5405(xtrabackup_apply_delta(char const*, char const*, char const*, void*))[0x5624aceb7081]
mariabackup/xtrabackup.cc:5790(xb_process_datadir(char const*, char const*, unsigned long (*)(char const*, char const*, char const*, void*), void*))[0x5624aceb8cd2]
mariabackup/xtrabackup.cc:5828(xtrabackup_apply_deltas())[0x5624aceb8dd8]
mariabackup/xtrabackup.cc:6060(xtrabackup_prepare_func(char**))[0x5624aceba582]
mariabackup/xtrabackup.cc:7065(main_low(char**))[0x5624acebd346]
mariabackup/xtrabackup.cc:6861(main)[0x5624acebcb5e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f8a177ca0b3]

Not reproducible on 10.3.

At least on my machine the failure is deterministic.



 Comments   
Comment by Marko Mäkelä [ 2022-09-14 ]

I was thinking that this might have been fixed by MDEV-28473, but it is not the case. 10.5 debug build:

10.5 fb70bb44d096f4117e9454740aa4fa0651a1a69e

[00] 2022-09-14 10:54:20 page size for /dev/shm/10.5d/mysql-test/var/tmp/backup_1//ibdata1.delta is 16384 bytes
[00] 2022-09-14 10:54:20 Applying /dev/shm/10.5d/mysql-test/var/tmp/backup_1//ibdata1.delta to ./ibdata1...
[00] 2022-09-14 10:54:20 page size for /dev/shm/10.5d/mysql-test/var/tmp/backup_1//test/t#P#p0.ibd.delta is 8192 bytes
mariabackup: /mariadb/10.5/extra/mariabackup/xtrabackup.cc:5168: pfs_os_file_t xb_delta_open_matching_space(const char*, const char*, const xb_delta_info_t&, char*, size_t, bool*): Assertion `fil_space_t::physical_size(flags) == info.page_size' failed.

The following would fix the failure:

diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc
index 2c4da11c860..03b8a1d3ea5 100644
--- a/extra/mariabackup/xtrabackup.cc
+++ b/extra/mariabackup/xtrabackup.cc
@@ -2364,7 +2364,8 @@ xb_read_delta_metadata(const char *filepath, xb_delta_info_t *info)
 		msg("page_size is required in %s", filepath);
 		r = FALSE;
 	} else {
-		info->page_size = zip_size ? zip_size : page_size;
+		info->page_size = page_size;
+		info->zip_size = zip_size;
 	}
 
 	if (info->space_id == ULINT_UNDEFINED) {

This bug was introduced by me in MDEV-18493.

Generated at Thu Feb 08 09:49:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.