[MDEV-18589] Assertion `fil_space_t::physical_size(flags) == info.page_size' failed in xb_delta_open_matching_space upon prepare on the incremental backup Created: 2019-02-15  Updated: 2022-09-14  Resolved: 2022-09-14

Status: Closed
Project: MariaDB Server
Component/s: mariabackup
Affects Version/s: 10.4.3, 10.5.0, 10.4
Fix Version/s: 10.4.27, 10.5.18, 10.6.11, 10.7.7, 10.8.6, 10.9.4, 10.10.2

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: regression-10.4

Attachments: File backup_before_prepare_0.tar.gz     File backup_before_prepare_1.tar.gz     File backup_before_prepare_2.tar.gz     File mbackup_backup_0.log     File mbackup_backup_1.log     File mbackup_backup_2.log     File mbackup_prepare_0.log     File mbackup_prepare_1.log    
Issue Links:
Duplicate
is duplicated by MDEV-22913 Error: xtrabackup_apply_delta() upon ... Closed
is duplicated by MDEV-22918 innodb_init() returned 11 (Generic er... Closed
is duplicated by MDEV-26983 Server crash in buf_is_zeroes or asse... Closed
Problem/Incident
is caused by MDEV-18493 Remove page_size_t Closed
Relates
relates to MDEV-22913 Error: xtrabackup_apply_delta() upon ... Closed

 Description   

10.4 a081a998

 
mariabackup: /data/src/10.4/extra/mariabackup/xtrabackup.cc:4784: pfs_os_file_t xb_delta_open_matching_space(const char*, const char*, const xb_delta_info_t&, char*, size_t, bool*): Assertion `fil_space_t::physical_size(flags) == info.page_size' failed.
 
190215 15:26:44 [ERROR] mysqld got signal 6 ;
 
 
 
#6  0x00007f9a85297ee2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
 
#7  0x000055c4639ab51a in xb_delta_open_matching_space (dbname=0x7ffdd1243a30 "test", name=0x7ffdd1242800 "t8.ibd", info=..., real_name=0x7ffdd1242400 "./test/t8.ibd", real_name_len=512, success=0x7ffdd1242126) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:4784
 
#8  0x000055c4639ab9c8 in xtrabackup_apply_delta (dirname=0x55c465397440 <xtrabackup_real_incremental_dir> "/data/bld/10.4/backup_before_prepare_1/", dbname=0x7ffdd1243a30 "test", filename=0x7ffdd1244a10 "t8.ibd.delta") at /data/src/10.4/extra/mariabackup/xtrabackup.cc:4883
 
#9  0x000055c4639acc18 in xb_process_datadir (path=0x55c465397440 <xtrabackup_real_incremental_dir> "/data/bld/10.4/backup_before_prepare_1/", suffix=0x55c4646d9efc ".delta", func=0x55c4639ab5d0 <xtrabackup_apply_delta(char const*, char const*, char const*, void*)>) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:5225
 
#10 0x000055c4639acd0e in xtrabackup_apply_deltas () at /data/src/10.4/extra/mariabackup/xtrabackup.cc:5258
 
#11 0x000055c4639ae207 in xtrabackup_prepare_func (argv=0x55c467601118) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:5502
 
#12 0x000055c4639b05a3 in main_low (argv=0x55c467601118) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:6365
 
#13 0x000055c4639afdb7 in main (argc=7, argv=0x7ffdd12463e8) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:6158

Backups (before prepare) and backup logs are attached.

To reproduce, extract backup_before_prepare_0.tar.gz and backup_before_prepare_1.tar.gz and run on a debug 10.4 build

bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0
 
bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_1

The second one fails.

On a non-debug build, there is no crash, only the second prepare throws the warning in the middle:

2019-02-15 15:34:43 0 [Warning] InnoDB: Tablespace 657 was not found at ./test/#sql-6009_f.ibd when restoring a (partial?) backup. All redo log for this file will be ignored!

(but still reports OK at the end).
An attempt to run the third prepare on backup_before_prepare_2.tar.gz , however, produces contradictory diagnostics:

bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_2


2019-02-15 15:36:34 0 [Note] InnoDB: Starting final batch to recover 225 pages from redo log.
 
2019-02-15 15:36:34 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
 
[00] 2019-02-15 15:36:34 mariabackup: innodb_init() returned 39 (Data structure corruption).
 
[00] 2019-02-15 15:36:34 completed OK!

I don't know if the underlying cause is the same as for the assertion failure or different, but at least the diagnostics is a problem in itself: if it complains about data corruption and plugin initialization error, it should report "completed OK" and return 0.

I have no information yet whether any of the above affects other versions.

To reproduce from scratch

git clone https://github.com/MariaDB/randgen --branch mdev18589 rqg-mdev18589
 
cd rqg-mdev18589
 
. ./mdev18589.cmd <your basedir>

As of 10.4 9e0ed0fa, it happens for me every time when I run the test, but it's still a concurrent non-deterministic test, re-run if it doesn't fail right away.

 Comments   
Comment by Elena Stepanova [ 2021-11-05 ]

It looks much like a newer bug MDEV-26983, but the latter has a test case, so I'm closing this one.

Comment by Marko Mäkelä [ 2022-09-14 ]

I confirmed that MDEV-26983 is a duplicate of this. I can recover the backup_before_prepare_0.tar.gz and backup_before_prepare_1.tar.gz as follows, after applying the patch that I posted in MDEV-26983:

diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc
 
index fa3bb2c0279..a270a3edd04 100644
 
--- a/extra/mariabackup/xtrabackup.cc
 
+++ b/extra/mariabackup/xtrabackup.cc
 
@@ -2411,7 +2411,8 @@ xb_read_delta_metadata(const char *filepath, xb_delta_info_t *info)
 
 		msg("page_size is required in %s", filepath);
 
 		r = FALSE;
 
 	} else {
 
-		info->page_size = zip_size ? zip_size : page_size;
 
+		info->page_size = page_size;
 
+		info->zip_size = zip_size;
 
 	}
 
 
 	if (info->space_id == ULINT_UNDEFINED) {


extra/mariabackup/mariabackup --prepare --innodb-file-io-threads=1 --target-dir=/dev/shm/backup_before_prepare_0
 
extra/mariabackup/mariabackup --prepare --innodb-file-io-threads=1 --target-dir=/dev/shm/backup_before_prepare_0 --incremental-dir=/dev/shm/backup_before_prepare_1

Without the patch, the latter command would crash like this:

10.4 289105e282fb5f7865bb7ddb6226149ac5bdf2b8

 
[00] 2022-09-14 11:22:25 Applying /dev/shm/backup_before_prepare_1//mysql/transaction_registry.ibd.delta to ./mysql/transaction_registry.ibd...
 
[00] 2022-09-14 11:22:25 page size for /dev/shm/backup_before_prepare_1//test/t8.ibd.delta is 8192 bytes
 
mariabackup: /mariadb/10.4/extra/mariabackup/xtrabackup.cc:5300: pfs_os_file_t xb_delta_open_matching_space(const char *, const char *, const xb_delta_info_t &, char *, size_t, bool *): Assertion `fil_space_t::physical_size(flags) == info.page_size' failed.

Comment by Marko Mäkelä [ 2022-09-14 ]

This only affects backups that contain tables in ROW_FORMAT=COMPRESSED.

Generated at Thu Feb 08 08:45:15 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.