Marko Mäkelä added a comment - 2019-01-17 11:23 I think that this should likely affect mariabackup 10.1 already. The code should be equivalent between 10.2, 10.3, and 10.4.

Thirunarayanan Balathandayuthapani added a comment - 2019-01-22 12:53 MLOG_FILE_RENAME2 doesn't replay in mariabackup during prepare phase. Because file path stored in mariabackup as "test/t1.ibd". But redo log records stores the file path as "./test/t1.ibd". So fil_op_replay_rename() does string matching and fails to do the renaming of the file.

Thirunarayanan Balathandayuthapani added a comment - 2019-01-22 12:57 After preparing full backup phase: Problematic space id is t8#P#ver_pn. Space id is 903. It has only 4 pages. While preparing incremental backup phase: t8#P#ver_pn.ibd has only .delta file. It is empty. Based on .meta file, mariabackup prepare creates new ibd file with new space id(1239) and name it as t8#P#ver_pn.ibd Redo log now tries to access 96th page. So it crashes. Basically backup phase went wrong during incremental backup i think.

Thirunarayanan Balathandayuthapani added a comment - 2019-01-22 17:18 The following test case repeats the issue: --source include/have_debug.inc --source include/have_innodb.inc --source include/have_partition.inc   call mtr.add_suppression("InnoDB: New log files created");   let $basedir=$MYSQLTEST_VARDIR/tmp/backup; let $incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1; let $backuplog=$MYSQLTEST_VARDIR/tmp/backup.log;   CREATE TABLE t1(i INT) ENGINE INNODB; alter table t1 partition by hash(i) partitions 4; CREATE TABLE t2(i INT PRIMARY KEY) ENGINE INNODB; CREATE TABLE t3(i INT) ENGINE INNODB;   INSERT INTO t1 values(1), (2), (3), (4), (5), (6), (7), (8), (9); insert into t1 select * from t1; insert into t1 select * from t1;   echo # Create full backup , modify table, then create incremental/differential backup; --disable_result_log exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$basedir; --enable_result_log drop table t1; create table t4(f1 int not null, f2 int not null)engine=innodb; insert into t4 values(1, 2), (2, 2), (3, 3), (5, 5), (6, 6), (4, 4), (9, 9); insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; alter table t4 partition by hash(f1) partitions 4; rename table t4 to t1;   --echo # XTRABACKUP INCREMENTAL exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$incremental_dir --incremental-basedir=$basedir;   --echo # XTRABACKUP PREPARE exec $XTRABACKUP --apply-log-only --prepare --target-dir=$basedir;   --echo # XTRABACKUP INCREMENTAL PREPARE exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir;   let $targetdir=$basedir; -- source include/restart_and_restore.inc --enable_result_log   select count(*) from t1; drop table t2, t1, t3;

Thirunarayanan Balathandayuthapani added a comment - 2019-01-23 20:24 --source include/have_debug.inc   call mtr.add_suppression("InnoDB: New log files created");   let $basedir=$MYSQLTEST_VARDIR/tmp/backup; let $incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1; let $backuplog=$MYSQLTEST_VARDIR/tmp/backup.log;   CREATE TABLE t1(i INT) ENGINE INNODB; CREATE TABLE t2(i INT PRIMARY KEY) ENGINE INNODB; CREATE TABLE t3(i INT) ENGINE INNODB;   --echo # FULL XTRABACKUP   --disable_result_log exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$basedir; --enable_result_log   create table t4(f1 int not null, f2 int not null)engine=innodb; insert into t4 values(1, 2), (2, 2), (3, 3), (5, 5), (6, 6), (4, 4), (9, 9); insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; insert into t4 select * from t4; rename table t4 to t7; select count(*) from t7;   --echo # XTRABACKUP INCREMENTAL exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$incremental_dir --incremental-basedir=$basedir   --echo # XTRABACKUP PREPARE exec $XTRABACKUP --apply-log-only --prepare --target-dir=$basedir;   --echo # XTRABACKUP INCREMENTAL PREPARE --error 0, 1, 134 exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir --dbug="+d,ib_log" 2> $backuplog;   let $targetdir=$basedir; -- source include/restart_and_restore.inc --enable_result_log   select count(*) from t7; drop table t1, t2, t7, t3; The above is the another test case to repeat the failure. There is no need for partition or so. INCREMENTAL BACKUP: =================== xtrabackup tries to copy delta page of t7 tablespace when all pages are in the buffer pool. So lsn is zero in the file. It stores nothing in .delta file. But t7 tablespace has 12 pages in buffer pool. At the end, incremental backup contains t7.delta which has nothing and t7.meta which has page_size, space id. INCREMENTAL PREPARE PHASE: =========================== Mariabackup tries to apply delta for all tablespace. When it encounters t7 tablespace. Target directory doesn't have t7. So it creates new tablespace as t7 with space id stored in .meta file. It sets the file size to 4. (Default file size). redo log for MLOG_FILE_NAME will fail to load the tablespace to memory for t4 tablespace. Because fil_ibd_discovery doesn't find any t4 tablespace. redo log for FSP_SIZE will try to search the tablespace for the respective id. Since tablespace is not loaded yet for MLOG_FILE_NAME. It will fail to set. In applying redo log phase, redo log for the tablespace t7 and its respective pages. (12 pages in our case). So it tries to fetch the pages at all once to apply the redo log. Since mariabackup prepare creates only 4 pages. It leads to assert while accessing the 5th page. Fix: ==== .meta file should contain file size info. Instead of setting the default file size, mariabackup prepare should create respective file size present in meta file.

Thirunarayanan Balathandayuthapani added a comment - 2019-01-24 07:51 Patch is in bb-10.2- MDEV-18194 https://github.com/MariaDB/server/commit/376f3a2ea6347397b6be9378042b8fd871eb8ea7

Marko Mäkelä added a comment - 2019-01-28 07:03 The fix looks OK to me. With the additional patch that effectively disables MDEV-12103 , it passed the additional testing by elenst . Alas, I believe that we must first fix the corner case of crash recovery that was broken in MDEV-12103 . Without fixing that, this fix could make matters worse. The issue that MDEV-12103 caused is: If multiple redo log checkpoints ‘overlap’ each other, then crash recovery would fail to start parsing the log records near the first available MLOG_CHECKPOINT record, and start from the last record instead. Due to this, some recently created, renamed or deleted tablespaces could be unavailable during redo log processing in crash recovery.