[MDEV-28446] mariabackup prepare fails for incrementals if a new schema is created after full backup is taken Created: 2022-04-22  Updated: 2022-11-09  Resolved: 2022-05-11

Status: Closed
Project: MariaDB Server
Component/s: mariabackup
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9
Fix Version/s: 10.2.44, 10.3.35, 10.4.25, 10.5.16, 10.6.8, 10.7.4, 10.8.3, 10.9.1

Type: Bug Priority: Critical
Reporter: Rick Pizzi Assignee: Alexander Barkov
Resolution: Fixed Votes: 7
Labels: None

Issue Links:
Relates
relates to MDEV-23335 MariaBackup Incremental Does Not Refl... Closed

 Description   

If a new schema with at least one table is created after a full backup is taken, but before the incremental is taken, the backup cannot be restored because incremental prepare fails.

mariabackup prepare fails with : Error on rename ..... (errno: 20 "Not a directory" )

The issue seems to be that mariabackup prepare expects the folder for the new schema to be already there, but it isn't because the schema is created after the full is taken, so can't be found in the full backup.

This is critical as it breaks backup restores in presence of incrementals when the above condition is met (which is all but rare).

How to repeat:

1. take a full backup
2. create a new schema and table
3. take an incremental backup
4. restore the full backup and then the incremental backup; prepare will crash

 Comments   
Comment by Alexander Barkov [ 2022-04-28 ]

The problem is not repeatable with versions between 10.2 and 10.5.

Stably repeatable with 10.6 and 10.9.
(Did not check 10.7 and 10.8).

Comment by Alexander Barkov [ 2022-04-29 ]

This MTR test demonstrates the problem:

--source include/have_innodb.inc
 
 
 
call mtr.add_suppression("InnoDB: New log files created");
 
 
 
--let basedir=$MYSQLTEST_VARDIR/tmp/backup
 
--let incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1
 
 
 
CREATE TABLE t1 (i INT) ENGINE=InnoDB;
 
INSERT INTO t1 VALUES(0);
 
 
 
--disable_result_log
 
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf  --backup --parallel=10 --target-dir=$basedir
 
--enable_result_log
 
 
 
CREATE DATABASE test1;
 
CREATE TABLE test1.t1 (a INT) ENGINE=InnoDB;
 
INSERT INTO test1.t1 VALUES (1000);
 
 
 
--disable_result_log
 
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --parallel=2 --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wai
 
t-query-type=all --target-dir=$incremental_dir --incremental-basedir=$basedir
 
--exec $XTRABACKUP --prepare --target-dir=$basedir
 
--exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir
 
--enable_result_log
 
 
 
echo # Restore and check results;
 
--let $targetdir=$basedir
 
--source include/restart_and_restore.inc
 
--enable_result_log
 
 
 
SELECT * FROM t1;
 
SELECT * FROM test1.t1;
 
 
 
DROP TABLE t1;
 
DROP DATABASE test1;
 
--rmdir $basedir
 
--rmdir $incremental_dir

The output is:

==============================================================================
 
 
 
TEST                                      RESULT   TIME (ms) or COMMENT
 
--------------------------------------------------------------------------
 
 
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
 
call mtr.add_suppression("InnoDB: New log files created");
 
CREATE TABLE t1 (i INT) ENGINE=InnoDB;
 
INSERT INTO t1 VALUES(0);
 
CREATE DATABASE test1;
 
CREATE TABLE test1.t1 (a INT) ENGINE=InnoDB;
 
INSERT INTO test1.t1 VALUES (1000);
 
mariabackup.incremental_backup_newdb_before_inc 'innodb' [ fail ]
 
        Test ended at 2022-04-29 17:22:01
 
 
 
CURRENT_TEST: mariabackup.incremental_backup_newdb_before_inc
 
mysqltest: At line 22: exec of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
 
Output from before failure:
 
/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup based on MariaDB server 10.6.8-MariaDB Linux (x86_64)
 
[00] 2022-04-29 17:22:00 incremental backup from 44203 is enabled.
 
[00] 2022-04-29 17:22:00 cd to /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup/
 
[00] 2022-04-29 17:22:00 open files limit requested 0, set to 1024
 
[00] 2022-04-29 17:22:00 Removing ./test/t1.ibd
 
[00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test/t1.new to ./test/t1.ibd
 
 
 
[00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
 
 
 
/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
 
[00] FATAL ERROR: 2022-04-29 17:22:00 Can't rename /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2
 
 
 
 - saving '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/' to '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/'
 
--------------------------------------------------------------------------
 
The servers were restarted 0 times
 
Spent 0.000 of 8 seconds executing testcases
 
 
 
Failure: Failed 1/1 tests, 0.00% were successful.
 
 
 
Failing test(s): mariabackup.incremental_backup_newdb_before_inc

Comment by Alexander Barkov [ 2022-04-29 ]

serg, please review a patch:
https://github.com/MariaDB/server/commit/fda1ccea3e1710ff35549431429872676699e578
Thanks.

Comment by Alexander Barkov [ 2022-04-30 ]

Repeatable with 10.2 with a different test:

--source include/have_innodb.inc
 
--source include/have_debug.inc
 
 
 
call mtr.add_suppression("InnoDB: New log files created");
 
 
 
--echo #
 
--echo # MDEV-28446 mariabackup prepare fails for incrementals if a new schema is created after full backup is taken
 
--echo #
 
 
 
--let $basedir=$MYSQLTEST_VARDIR/tmp/backup
 
--let $incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1
 
 
 
CREATE TABLE t1(i INT PRIMARY KEY) ENGINE INNODB;
 
 
 
--disable_result_log
 
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf  --backup --target-dir=$basedir
 
--enable_result_log
 
 
 
--let after_load_tablespaces=BEGIN NOT ATOMIC CREATE DATABASE test1; CREATE TABLE test1.t1 ENGINE=INNODB SELECT UUID() from test.seq_1_to_10000; END
 
 
 
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf  --backup --target-dir=$incremental_dir --incremental-basedir=$basedir --dbug=+d,mariabackup_events,mari
 
--let after_load_tablespaces=
 
--disable_result_log
 
--echo # Prepare full backup, apply incremental one
 
--exec $XTRABACKUP --apply-log-only --prepare --target-dir=$basedir
 
--exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir
 
--enable_result_log
 
 
 
--let $targetdir=$basedir
 
--source include/restart_and_restore.inc
 
--enable_result_log
 
 
 
SELECT COUNT(*) FROM test.t1;
 
SELECT COUNT(*) FROM test1.t1;
 
 
 
DROP TABLE t1;
 
DROP DATABASE test1;
 
--rmdir $basedir
 
--rmdir $incremental_dir


....
 
[00] 2022-04-30 08:35:32 Redo log (from LSN 1631081 to 2536974) was copied.
 
[00] 2022-04-30 08:35:32 completed OK!
 
mysqltest: At line 26: exec of '/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
 
Output from before failure:
 
/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup based on MariaDB server 10.2.44-MariaDB Linux (x86_64)
 
[00] 2022-04-30 08:35:33 incremental backup from 1631081 is enabled.
 
[00] 2022-04-30 08:35:33 cd to /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup/
 
[00] 2022-04-30 08:35:33 open files limit requested 0, set to 1024
 
[00] 2022-04-30 08:35:33 Renaming /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
 
 
 
/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
 
[00] FATAL ERROR: 2022-04-30 08:35:33 Can't rename /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2
 
 
 
 - saving '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/' to '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/'
 
--------------------------------------------------------------------------
 
The servers were restarted 0 times
 
Spent 0.000 of 9 seconds executing testcases
 
 
 
Failure: Failed 1/1 tests, 0.00% were successful.

Comment by Alexander Barkov [ 2022-04-30 ]

serg, this is a patch for 10.2:

https://github.com/MariaDB/server/commit/5d1ae4f4c71743ce60e7ab218b33deecec4c4050

Comment by Sergei Golubchik [ 2022-05-01 ]

5d1ae4f4c71743ce60e7ab218b33deecec4c4050 is ok to push

Comment by Alexander Barkov [ 2022-05-02 ]

Pushed to 10.2.
Waiting for a merge to add a 10.6 specific test.

Generated at Thu Feb 08 10:00:48 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.