If a new schema with at least one table is created after a full backup is taken, but before the incremental is taken, the backup cannot be restored because incremental prepare fails.
mariabackup prepare fails with : Error on rename ..... (errno: 20 "Not a directory" )
The issue seems to be that mariabackup prepare expects the folder for the new schema to be already there, but it isn't because the schema is created after the full is taken, so can't be found in the full backup.
This is critical as it breaks backup restores in presence of incrementals when the above condition is met (which is all but rare).
How to repeat:
1. take a full backup
2. create a new schema and table
3. take an incremental backup
4. restore the full backup and then the incremental backup; prepare will crash
Attachments
Issue Links
relates to
MDEV-23335MariaBackup Incremental Does Not Reflect Dropped/Created Databases
If a new schema with at least one table is created after a full backup is taken, but before the incremental is taken, the backup cannot be restored because incremental prepare fails.
mariabackup prepare fails with : Error on rename ..... (errno: 20 "Not a directory" )
The issue seems to be that mariabackup prepare expects the folder for the new schema to be already there, but it isn't because the schema is created after the full is taken, so can't be found in the full backup.
This is critical as it breaks backup restores in presence of incrementals when the above condition is met (which is all but rare).
How to repeat:
1. take a full backup
2. create a new schema and table
3. take an incremental backup
4. restore the full backup and then the incremental backup; prepare will crash
cannot rename table: "errno: 20 "Not a directory" on the schema and table created after the full.
If a new schema with at least one table is created after a full backup is taken, but before the incremental is taken, the backup cannot be restored because incremental prepare fails.
mariabackup prepare fails with : Error on rename ..... (errno: 20 "Not a directory" )
The issue seems to be that mariabackup prepare expects the folder for the new schema to be already there, but it isn't because the schema is created after the full is taken, so can't be found in the full backup.
This is critical as it breaks backup restores in presence of incrementals when the above condition is met (which is all but rare).
How to repeat:
1. take a full backup
2. create a new schema and table
3. take an incremental backup
4. restore the full backup and then the incremental backup; prepare will crash
The problem is not repeatable with versions between 10.2 and 10.5.
Stably repeatable with 10.6 and 10.9.
(Did not check 10.7 and 10.8).
Alexander Barkov
added a comment - - edited The problem is not repeatable with versions between 10.2 and 10.5.
Stably repeatable with 10.6 and 10.9.
(Did not check 10.7 and 10.8).
mysqltest: At line 22: exec of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
Output from before failure:
/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup based on MariaDB server 10.6.8-MariaDB Linux (x86_64)
[00] 2022-04-29 17:22:00 incremental backup from 44203 is enabled.
[00] 2022-04-29 17:22:00 cd to /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup/
[00] 2022-04-29 17:22:00 open files limit requested 0, set to 1024
[00] 2022-04-29 17:22:00 Removing ./test/t1.ibd
[00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test/t1.new to ./test/t1.ibd
[00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
- saving '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/' to '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/'
Alexander Barkov
added a comment - This MTR test demonstrates the problem:
--source include/have_innodb.inc
call mtr.add_suppression( "InnoDB: New log files created" );
--let basedir=$MYSQLTEST_VARDIR/tmp/backup
--let incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1
CREATE TABLE t1 (i INT ) ENGINE=InnoDB;
INSERT INTO t1 VALUES (0);
--disable_result_log
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --parallel=10 --target-dir=$basedir
--enable_result_log
CREATE DATABASE test1;
CREATE TABLE test1.t1 (a INT ) ENGINE=InnoDB;
INSERT INTO test1.t1 VALUES (1000);
--disable_result_log
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --parallel=2 --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wai
t-query-type= all --target-dir=$incremental_dir --incremental-basedir=$basedir
--exec $XTRABACKUP --prepare --target-dir=$basedir
--exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir
--enable_result_log
echo # Restore and check results;
--let $targetdir=$basedir
--source include/restart_and_restore.inc
--enable_result_log
SELECT * FROM t1;
SELECT * FROM test1.t1;
DROP TABLE t1;
DROP DATABASE test1;
--rmdir $basedir
--rmdir $incremental_dir
The output is:
==============================================================================
TEST RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
call mtr.add_suppression("InnoDB: New log files created");
CREATE TABLE t1 (i INT) ENGINE=InnoDB;
INSERT INTO t1 VALUES(0);
CREATE DATABASE test1;
CREATE TABLE test1.t1 (a INT) ENGINE=InnoDB;
INSERT INTO test1.t1 VALUES (1000);
mariabackup.incremental_backup_newdb_before_inc 'innodb' [ fail ]
Test ended at 2022-04-29 17:22:01
CURRENT_TEST: mariabackup.incremental_backup_newdb_before_inc
mysqltest: At line 22: exec of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
Output from before failure:
/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup based on MariaDB server 10.6.8-MariaDB Linux (x86_64)
[00] 2022-04-29 17:22:00 incremental backup from 44203 is enabled.
[00] 2022-04-29 17:22:00 cd to /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup/
[00] 2022-04-29 17:22:00 open files limit requested 0, set to 1024
[00] 2022-04-29 17:22:00 Removing ./test/t1.ibd
[00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test/t1.new to ./test/t1.ibd
[00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
[00] FATAL ERROR: 2022-04-29 17:22:00 Can't rename /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2
- saving '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/' to '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/'
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 0.000 of 8 seconds executing testcases
Failure: Failed 1/1 tests, 0.00% were successful.
Failing test(s): mariabackup.incremental_backup_newdb_before_inc
Alexander Barkov
added a comment - serg , please review a patch:
https://github.com/MariaDB/server/commit/fda1ccea3e1710ff35549431429872676699e578
Thanks.
[00] 2022-04-30 08:35:32 Redo log (from LSN 1631081 to 2536974) was copied.
[00] 2022-04-30 08:35:32 completed OK!
mysqltest: At line 26: exec of '/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
Output from before failure:
/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup based on MariaDB server 10.2.44-MariaDB Linux (x86_64)
[00] 2022-04-30 08:35:33 incremental backup from 1631081 is enabled.
[00] 2022-04-30 08:35:33 cd to /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup/
[00] 2022-04-30 08:35:33 open files limit requested 0, set to 1024
[00] 2022-04-30 08:35:33 Renaming /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
- saving '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/' to '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/'
Alexander Barkov
added a comment - Repeatable with 10.2 with a different test:
--source include/have_innodb.inc
--source include/have_debug.inc
call mtr.add_suppression( "InnoDB: New log files created" );
--echo #
--echo # MDEV-28446 mariabackup prepare fails for incrementals if a new schema is created after full backup is taken
--echo #
--let $basedir=$MYSQLTEST_VARDIR/tmp/backup
--let $incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1
CREATE TABLE t1(i INT PRIMARY KEY ) ENGINE INNODB;
--disable_result_log
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$basedir
--enable_result_log
--let after_load_tablespaces=BEGIN NOT ATOMIC CREATE DATABASE test1; CREATE TABLE test1.t1 ENGINE=INNODB SELECT UUID() from test.seq_1_to_10000; END
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$incremental_dir --incremental-basedir=$basedir --dbug=+d,mariabackup_events,mari
--let after_load_tablespaces=
--disable_result_log
--echo # Prepare full backup, apply incremental one
--exec $XTRABACKUP --apply-log-only --prepare --target-dir=$basedir
--exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir
--enable_result_log
--let $targetdir=$basedir
--source include/restart_and_restore.inc
--enable_result_log
SELECT COUNT (*) FROM test.t1;
SELECT COUNT (*) FROM test1.t1;
DROP TABLE t1;
DROP DATABASE test1;
--rmdir $basedir
--rmdir $incremental_dir
....
[00] 2022-04-30 08:35:32 Redo log (from LSN 1631081 to 2536974) was copied.
[00] 2022-04-30 08:35:32 completed OK!
mysqltest: At line 26: exec of '/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
Output from before failure:
/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup based on MariaDB server 10.2.44-MariaDB Linux (x86_64)
[00] 2022-04-30 08:35:33 incremental backup from 1631081 is enabled.
[00] 2022-04-30 08:35:33 cd to /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup/
[00] 2022-04-30 08:35:33 open files limit requested 0, set to 1024
[00] 2022-04-30 08:35:33 Renaming /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
[00] FATAL ERROR: 2022-04-30 08:35:33 Can't rename /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2
- saving '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/' to '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/'
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 0.000 of 9 seconds executing testcases
Failure: Failed 1/1 tests, 0.00% were successful.
The problem is not repeatable with versions between 10.2 and 10.5.
Stably repeatable with 10.6 and 10.9.
(Did not check 10.7 and 10.8).