Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28446

mariabackup prepare fails for incrementals if a new schema is created after full backup is taken

Details

    Description

      If a new schema with at least one table is created after a full backup is taken, but before the incremental is taken, the backup cannot be restored because incremental prepare fails.

      mariabackup prepare fails with : Error on rename ..... (errno: 20 "Not a directory" )

      The issue seems to be that mariabackup prepare expects the folder for the new schema to be already there, but it isn't because the schema is created after the full is taken, so can't be found in the full backup.

      This is critical as it breaks backup restores in presence of incrementals when the above condition is met (which is all but rare).

      How to repeat:

      1. take a full backup
      2. create a new schema and table
      3. take an incremental backup
      4. restore the full backup and then the incremental backup; prepare will crash

      Attachments

        Issue Links

          Activity

            bar Alexander Barkov added a comment - - edited

            The problem is not repeatable with versions between 10.2 and 10.5.

            Stably repeatable with 10.6 and 10.9.
            (Did not check 10.7 and 10.8).

            bar Alexander Barkov added a comment - - edited The problem is not repeatable with versions between 10.2 and 10.5. Stably repeatable with 10.6 and 10.9. (Did not check 10.7 and 10.8).

            This MTR test demonstrates the problem:

            --source include/have_innodb.inc
             
            call mtr.add_suppression("InnoDB: New log files created");
             
            --let basedir=$MYSQLTEST_VARDIR/tmp/backup
            --let incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1
             
            CREATE TABLE t1 (i INT) ENGINE=InnoDB;
            INSERT INTO t1 VALUES(0);
             
            --disable_result_log
            --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf  --backup --parallel=10 --target-dir=$basedir
            --enable_result_log
             
            CREATE DATABASE test1;
            CREATE TABLE test1.t1 (a INT) ENGINE=InnoDB;
            INSERT INTO test1.t1 VALUES (1000);
             
            --disable_result_log
            --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --parallel=2 --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wai
            t-query-type=all --target-dir=$incremental_dir --incremental-basedir=$basedir
            --exec $XTRABACKUP --prepare --target-dir=$basedir
            --exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir
            --enable_result_log
             
            echo # Restore and check results;
            --let $targetdir=$basedir
            --source include/restart_and_restore.inc
            --enable_result_log
             
            SELECT * FROM t1;
            SELECT * FROM test1.t1;
             
            DROP TABLE t1;
            DROP DATABASE test1;
            --rmdir $basedir
            --rmdir $incremental_dir
            

            The output is:

            ==============================================================================
             
            TEST                                      RESULT   TIME (ms) or COMMENT
            --------------------------------------------------------------------------
             
            worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
            call mtr.add_suppression("InnoDB: New log files created");
            CREATE TABLE t1 (i INT) ENGINE=InnoDB;
            INSERT INTO t1 VALUES(0);
            CREATE DATABASE test1;
            CREATE TABLE test1.t1 (a INT) ENGINE=InnoDB;
            INSERT INTO test1.t1 VALUES (1000);
            mariabackup.incremental_backup_newdb_before_inc 'innodb' [ fail ]
                    Test ended at 2022-04-29 17:22:01
             
            CURRENT_TEST: mariabackup.incremental_backup_newdb_before_inc
            mysqltest: At line 22: exec of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
            Output from before failure:
            /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup based on MariaDB server 10.6.8-MariaDB Linux (x86_64)
            [00] 2022-04-29 17:22:00 incremental backup from 44203 is enabled.
            [00] 2022-04-29 17:22:00 cd to /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup/
            [00] 2022-04-29 17:22:00 open files limit requested 0, set to 1024
            [00] 2022-04-29 17:22:00 Removing ./test/t1.ibd
            [00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test/t1.new to ./test/t1.ibd
             
            [00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
             
            /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
            [00] FATAL ERROR: 2022-04-29 17:22:00 Can't rename /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2
             
             - saving '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/' to '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/'
            --------------------------------------------------------------------------
            The servers were restarted 0 times
            Spent 0.000 of 8 seconds executing testcases
             
            Failure: Failed 1/1 tests, 0.00% were successful.
             
            Failing test(s): mariabackup.incremental_backup_newdb_before_inc
            

            bar Alexander Barkov added a comment - This MTR test demonstrates the problem: --source include/have_innodb.inc   call mtr.add_suppression( "InnoDB: New log files created" );   --let basedir=$MYSQLTEST_VARDIR/tmp/backup --let incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1   CREATE TABLE t1 (i INT ) ENGINE=InnoDB; INSERT INTO t1 VALUES (0);   --disable_result_log --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --parallel=10 --target-dir=$basedir --enable_result_log   CREATE DATABASE test1; CREATE TABLE test1.t1 (a INT ) ENGINE=InnoDB; INSERT INTO test1.t1 VALUES (1000);   --disable_result_log --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --parallel=2 --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wai t-query-type= all --target-dir=$incremental_dir --incremental-basedir=$basedir --exec $XTRABACKUP --prepare --target-dir=$basedir --exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir --enable_result_log   echo # Restore and check results; --let $targetdir=$basedir --source include/restart_and_restore.inc --enable_result_log   SELECT * FROM t1; SELECT * FROM test1.t1;   DROP TABLE t1; DROP DATABASE test1; --rmdir $basedir --rmdir $incremental_dir The output is: ==============================================================================   TEST RESULT TIME (ms) or COMMENT --------------------------------------------------------------------------   worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 call mtr.add_suppression("InnoDB: New log files created"); CREATE TABLE t1 (i INT) ENGINE=InnoDB; INSERT INTO t1 VALUES(0); CREATE DATABASE test1; CREATE TABLE test1.t1 (a INT) ENGINE=InnoDB; INSERT INTO test1.t1 VALUES (1000); mariabackup.incremental_backup_newdb_before_inc 'innodb' [ fail ] Test ended at 2022-04-29 17:22:01   CURRENT_TEST: mariabackup.incremental_backup_newdb_before_inc mysqltest: At line 22: exec of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11 Output from before failure: /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup based on MariaDB server 10.6.8-MariaDB Linux (x86_64) [00] 2022-04-29 17:22:00 incremental backup from 44203 is enabled. [00] 2022-04-29 17:22:00 cd to /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup/ [00] 2022-04-29 17:22:00 open files limit requested 0, set to 1024 [00] 2022-04-29 17:22:00 Removing ./test/t1.ibd [00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test/t1.new to ./test/t1.ibd   [00] 2022-04-29 17:22:00 Renaming /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd   /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory") [00] FATAL ERROR: 2022-04-29 17:22:00 Can't rename /home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2   - saving '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/' to '/home/bar/maria-git/server.10.6/BUILD-DEB-ASAN/mysql-test/var/log/mariabackup.incremental_backup_newdb_before_inc-innodb/' -------------------------------------------------------------------------- The servers were restarted 0 times Spent 0.000 of 8 seconds executing testcases   Failure: Failed 1/1 tests, 0.00% were successful.   Failing test(s): mariabackup.incremental_backup_newdb_before_inc
            bar Alexander Barkov added a comment - serg , please review a patch: https://github.com/MariaDB/server/commit/fda1ccea3e1710ff35549431429872676699e578 Thanks.

            Repeatable with 10.2 with a different test:

            --source include/have_innodb.inc
            --source include/have_debug.inc
             
            call mtr.add_suppression("InnoDB: New log files created");
             
            --echo #
            --echo # MDEV-28446 mariabackup prepare fails for incrementals if a new schema is created after full backup is taken
            --echo #
             
            --let $basedir=$MYSQLTEST_VARDIR/tmp/backup
            --let $incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1
             
            CREATE TABLE t1(i INT PRIMARY KEY) ENGINE INNODB;
             
            --disable_result_log
            --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf  --backup --target-dir=$basedir
            --enable_result_log
             
            --let after_load_tablespaces=BEGIN NOT ATOMIC CREATE DATABASE test1; CREATE TABLE test1.t1 ENGINE=INNODB SELECT UUID() from test.seq_1_to_10000; END
             
            --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf  --backup --target-dir=$incremental_dir --incremental-basedir=$basedir --dbug=+d,mariabackup_events,mari
            --let after_load_tablespaces=
            --disable_result_log
            --echo # Prepare full backup, apply incremental one
            --exec $XTRABACKUP --apply-log-only --prepare --target-dir=$basedir
            --exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir
            --enable_result_log
             
            --let $targetdir=$basedir
            --source include/restart_and_restore.inc
            --enable_result_log
             
            SELECT COUNT(*) FROM test.t1;
            SELECT COUNT(*) FROM test1.t1;
             
            DROP TABLE t1;
            DROP DATABASE test1;
            --rmdir $basedir
            --rmdir $incremental_dir
            

            ....
            [00] 2022-04-30 08:35:32 Redo log (from LSN 1631081 to 2536974) was copied.
            [00] 2022-04-30 08:35:32 completed OK!
            mysqltest: At line 26: exec of '/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11
            Output from before failure:
            /home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup based on MariaDB server 10.2.44-MariaDB Linux (x86_64)
            [00] 2022-04-30 08:35:33 incremental backup from 1631081 is enabled.
            [00] 2022-04-30 08:35:33 cd to /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup/
            [00] 2022-04-30 08:35:33 open files limit requested 0, set to 1024
            [00] 2022-04-30 08:35:33 Renaming /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd
             
            /home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory")
            [00] FATAL ERROR: 2022-04-30 08:35:33 Can't rename /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2
             
             - saving '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/' to '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/'
            --------------------------------------------------------------------------
            The servers were restarted 0 times
            Spent 0.000 of 9 seconds executing testcases
             
            Failure: Failed 1/1 tests, 0.00% were successful.
            

            bar Alexander Barkov added a comment - Repeatable with 10.2 with a different test: --source include/have_innodb.inc --source include/have_debug.inc   call mtr.add_suppression( "InnoDB: New log files created" );   --echo # --echo # MDEV-28446 mariabackup prepare fails for incrementals if a new schema is created after full backup is taken --echo #   --let $basedir=$MYSQLTEST_VARDIR/tmp/backup --let $incremental_dir=$MYSQLTEST_VARDIR/tmp/backup_inc1   CREATE TABLE t1(i INT PRIMARY KEY ) ENGINE INNODB;   --disable_result_log --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$basedir --enable_result_log   --let after_load_tablespaces=BEGIN NOT ATOMIC CREATE DATABASE test1; CREATE TABLE test1.t1 ENGINE=INNODB SELECT UUID() from test.seq_1_to_10000; END   --exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$incremental_dir --incremental-basedir=$basedir --dbug=+d,mariabackup_events,mari --let after_load_tablespaces= --disable_result_log --echo # Prepare full backup, apply incremental one --exec $XTRABACKUP --apply-log-only --prepare --target-dir=$basedir --exec $XTRABACKUP --prepare --target-dir=$basedir --incremental-dir=$incremental_dir --enable_result_log   --let $targetdir=$basedir --source include/restart_and_restore.inc --enable_result_log   SELECT COUNT (*) FROM test.t1; SELECT COUNT (*) FROM test1.t1;   DROP TABLE t1; DROP DATABASE test1; --rmdir $basedir --rmdir $incremental_dir .... [00] 2022-04-30 08:35:32 Redo log (from LSN 1631081 to 2536974) was copied. [00] 2022-04-30 08:35:32 completed OK! mysqltest: At line 26: exec of '/home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup --prepare --target-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup --incremental-dir=/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1 2>&1' failed, error: 256, status: 1, errno: 11 Output from before failure: /home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup based on MariaDB server 10.2.44-MariaDB Linux (x86_64) [00] 2022-04-30 08:35:33 incremental backup from 1631081 is enabled. [00] 2022-04-30 08:35:33 cd to /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup/ [00] 2022-04-30 08:35:33 open files limit requested 0, set to 1024 [00] 2022-04-30 08:35:33 Renaming /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd   /home/bar/maria-git/server.10.2/BUILD-DEB/extra/mariabackup/mariabackup: Error on rename of '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new' to './test1/t1.ibd' (errno: 20 "Not a directory") [00] FATAL ERROR: 2022-04-30 08:35:33 Can't rename /home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/tmp/backup_inc1//test1/t1.new to ./test1/t1.ibd errno 2   - saving '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/' to '/home/bar/maria-git/server.10.2/BUILD-DEB/mysql-test/var/log/mariabackup.AAA-innodb/' -------------------------------------------------------------------------- The servers were restarted 0 times Spent 0.000 of 9 seconds executing testcases   Failure: Failed 1/1 tests, 0.00% were successful.
            bar Alexander Barkov added a comment - serg , this is a patch for 10.2: https://github.com/MariaDB/server/commit/5d1ae4f4c71743ce60e7ab218b33deecec4c4050

            5d1ae4f4c71743ce60e7ab218b33deecec4c4050 is ok to push

            serg Sergei Golubchik added a comment - 5d1ae4f4c71743ce60e7ab218b33deecec4c4050 is ok to push

            Pushed to 10.2.
            Waiting for a merge to add a 10.6 specific test.

            bar Alexander Barkov added a comment - Pushed to 10.2. Waiting for a merge to add a 10.6 specific test.

            People

              bar Alexander Barkov
              rpizzi Rick Pizzi (Inactive)
              Votes:
              7 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.