Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16519

mariabackup --backup fails with concurrent RENAME TABLE

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1, 10.2, 10.3
    • 10.2.16, 10.3.8
    • Backup
    • None

    Description

      If a table is being both renamed and written to while Mariabackup is trying to back up the server, the backup may fail. Here is a simple test case:

      let $n=10000000;
       
      exec echo $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf  --backup --target-dir=$basedir;
      eval create table t$n (a serial) engine=innodb;
       
      --disable_query_log
      while ($n) {
      eval insert into t$n values();
      let $o=$n;
      dec $n;
      eval rename table t$o to t$n;
      }
       
      --enable_query_log
      drop table t0;
      

      You will have to run mariabackup --backup concurrently while the test is running, say, ./mtr mariabackup.rename. For me, the backup fails in the file copying phase, both in 10.1 and 10.2. Here is a 10.2 invocation, with the lock_ddl_per_table parameter, which is supposed to prevent DDL operations:

      /dev/shm/10.2/extra/mariabackup/mariabackup --defaults-file=/dev/shm/10.2/mysql-test/var/my.cnf --backup --lock-ddl-per-table --target-dir=/dev/shm/bu
      

      10.2 c55de8d40bba29503773a6a56d6f13f19ca7e339

      180619 10:30:24 Connecting to MySQL server host: localhost, user: root, password: set, port: 16000, socket: /dev/shm/10.2/mysql-test/var/tmp/mysqld.1.sock
      180619 10:30:24 Locking MDL for `mysql`.`innodb_table_stats`
      180619 10:30:24 Locking MDL for `mysql`.`innodb_index_stats`
      180619 10:30:24 Locking MDL for `test`.`t9997860`
      180619 10:30:24 [01] Copying ibdata1 to /dev/shm/bu/ibdata1
      180619 10:30:24 [01]        ...done
      180619 10:30:24 [01] Copying ./mysql/innodb_table_stats.ibd to /dev/shm/bu/mysql/innodb_table_stats.ibd
      180619 10:30:24 [01]        ...done
      180619 10:30:24 [01] Copying ./mysql/innodb_index_stats.ibd to /dev/shm/bu/mysql/innodb_index_stats.ibd
      180619 10:30:24 [01]        ...done
      [01] mariabackup: error: cannot stat ./test/t9997901.ibd
      [01] mariabackup: Error: xtrabackup_copy_datafile() failed.
      [01] mariabackup: Error: failed to copy datafile.
      

      Note that an MDL was claimed to have been acquired on the table t9997860. Why did we not attempt to copy t9997860.ibd instead of trying to copy the file with a much older name t9997901.ibd? (The number counts backwards.) Also, maybe we should actually check that the MDL acquisition worked? I would not be surprised if the table had already been renamed to something else while we were trying to acquire the MDL.

      Even better, backup with concurrent RENAME TABLE should work without any --lock-ddl-per-table option. The file copying should not lead to termination this easily, and there could be some connection to the MLOG_FILE_RENAME2 records that are being parsed from the redo log.

      The backup could fail in different ways. The reason I tested this was that in one case, mariabackup --prepare reported that some files were missing. That is, the backup did not terminate with an error, but it failed to produce a complete result.

      Attachments

        Issue Links

          Activity

            People

              wlad Vladislav Vaintroub
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.