Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22913

Error: xtrabackup_apply_delta() upon preparing incremental backup

Details

    Description

      This report is about a non-debug 10.5 build. Debug builds fail with an assertion failure from MDEV-18589. However, according to MDEV-18589, on non-debug builds it didn't have the failure described here, so I'm not sure it's the same problem.

      10.5 e290e5a7

      [00] 2020-06-16 15:12:21 page size for /dev/shm/var_xbackup_apply/backup_2//ibdata1.delta is 16384 bytes
      [00] 2020-06-16 15:12:21 Applying /dev/shm/var_xbackup_apply/backup_2//ibdata1.delta to ./ibdata1...
      [00] 2020-06-16 15:12:21 page size for /dev/shm/var_xbackup_apply/backup_2//test/alt_t1.ibd.delta is 8192 bytes
      [00] 2020-06-16 15:12:21 Can't create file ./test/alt_t1.ibd
      [00] 2020-06-16 15:12:21 error: can't open ./test/alt_t1.ibd
      [00] 2020-06-16 15:12:21 Error: xtrabackup_apply_delta(): failed to apply /dev/shm/var_xbackup_apply/backup_2//test/alt_t1.ibd.delta to ./test/alt_t1.ibd.
      

      To reproduce:

      git clone https://github.com/MariaDB/randgen --branch mdev22913 rqg-mdev22913
      rqg-mdev22913
      perl ./runall-trials.pl --basedir=<basedir> --trials=5 --skip-gendata --duration=150 --threads=4 --scenario=MariaBackupIncremental --grammar=mdev22913.yy --vardir=/dev/shm/var_mdev22913
      

      Remember to set the basedir on the command line.
      For reproducing, use a 10.5 non-debug build.

      The grammar in the test above only contains single query:

      query:
        CREATE OR REPLACE TABLE { 't'.$prng->int(1,9) } (b BIT) ENGINE=InnoDB ROW_FORMAT COMPRESSED;
      

      It currently fails every time for me, but there is always room for non-determinism.
      The command line above runs the test 5 times, until it fails (in any way). Given the number of problems with incremental backup (or even any backup), it can occasionally fail with a different problem than the one described here. If you want to ignore all other ones and only focus on this one, run with the extra option

      --output="Error: xtrabackup_apply_delta"
      

      Then it will ignore other errors and automatically continue running the configured 5 trials until it hits this exact failure.

      Logs are attached.

      In the community server, this failure was only observed on 10.5 non-debug.

      10.4 and 10.5 debug and sometimes 10.4 non-debug fail with MDEV-18589 instead.
      10.4 non-debug also fails non-deterministically with

      2020-06-16 20:13:22 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
      [00] FATAL ERROR: 2020-06-16 20:13:23 mariabackup: innodb_init() returned 11 (Generic error).
      

      (to be filed separately).

      10.2 non-debug fails non-deterministically with MDEV-18550.
      10.3 non-debug fails non-deterministically with

      [00] FATAL ERROR: 2020-06-16 20:03:57 xtrabackup_copy_logfile() failed.
      

      Attachments

        1. mbackup_backup_0.log
          2.01 MB
        2. mbackup_backup_1.log
          1.48 MB
        3. mbackup_backup_2.log
          568 kB
        4. mbackup_backup_3.log
          2.60 MB
        5. mbackup_prepare_0.log
          2.22 MB
        6. mbackup_prepare_1.log
          1.65 MB
        7. mbackup_prepare_2.log
          14 kB

        Issue Links

          Activity

            I suspected that this might have been fixed by MDEV-28473, but I was able to reproduce the following:

            10.5 fb70bb44d096f4117e9454740aa4fa0651a1a69e

            # 2022-09-14T10:30:37 [462207] | [00] 2022-09-14 10:30:36 mariabackup: Generating a list of tablespaces
            # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
            # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
            # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
            # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: Cannot open datafile for read-only: './test/t5.ibd' OS error: 71
            # 2022-09-14T10:30:37 [462207] | [00] FATAL ERROR: 2022-09-14 10:30:36 Can't open datafile test/t5
            # 2022-09-14T10:30:37 [462207] ----------------------------------
            # 2022-09-14T10:30:37 [462287][ERROR] Executor::MySQL::execute: Failed to reconnect after getting STATUS_SERVER_CRASHED
            # 2022-09-14T10:30:37 [462287] Executor::MySQL::execute: Query: CREATE /* QNO 6205 CON_ID 9 */ OR REPLACE TABLE t8 (b BIT) ENGINE=InnoDB ROW_FORMAT COMPRESSED failed: 2013 Lost connection to server during query (STATUS_SERVER_CRASHED)
            

            marko Marko Mäkelä added a comment - I suspected that this might have been fixed by MDEV-28473 , but I was able to reproduce the following: 10.5 fb70bb44d096f4117e9454740aa4fa0651a1a69e # 2022-09-14T10:30:37 [462207] | [00] 2022-09-14 10:30:36 mariabackup: Generating a list of tablespaces # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: Operating system error number 2 in a file operation. # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: The error means the system cannot find the path specified. # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them. # 2022-09-14T10:30:37 [462207] | 2022-09-14 10:30:36 0 [ERROR] InnoDB: Cannot open datafile for read-only: './test/t5.ibd' OS error: 71 # 2022-09-14T10:30:37 [462207] | [00] FATAL ERROR: 2022-09-14 10:30:36 Can't open datafile test/t5 # 2022-09-14T10:30:37 [462207] ---------------------------------- # 2022-09-14T10:30:37 [462287][ERROR] Executor::MySQL::execute: Failed to reconnect after getting STATUS_SERVER_CRASHED # 2022-09-14T10:30:37 [462287] Executor::MySQL::execute: Query: CREATE /* QNO 6205 CON_ID 9 */ OR REPLACE TABLE t8 (b BIT) ENGINE=InnoDB ROW_FORMAT COMPRESSED failed: 2013 Lost connection to server during query (STATUS_SERVER_CRASHED)

            I could reproduce a failure of this on 10.5 without the fix of MDEV-18589. After the fix, the test passed:

            10.5 fb70bb44d096f4117e9454740aa4fa0651a1a69e + MDEV-18589

            # 2022-09-14T12:28:37 [511102] For table `test`.`t9` : status : OK
            # 2022-09-14T12:28:37 [511102] #-------------------------#
            # 2022-09-14T12:28:37 [511102] #-- Stopping the server --#
            # 2022-09-14T12:28:37 [511102] #-------------------------#
            # 2022-09-14T12:28:37 [511102] Stopping server on port 19300
            # 2022-09-14T12:28:38 [511102] Server has been stopped
            # 2022-09-14T12:28:38 [511102] [511102] /mariadb/rqg-mdev22913/runall-new.pl will exit with exit status STATUS_OK (0)
            # 2022-09-14T12:28:38 [509225] Trial 5 ended with exit status STATUS_OK (0)
            # 2022-09-14T12:28:38 [509225] Exit status STATUS_OK and the list of desired status codes is empty, result will be ignored
            # 2022-09-14T12:28:38 [509225] ./runall-trials.pl will exit with exit status FAILED TO REPRODUCE (0)
            

            marko Marko Mäkelä added a comment - I could reproduce a failure of this on 10.5 without the fix of MDEV-18589 . After the fix, the test passed: 10.5 fb70bb44d096f4117e9454740aa4fa0651a1a69e + MDEV-18589 # 2022-09-14T12:28:37 [511102] For table `test`.`t9` : status : OK # 2022-09-14T12:28:37 [511102] #-------------------------# # 2022-09-14T12:28:37 [511102] #-- Stopping the server --# # 2022-09-14T12:28:37 [511102] #-------------------------# # 2022-09-14T12:28:37 [511102] Stopping server on port 19300 # 2022-09-14T12:28:38 [511102] Server has been stopped # 2022-09-14T12:28:38 [511102] [511102] /mariadb/rqg-mdev22913/runall-new.pl will exit with exit status STATUS_OK (0) # 2022-09-14T12:28:38 [509225] Trial 5 ended with exit status STATUS_OK (0) # 2022-09-14T12:28:38 [509225] Exit status STATUS_OK and the list of desired status codes is empty, result will be ignored # 2022-09-14T12:28:38 [509225] ./runall-trials.pl will exit with exit status FAILED TO REPRODUCE (0)

            People

              marko Marko Mäkelä
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.