Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18589

Assertion `fil_space_t::physical_size(flags) == info.page_size' failed in xb_delta_open_matching_space upon prepare on the incremental backup

    XMLWordPrintable

Details

    Description

      10.4 a081a998

      mariabackup: /data/src/10.4/extra/mariabackup/xtrabackup.cc:4784: pfs_os_file_t xb_delta_open_matching_space(const char*, const char*, const xb_delta_info_t&, char*, size_t, bool*): Assertion `fil_space_t::physical_size(flags) == info.page_size' failed.
      190215 15:26:44 [ERROR] mysqld got signal 6 ;
       
      #6  0x00007f9a85297ee2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
      #7  0x000055c4639ab51a in xb_delta_open_matching_space (dbname=0x7ffdd1243a30 "test", name=0x7ffdd1242800 "t8.ibd", info=..., real_name=0x7ffdd1242400 "./test/t8.ibd", real_name_len=512, success=0x7ffdd1242126) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:4784
      #8  0x000055c4639ab9c8 in xtrabackup_apply_delta (dirname=0x55c465397440 <xtrabackup_real_incremental_dir> "/data/bld/10.4/backup_before_prepare_1/", dbname=0x7ffdd1243a30 "test", filename=0x7ffdd1244a10 "t8.ibd.delta") at /data/src/10.4/extra/mariabackup/xtrabackup.cc:4883
      #9  0x000055c4639acc18 in xb_process_datadir (path=0x55c465397440 <xtrabackup_real_incremental_dir> "/data/bld/10.4/backup_before_prepare_1/", suffix=0x55c4646d9efc ".delta", func=0x55c4639ab5d0 <xtrabackup_apply_delta(char const*, char const*, char const*, void*)>) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:5225
      #10 0x000055c4639acd0e in xtrabackup_apply_deltas () at /data/src/10.4/extra/mariabackup/xtrabackup.cc:5258
      #11 0x000055c4639ae207 in xtrabackup_prepare_func (argv=0x55c467601118) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:5502
      #12 0x000055c4639b05a3 in main_low (argv=0x55c467601118) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:6365
      #13 0x000055c4639afdb7 in main (argc=7, argv=0x7ffdd12463e8) at /data/src/10.4/extra/mariabackup/xtrabackup.cc:6158
      

      Backups (before prepare) and backup logs are attached.

      To reproduce, extract backup_before_prepare_0.tar.gz and backup_before_prepare_1.tar.gz and run on a debug 10.4 build

      bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0
      bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_1
      

      The second one fails.

      On a non-debug build, there is no crash, only the second prepare throws the warning in the middle:

      2019-02-15 15:34:43 0 [Warning] InnoDB: Tablespace 657 was not found at ./test/#sql-6009_f.ibd when restoring a (partial?) backup. All redo log for this file will be ignored!
      

      (but still reports OK at the end).
      An attempt to run the third prepare on backup_before_prepare_2.tar.gz , however, produces contradictory diagnostics:

      bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_2
      

      2019-02-15 15:36:34 0 [Note] InnoDB: Starting final batch to recover 225 pages from redo log.
      2019-02-15 15:36:34 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
      [00] 2019-02-15 15:36:34 mariabackup: innodb_init() returned 39 (Data structure corruption).
      [00] 2019-02-15 15:36:34 completed OK!
      

      I don't know if the underlying cause is the same as for the assertion failure or different, but at least the diagnostics is a problem in itself: if it complains about data corruption and plugin initialization error, it should report "completed OK" and return 0.

      I have no information yet whether any of the above affects other versions.


      To reproduce from scratch

      git clone https://github.com/MariaDB/randgen --branch mdev18589 rqg-mdev18589
      cd rqg-mdev18589
      . ./mdev18589.cmd <your basedir>
      

      As of 10.4 9e0ed0fa, it happens for me every time when I run the test, but it's still a concurrent non-deterministic test, re-run if it doesn't fail right away.

      Attachments

        1. mbackup_prepare_1.log
          8 kB
        2. mbackup_prepare_0.log
          15 kB
        3. mbackup_backup_2.log
          29 kB
        4. mbackup_backup_1.log
          45 kB
        5. mbackup_backup_0.log
          56 kB
        6. backup_before_prepare_2.tar.gz
          1.00 MB
        7. backup_before_prepare_1.tar.gz
          830 kB
        8. backup_before_prepare_0.tar.gz
          808 kB

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.