Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21109

Table corruption not detected with CHECK TABLE or innochecksum, only with mariabackup

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 10.3.17, 10.3.21, 10.3.23
    • N/A
    • mariabackup

    Description

      It is observed that, while taking the backup using mariabackup(10.3.17) it's keep failing with following error:

      Backup command used:


      /var/lib/mysql/bin/mariabackup --defaults-file=/etc/my.cnf --user=**** --password=******* --backup --skip-encrypted-backup --compress --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wait-query-type=all --target-dir=/tmp/backup/full_backup_2019
      
      


      mariabackup output:


      [00] 2019-11-16 06:29:24 Connecting to MySQL server host: localhost, user: xxx, password: xxx, port: 3306, socket: /var/lib/mysql/mysql.sock
      [00] 2019-11-16 06:29:24 Using server version 10.3.17-MariaDB-log
      /var/lib/mysql/bin/mariabackup based on MariaDB server 10.3.17-MariaDB Linux (x86_64)
      ....
      ....
      [01] 2019-11-16 06:29:27 Compressing ./foo/bar.ibd to /tmp/backup/full_backup_2019/foo/bar.ibd.qp
      [01] 2019-11-16 06:29:27         ...done
      [01] 2019-11-16 06:29:27 Compressing ./foo/foobar.ibd to /tmp/backup/full_backup_2019/foo/foobar.ibd.qp
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
      [01] 2019-11-16 06:29:28 Database page corruption detected at page 5, retrying...
      [00] 2019-11-16 06:29:28 >> log scanned up to (30851334774)
      [01] 2019-11-16 06:29:28 Error: failed to read page after 10 retries. File ./foo/foobar.ibd seems to be corrupted.
      2019-11-16 6:29:28 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
      .......
      .......
      InnoDB: End of page dump
      2019-11-16 6:29:28 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 3454859770, calculated checksums for field1: crc32 3454859770, innodb 3654618756, page type 17855 == INDEX.none 3735928559, stored checksum in field2 3454859770, calculated checksums for field2: crc32 3454859770, innodb 4252287317, none 3735928559, page LSN 7 450897511, low 4 bytes of LSN at page end 450897511, page number (if stored to page already) 7, space id (if created with >= MySQL-4.1.1 and stored already) 1962
      2019-11-16 6:29:28 0 [Note] InnoDB: Page may be an index page where index id is 5185
      [01] 2019-11-16 06:29:28 mariabackup: xtrabackup_copy_datafile() failed.
      [00] FATAL ERROR: 2019-11-16 06:29:28 failed to copy datafile.
      
      


      Did tried, fixing the table (pointed as corrupted) with following, but the same issue is occurring.


      set OLD_ALTER_TABLE=1
      Alter table table_name engine=InnoDB
      Alter table table_name FORCE
      Take mysqldump of table and restore it to Database


      Attachments

        1. break_down_rate.ddl
          0.8 kB
        2. break_down_rate.ibd
          112 kB
        3. corrupt20200921.tgz.enc
          10 kB
        4. cs0093460_20200807_core.21139.gza
          10.00 MB
        5. cs0093460_20200807_core.21139.gzb
          10.00 MB
        6. cs0093460_20200807_core.21139.gzc
          10.00 MB
        7. cs0093460_20200807_core.21139.gzd
          10.00 MB
        8. cs0093460_20200807_core.21139.gze
          10.00 MB
        9. cs0093460_20200807_core.21139.gzf
          10.00 MB
        10. cs0093460_20200807_core.21139.gzg
          10.00 MB
        11. cs0093460_20200807_core.21139.gzh
          10.00 MB
        12. cs0093460_20200807_core.21139.gzi
          10.00 MB
        13. cs0093460_20200807_core.21139.gzj
          10.00 MB
        14. cs0093460_20200807_core.21139.gzk
          10.00 MB
        15. cs0093460_20200807_core.21139.gzl
          10.00 MB
        16. cs0093460_20200807_core.21139.gzm
          8.14 MB
        17. cs0093460_20200807_error_mariadb_x01bofiddb1a.log
          78 kB
        18. cs0093460_20200807_lib.tar.gza
          10.00 MB
        19. cs0093460_20200807_lib.tar.gzb
          4.03 MB
        20. cs0093460_20200917_error_mariadb_x01bstredb1a.log
          14 kB
        21. cs0093460_20200917_full_backup_20200916_054536.log
          30 kB
        22. cs0093460_20200917_full_backup_20200917_120913.log
          29 kB
        23. mdev-21109-rr.sh
          4 kB
        24. show_global_status_x07gisiddb3a.log
          62 kB
        25. show_global_variables_x07gisiddb3a.log
          475 kB
        26. t33914-202006252311.log
          104 kB
        27. t33914-202006252313.tgz.enc.a
          9.00 MB
        28. t33914-202006252313.tgz.enc.b
          9.00 MB
        29. t33914-202006252313.tgz.enc.c
          9.00 MB
        30. t33914-202006252313.tgz.enc.d
          8.09 MB

        Issue Links

          Activity

            suresh.ramagiri@mariadb.com suresh ramagiri created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Assignee Vladislav Lesin [ vlad.lesin ]
            Labels corruption
            Priority Major [ 3 ] Critical [ 2 ]
            vlad.lesin Vladislav Lesin made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            pramod.mahto@mariadb.com Pramod Mahto made changes -
            Description It is observed that, while taking the backup using mariabackup(10.3.17) it's keep failing with following error:

            Backup command used:


            ----

            /var/lib/mysql/bin/mariabackup --defaults-file=/etc/my.cnf --user=**** --password=******* --backup --skip-encrypted-backup --compress --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wait-query-type=all --target-dir=/tmp/backup/full_backup_2019

            ----


            mariabackup output:

            ----
            [00] 2019-11-16 06:29:24 Connecting to MySQL server host: localhost, user: xxx, password: xxx, port: 3306, socket: /var/lib/mysql/mysql.sock
            [00] 2019-11-16 06:29:24 Using server version 10.3.17-MariaDB-log
            /var/lib/mysql/bin/mariabackup based on MariaDB server 10.3.17-MariaDB Linux (x86_64)
            ....
            ....
            [01] 2019-11-16 06:29:27 Compressing ./foo/bar.ibd to /tmp/backup/full_backup_2019/foo/bar.ibd.qp
            [01] 2019-11-16 06:29:27 ...done
            [01] 2019-11-16 06:29:27 Compressing ./foo/foobar.ibd to /tmp/backup/full_backup_2019/foo/foobar.ibd.qp
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:28 Database page corruption detected at page 5, retrying...
            [00] 2019-11-16 06:29:28 >> log scanned up to (30851334774)
            [01] 2019-11-16 06:29:28 Error: failed to read page after 10 retries. File ./foo/foobar.ibd seems to be corrupted.
            2019-11-16 6:29:28 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
            .......
            .......
            InnoDB: End of page dump
            2019-11-16 6:29:28 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 3454859770, calculated checksums for field1: crc32 3454859770, innodb 3654618756, page type 17855 == INDEX.none 3735928559, stored checksum in field2 3454859770, calculated checksums for field2: crc32 3454859770, innodb 4252287317, none 3735928559, page LSN 7 450897511, low 4 bytes of LSN at page end 450897511, page number (if stored to page already) 7, space id (if created with >= MySQL-4.1.1 and stored already) 1962
            2019-11-16 6:29:28 0 [Note] InnoDB: Page may be an index page where index id is 5185
            [01] 2019-11-16 06:29:28 mariabackup: xtrabackup_copy_datafile() failed.
            [00] FATAL ERROR: 2019-11-16 06:29:28 failed to copy datafile.

            ----


            Did tried, fixing the table (pointed as corrupted) with following, but the same issue is occurring.

            ----
            set OLD_ALTER_TABLE=1
            Alter table table_name engine=InnoDB
            Alter table table_name FORCE
            Take mysqldump of table and restore it to Database
            ----

            It is observed that, while taking the backup using mariabackup(10.3.17) it's keep failing with following error:

            Backup command used:


            ----
            {code}
            /var/lib/mysql/bin/mariabackup --defaults-file=/etc/my.cnf --user=**** --password=******* --backup --skip-encrypted-backup --compress --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wait-query-type=all --target-dir=/tmp/backup/full_backup_2019

            {code}
            ----


            mariabackup output:

            ----
            {code}
            [00] 2019-11-16 06:29:24 Connecting to MySQL server host: localhost, user: xxx, password: xxx, port: 3306, socket: /var/lib/mysql/mysql.sock
            [00] 2019-11-16 06:29:24 Using server version 10.3.17-MariaDB-log
            /var/lib/mysql/bin/mariabackup based on MariaDB server 10.3.17-MariaDB Linux (x86_64)
            ....
            ....
            [01] 2019-11-16 06:29:27 Compressing ./foo/bar.ibd to /tmp/backup/full_backup_2019/foo/bar.ibd.qp
            [01] 2019-11-16 06:29:27 ...done
            [01] 2019-11-16 06:29:27 Compressing ./foo/foobar.ibd to /tmp/backup/full_backup_2019/foo/foobar.ibd.qp
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
            [01] 2019-11-16 06:29:28 Database page corruption detected at page 5, retrying...
            [00] 2019-11-16 06:29:28 >> log scanned up to (30851334774)
            [01] 2019-11-16 06:29:28 Error: failed to read page after 10 retries. File ./foo/foobar.ibd seems to be corrupted.
            2019-11-16 6:29:28 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
            .......
            .......
            InnoDB: End of page dump
            2019-11-16 6:29:28 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 3454859770, calculated checksums for field1: crc32 3454859770, innodb 3654618756, page type 17855 == INDEX.none 3735928559, stored checksum in field2 3454859770, calculated checksums for field2: crc32 3454859770, innodb 4252287317, none 3735928559, page LSN 7 450897511, low 4 bytes of LSN at page end 450897511, page number (if stored to page already) 7, space id (if created with >= MySQL-4.1.1 and stored already) 1962
            2019-11-16 6:29:28 0 [Note] InnoDB: Page may be an index page where index id is 5185
            [01] 2019-11-16 06:29:28 mariabackup: xtrabackup_copy_datafile() failed.
            [00] FATAL ERROR: 2019-11-16 06:29:28 failed to copy datafile.

            {code}
            ----


            Did tried, fixing the table (pointed as corrupted) with following, but the same issue is occurring.

            ----
            set OLD_ALTER_TABLE=1
            Alter table table_name engine=InnoDB
            Alter table table_name FORCE
            Take mysqldump of table and restore it to Database
            ----

            vlad.lesin Vladislav Lesin made changes -
            julien.fritsch Julien Fritsch made changes -
            Labels corruption corruption need_feedback
            vlad.lesin Vladislav Lesin made changes -
            valerii Valerii Kravchuk made changes -
            Labels corruption need_feedback corruption
            julien.fritsch Julien Fritsch made changes -
            Labels corruption corruption need_feedback
            valerii Valerii Kravchuk made changes -
            Affects Version/s 10.3.21 [ 24014 ]
            vlad.lesin Vladislav Lesin made changes -
            juan.vera Juan made changes -
            Attachment break_down_rate.ddl [ 51826 ]
            Attachment break_down_rate.ibd [ 51827 ]
            serg Sergei Golubchik made changes -
            Labels corruption need_feedback corruption
            maxmether Max Mether made changes -
            Summary Mariabackup failed with table corruption Table corruption not detected with CHECK TABLE or innochecksum, only with mariabackup
            maxmether Max Mether made changes -
            Assignee Vladislav Lesin [ vlad.lesin ] Marko Mäkelä [ marko ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Vladislav Lesin [ vlad.lesin ]
            vlad.lesin Vladislav Lesin made changes -
            Comment [ [~marko] I have a couple of questions.

            > I remember diagnosing a similar issue with the same support customer about a year ago, but we failed to reach a conclusion then. On July 1, 2019, we added a debug check to InnoDB in MDEV-19871 to specifically catch such tablespace ID mismatch. (The commit message uses the wrong MDEV number and is misleadingly mentioning the innochecksum tool, which was not changed.). That check has never failed in our internal testing since then. Based on that, I firmly believe that a bug must exist in the interaction between mariabackup --backup and the server process.

            What interaction do you mean? The backup was never restored, mariabackup does not change data files during copying. The only problem I can assume is that mariabackup does some checks during copying, which must not be done. At the other hand, it uses innodb code to read pages. But I am checking this version.

            > On July 1, 2019, we added a debug check to InnoDB in MDEV-19871 to specifically catch such tablespace ID mismatch."

            Yes, we did it. The following code was added in fil_io():

            {code:java}
                    ut_ad(!req_type.is_write()
                          || page_id.space() == SRV_LOG_SPACE_FIRST_ID
                          || !fil_is_user_tablespace_id(page_id.space())
                          || offset == page_id.page_no() * page_size.physical());
            {code}

            This code does not check the content of the page, it just checks the correctness of passed to fil_io() parameters. I have added the following check:

            {code:java}
              if (req_type.is_write()⏎
                && page_id.space() != SRV_LOG_SPACE_FIRST_ID⏎
                && fil_is_user_tablespace_id(page_id.space())⏎
                && (offset != page_id.page_no() * page_size.physical()⏎
                  || mach_read_from_4(static_cast<const byte *>(buf) + FIL_PAGE_OFFSET)⏎
                    != page_id.page_no()⏎
                  || mach_read_from_4(static_cast<const byte *>(buf)⏎
                   + FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID) != page_id.space())) {⏎
            {code}

            in my custom build. If this check is correct from your perspective, should not we add it in trunk to catch the bug during testing? ]
            juan.vera Juan made changes -
            vlad.lesin Vladislav Lesin made changes -
            juan.vera Juan made changes -
            Attachment t33914-202006252215.tgz.enc [ 52375 ]
            juan.vera Juan made changes -
            Attachment t33914-202006252215.tgz.enc [ 52375 ]
            juan.vera Juan made changes -
            Attachment t33914-202006252311.log [ 52376 ]
            juan.vera Juan made changes -
            Attachment t33914-202006252313.tgz.enc.a [ 52377 ]
            Attachment t33914-202006252313.tgz.enc.b [ 52378 ]
            Attachment t33914-202006252313.tgz.enc.c [ 52379 ]
            Attachment t33914-202006252313.tgz.enc.d [ 52380 ]
            vlad.lesin Vladislav Lesin made changes -
            Comment [ A comment with security level 'Developers' was removed. ]
            julien.fritsch Julien Fritsch made changes -
            Labels corruption corruption need_feedback
            valerii Valerii Kravchuk made changes -
            Comment [ A comment with security level 'Developers' was removed. ]
            juan.vera Juan made changes -
            Attachment cs0093460_20200807_core.21139.gzm [ 53271 ]
            Attachment cs0093460_20200807_core.21139.gzl [ 53272 ]
            Attachment cs0093460_20200807_core.21139.gzk [ 53273 ]
            Attachment cs0093460_20200807_core.21139.gzj [ 53274 ]
            Attachment cs0093460_20200807_core.21139.gzi [ 53275 ]
            Attachment cs0093460_20200807_core.21139.gzh [ 53276 ]
            Attachment cs0093460_20200807_core.21139.gzg [ 53277 ]
            Attachment cs0093460_20200807_core.21139.gzf [ 53278 ]
            Attachment cs0093460_20200807_core.21139.gze [ 53279 ]
            Attachment cs0093460_20200807_core.21139.gzd [ 53280 ]
            Attachment cs0093460_20200807_core.21139.gzc [ 53281 ]
            Attachment cs0093460_20200807_core.21139.gzb [ 53282 ]
            Attachment cs0093460_20200807_core.21139.gza [ 53283 ]
            juan.vera Juan made changes -
            juan.vera Juan made changes -
            Attachment cs0093460_20200807_lib.tar.gzb [ 53285 ]
            Attachment cs0093460_20200807_lib.tar.gza [ 53286 ]
            vlad.lesin Vladislav Lesin made changes -
            vlad.lesin Vladislav Lesin made changes -
            vlad.lesin Vladislav Lesin made changes -
            Attachment mdev-21109-rr.sh [ 53625 ]
            vlad.lesin Vladislav Lesin made changes -
            Attachment mdev-21109-rr.sh [ 53625 ]
            vlad.lesin Vladislav Lesin made changes -
            Attachment mdev-21109-rr.sh [ 53628 ]
            juan.vera Juan made changes -
            juan.vera Juan made changes -
            juan.vera Juan made changes -
            juan.vera Juan made changes -
            Attachment stre_run_status.frm [ 53899 ]
            juan.vera Juan made changes -
            Attachment stre_run_status.ibd [ 53900 ]
            juan.vera Juan made changes -
            Attachment stre_run_status.frm [ 53899 ]
            juan.vera Juan made changes -
            Attachment stre_run_status.ibd [ 53900 ]
            juan.vera Juan made changes -
            Attachment corrupt20200921.tgz.enc [ 53901 ]
            nicklamb Nick (Inactive) made changes -
            Labels corruption need_feedback corruption
            marko Marko Mäkelä made changes -
            vlad.lesin Vladislav Lesin made changes -
            allen.lee@mariadb.com Allen Lee (Inactive) made changes -
            Affects Version/s 10.3.23 [ 24222 ]
            allen.lee@mariadb.com Allen Lee (Inactive) made changes -
            6tasticMDB Frank Olsen made changes -
            vlad.lesin Vladislav Lesin made changes -
            julien.fritsch Julien Fritsch made changes -
            julien.fritsch Julien Fritsch made changes -
            julien.fritsch Julien Fritsch made changes -
            julien.fritsch Julien Fritsch made changes -
            Labels corruption corruption need_feedback
            julien.fritsch Julien Fritsch made changes -
            Assignee Vladislav Lesin [ vlad.lesin ] Chris Calender [ ccalender ]
            ccalender Chris Calender (Inactive) made changes -
            Labels corruption need_feedback corruption
            julien.fritsch Julien Fritsch made changes -
            Assignee Chris Calender [ ccalender ] Vladislav Lesin [ vlad.lesin ]
            julien.fritsch Julien Fritsch made changes -
            Assignee Vladislav Lesin [ vlad.lesin ] Sergei Golubchik [ serg ]
            julien.fritsch Julien Fritsch made changes -
            Assignee Sergei Golubchik [ serg ] Marko Mäkelä [ marko ]
            marko Marko Mäkelä made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            julien.fritsch Julien Fritsch made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            serg Sergei Golubchik made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            Labels corruption ServiceNow corruption
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            Labels ServiceNow corruption 76qDvLB8Gju6Hs7nk3VY3EX42G795W5z corruption
            serg Sergei Golubchik made changes -
            Labels 76qDvLB8Gju6Hs7nk3VY3EX42G795W5z corruption corruption
            julien.fritsch Julien Fritsch made changes -
            Labels corruption corruption need_feedback
            valerii Valerii Kravchuk made changes -
            Labels corruption need_feedback corruption
            marko Marko Mäkelä made changes -
            Labels corruption corruption need_feedback
            valerii Valerii Kravchuk made changes -
            Labels corruption need_feedback corruption
            julien.fritsch Julien Fritsch made changes -
            Labels corruption corruption need_feedback
            marko Marko Mäkelä made changes -
            Fix Version/s N/A [ 14700 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Cannot Reproduce [ 5 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            pramod.mahto@mariadb.com Pramod Mahto made changes -
            Resolution Cannot Reproduce [ 5 ]
            Status Closed [ 6 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 101312 ] MariaDB v4 [ 143620 ]
            julien.fritsch Julien Fritsch made changes -
            Status Stalled [ 10000 ] Open [ 1 ]
            julien.fritsch Julien Fritsch made changes -
            Status Open [ 1 ] Needs Feedback [ 10501 ]
            julien.fritsch Julien Fritsch made changes -
            Labels corruption need_feedback corruption
            elenst Elena Stepanova made changes -
            Status Needs Feedback [ 10501 ] Open [ 1 ]
            elenst Elena Stepanova made changes -
            Resolution Incomplete [ 4 ]
            Status Open [ 1 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 201658 110821 112315 173466 179943 123943 187534 176114 119382 185032 126984 132703 167577 161518
            Zendesk active tickets 201658

            People

              marko Marko Mäkelä
              suresh.ramagiri@mariadb.com suresh ramagiri
              Votes:
              4 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.