Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11939

innochecksum mistakes a file for an encrypted one (page 0 invalid)

Details

    • 10.1.22, 10.2.6-2, 10.2.6-3

    Description

      innochecksum returns an error message when ran on ibdata1 on a server that is working fine (a check on all tables has been done without finding anything) :

      # innochecksum -d /var/lib/mysql/ibdata1 
      InnoDB offline file checksum utility.
       
      Variables (--variable-name=value)
      and boolean options {FALSE|TRUE}  Value (after reading options)
      --------------------------------- ----------------------------------------
      verbose                           TRUE
      debug                             TRUE
      skip-corrupt                      FALSE
      count                             FALSE
      start-page                        0
      end-page                          0
      page                              0
      per-page-details                  FALSE
      leaf                              FALSE
      merge                             0
      Table is uncompressed
      Page size is 16384
      file /var/lib/mysql/ibdata1 = 3844079616 bytes (234624 pages)...
      InnoChecksum; checking pages in range 0 to 234623
      Page 0 encrypted key_version 5 checksum 1135705172
      page 0: log sequence number: first = 1136557346; second = 1136557346
      page 0: old style: calculated = 2609070081; recorded = 2609070081
      page 0: new style: calculated = 3631443077; crc32 = 286255075; recorded = 1135705172
      Fail; page 0 invalid (fails innodb and crc32 checksum)
      

      This server was previously running on 10.0 and was just upgraded to 10.1 and it is not using tablespace encryption.

      Attachments

        Issue Links

          Activity

            jb-boin Jean Weisbuch created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Fix Version/s 10.1 [ 16100 ]
            Assignee Marko Mäkelä [ marko ]

            jb-boin Which 10.1 version was the 10.0 server upgraded to, and which version of innochecksum was invoked?
            I suspect that this is related to MDEV-11623 in some way.
            I cannot repeat this when invoking the latest 10.1 innochecksum on a freshly created ibdata1 file with the latest 10.1. I tried both the default innodb_page_size=16k and the non-default setting innodb_page_size=4k.

            marko Marko Mäkelä added a comment - jb-boin Which 10.1 version was the 10.0 server upgraded to, and which version of innochecksum was invoked? I suspect that this is related to MDEV-11623 in some way. I cannot repeat this when invoking the latest 10.1 innochecksum on a freshly created ibdata1 file with the latest 10.1. I tried both the default innodb_page_size=16k and the non-default setting innodb_page_size=4k.
            jb-boin Jean Weisbuch added a comment -

            It was previously running 10.0.29-MariaDB-1~wheezy.

            # innochecksum --version
            InnoDB offline file checksum utility.
            innochecksum Ver 5.6.35, for debian-linux-gnu (x86_64)
             
            # which innochecksum 
            /usr/bin/innochecksum
             
            # dpkg -S /usr/bin/innochecksum
            mariadb-client-10.1: /usr/bin/innochecksum
             
            # apt-cache policy mariadb-client-10.1 
            mariadb-client-10.1:
              Installed: 10.1.21+maria-1~wheezy
              Candidate: 10.1.21+maria-1~wheezy
              Version table:
             *** 10.1.21+maria-1~wheezy 0
                    500 http://ftp.igh.cnrs.fr/pub/mariadb/repo/10.1/debian/ wheezy/main amd64 Packages
                    100 /var/lib/dpkg/status
            

            The innodb_page_size value has never been modified on this server.

            jb-boin Jean Weisbuch added a comment - It was previously running 10.0.29-MariaDB-1~wheezy. # innochecksum --version InnoDB offline file checksum utility. innochecksum Ver 5.6.35, for debian-linux-gnu (x86_64)   # which innochecksum /usr/bin/innochecksum   # dpkg -S /usr/bin/innochecksum mariadb-client-10.1: /usr/bin/innochecksum   # apt-cache policy mariadb-client-10.1 mariadb-client-10.1: Installed: 10.1.21+maria-1~wheezy Candidate: 10.1.21+maria-1~wheezy Version table: *** 10.1.21+maria-1~wheezy 0 500 http://ftp.igh.cnrs.fr/pub/mariadb/repo/10.1/debian/ wheezy/main amd64 Packages 100 /var/lib/dpkg/status The innodb_page_size value has never been modified on this server.

            Can you attach the first 16384 bytes of the ibdata1 file?

            dd bs=16384 count=1 if=ibdata1 of=/tmp/page0.bin
            

            marko Marko Mäkelä added a comment - Can you attach the first 16384 bytes of the ibdata1 file? dd bs=16384 count=1 if=ibdata1 of=/tmp/page0.bin
            jb-boin Jean Weisbuch made changes -
            Attachment MDEV11939-page0.bin [ 43272 ]
            jb-boin Jean Weisbuch added a comment - MDEV11939-page0.bin

            I can repeat the failure with the attached file and with the Debian 9.0 packaged innochecksum (mariadb-server-core-10.1). Curiously, the 10.2 innochecksum is silent, but that is probably due to an unrelated issue: because the FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field is nonzero (it contains the shutdown LSN), the 10.2 innochecksum wrongly thinks that the file is encrypted and does not report anything.

            The cause must be something else than MDEV-11623, because at byte offset 54..57 the FSP_SPACE_FLAGS are 0, and thus the upgrade would have been unaffected by the bug that MDEV-11623 addressed.
            Which innodb_checksum_algorithm did you use in 10.0?

            marko Marko Mäkelä added a comment - I can repeat the failure with the attached file and with the Debian 9.0 packaged innochecksum (mariadb-server-core-10.1). Curiously, the 10.2 innochecksum is silent, but that is probably due to an unrelated issue: because the FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field is nonzero (it contains the shutdown LSN), the 10.2 innochecksum wrongly thinks that the file is encrypted and does not report anything. The cause must be something else than MDEV-11623 , because at byte offset 54..57 the FSP_SPACE_FLAGS are 0, and thus the upgrade would have been unaffected by the bug that MDEV-11623 addressed. Which innodb_checksum_algorithm did you use in 10.0?

            According to jplindst, the innochecksum fix in 10.2 is waiting for MDEV-11738 to be fixed.

            marko Marko Mäkelä added a comment - According to jplindst , the innochecksum fix in 10.2 is waiting for MDEV-11738 to be fixed.
            marko Marko Mäkelä made changes -

            Because the field FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is not protected by the page checksum, I did a simple experiment, zeroing out the field:

            dd if=/dev/zero bs=1 count=8 seek=26 conv=notrunc if=/dev/zero of=page0.bin
            innochecksum page0.bin
            

            After I zeroed out the FLUSH_LSN field, the 10.1 innochecksum will treat the page as valid.
            In the innochecksum of the MariaDB Server 10.2 source tree branch, the bug is different: a corrupted page would be silently accepted, as long as this field is nonzero.

            marko Marko Mäkelä added a comment - Because the field FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is not protected by the page checksum, I did a simple experiment, zeroing out the field: dd if=/dev/zero bs=1 count=8 seek=26 conv=notrunc if=/dev/zero of=page0.bin innochecksum page0.bin After I zeroed out the FLUSH_LSN field, the 10.1 innochecksum will treat the page as valid. In the innochecksum of the MariaDB Server 10.2 source tree branch, the bug is different: a corrupted page would be silently accepted, as long as this field is nonzero.
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Jan Lindström [ jplindst ]
            marko Marko Mäkelä made changes -
            Summary innochecksum is not working on 10.1 (page 0 invalid) innochecksum mistakes a file for an encrypted one (page 0 invalid)
            jplindst Jan Lindström (Inactive) made changes -
            jplindst Jan Lindström (Inactive) made changes -
            jplindst Jan Lindström (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Sprint 10.1.22 [ 143 ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Rank Ranked higher

            I used attached page to verify that we do not anymore mistake.

            https://github.com/MariaDB/server/commit/43815fe08fecd4dcdb45c71e2d0906b91ab0ea9d

            jplindst Jan Lindström (Inactive) added a comment - I used attached page to verify that we do not anymore mistake. https://github.com/MariaDB/server/commit/43815fe08fecd4dcdb45c71e2d0906b91ab0ea9d
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Jan Lindström [ jplindst ] Marko Mäkelä [ marko ]
            Status In Progress [ 3 ] In Review [ 10002 ]

            Please submit the 10.2 patch as well. In 10.2, the bug is different: a corrupted (unencrypted) page is not reported as such, provided that the FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field is nonzero.

            marko Marko Mäkelä added a comment - Please submit the 10.2 patch as well. In 10.2, the bug is different: a corrupted (unencrypted) page is not reported as such, provided that the FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION field is nonzero.
            ratzpo Rasmus Johansson (Inactive) made changes -
            Sprint 10.1.22 [ 143 ] 10.1.22, 10.2.6-2 [ 143, 148 ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Rank Ranked lower
            ratzpo Rasmus Johansson (Inactive) made changes -
            Sprint 10.1.22, 10.2.6-2 [ 143, 148 ] 10.1.22, 10.2.6-2, 10.2.6-3 [ 143, 148, 150 ]
            marko Marko Mäkelä made changes -
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Marko Mäkelä [ marko ] Jan Lindström [ jplindst ]
            jplindst Jan Lindström (Inactive) made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            jplindst Jan Lindström (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]

            10.1: commit 8b019f87dd51b1d2755655d2f136d5429fc47e1f
            Author: Jan Lindström <jan.lindstrom@mariadb.com>
            Date: Thu Aug 3 08:29:36 2017 +0300

            MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid)

            Always read full page 0 to determine does tablespace contain
            encryption metadata. Tablespaces that are page compressed or
            page compressed and encrypted do not compare checksum as
            it does not exists. For encrypted tables use checksum
            verification written for encrypted tables and normal tables
            use normal method.

            buf_page_is_checksum_valid_crc32
            buf_page_is_checksum_valid_innodb
            buf_page_is_checksum_valid_none
            Add Innochecksum logging to file

            buf_page_is_corrupted
            Remove ib_logf and page_warn_strict_checksum
            calls in innochecksum compilation. Add innochecksum
            logging to file.

            fil0crypt.cc fil0crypt.h
            Modify to be able to use in innochecksum compilation and
            move fil_space_verify_crypt_checksum to end of the file.
            Add innochecksum logging to file.

            univ.i
            Add innochecksum strict_verify, log_file and cur_page_num
            variables as extern.

            page_zip_verify_checksum
            Add innochecksum logging to file.

            innochecksum.cc
            Lot of changes most notable able to read encryption
            metadata from page 0 of the tablespace.

            Added test case where we corrupt intentionally
            FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version)
            FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum)
            FIL_DATA+10 (data)

            jplindst Jan Lindström (Inactive) added a comment - 10.1: commit 8b019f87dd51b1d2755655d2f136d5429fc47e1f Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Thu Aug 3 08:29:36 2017 +0300 MDEV-11939 : innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Add Innochecksum logging to file buf_page_is_corrupted Remove ib_logf and page_warn_strict_checksum calls in innochecksum compilation. Add innochecksum logging to file. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data)
            jplindst Jan Lindström (Inactive) made changes -

            10.2:

            commit 34eef269eb93ee42c0e5e1894781c61f66cb6f14
            Author: Jan Lindström <jan.lindstrom@mariadb.com>
            Date: Fri Aug 4 13:57:26 2017 +0300

            MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid)

            Always read full page 0 to determine does tablespace contain
            encryption metadata. Tablespaces that are page compressed or
            page compressed and encrypted do not compare checksum as
            it does not exists. For encrypted tables use checksum
            verification written for encrypted tables and normal tables
            use normal method.

            buf_page_is_checksum_valid_crc32
            buf_page_is_checksum_valid_innodb
            buf_page_is_checksum_valid_none
            Modify Innochecksum logging to file to avoid compilation
            warnings.

            fil0crypt.cc fil0crypt.h
            Modify to be able to use in innochecksum compilation and
            move fil_space_verify_crypt_checksum to end of the file.
            Add innochecksum logging to file.

            univ.i
            Add innochecksum strict_verify, log_file and cur_page_num
            variables as extern.

            page_zip_verify_checksum
            Add innochecksum logging to file and remove unnecessary code.

            innochecksum.cc
            Lot of changes most notable able to read encryption
            metadata from page 0 of the tablespace.

            Added test case where we corrupt intentionally
            FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version)
            FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum)
            FIL_DATA+10 (data

            jplindst Jan Lindström (Inactive) added a comment - 10.2: commit 34eef269eb93ee42c0e5e1894781c61f66cb6f14 Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Fri Aug 4 13:57:26 2017 +0300 MDEV-11939 : innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Modify Innochecksum logging to file to avoid compilation warnings. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file and remove unnecessary code. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data
            jplindst Jan Lindström (Inactive) made changes -
            issue.field.resolutiondate 2017-08-08 06:43:14.0 2017-08-08 06:43:14.322
            jplindst Jan Lindström (Inactive) made changes -
            Fix Version/s 10.1.26 [ 22553 ]
            Fix Version/s 10.2.8 [ 22544 ]
            Fix Version/s 10.1 [ 16100 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 79434 ] MariaDB v4 [ 151626 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -

            People

              jplindst Jan Lindström (Inactive)
              jb-boin Jean Weisbuch
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.