Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25361

innochecksum must not report errors for freed pages

Details

    Description

      CHECK TABLE needs to be able to detect at least the same corruption as innochecksum. This is necessary because the server must be stopped for innochecksum whereas CHECK TABLE does not.

      This is related to MDEV-21109, but distinct in that MDEV-21109 is specific to mariabackup (at least per the "Component" field whereas this was a feature request to enhance CHECK TABLE.

      Edit: Marko Mäkelä thinks that it is more feasible to improve innochecksum so that it will ignore data pages that have been marked freed in the tablespace files.

      The innochecksum tool is mostly operating on single pages only and can only find pages with invalid checksum. CHECK TABLE can find more, such as broken links between B+ tree pages, or some inconsistency between B+ tree indexes.

      Freed pages should never be accessed by the server, be it by CHECK TABLE or elsewhere. We do not want CHECK TABLE to load every page of the data file into the buffer pool, because that could make CHECK TABLE extremely slow for tables that are stored in the InnoDB system tablespace. Furthermore, loading freed pages to the buffer pool could cause MariaDB tablespace encryption to mark corrupted pages as dirty. We should simply tolerate corrupted pages that are freed.

      As a further enhancement to innochecksum, we could implement an option that would zero out pages that have been marked as freed (similar to what MariaDB 10.5 would do if innodb_immediate_scrub_data_uncompressed=ON). That would also make data files compress better.

      Note: innochecksum already includes a call to xdes_is_free(), but it is currently only invoked for the -S and -D options that output data on page types.

      Attachments

        Issue Links

          Activity

            This needs to be non-blocking.

            ccalender Chris Calender (Inactive) added a comment - This needs to be non-blocking.

            I think that we must identify the allocation bitmap pages based on the page number and page size.

            marko Marko Mäkelä added a comment - I think that we must identify the allocation bitmap pages based on the page number and page size.

            Page number is now used to find xdes bitmap page.

            But I left the condition to check for the first page because innochecksum requires it anyways.

            kevg Eugene Kosov (Inactive) added a comment - Page number is now used to find xdes bitmap page. But I left the condition to check for the first page because innochecksum requires it anyways.
            kevg Eugene Kosov (Inactive) added a comment - The code is in https://github.com/MariaDB/server/tree/bb-10.2-MDEV-25361-innochecksum

            The first page check appears to be broken in later versions of innochecksum, possibly due to innodb_checksum_algorithm=full_crc32 (MDEV-12026). In 10.2, the following works fine without your change, which insists that the first page have a specially initialized FIL_PAGE_TYPE. Such FIL_PAGE_TYPE value is not even guaranteed if the data file had been created by an old enough version of MySQL or MariaDB.

            dd bs=16384 skip=3 count=1 if=var/install.db/ibdata1 of=page3.bin
            ../extra/innochecksum page3.bin
            

            marko Marko Mäkelä added a comment - The first page check appears to be broken in later versions of innochecksum , possibly due to innodb_checksum_algorithm=full_crc32 ( MDEV-12026 ). In 10.2, the following works fine without your change, which insists that the first page have a specially initialized FIL_PAGE_TYPE . Such FIL_PAGE_TYPE value is not even guaranteed if the data file had been created by an old enough version of MySQL or MariaDB. dd bs=16384 skip=3 count=1 if=var/install.db/ibdata1 of=page3.bin ../extra/innochecksum page3.bin
            kevg Eugene Kosov (Inactive) added a comment - You're right. I've removed the condition completely. https://github.com/MariaDB/server/commits/bb-10.2-MDEV-25361-innochecksum

            Thank you, the fix looks correct to me now.

            marko Marko Mäkelä added a comment - Thank you, the fix looks correct to me now.

            People

              kevg Eugene Kosov (Inactive)
              ccalender Chris Calender (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.