Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18097

Indicate which InnoDB data files are zero-initialized

Details

    Description

      Originally, InnoDB did not zero-initialize buffer pool blocks that were allocated to a file page. Because of this, unused bytes on data pages could contain any garbage, belonging to the previous contents of the buffer pool block, such as another data page, or a generic memory allocation (BUF_BLOCK_MEMORY, buf_block_alloc(), MEM_HEAP_BUFFER) that is used by the adaptive hash index, crash recovery, record locks.

      The page initialization was finally added in MySQL 5.1.48 for all data files. The original author of InnoDB opposed the initialization for many years, because he feared that it might cost some performance.

      To make it easier to repurpose unused data fields in the future, we should introduce a flag in FSP_SPACE_FLAGS to indicate that any unused fields in the data file are zero-initialized.

      At least since MDEV-11623 in MariaDB 10.1.21, InnoDB is ignoring the useless tablespace flag that was introduced in MySQL 5.6 to indicate that the DATA DIRECTORY attribute was specified when the table was created or rebuilt. Let us repurpose this old flag to indicate that unused data fields have been zero-initialized. (MySQL 5.6 and later always zero-initialized all unused data fields, so it is safe to repurpose this bit. It can never be set on files that were created before MySQL 5.6.)

      We must be careful to never set this flag on old data files without first having rewritten all pages. This could be implemented in a conversion tool later.

      Attachments

        Issue Links

          Activity

            marko Marko Mäkelä created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Attachment 10.1.patch [ 46903 ]

            10.1.patch attempts to implement this. There are a few failing tests.

            There also is a downgrade problem. A prerequisite for implementing this should be to ensure that IMPORT TABLESPACE to older versions works.

            marko Marko Mäkelä added a comment - 10.1.patch attempts to implement this. There are a few failing tests. There also is a downgrade problem. A prerequisite for implementing this should be to ensure that IMPORT TABLESPACE to older versions works.
            marko Marko Mäkelä made changes -
            Fix Version/s 10.1 [ 16100 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.1.38 [ 23209 ]
            Fix Version/s 10.2.21 [ 23213 ]
            Fix Version/s 10.3.12 [ 23214 ]
            Fix Version/s 10.4.2 [ 23229 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -

            After to MDEV-12026 in MariaDB 10.4, data files created with innodb_checksum_algorithm=full_crc32 will be guaranteed to be zero-initialized. The tablespace flag that indicates full_crc32 format is also indicating that the file contains no uninitialized garbage. (We do not support in-place update of old data files (which may contain uninitialized garbage) to the full_crc32 format.)

            marko Marko Mäkelä added a comment - After to MDEV-12026 in MariaDB 10.4, data files created with innodb_checksum_algorithm=full_crc32 will be guaranteed to be zero-initialized. The tablespace flag that indicates full_crc32 format is also indicating that the file contains no uninitialized garbage. (We do not support in-place update of old data files (which may contain uninitialized garbage) to the full_crc32 format.)
            marko Marko Mäkelä made changes -
            Fix Version/s 10.4.3 [ 23230 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.1 [ 16100 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]

            I updated the docs here:

            https://mariadb.com/kb/en/library/innodb-encryption-troubleshooting/#tablespaces-created-on-mysql-5147-or-earlier

            Are pages for tables with ROW_FORMAT=COMPRESSED guaranteed to be zero-initialized if MDEV-18644 is still open?

            GeoffMontee Geoff Montee (Inactive) added a comment - I updated the docs here: https://mariadb.com/kb/en/library/innodb-encryption-troubleshooting/#tablespaces-created-on-mysql-5147-or-earlier Are pages for tables with ROW_FORMAT=COMPRESSED guaranteed to be zero-initialized if MDEV-18644 is still open?
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 91423 ] MariaDB v4 [ 155418 ]

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.