Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32150

InnoDB reports corruption on 32-bit platforms with ibd files sizes > 4GB

Details

    Description

      Mariadb reports a space/page mismatch and index corruption on 32-bit platforms when loading large InnoDB files. This started in the 10.5 series and is reproducible on each version I have tested since that series.

      I tracked the issue down to a 32-bit truncation when the file offset is calculated in storage/innobase/buf/buf0rea.cc.

      Casting this to os_offset_t solves the issue (patch attached).

      On 64-bit platforms this calculation would naturally occur in a 64-bit register so it is not reproducible there.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23855 InnoDB log checkpointing causes regression for write-heavy OLTP

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            danwood76 Danny Wood created issue -
            marko Marko Mäkelä added a comment -

            Thank you for the report. We had something similar in MDEV-24412, but that affected also 64-bit environments.

            marko Marko Mäkelä added a comment - Thank you for the report. We had something similar in MDEV-24412 , but that affected also 64-bit environments.
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.10 [ 27530 ]
            Fix Version/s 10.11 [ 27614 ]
            Fix Version/s 11.0 [ 28320 ]
            Fix Version/s 11.1 [ 28549 ]
            Fix Version/s 11.2 [ 28603 ]
            Assignee Marko Mäkelä [ marko ]
            danwood76 Danny Wood added a comment -

            Thanks @Marko, I did have a thorough look through the bug database before I reported this.
            Although that looks a similar issue the versions I have tested are marked as "fixed" in that bug report.

            The issue I have found specifically affects 32-bit builds as the compiler will use 32-bit registers in the offset calculation as page_no() returns a 32-bit uint (it is 32-bit in the InnoDB file structure AFAIK), casting this to os_offset_t causes the compiler to do 64-bit maths as os_offset_t is typedef uint64_t.

            I first experienced this on my embedded linux platform (armhf) running mariadb 10.11.5 but was able to reproduce this on Debian armhf and x86 stock installs and the same database. With my patch my embedded platform loads my large database.

            I have tested the same database on Debian 12 64-bit (mariadb 10.11.3) and it works without issue.

            Do you want me to create an example database/table to test against?

            danwood76 Danny Wood added a comment - Thanks @Marko, I did have a thorough look through the bug database before I reported this. Although that looks a similar issue the versions I have tested are marked as "fixed" in that bug report. The issue I have found specifically affects 32-bit builds as the compiler will use 32-bit registers in the offset calculation as page_no() returns a 32-bit uint (it is 32-bit in the InnoDB file structure AFAIK), casting this to os_offset_t causes the compiler to do 64-bit maths as os_offset_t is typedef uint64_t. I first experienced this on my embedded linux platform (armhf) running mariadb 10.11.5 but was able to reproduce this on Debian armhf and x86 stock installs and the same database. With my patch my embedded platform loads my large database. I have tested the same database on Debian 12 64-bit (mariadb 10.11.3) and it works without issue. Do you want me to create an example database/table to test against?
            marko Marko Mäkelä added a comment -

            Luckily, this was the only call to fil_space_t::io() that would fail to use 64-bit arithmetics.

            marko Marko Mäkelä added a comment - Luckily, this was the only call to fil_space_t::io() that would fail to use 64-bit arithmetics.
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2023-09-12 13:19:37.0 2023-09-12 13:19:37.198
            marko Marko Mäkelä made changes -
            Fix Version/s 10.5.23 [ 29012 ]
            Fix Version/s 10.6.16 [ 29014 ]
            Fix Version/s 10.10.7 [ 29018 ]
            Fix Version/s 10.11.6 [ 29020 ]
            Fix Version/s 11.0.4 [ 29021 ]
            Fix Version/s 11.1.3 [ 29023 ]
            Fix Version/s 11.2.2 [ 29035 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.10 [ 27530 ]
            Fix Version/s 10.11 [ 27614 ]
            Fix Version/s 11.0 [ 28320 ]
            Fix Version/s 11.1 [ 28549 ]
            Fix Version/s 11.2 [ 28603 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            danwood76 Danny Wood added a comment -

            That's good!

            Thanks for the quick response on this.

            danwood76 Danny Wood added a comment - That's good! Thanks for the quick response on this.
            marko Marko Mäkelä added a comment -

            danwood76, sorry, I did not notice your update before closing this. I think that we’re good. In my patch, I used a C++11 style widening cast. The tablespace ID and page number are stored as 32-bit in each InnoDB data page; that is why they are being returned as 32-bit by the access methods. Also, the commonly used ulint (nowadays a typedef alias of size_t) would be 32 bits on 32-bit systems.

            marko Marko Mäkelä added a comment - danwood76 , sorry, I did not notice your update before closing this. I think that we’re good. In my patch, I used a C++11 style widening cast. The tablespace ID and page number are stored as 32-bit in each InnoDB data page; that is why they are being returned as 32-bit by the access methods. Also, the commonly used ulint (nowadays a typedef alias of size_t ) would be 32 bits on 32-bit systems.
            danwood76 Danny Wood added a comment -

            Hi @Marko, no problem. Thanks for the explanation.

            danwood76 Danny Wood added a comment - Hi @Marko, no problem. Thanks for the explanation.

            People

              marko Marko Mäkelä
              danwood76 Danny Wood
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.