Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-38143

Innodb data file considered curropted after mariadb 10.11 to 11.4 upgrade

    XMLWordPrintable

Details

    Description

      Hello team,

      I've encountered a corruption issue with the InnoDB data file following an upgrade from MariaDB 10.11 to 11.4 (11.4.9-MariaDB-ubu2404 to be precise). When running innochecksum on /var/lib/mysql/ibdata1
      after upgrade, the following error is reported:

      > /usr/bin/innochecksum /var/lib/mysql/ibdata1
      Fail: page::64 invalid
      Exceeded the maximum allowed checksum mismatch count::1 current::0
      

      Upon further inspection, I identified that pages 64 through 76 in the ibdata1 file identified as corrupted, while the remaining pages appear intact and unaffected.

      Although MariaDB appears to function normally after the upgrade, there is no error messages or error logs created, no issue recognized by `mysqlcheck -A`. However the issue is concerning. As far as I know, the innochecksum failure typically signals serious data store corruption, which raises questions about the integrity of the InnoDB files despite no immediate operational impact.

      The issue has been consistently reproduced across multiple environments:

      • AlmaLinux 8: upgrade from 10.3.39 → 11.4.9
      • Ubuntu 20.04: upgrade from 10.3.39 → 11.4.9
      • Ubuntu 24.04: upgrade from 10.11.13 → 11.4.9
        The simplest way to reproduce is using a default Ubuntu 24 AWS t3 small instance:

      # cat /etc/os-release
      PRETTY_NAME="Ubuntu 24.04.3 LTS"
      NAME="Ubuntu"
      VERSION_ID="24.04"
      VERSION="24.04.3 LTS (Noble Numbat)"
      VERSION_CODENAME=noble
      ID=ubuntu
      ID_LIKE=debian
      HOME_URL="https://www.ubuntu.com/"
      SUPPORT_URL="https://help.ubuntu.com/"
      BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
      PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
      UBUNTU_CODENAME=noble
      LOGO=ubuntu-logo
      

      Steps to Reproduce:
      1. `apt -y update && apt install -y mariadb-server mariadb-backup`
      2. `mariadb-backup --backup --user=root --target-dir=/tmp/preupgrade_backup
      3. `mariadb-backup --prepare --target-dir=/tmp/preupgrade_backup
      4. Disable mariadb audit plugin (it's not enabled actually): ` UNINSTALL SONAME 'server_audit';`
      5. Make sure there is no COMPRESSED tables
      6. Prepare mariadb for shut down: `SET GLOBAL innodb_fast_shutdown = 1; XA RECOVER;`
      7. `systemctl stop mariadb`
      8. `apt-get remove "mariadb-*"`
      9. Configure `/etc/apt/sources.list.d/mariadb.sources`:
      10. Disable pre-installed by OS-provided packages plugins: `rm /etc/mysql/mariadb.conf.d/provider_*` (Otherwise MariaDB fails to start after mariadb-server package installation).
      11. `apt update && apt -y remove mariadb-server`
      12. `apt -y install mariadb-server`
      13. `innochecksum /var/lib/mysql/ibdata1`

      I reviewed the MariaDB configuration files and found no specific InnoDB-related settings that could explain the issue. The system is a default AWS Ubuntu 24 instance with no custom pre-configuration, and the database contains no user data:

      > egrep innodb -R /etc/mysql/
      /etc/mysql/mariadb.conf.d/99-enable-encryption.cnf.preset/enable_encryption.preset:loose-innodb-encrypt-log
      /etc/mysql/mariadb.conf.d/99-enable-encryption.cnf.preset/enable_encryption.preset:loose-innodb-encrypt-tables
      /etc/mysql/mariadb.conf.d/50-server.cnf:# https://mariadb.com/kb/en/innodb-system-variables/#innodb_buffer_pool_size
      /etc/mysql/mariadb.conf.d/50-server.cnf:#innodb_buffer_pool_size = 8G
      /etc/mysql/mariadb.conf.d/60-galera.cnf:#innodb_autoinc_lock_mode = 2
      /etc/mysql/mariadb.conf.d/60-galera.cnf:#innodb_flush_log_at_trx_commit = 0
      

      Please note: I attempted to recover the corrupted pages using innodb_force_recovery with values 1, 3, and 6, but none of them resolved the issue.

      I've reviewed MDEV-36556, MDEV-30129 as well as the documentation for upgrade from 10.11 to 11.4 and upgrade community server to 11.4. Unfortunately, I haven't found a way to mitigate the issue.
      Could you please advise on how to avoid this corruption scenario during upgrade?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              msandakov Mikhail Sandakov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.