Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25981

Upgrade results in InnoDB failures

Details

    Description

      Details

      This relates to MDEV-15912, which has some nightly rpms built but is not available in public repos yet.

      The failures occur even with innodb_fast_shutdown disable (set to 0).

      I have two sample datadirs which can be used to reproduce the issue – one from mysql 5.7 directly, and one which went from 5.7 -> 10.3.29.
      Both of these datadirs fail to start with 10.3.30.

      Step-by-Step Instructions

      1. Start with a CentOS 7 server – this will be relevant for step 4 later.

      2. Install MariaDB 10.3.29.

      3. Extract the datadir from mariadb-10.3.29-datadir.tar.xz – start up MariaDB using this datadir and note that it runs without issues.

      4. Update to 10.3.30 (not in yum repos yet, so installing via direct downloads):

      yum -y install http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-client-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-common-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-compat-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm

      5. Observe failures:

      [root@server ~]# systemctl status mariadb.service
      		 
      ● mariadb.service - MariaDB 10.3.30 database server
      		 
         Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
      		 
        Drop-In: /etc/systemd/system/mariadb.service.d
      		 
                 └─migrated-from-my.cnf-settings.conf
      		 
         Active: failed (Result: exit-code) since Mon 2021-06-21 23:56:01 UTC; 26s ago
      		 
           Docs: man:mysqld(8)
      		 
                 https://mariadb.com/kb/en/library/systemd/
      		 
        Process: 27604 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      		 
        Process: 27578 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
      		 
        Process: 27576 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      		 
       Main PID: 27604 (code=exited, status=1/FAILURE)
      		 
         Status: "MariaDB server is down"
      		 
       
      		 
      Jun 21 23:56:00 server systemd[1]: Starting MariaDB 10.3.30 database server...
      		 
      Jun 21 23:56:00 server mysqld[27604]: 2021-06-21 23:56:00 0 [Note] /usr/sbin/mysqld (mysqld 10.3.30-MariaDB) starting as process 27604 ...
      		 
      Jun 21 23:56:01 server systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
      		 
      Jun 21 23:56:01 server systemd[1]: Failed to start MariaDB 10.3.30 database server.
      		 
      Jun 21 23:56:01 server systemd[1]: Unit mariadb.service entered failed state.
      		 
      Jun 21 23:56:01 server systemd[1]: mariadb.service failed.
      		 
       
      		 
       
      		 
      [root@server ~]# tail -n 30 /var/log/mysqld.log
      		 
      ...
      		 
      2021-06-21 21:41:08 0 [Note] InnoDB: Completed initialization of buffer pool
      		 
      2021-06-21 21:41:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
      		 
      2021-06-21 21:41:08 0 [ERROR] InnoDB: upgrade from older version than MariaDB 10.3 requires clean shutdown
      		 
      2021-06-21 21:41:08 0 [Note] InnoDB: Retry with innodb_force_recovery=5
      		 
      2021-06-21 21:41:08 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
      		 
      2021-06-21 21:41:09 0 [Note] InnoDB: Starting shutdown...
      		 
      2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' init function returned error.
      		 
      2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
      		 
      2021-06-21 21:41:09 0 [Note] Plugin 'FEEDBACK' is disabled.
      		 
      2021-06-21 21:41:09 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
      		 
      2021-06-21 21:41:09 0 [ERROR] Unknown/unsupported storage engine: InnoDB
      		 
      2021-06-21 21:41:09 0 [ERROR] Aborting

      Suggested Resolution / Expected Behavior

      I would not expect a minor version update to suddenly fail with innodb errors that require "innodb_force_recovery=5".

      Known work around(s)

      Rolling back to 10.3.29 works and seems like the best option.

      If an average end-user sees the suggestion in the log and tries "innodb_force_recovery=5" I think they'd be more likely to do harm than good.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15912 InnoDB: Failing assertion: purge_sys.tail.commit <= purge_sys.rseg->last_commit upon upgrade from 10.0 or 10.1 to 10.3

          • Major - Major loss of function.
          • Closed
          mentioned in

          Page Failed to load

          Page Failed to load

          Page Failed to load

          Page Failed to load

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            Ascending order - Click to sort in descending order
            marko Marko Mäkelä added a comment -

            Thank you for filing this. I should have tested this better upfront. The problem also manifests itself as upgrade failures on our CI system, and I intended to work on this as the first thing today.

            The necessary fix should be simple. We must allow the type code TRX_UNDO_INSERT=1 in TRX_UNDO_CACHED pages. The function trx_undo_reuse_cached() will reset the type code to 0.

            I have pushed a fix to a staging branch, to see if the upgrade tests would now pass. The fix will need some further refinement, because the data directories attached to MDEV-15912 and MDEV-15464 will result in crashes.

            marko Marko Mäkelä added a comment - Thank you for filing this. I should have tested this better upfront. The problem also manifests itself as upgrade failures on our CI system, and I intended to work on this as the first thing today. The necessary fix should be simple. We must allow the type code TRX_UNDO_INSERT=1 in TRX_UNDO_CACHED pages. The function trx_undo_reuse_cached() will reset the type code to 0. I have pushed a fix to a staging branch, to see if the upgrade tests would now pass. The fix will need some further refinement, because the data directories attached to MDEV-15912 and MDEV-15464 will result in crashes.
            marko Marko Mäkelä added a comment -

            The VERSION file in the source code repository (as well as the version number reported by the server) is only tentative. It turns out that there will be an unscheduled 10.3.30 release, which will exclude MDEV-15912. Thus, neither the 10.3.30 nor the 10.3.31 releases should be affected by this.

            I tested the upgrade locally from a freshly created 10.2 data directory in mysql-test/var/install.db, as well as the data directories from MDEV-15912 and MDEV-15464, and your attached mysql-5.7.34-datadir.tar.xz and mariadb-10.3.29-datadir.tar.xz.

            If upgrade from mariadb-10.3.29-datadir.tar.xz failed with MDEV-15912 before this fix, then that data directory must have been originally created with something earlier than the first GA release of MariaDB Server 10.3.

            marko Marko Mäkelä added a comment - The VERSION file in the source code repository (as well as the version number reported by the server) is only tentative. It turns out that there will be an unscheduled 10.3.30 release, which will exclude MDEV-15912 . Thus, neither the 10.3.30 nor the 10.3.31 releases should be affected by this. I tested the upgrade locally from a freshly created 10.2 data directory in mysql-test/var/install.db , as well as the data directories from MDEV-15912 and MDEV-15464 , and your attached mysql-5.7.34-datadir.tar.xz and mariadb-10.3.29-datadir.tar.xz . If upgrade from mariadb-10.3.29-datadir.tar.xz failed with MDEV-15912 before this fix, then that data directory must have been originally created with something earlier than the first GA release of MariaDB Server 10.3.

            People

              marko Marko Mäkelä
              hborresen Hans Borresen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.