Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25981

Upgrade results in InnoDB failures

    XMLWordPrintable

    Details

      Description

      Details

      This relates to MDEV-15912, which has some nightly rpms built but is not available in public repos yet.

      The failures occur even with innodb_fast_shutdown disable (set to 0).

      I have two sample datadirs which can be used to reproduce the issue – one from mysql 5.7 directly, and one which went from 5.7 -> 10.3.29.
      Both of these datadirs fail to start with 10.3.30.


      Step-by-Step Instructions

      1. Start with a CentOS 7 server – this will be relevant for step 4 later.

      2. Install MariaDB 10.3.29.

      3. Extract the datadir from mariadb-10.3.29-datadir.tar.xz – start up MariaDB using this datadir and note that it runs without issues.

      4. Update to 10.3.30 (not in yum repos yet, so installing via direct downloads):

      yum -y install http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-client-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-common-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-compat-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm
      

      5. Observe failures:

      [root@server ~]# systemctl status mariadb.service
      ● mariadb.service - MariaDB 10.3.30 database server
         Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
        Drop-In: /etc/systemd/system/mariadb.service.d
                 └─migrated-from-my.cnf-settings.conf
         Active: failed (Result: exit-code) since Mon 2021-06-21 23:56:01 UTC; 26s ago
           Docs: man:mysqld(8)
                 https://mariadb.com/kb/en/library/systemd/
        Process: 27604 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
        Process: 27578 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
        Process: 27576 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
       Main PID: 27604 (code=exited, status=1/FAILURE)
         Status: "MariaDB server is down"
       
      Jun 21 23:56:00 server systemd[1]: Starting MariaDB 10.3.30 database server...
      Jun 21 23:56:00 server mysqld[27604]: 2021-06-21 23:56:00 0 [Note] /usr/sbin/mysqld (mysqld 10.3.30-MariaDB) starting as process 27604 ...
      Jun 21 23:56:01 server systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
      Jun 21 23:56:01 server systemd[1]: Failed to start MariaDB 10.3.30 database server.
      Jun 21 23:56:01 server systemd[1]: Unit mariadb.service entered failed state.
      Jun 21 23:56:01 server systemd[1]: mariadb.service failed.
       
       
      [root@server ~]# tail -n 30 /var/log/mysqld.log
      ...
      2021-06-21 21:41:08 0 [Note] InnoDB: Completed initialization of buffer pool
      2021-06-21 21:41:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
      2021-06-21 21:41:08 0 [ERROR] InnoDB: upgrade from older version than MariaDB 10.3 requires clean shutdown
      2021-06-21 21:41:08 0 [Note] InnoDB: Retry with innodb_force_recovery=5
      2021-06-21 21:41:08 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
      2021-06-21 21:41:09 0 [Note] InnoDB: Starting shutdown...
      2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' init function returned error.
      2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
      2021-06-21 21:41:09 0 [Note] Plugin 'FEEDBACK' is disabled.
      2021-06-21 21:41:09 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
      2021-06-21 21:41:09 0 [ERROR] Unknown/unsupported storage engine: InnoDB
      2021-06-21 21:41:09 0 [ERROR] Aborting
      


      Suggested Resolution / Expected Behavior

      I would not expect a minor version update to suddenly fail with innodb errors that require "innodb_force_recovery=5".


      Known work around(s)

      Rolling back to 10.3.29 works and seems like the best option.

      If an average end-user sees the suggestion in the log and tries "innodb_force_recovery=5" I think they'd be more likely to do harm than good.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marko Marko Mäkelä
              Reporter:
              hborresen Hans Borresen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration