[MDEV-25981] Upgrade results in InnoDB failures Created: 2021-06-22  Updated: 2023-11-08  Resolved: 2021-06-22

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3, 10.4, 10.5, 10.6
Fix Version/s: 10.3.31, 10.4.21, 10.5.12, 10.6.3

Type: Bug Priority: Critical
Reporter: Hans Borresen Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS Linux release 7.9.2009 (Core)
http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm

Attachments: File mariadb-10.3.29-datadir.tar.xz     File mysql-5.7.34-datadir.tar.xz    
Issue Links:
Problem/Incident
is caused by MDEV-15912 InnoDB: Failing assertion: purge_sys.... Closed

 Description   
Details

This relates to MDEV-15912, which has some nightly rpms built but is not available in public repos yet.

The failures occur even with innodb_fast_shutdown disable (set to 0).

I have two sample datadirs which can be used to reproduce the issue – one from mysql 5.7 directly, and one which went from 5.7 -> 10.3.29.
Both of these datadirs fail to start with 10.3.30.

Step-by-Step Instructions

1. Start with a CentOS 7 server – this will be relevant for step 4 later.

2. Install MariaDB 10.3.29.

3. Extract the datadir from mariadb-10.3.29-datadir.tar.xz – start up MariaDB using this datadir and note that it runs without issues.

4. Update to 10.3.30 (not in yum repos yet, so installing via direct downloads):

yum -y install http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-client-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-common-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-compat-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm

5. Observe failures:

[root@server ~]# systemctl status mariadb.service
 
● mariadb.service - MariaDB 10.3.30 database server
 
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
 
  Drop-In: /etc/systemd/system/mariadb.service.d
 
           └─migrated-from-my.cnf-settings.conf
 
   Active: failed (Result: exit-code) since Mon 2021-06-21 23:56:01 UTC; 26s ago
 
     Docs: man:mysqld(8)
 
           https://mariadb.com/kb/en/library/systemd/
 
  Process: 27604 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
 
  Process: 27578 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
 
  Process: 27576 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 
 Main PID: 27604 (code=exited, status=1/FAILURE)
 
   Status: "MariaDB server is down"
 
 
 
Jun 21 23:56:00 server systemd[1]: Starting MariaDB 10.3.30 database server...
 
Jun 21 23:56:00 server mysqld[27604]: 2021-06-21 23:56:00 0 [Note] /usr/sbin/mysqld (mysqld 10.3.30-MariaDB) starting as process 27604 ...
 
Jun 21 23:56:01 server systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
 
Jun 21 23:56:01 server systemd[1]: Failed to start MariaDB 10.3.30 database server.
 
Jun 21 23:56:01 server systemd[1]: Unit mariadb.service entered failed state.
 
Jun 21 23:56:01 server systemd[1]: mariadb.service failed.
 
 
 
 
 
[root@server ~]# tail -n 30 /var/log/mysqld.log
 
...
 
2021-06-21 21:41:08 0 [Note] InnoDB: Completed initialization of buffer pool
 
2021-06-21 21:41:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2021-06-21 21:41:08 0 [ERROR] InnoDB: upgrade from older version than MariaDB 10.3 requires clean shutdown
 
2021-06-21 21:41:08 0 [Note] InnoDB: Retry with innodb_force_recovery=5
 
2021-06-21 21:41:08 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
 
2021-06-21 21:41:09 0 [Note] InnoDB: Starting shutdown...
 
2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' init function returned error.
 
2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 
2021-06-21 21:41:09 0 [Note] Plugin 'FEEDBACK' is disabled.
 
2021-06-21 21:41:09 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
 
2021-06-21 21:41:09 0 [ERROR] Unknown/unsupported storage engine: InnoDB
 
2021-06-21 21:41:09 0 [ERROR] Aborting

Suggested Resolution / Expected Behavior

I would not expect a minor version update to suddenly fail with innodb errors that require "innodb_force_recovery=5".

Known work around(s)

Rolling back to 10.3.29 works and seems like the best option.

If an average end-user sees the suggestion in the log and tries "innodb_force_recovery=5" I think they'd be more likely to do harm than good.

 Comments   
Comment by Marko Mäkelä [ 2021-06-22 ]

Thank you for filing this. I should have tested this better upfront. The problem also manifests itself as upgrade failures on our CI system, and I intended to work on this as the first thing today.

The necessary fix should be simple. We must allow the type code TRX_UNDO_INSERT=1 in TRX_UNDO_CACHED pages. The function trx_undo_reuse_cached() will reset the type code to 0.

I have pushed a fix to a staging branch, to see if the upgrade tests would now pass. The fix will need some further refinement, because the data directories attached to MDEV-15912 and MDEV-15464 will result in crashes.

Comment by Marko Mäkelä [ 2021-06-22 ]

The VERSION file in the source code repository (as well as the version number reported by the server) is only tentative. It turns out that there will be an unscheduled 10.3.30 release, which will exclude MDEV-15912. Thus, neither the 10.3.30 nor the 10.3.31 releases should be affected by this.

I tested the upgrade locally from a freshly created 10.2 data directory in mysql-test/var/install.db, as well as the data directories from MDEV-15912 and MDEV-15464, and your attached mysql-5.7.34-datadir.tar.xz and mariadb-10.3.29-datadir.tar.xz.

If upgrade from mariadb-10.3.29-datadir.tar.xz failed with MDEV-15912 before this fix, then that data directory must have been originally created with something earlier than the first GA release of MariaDB Server 10.3.

Generated at Thu Feb 08 09:41:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.