Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.3(EOL), 10.4(EOL), 10.5, 10.6
-
None
-
CentOS Linux release 7.9.2009 (Core)
http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm
Description
DetailsThis relates to MDEV-15912, which has some nightly rpms built but is not available in public repos yet.
The failures occur even with innodb_fast_shutdown disable (set to 0).
I have two sample datadirs which can be used to reproduce the issue – one from mysql 5.7 directly, and one which went from 5.7 -> 10.3.29.
Both of these datadirs fail to start with 10.3.30.
Step-by-Step Instructions1. Start with a CentOS 7 server – this will be relevant for step 4 later.
2. Install MariaDB 10.3.29.
3. Extract the datadir from mariadb-10.3.29-datadir.tar.xz – start up MariaDB using this datadir and note that it runs without issues.
4. Update to 10.3.30 (not in yum repos yet, so installing via direct downloads):
yum -y install http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-client-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-common-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-compat-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm
|
5. Observe failures:
[root@server ~]# systemctl status mariadb.service
|
● mariadb.service - MariaDB 10.3.30 database server
|
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
|
Drop-In: /etc/systemd/system/mariadb.service.d
|
└─migrated-from-my.cnf-settings.conf
|
Active: failed (Result: exit-code) since Mon 2021-06-21 23:56:01 UTC; 26s ago
|
Docs: man:mysqld(8)
|
https://mariadb.com/kb/en/library/systemd/
|
Process: 27604 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
|
Process: 27578 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
|
Process: 27576 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Main PID: 27604 (code=exited, status=1/FAILURE)
|
Status: "MariaDB server is down"
|
|
|
Jun 21 23:56:00 server systemd[1]: Starting MariaDB 10.3.30 database server...
|
Jun 21 23:56:00 server mysqld[27604]: 2021-06-21 23:56:00 0 [Note] /usr/sbin/mysqld (mysqld 10.3.30-MariaDB) starting as process 27604 ...
|
Jun 21 23:56:01 server systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
|
Jun 21 23:56:01 server systemd[1]: Failed to start MariaDB 10.3.30 database server.
|
Jun 21 23:56:01 server systemd[1]: Unit mariadb.service entered failed state.
|
Jun 21 23:56:01 server systemd[1]: mariadb.service failed.
|
|
|
|
|
[root@server ~]# tail -n 30 /var/log/mysqld.log
|
...
|
2021-06-21 21:41:08 0 [Note] InnoDB: Completed initialization of buffer pool
|
2021-06-21 21:41:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
|
2021-06-21 21:41:08 0 [ERROR] InnoDB: upgrade from older version than MariaDB 10.3 requires clean shutdown
|
2021-06-21 21:41:08 0 [Note] InnoDB: Retry with innodb_force_recovery=5
|
2021-06-21 21:41:08 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
|
2021-06-21 21:41:09 0 [Note] InnoDB: Starting shutdown...
|
2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' init function returned error.
|
2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
|
2021-06-21 21:41:09 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2021-06-21 21:41:09 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
|
2021-06-21 21:41:09 0 [ERROR] Unknown/unsupported storage engine: InnoDB
|
2021-06-21 21:41:09 0 [ERROR] Aborting
|
Suggested Resolution / Expected BehaviorI would not expect a minor version update to suddenly fail with innodb errors that require "innodb_force_recovery=5".
Known work around(s)Rolling back to 10.3.29 works and seems like the best option.
If an average end-user sees the suggestion in the log and tries "innodb_force_recovery=5" I think they'd be more likely to do harm than good.
Attachments
Issue Links
- is caused by
-
-
- Closed
-
- mentioned in
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Loading...
-
-
Activity
The VERSION file in the source code repository (as well as the version number reported by the server) is only tentative. It turns out that there will be an unscheduled 10.3.30 release, which will exclude MDEV-15912. Thus, neither the 10.3.30 nor the 10.3.31 releases should be affected by this.
I tested the upgrade locally from a freshly created 10.2 data directory in mysql-test/var/install.db, as well as the data directories from MDEV-15912 and MDEV-15464, and your attached mysql-5.7.34-datadir.tar.xz
and mariadb-10.3.29-datadir.tar.xz.
If upgrade from mariadb-10.3.29-datadir.tar.xz failed with MDEV-15912 before this fix, then that data directory must have been originally created with something earlier than the first GA release of MariaDB Server 10.3.
Thank you for filing this. I should have tested this better upfront. The problem also manifests itself as upgrade failures on our CI system, and I intended to work on this as the first thing today.
The necessary fix should be simple. We must allow the type code TRX_UNDO_INSERT=1 in TRX_UNDO_CACHED pages. The function trx_undo_reuse_cached() will reset the type code to 0.
I have pushed a fix to a staging branch, to see if the upgrade tests would now pass. The fix will need some further refinement, because the data directories attached to
MDEV-15912andMDEV-15464will result in crashes.