Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.5, 10.6, 10.3(EOL), 10.4(EOL)
-
None
-
CentOS Linux release 7.9.2009 (Core)
http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm
Description
DetailsThis relates to MDEV-15912, which has some nightly rpms built but is not available in public repos yet.
The failures occur even with innodb_fast_shutdown disable (set to 0).
I have two sample datadirs which can be used to reproduce the issue – one from mysql 5.7 directly, and one which went from 5.7 -> 10.3.29.
Both of these datadirs fail to start with 10.3.30.
Step-by-Step Instructions1. Start with a CentOS 7 server – this will be relevant for step 4 later.
2. Install MariaDB 10.3.29.
3. Extract the datadir from mariadb-10.3.29-datadir.tar.xz – start up MariaDB using this datadir and note that it runs without issues.
4. Update to 10.3.30 (not in yum repos yet, so installing via direct downloads):
yum -y install http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-client-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-common-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-compat-10.3.30-1.el7.centos.x86_64.rpm http://hasky.askmonty.org/archive/10.3/build-41050/kvm-rpm-centos74-amd64/rpms/MariaDB-server-10.3.30-1.el7.centos.x86_64.rpm
|
5. Observe failures:
[root@server ~]# systemctl status mariadb.service
|
● mariadb.service - MariaDB 10.3.30 database server
|
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
|
Drop-In: /etc/systemd/system/mariadb.service.d
|
└─migrated-from-my.cnf-settings.conf
|
Active: failed (Result: exit-code) since Mon 2021-06-21 23:56:01 UTC; 26s ago
|
Docs: man:mysqld(8)
|
https://mariadb.com/kb/en/library/systemd/
|
Process: 27604 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
|
Process: 27578 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
|
Process: 27576 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
|
Main PID: 27604 (code=exited, status=1/FAILURE)
|
Status: "MariaDB server is down"
|
|
|
Jun 21 23:56:00 server systemd[1]: Starting MariaDB 10.3.30 database server...
|
Jun 21 23:56:00 server mysqld[27604]: 2021-06-21 23:56:00 0 [Note] /usr/sbin/mysqld (mysqld 10.3.30-MariaDB) starting as process 27604 ...
|
Jun 21 23:56:01 server systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
|
Jun 21 23:56:01 server systemd[1]: Failed to start MariaDB 10.3.30 database server.
|
Jun 21 23:56:01 server systemd[1]: Unit mariadb.service entered failed state.
|
Jun 21 23:56:01 server systemd[1]: mariadb.service failed.
|
|
|
|
|
[root@server ~]# tail -n 30 /var/log/mysqld.log
|
...
|
2021-06-21 21:41:08 0 [Note] InnoDB: Completed initialization of buffer pool
|
2021-06-21 21:41:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
|
2021-06-21 21:41:08 0 [ERROR] InnoDB: upgrade from older version than MariaDB 10.3 requires clean shutdown
|
2021-06-21 21:41:08 0 [Note] InnoDB: Retry with innodb_force_recovery=5
|
2021-06-21 21:41:08 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
|
2021-06-21 21:41:09 0 [Note] InnoDB: Starting shutdown...
|
2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' init function returned error.
|
2021-06-21 21:41:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
|
2021-06-21 21:41:09 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2021-06-21 21:41:09 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
|
2021-06-21 21:41:09 0 [ERROR] Unknown/unsupported storage engine: InnoDB
|
2021-06-21 21:41:09 0 [ERROR] Aborting
|
Suggested Resolution / Expected BehaviorI would not expect a minor version update to suddenly fail with innodb errors that require "innodb_force_recovery=5".
Known work around(s)Rolling back to 10.3.29 works and seems like the best option.
If an average end-user sees the suggestion in the log and tries "innodb_force_recovery=5" I think they'd be more likely to do harm than good.
