Centos8 server with WordPress. Appstream packages from CentOS8: [root@server1 ~]# rpm -qa | grep maria
mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-3.0.7-1.el8.x86_64
mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-config-3.0.7-1.el8.noarch
mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64

The stack traces in journal.txt suggest that an InnoDB undo log page is corrupted. I am afraid that you will have to resort to starting up the server with innodb_force_recovery=5 and taking a logical dump of all data (mysqldump) so that you can reinitialize the database.

Note: innodb_force_recovery=5 will hard-wire the transaction isolation level to READ UNCOMMITTED. Also note that before version 10.5 (with MDEV-19514), setting innodb_force_recovery to the value 4 or 5 may corrupt secondary indexes. (Setting the parameter to the value 6 can corrupt anything.)

If you have a repeatable test case (using SQL commands, and not using this corrupted data directory as a starting point), I am happy to analyze and fix it.

Thanks for the quick reply Marko!

I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.

Can you please tell me more specifically what I need to do?
FYI, I have erased and reinstalled all mariadb packages several times,
hoping this would somehow put everything back to default, but this
didn't help...

So, do I need to:

# systemctl start mariadb.service

, with an option like
innodb_force_recovery=5 (how do I do that?) or exactly what
command do I have to give?
I am running 10.3.17, so giving the value 5, as you suggest, "will
corrupt secondary indexes". What does this mean?
I do have backups of my wordpress database, which is why I am using
mariadb, so I am more than happy to remove things, if this could solve
the problem. I can always re-install my wordpress database from my
backup.

I am very grateful for your help here! Can you please give more
specific instructions?

/Ralf

Ralf Hartings kommenterade MDEV-22624:
--------------------------------------

Thanks for the quick reply Marko!

I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.

Can you please tell me more specifically what I need to do?
FYI, I have erased and reinstalled all mariadb packages several times,
hoping this would somehow put everything back to default, but this
didn't help...

So, do I need to:

1. systemctl start mariadb.service , with an option like
   innodb_force_recovery=5 (how do I do that?) or exactly what
   command do I have to give?
   I am running 10.3.17, so giving the value 5, as you suggest, "will
   corrupt secondary indexes". What does this mean?
   I do have backups of my wordpress database, which is why I am using
   mariadb, so I am more than happy to remove things, if this could solve
   the problem. I can always re-install my wordpress database from my
   backup.

I am very grateful for your help here! Can you please give more
specific instructions?

/Ralf

Marko, sorry for messing up this reply function...

In the mean time I tried to trace back anything related to InnoDB, as this seems to be the reason/indication for/of the crash according to your reply.

In Centos8 I found this directory /var/lib/mysql/mysql, where all but two files are dated Oct 16 (when I installed it).
The two other files are dated May 16th, which is when the whole server locked itself.
These files are:

rw-rw---. 1 mysql mysql 163840 May 16 18:35 innodb_index_stats.ibd
rw-rw---. 1 mysql mysql 98304 May 16 18:35 innodb_table_stats.ibd

Can it be as simple as to delete this whole directory or just the two files and reinstall all mariadb packages?

Or, shoud I re-install the below two packages, which include something like "mysql" (as the files above are owned by mysql).
I am assuming the files above in /var/lib/mysql/mysql have been created by a package that deals with mysql.

[root@server1 mysql]# rpm -qa | grep mysql
grafana-mysql-6.3.6-1.el8.x86_64**php-mysqlnd-7.2.31-1.el8.remi.x86_64

Thanks again for looking into this!

/Ralf

Ralf Hartings kommenterade MDEV-22624:
--------------------------------------

Thanks for the quick reply Marko!

I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.

Can you please tell me more specifically what I need to do?
FYI, I have erased and reinstalled all mariadb packages several times,
hoping this would somehow put everything back to default, but this
didn't help...

So, do I need to:

1. systemctl start mariadb.service , with an option like
   innodb_force_recovery=5 (how do I do that?) or exactly what
   command do I have to give?
   I am running 10.3.17, so giving the value 5, as you suggest, "will
   corrupt secondary indexes". What does this mean?
   I do have backups of my wordpress database, which is why I am using
   mariadb, so I am more than happy to remove things, if this could solve
   the problem. I can always re-install my wordpress database from my
   backup.

I am very grateful for your help here! Can you please give more
specific instructions?

/Ralf

ralf, sorry, I am not that familiar with systemd. I am an abnormal user of the database, because I only use it to run tests, to test the code that I my colleagues are modifying.

Our support engineers would be happy to answer such questions to our customers.

If you can repeat this problem by starting from an empty database (loading a logical dump or something), then I would be happy to analyze it. In our internal stress testing with Random Query Generator, we have not encountered any bugs in the undo logs recently. We have encountered and fixed some bugs in crash recovery, and theoretically those fixes might prevent this from reoccurring in the future.

Lets close this case
I did a fresh install instead