Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Incomplete
-
10.3.17
-
Centos8 server with WordPress. Appstream packages from CentOS8: [root@server1 ~]# rpm -qa | grep maria
mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-3.0.7-1.el8.x86_64
mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-config-3.0.7-1.el8.noarch
mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64Centos8 server with WordPress. Appstream packages from CentOS8: [ root@server1 ~]# rpm -qa | grep maria mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-connector-c-3.0.7-1.el8.x86_64 mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-connector-c-config-3.0.7-1.el8.noarch mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
Description
Yesterday, mariadb started to crash and restart continuously. No changes done to server in last 2 weeks, no updates done i these 2 weeks:
May 18 13:46:24 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
|
May 18 13:46:24 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.
|
May 18 13:46:29 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
|
May 18 13:46:29 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1424.
|
May 18 13:46:31 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
|
May 18 13:46:31 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.
|
May 18 13:46:36 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
|
May 18 13:46:36 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1425.
|
Forum told me to report this bug here.
Journalctl output in attached file
Recently, I experienced many attempts to access phpmyadmin and the database.
From mariadb logfile:
[root@server1 mariadb]# more mariadb.log
|
2020-05-16 3:16:31 61116 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
|
2020-05-16 3:55:05 61123 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
|
2020-05-16 5:13:37 61127 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
|
2020-05-16 5:55:02 61133 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
|
2020-05-16 6:34:25 61136 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
|
2020-05-16 7:19:49 61142 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
|
2020-05-16 7:28:10 61144 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
|
2020-05-16 8:01:01 61147 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
|
2020-05-16 8:38:41 61151 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
|
2020-05-16 9:56:38 61157 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
|
2020-05-16 10:30:18 61164 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
|
2020-05-16 11:02:05 61167 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
|
2020-05-16 11:35:58 61171 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
|
2020-05-16 12:09:35 61188 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
|
2020-05-16 12:39:52 61205 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
|
2020-05-16 13:11:01 61230 [Warning] Access denied for user 'admin'@'localhost' (using password: YES)
|
2020-05-16 13:29:02 61247 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
|
2020-05-16 14:07:09 61283 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
|
2020-05-16 14:32:38 61298 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
|
2020-05-16 15:02:23 61528 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
|
2020-05-16 16:05:05 61585 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
|
2020-05-16 16:29:38 61612 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
|
2020-05-16 17:01:17 61637 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
|
2020-05-16 17:29:22 61664 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
|
2020-05-16 17:56:55 61689 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
|
2020-05-16 18:25:28 61713 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
|
2020-05-16 19:02:20 61745 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
|
2020-05-17 10:18:03 0 [Note] InnoDB: Using Linux native AIO
|
2020-05-17 10:18:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2020-05-17 10:18:03 0 [Note] InnoDB: Uses event mutexes
|
2020-05-17 10:18:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2020-05-17 10:18:03 0 [Note] InnoDB: Number of pools: 1
|
2020-05-17 10:18:03 0 [Note] InnoDB: Using SSE2 crc32 instructions
|
2020-05-17 10:18:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
|
2020-05-17 10:18:03 0 [Note] InnoDB: Completed initialization of buffer pool
|
2020-05-17 10:18:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man pa
|
ge of setpriority().
|
2020-05-17 10:18:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=79381745
|
2020-05-17 10:18:03 0 [ERROR] [FATAL] InnoDB: Trying to read page number 4294934527 in space 0, space name innodb_system, which is outside t
|
he tablespace bounds. Byte offset 0, len 16384Please check that the configuration matches the InnoDB system tablespace location (ibdata file
|
s)
|
200517 10:18:03 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
[root@server1 mariadb]#
|
See more info on CENTOS forum:
https://forums.centos.org/viewtopic.php?f=54&t=74436&p=313469#p313469
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
Thanks for the quick reply Marko!
I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.
Can you please tell me more specifically what I need to do?
FYI, I have erased and reinstalled all mariadb packages several times,
hoping this would somehow put everything back to default, but this
didn't help...
So, do I need to:
# systemctl start mariadb.service
|
, with an option like
innodb_force_recovery=5 (how do I do that?) or exactly what
command do I have to give?
I am running 10.3.17, so giving the value 5, as you suggest, "will
corrupt secondary indexes". What does this mean?
I do have backups of my wordpress database, which is why I am using
mariadb, so I am more than happy to remove things, if this could solve
the problem. I can always re-install my wordpress database from my
backup.
I am very grateful for your help here! Can you please give more
specific instructions?
/Ralf
Ralf Hartings kommenterade MDEV-22624:
--------------------------------------
Thanks for the quick reply Marko!
I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.
Can you please tell me more specifically what I need to do?
FYI, I have erased and reinstalled all mariadb packages several times,
hoping this would somehow put everything back to default, but this
didn't help...
So, do I need to:
- systemctl start mariadb.service , with an option like
innodb_force_recovery=5 (how do I do that?) or exactly what
command do I have to give?
I am running 10.3.17, so giving the value 5, as you suggest, "will
corrupt secondary indexes". What does this mean?
I do have backups of my wordpress database, which is why I am using
mariadb, so I am more than happy to remove things, if this could solve
the problem. I can always re-install my wordpress database from my
backup.
I am very grateful for your help here! Can you please give more
specific instructions?
/Ralf
Marko, sorry for messing up this reply function...
In the mean time I tried to trace back anything related to InnoDB, as this seems to be the reason/indication for/of the crash according to your reply.
In Centos8 I found this directory /var/lib/mysql/mysql, where all but two files are dated Oct 16 (when I installed it).
The two other files are dated May 16th, which is when the whole server locked itself.
These files are:
rw-rw---. 1 mysql mysql 163840 May 16 18:35 innodb_index_stats.ibd
rw-rw---. 1 mysql mysql 98304 May 16 18:35 innodb_table_stats.ibd
Can it be as simple as to delete this whole directory or just the two files and reinstall all mariadb packages?
Or, shoud I re-install the below two packages, which include something like "mysql" (as the files above are owned by mysql).
I am assuming the files above in /var/lib/mysql/mysql have been created by a package that deals with mysql. 
[root@server1 mysql]# rpm -qa | grep mysql
grafana-mysql-6.3.6-1.el8.x86_64**php-mysqlnd-7.2.31-1.el8.remi.x86_64
Thanks again for looking into this!
/Ralf
Ralf Hartings kommenterade MDEV-22624:
--------------------------------------
Thanks for the quick reply Marko!
I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.
Can you please tell me more specifically what I need to do?
FYI, I have erased and reinstalled all mariadb packages several times,
hoping this would somehow put everything back to default, but this
didn't help...
So, do I need to:
- systemctl start mariadb.service , with an option like
innodb_force_recovery=5 (how do I do that?) or exactly what
command do I have to give?
I am running 10.3.17, so giving the value 5, as you suggest, "will
corrupt secondary indexes". What does this mean?
I do have backups of my wordpress database, which is why I am using
mariadb, so I am more than happy to remove things, if this could solve
the problem. I can always re-install my wordpress database from my
backup.
I am very grateful for your help here! Can you please give more
specific instructions?
/Ralf
ralf, sorry, I am not that familiar with systemd. I am an abnormal user of the database, because I only use it to run tests, to test the code that I my colleagues are modifying.
Our support engineers would be happy to answer such questions to our customers.
If you can repeat this problem by starting from an empty database (loading a logical dump or something), then I would be happy to analyze it. In our internal stress testing with Random Query Generator, we have not encountered any bugs in the undo logs recently. We have encountered and fixed some bugs in crash recovery, and theoretically those fixes might prevent this from reoccurring in the future.
The stack traces in journal.txt
suggest that an InnoDB undo log page is corrupted. I am afraid that you will have to resort to starting up the server with innodb_force_recovery=5 and taking a logical dump of all data (mysqldump) so that you can reinitialize the database.
Note: innodb_force_recovery=5 will hard-wire the transaction isolation level to READ UNCOMMITTED. Also note that before version 10.5 (with
MDEV-19514), setting innodb_force_recovery to the value 4 or 5 may corrupt secondary indexes. (Setting the parameter to the value 6 can corrupt anything.)If you have a repeatable test case (using SQL commands, and not using this corrupted data directory as a starting point), I am happy to analyze and fix it.