[MDEV-22624] mariadb used for wordpress restarts and crashes continuously - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Incomplete
Affects Version/s: 10.3.17
Fix Version/s: N/A
Component/s: Platform RedHat
Labels:
Environment:

Hide
Centos8 server with WordPress. Appstream packages from CentOS8: [root@server1 ~]# rpm -qa | grep maria
mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-3.0.7-1.el8.x86_64
mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-config-3.0.7-1.el8.noarch
mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64

Show
Centos8 server with WordPress. Appstream packages from CentOS8: [ root@server1 ~]# rpm -qa | grep maria mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-connector-c-3.0.7-1.el8.x86_64 mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-connector-c-config-3.0.7-1.el8.noarch mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64 mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64

Description

Yesterday, mariadb started to crash and restart continuously. No changes done to server in last 2 weeks, no updates done i these 2 weeks:

May 18 13:46:24 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT

May 18 13:46:24 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.

May 18 13:46:29 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.

May 18 13:46:29 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1424.

May 18 13:46:31 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT

May 18 13:46:31 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.

May 18 13:46:36 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.

May 18 13:46:36 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1425.

Forum told me to report this bug here.
Journalctl output in attached file

Recently, I experienced many attempts to access phpmyadmin and the database.
From mariadb logfile:

[root@server1 mariadb]# more mariadb.log

2020-05-16  3:16:31 61116 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)

2020-05-16  3:55:05 61123 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)

2020-05-16  5:13:37 61127 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)

2020-05-16  5:55:02 61133 [Warning] Access denied for user 'user'@'localhost' (using password: YES)

2020-05-16  6:34:25 61136 [Warning] Access denied for user 'user'@'localhost' (using password: YES)

2020-05-16  7:19:49 61142 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)

2020-05-16  7:28:10 61144 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)

2020-05-16  8:01:01 61147 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)

2020-05-16  8:38:41 61151 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)

2020-05-16  9:56:38 61157 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)

2020-05-16 10:30:18 61164 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)

2020-05-16 11:02:05 61167 [Warning] Access denied for user 'root'@'localhost' (using password: YES)

2020-05-16 11:35:58 61171 [Warning] Access denied for user 'root'@'localhost' (using password: YES)

2020-05-16 12:09:35 61188 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)

2020-05-16 12:39:52 61205 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)

2020-05-16 13:11:01 61230 [Warning] Access denied for user 'admin'@'localhost' (using password: YES)

2020-05-16 13:29:02 61247 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)

2020-05-16 14:07:09 61283 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)

2020-05-16 14:32:38 61298 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)

2020-05-16 15:02:23 61528 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)

2020-05-16 16:05:05 61585 [Warning] Access denied for user 'user'@'localhost' (using password: YES)

2020-05-16 16:29:38 61612 [Warning] Access denied for user 'user'@'localhost' (using password: YES)

2020-05-16 17:01:17 61637 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)

2020-05-16 17:29:22 61664 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)

2020-05-16 17:56:55 61689 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)

2020-05-16 18:25:28 61713 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)

2020-05-16 19:02:20 61745 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)

2020-05-17 10:18:03 0 [Note] InnoDB: Using Linux native AIO

2020-05-17 10:18:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2020-05-17 10:18:03 0 [Note] InnoDB: Uses event mutexes

2020-05-17 10:18:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2020-05-17 10:18:03 0 [Note] InnoDB: Number of pools: 1

2020-05-17 10:18:03 0 [Note] InnoDB: Using SSE2 crc32 instructions

2020-05-17 10:18:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M

2020-05-17 10:18:03 0 [Note] InnoDB: Completed initialization of buffer pool

2020-05-17 10:18:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man pa

ge of setpriority().

2020-05-17 10:18:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=79381745

2020-05-17 10:18:03 0 [ERROR] [FATAL] InnoDB: Trying to read page number 4294934527 in space 0, space name innodb_system, which is outside t

he tablespace bounds. Byte offset 0, len 16384Please check that the configuration matches the InnoDB system tablespace location (ibdata file

s)

200517 10:18:03 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

[root@server1 mariadb]#

See more info on CENTOS forum:
https://forums.centos.org/viewtopic.php?f=54&t=74436&p=313469#p313469

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

2020-05-18Mariadb_my.cnf.odt
14 kB
2020-05-19 06:15
journal.txt
819 kB
2020-05-19 06:15
status2.txt
1 kB
2020-05-19 06:21
statusdb.txt
1.0 kB
2020-05-19 06:15

Issue Links

relates to

MDEV-13542 Crashing on a corrupted page is unhelpful

Closed

Activity

Ascending order - Click to sort in descending order

Ralf Hartings created issue - 2020-05-19 06:39

Elena Stepanova made changes - 2020-05-19 12:22

Field	Original Value	New Value
Epic/Theme	server

Elena Stepanova made changes - 2020-05-19 12:22

Assignee

Marko Mäkelä [ marko ]

Marko Mäkelä added a comment - 2020-05-19 14:34

The stack traces in journal.txt suggest that an InnoDB undo log page is corrupted. I am afraid that you will have to resort to starting up the server with innodb_force_recovery=5 and taking a logical dump of all data (mysqldump) so that you can reinitialize the database.

Note: innodb_force_recovery=5 will hard-wire the transaction isolation level to READ UNCOMMITTED. Also note that before version 10.5 (with ~~MDEV-19514~~), setting innodb_force_recovery to the value 4 or 5 may corrupt secondary indexes. (Setting the parameter to the value 6 can corrupt anything.)

If you have a repeatable test case (using SQL commands, and not using this corrupted data directory as a starting point), I am happy to analyze and fix it.

Marko Mäkelä added a comment - 2020-05-19 14:34 The stack traces in journal.txt suggest that an InnoDB undo log page is corrupted. I am afraid that you will have to resort to starting up the server with innodb_force_recovery=5 and taking a logical dump of all data ( mysqldump ) so that you can reinitialize the database. Note: innodb_force_recovery=5 will hard-wire the transaction isolation level to READ UNCOMMITTED . Also note that before version 10.5 (with MDEV-19514 ), setting innodb_force_recovery to the value 4 or 5 may corrupt secondary indexes. (Setting the parameter to the value 6 can corrupt anything.) If you have a repeatable test case (using SQL commands, and not using this corrupted data directory as a starting point), I am happy to analyze and fix it.

Marko Mäkelä made changes - 2020-05-19 14:34

Link

This issue relates to ~~MDEV-13542~~ [ ~~MDEV-13542~~ ]

Marko Mäkelä made changes - 2020-05-19 14:34

Labels

crash innodb

crash innodb need_feedback

Ralf Hartings added a comment - 2020-05-19 16:10 - edited

Thanks for the quick reply Marko!

I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.

Can you please tell me more specifically what I need to do?
FYI, I have erased and reinstalled all mariadb packages several times,
hoping this would somehow put everything back to default, but this
didn't help...

So, do I need to:

# systemctl start mariadb.service

, with an option like
innodb_force_recovery=5 (how do I do that?) or exactly what
command do I have to give?
I am running 10.3.17, so giving the value 5, as you suggest, "will
corrupt secondary indexes". What does this mean?
I do have backups of my wordpress database, which is why I am using
mariadb, so I am more than happy to remove things, if this could solve
the problem. I can always re-install my wordpress database from my
backup.

I am very grateful for your help here! Can you please give more
specific instructions?

/Ralf

Ralf Hartings added a comment - 2020-05-19 16:10 - edited Thanks for the quick reply Marko! I am afraid, I'll need some step by step help here, as I am no expert whatsoever in mariadb. Can you please tell me more specifically what I need to do? FYI, I have erased and reinstalled all mariadb packages several times, hoping this would somehow put everything back to default, but this didn't help... So, do I need to: # systemctl start mariadb.service , with an option like innodb_force_recovery=5 (how do I do that?) or exactly what command do I have to give? I am running 10.3.17, so giving the value 5, as you suggest, "will corrupt secondary indexes". What does this mean? I do have backups of my wordpress database, which is why I am using mariadb, so I am more than happy to remove things, if this could solve the problem. I can always re-install my wordpress database from my backup. I am very grateful for your help here! Can you please give more specific instructions? /Ralf

Ralf Hartings added a comment - 2020-05-19 16:25

Ralf Hartings kommenterade ~~MDEV-22624~~:
--------------------------------------

Thanks for the quick reply Marko!

I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.

So, do I need to:

systemctl start mariadb.service , with an option like
innodb_force_recovery=5 (how do I do that?) or exactly what
command do I have to give?
I am running 10.3.17, so giving the value 5, as you suggest, "will
corrupt secondary indexes". What does this mean?
I do have backups of my wordpress database, which is why I am using
mariadb, so I am more than happy to remove things, if this could solve
the problem. I can always re-install my wordpress database from my
backup.

I am very grateful for your help here! Can you please give more
specific instructions?

/Ralf

Ralf Hartings added a comment - 2020-05-19 16:25 Ralf Hartings kommenterade MDEV-22624 : -------------------------------------- Thanks for the quick reply Marko! I am afraid, I'll need some step by step help here, as I am no expert whatsoever in mariadb. Can you please tell me more specifically what I need to do? FYI, I have erased and reinstalled all mariadb packages several times, hoping this would somehow put everything back to default, but this didn't help... So, do I need to: systemctl start mariadb.service , with an option like innodb_force_recovery=5 (how do I do that?) or exactly what command do I have to give? I am running 10.3.17, so giving the value 5, as you suggest, "will corrupt secondary indexes". What does this mean? I do have backups of my wordpress database, which is why I am using mariadb, so I am more than happy to remove things, if this could solve the problem. I can always re-install my wordpress database from my backup. I am very grateful for your help here! Can you please give more specific instructions? /Ralf

Ralf Hartings added a comment - 2020-05-19 17:55

Marko, sorry for messing up this reply function...

In the mean time I tried to trace back anything related to InnoDB, as this seems to be the reason/indication for/of the crash according to your reply.

In Centos8 I found this directory /var/lib/mysql/mysql, where all but two files are dated Oct 16 (when I installed it).
The two other files are dated May 16th, which is when the whole server locked itself.
These files are:

~~rw-rw~~---. 1 mysql mysql 163840 May 16 18:35 innodb_index_stats.ibd
~~rw-rw~~---. 1 mysql mysql 98304 May 16 18:35 innodb_table_stats.ibd

Can it be as simple as to delete this whole directory or just the two files and reinstall all mariadb packages?

Or, shoud I re-install the below two packages, which include something like "mysql" (as the files above are owned by mysql).
I am assuming the files above in /var/lib/mysql/mysql have been created by a package that deals with mysql.

[root@server1 mysql]# rpm -qa | grep mysql
grafana-mysql-6.3.6-1.el8.x86_64**php-mysqlnd-7.2.31-1.el8.remi.x86_64

Thanks again for looking into this!

/Ralf

Ralf Hartings kommenterade ~~MDEV-22624~~:
--------------------------------------

Thanks for the quick reply Marko!

I am afraid, I'll need some step by step help here, as I am no expert
whatsoever in mariadb.

So, do I need to:

systemctl start mariadb.service , with an option like
innodb_force_recovery=5 (how do I do that?) or exactly what
command do I have to give?
I am running 10.3.17, so giving the value 5, as you suggest, "will
corrupt secondary indexes". What does this mean?
I do have backups of my wordpress database, which is why I am using
mariadb, so I am more than happy to remove things, if this could solve
the problem. I can always re-install my wordpress database from my
backup.

I am very grateful for your help here! Can you please give more
specific instructions?

/Ralf

Ralf Hartings added a comment - 2020-05-19 17:55 Marko, sorry for messing up this reply function... In the mean time I tried to trace back anything related to InnoDB, as this seems to be the reason/indication for/of the crash according to your reply. In Centos8 I found this directory /var/lib/mysql/mysql, where all but two files are dated Oct 16 (when I installed it). The two other files are dated May 16th, which is when the whole server locked itself. These files are: rw-rw ---. 1 mysql mysql 163840 May 16 18:35 innodb_index_stats.ibd rw-rw ---. 1 mysql mysql 98304 May 16 18:35 innodb_table_stats.ibd Can it be as simple as to delete this whole directory or just the two files and reinstall all mariadb packages? Or, shoud I re-install the below two packages, which include something like "mysql" (as the files above are owned by mysql). I am assuming the files above in /var/lib/mysql/mysql have been created by a package that deals with mysql. [root@server1 mysql] # rpm -qa | grep mysql grafana-mysql-6.3.6-1.el8.x86_64 ** php-mysqlnd-7.2.31-1.el8.remi.x86_64 Thanks again for looking into this! /Ralf Ralf Hartings kommenterade MDEV-22624 : -------------------------------------- Thanks for the quick reply Marko! I am afraid, I'll need some step by step help here, as I am no expert whatsoever in mariadb. Can you please tell me more specifically what I need to do? FYI, I have erased and reinstalled all mariadb packages several times, hoping this would somehow put everything back to default, but this didn't help... So, do I need to: systemctl start mariadb.service , with an option like innodb_force_recovery=5 (how do I do that?) or exactly what command do I have to give? I am running 10.3.17, so giving the value 5, as you suggest, "will corrupt secondary indexes". What does this mean? I do have backups of my wordpress database, which is why I am using mariadb, so I am more than happy to remove things, if this could solve the problem. I can always re-install my wordpress database from my backup. I am very grateful for your help here! Can you please give more specific instructions? /Ralf

Sergei Golubchik made changes - 2020-05-27 16:10

Labels

crash innodb need_feedback

crash innodb

Sergei Golubchik made changes - 2020-05-27 16:10

Description

Yesterday, mariadb started to crash and restart continuously. No changes done to server in last 2 weeks, no updates done i these 2 weeks:

May 18 13:46:24 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 18 13:46:24 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 18 13:46:29 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
May 18 13:46:29 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1424.
May 18 13:46:31 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 18 13:46:31 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 18 13:46:36 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
May 18 13:46:36 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1425.

Forum told me to report this bug here.
Journalctl output in attached file

Recently, I experienced many attempts to access phpmyadmin and the database.
From mariadb logfile:

[root@server1 mariadb]# more mariadb.log
2020-05-16 3:16:31 61116 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 3:55:05 61123 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 5:13:37 61127 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16 5:55:02 61133 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 6:34:25 61136 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 7:19:49 61142 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 7:28:10 61144 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 8:01:01 61147 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 8:38:41 61151 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 9:56:38 61157 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 10:30:18 61164 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 11:02:05 61167 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 11:35:58 61171 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 12:09:35 61188 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 12:39:52 61205 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 13:11:01 61230 [Warning] Access denied for user 'admin'@'localhost' (using password: YES)
2020-05-16 13:29:02 61247 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 14:07:09 61283 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 14:32:38 61298 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 15:02:23 61528 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16 16:05:05 61585 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 16:29:38 61612 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 17:01:17 61637 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:29:22 61664 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:56:55 61689 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 18:25:28 61713 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 19:02:20 61745 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-17 10:18:03 0 [Note] InnoDB: Using Linux native AIO
2020-05-17 10:18:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-05-17 10:18:03 0 [Note] InnoDB: Uses event mutexes
2020-05-17 10:18:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-05-17 10:18:03 0 [Note] InnoDB: Number of pools: 1
2020-05-17 10:18:03 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-05-17 10:18:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-05-17 10:18:03 0 [Note] InnoDB: Completed initialization of buffer pool
2020-05-17 10:18:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man pa
ge of setpriority().
2020-05-17 10:18:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=79381745
2020-05-17 10:18:03 0 [ERROR] [FATAL] InnoDB: Trying to read page number 4294934527 in space 0, space name innodb_system, which is outside t
he tablespace bounds. Byte offset 0, len 16384Please check that the configuration matches the InnoDB system tablespace location (ibdata file
s)
200517 10:18:03 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
[root@server1 mariadb]#

See more info on CENTOS forum:
https://forums.centos.org/viewtopic.php?f=54&t=74436&p=313469#p313469

Yesterday, mariadb started to crash and restart continuously. No changes done to server in last 2 weeks, no updates done i these 2 weeks:
{noformat}
May 18 13:46:24 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 18 13:46:24 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 18 13:46:29 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
May 18 13:46:29 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1424.
May 18 13:46:31 server1 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 18 13:46:31 server1 systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 18 13:46:36 server1 systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
May 18 13:46:36 server1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1425.
{noformat}
Forum told me to report this bug here.
Journalctl output in attached file

Recently, I experienced many attempts to access phpmyadmin and the database.
From mariadb logfile:
{noformat}
[root@server1 mariadb]# more mariadb.log
2020-05-16 3:16:31 61116 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 3:55:05 61123 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 5:13:37 61127 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16 5:55:02 61133 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 6:34:25 61136 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 7:19:49 61142 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 7:28:10 61144 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 8:01:01 61147 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 8:38:41 61151 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 9:56:38 61157 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 10:30:18 61164 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 11:02:05 61167 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 11:35:58 61171 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 12:09:35 61188 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 12:39:52 61205 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 13:11:01 61230 [Warning] Access denied for user 'admin'@'localhost' (using password: YES)
2020-05-16 13:29:02 61247 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 14:07:09 61283 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 14:32:38 61298 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 15:02:23 61528 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16 16:05:05 61585 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 16:29:38 61612 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 17:01:17 61637 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:29:22 61664 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:56:55 61689 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 18:25:28 61713 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 19:02:20 61745 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-17 10:18:03 0 [Note] InnoDB: Using Linux native AIO
2020-05-17 10:18:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-05-17 10:18:03 0 [Note] InnoDB: Uses event mutexes
2020-05-17 10:18:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-05-17 10:18:03 0 [Note] InnoDB: Number of pools: 1
2020-05-17 10:18:03 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-05-17 10:18:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-05-17 10:18:03 0 [Note] InnoDB: Completed initialization of buffer pool
2020-05-17 10:18:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man pa
ge of setpriority().
2020-05-17 10:18:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=79381745
2020-05-17 10:18:03 0 [ERROR] [FATAL] InnoDB: Trying to read page number 4294934527 in space 0, space name innodb_system, which is outside t
he tablespace bounds. Byte offset 0, len 16384Please check that the configuration matches the InnoDB system tablespace location (ibdata file
s)
200517 10:18:03 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
[root@server1 mariadb]#
{noformat}
See more info on CENTOS forum:
https://forums.centos.org/viewtopic.php?f=54&t=74436&p=313469#p313469

Elena Stepanova made changes - 2020-06-28 01:00

Fix Version/s

10.3 [ 22126 ]

Marko Mäkelä added a comment - 2020-07-24 14:40

ralf, sorry, I am not that familiar with systemd. I am an abnormal user of the database, because I only use it to run tests, to test the code that I my colleagues are modifying.

Our support engineers would be happy to answer such questions to our customers.

If you can repeat this problem by starting from an empty database (loading a logical dump or something), then I would be happy to analyze it. In our internal stress testing with Random Query Generator, we have not encountered any bugs in the undo logs recently. We have encountered and fixed some bugs in crash recovery, and theoretically those fixes might prevent this from reoccurring in the future.

Marko Mäkelä added a comment - 2020-07-24 14:40 ralf , sorry, I am not that familiar with systemd. I am an abnormal user of the database, because I only use it to run tests, to test the code that I my colleagues are modifying. Our support engineers would be happy to answer such questions to our customers. If you can repeat this problem by starting from an empty database (loading a logical dump or something), then I would be happy to analyze it. In our internal stress testing with Random Query Generator, we have not encountered any bugs in the undo logs recently. We have encountered and fixed some bugs in crash recovery, and theoretically those fixes might prevent this from reoccurring in the future.

Marko Mäkelä made changes - 2020-07-24 14:40

Labels

crash innodb

crash innodb need_feedback

Sergei Golubchik made changes - 2020-08-24 09:50

Assignee

Marko Mäkelä [ marko ]

Sergei Golubchik made changes - 2020-08-24 09:50

Fix Version/s

10.3 [ 22126 ]

Ralf Hartings added a comment - 2020-08-24 10:13

Lets close this case
I did a fresh install instead

Ralf Hartings added a comment - 2020-08-24 10:13 Lets close this case I did a fresh install instead

Elena Stepanova made changes - 2020-09-21 13:21

Fix Version/s		N/A [ 14700 ]
Resolution		Incomplete [ 4 ]
Status	Open [ 1 ]	Closed [ 6 ]

Sergei Golubchik made changes - 2021-12-06 21:51

Workflow

MariaDB v3 [ 108880 ]

MariaDB v4 [ 157805 ]

People

Assignee:: Unassigned

Reporter:: Ralf Hartings

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2020-05-19 06:39

Updated:: 2020-09-21 13:21

Resolved:: 2020-09-21 13:21

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration