[MDEV-23344] InnoDB: Fatal: Trying to access page number 57 in space 182 space name zabbix/items. Created: 2020-07-30  Updated: 2021-05-17  Resolved: 2021-05-17

Status: Closed
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB
Affects Version/s: 5.5, 10.0, 10.1, 10.1.38, 10.2, 10.3, 10.4, 10.5, 10.6
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Everton Barbosa Kopec Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: crash, need_feedback
Environment:

Debian release 9.9
4 core: Version: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
Storage:
Size Used Avail Use% Mounted on
484G 272G 190G 59% /


Attachments: File Mysql_log.rar    
Issue Links:
Relates
relates to MDEV-18194 Incremental prepare tries to access p... Closed
relates to MDEV-13542 Crashing on a corrupted page is unhel... Closed

 Description   

Hi guys, I was trying to solve “outside tablespace bounds” MySQL error but I could not up the even the MySQL server.
I got this message:
Status: "InnoDB: Fatal: Trying to access page number 57 in space 182 space name zabbix/items, which is outside the tablespace bounds. Byte offset 0, len 16384 i/o ty...

I tried to set innodb_force_recovery=1 and then up to 6 and but still got the same error.
When I try to start mysql service I got the message:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")

But there is mysqld.sock in this folder.

Could someone show me a light?



 Comments   
Comment by jingbo zhao [ 2020-07-31 ]

hi Everton Barbosa Kopec, what do on your MySQL server to cause this problem? and please paste the complete mysql error log.

Comment by Everton Barbosa Kopec [ 2020-07-31 ]

Hi Jingbo Zhao,
thank you for answering me. Follow the files.
The situation was this way, at my work there is a Zabbix and no one could not explain correctly what's happened. Simply the server crashed and they do not have a backup for database only a server snapshot. So they restored the snapshot but the MySql could not startup. After all the problems they called for my help so far so good, but I am stuck in this situation.

Comment by Marko Mäkelä [ 2020-08-05 ]

According to the message, something seems to attempt accessing a page right after the end of the data file.

Be aware that setting innodb_force_recovery to 4 or higher may corrupt the database further. The setting 6 is the worst. Do not expect anything to be consistent.

Setting innodb_force_recovery to 4 should prevent any background access to the table. But it may also corrupt secondary indexes in tables. But you may attempt to rescue data by mysqldump or similar.

A simple trick could be to remove or rename that data file so that it cannot be accessed.

I cannot help with the socket problem. I would suggest to check the server error log and the process table. Maybe an old server process did not die and is preventing a new one from acquiring advisory locks on the files?

I wonder if this could be something that we fixed in MDEV-23190?

Comment by Elena Stepanova [ 2020-09-06 ]

For the socket problem – the socket is removed upon normal shutdown. Since your server keeps crashing, it doesn't get to remove the socket, that's why it is still there.

Comment by Marko Mäkelä [ 2020-10-28 ]

This is not exactly a duplicate of MDEV-13542. The check for out-of-bounds accesses could be moved earlier and we could return an error to the operation that causes the problematic access, before even invoking buf_page_get_gen().

Starting with MDEV-12545 in 10.3, we actually do have the fil_space_t::get_size() conveniently available in the higher-level code. MDEV-23855 in 10.5 cleaned up the low-level I/O code quite a bit, but the low-level function fil_report_invalid_page_access() was not removed yet.

Comment by Marko Mäkelä [ 2021-04-13 ]

We have not seen anything like this in our internal testing. Apparently the data file was truncated at some point. What happened before the time the log started? Was there some previous crash recovery or restore from a backup? What would SHOW CREATE TABLE report for the table definition? Can you get a fully resolved stack trace of the crash that would identify the current index? If the table contains a secondary index, then this might have been fixed by MDEV-24449.

Generated at Thu Feb 08 09:21:42 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.