[MDEV-28786] InnoDB crash leads to pagesize comparison failure Created: 2022-06-09  Updated: 2022-07-26  Resolved: 2022-07-26

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6.5
Fix Version/s: 10.6.9, 10.7.5, 10.8.4, 10.9.2, 10.10.1

Type: Bug Priority: Critical
Reporter: Christopher von Dambrowski Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 0
Labels: crash, innodb
Environment:

Debian Stretch on LXC


Issue Links:
Duplicate
duplicates MDEV-13542 Crashing on a corrupted page is unhel... Closed
Relates
relates to MDEV-26141 Crash in instant ALTER TABLE after IM... Confirmed
relates to MDEV-19783 Random crashes and corrupt data in IN... Closed

 Description   

Our database seems to crash for no apparent reason. This has happend to us 4 times on different nodes within the last 3 weeks. There is no log entry system-wide other than the mysql error.log. Resources, storage capacity, hardware - everything checks out just fine. We run Version 10.6.5 for almost 6 months now, so we find it odd that a problem within MariaDB hits us now in such succession.

The problem seems to occur on one specific table only for all occurences. Every query, regardless if it is a select, update or just a check table against the table stalls the connection and it hangs.
We don't have a replication setup running, nor have we changed much within the default configuration for MariaDB.
How can we help to narrow down the problem?

Update:
Somehow we guess, MDEV-19783 may be part of the problem, but then again the servers in question were rebuild from scratch while migrating to version 10.6.5. After clarification that was not the case, in fact we hardcopied the datafles like ibdata1 etc to the new server, so we might have copied some corruption with it aswell.

Errorlog:

2022-06-09  8:35:51 0 [ERROR] [FATAL] InnoDB: Page old data size 8700 new data size 11026, page old max ins size 7488 new max ins size 5162
220609  8:35:51 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.6.5-MariaDB-1:10.6.5+maria~stretch-log
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=154
max_threads=242
thread_count=17
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 549251 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x555779532a18
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f037f7fdbb8 thread_stack 0x30000
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x555776d9c5be]
/usr/sbin/mariadbd(handle_fatal_signal+0x38f)[0x55577680bb7f]



 Comments   
Comment by Marko Mäkelä [ 2022-07-26 ]

This crash has been replaced with an error code return in MDEV-13542.

To try to fix the corruption, you can disable the change buffer (see MDEV-27734 why) and try to rebuild the affected table, something like this:

SET GLOBAL innodb_change_buffering=none;
ALTER TABLE tablename FORCE;

Generated at Thu Feb 08 10:03:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.