[MDEV-29276] [FATAL] InnoDB: Page old data size 7898 new data size 7922, page old max ins size 8288 new max ins size 8264 Created: 2022-08-08 Updated: 2023-03-23 Resolved: 2023-03-13 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Storage Engine - InnoDB |
| Affects Version/s: | 10.5.13, 10.5.16 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | Richard Stanway | Assignee: | Marko Mäkelä |
| Resolution: | Incomplete | Votes: | 2 |
| Labels: | None | ||
| Environment: |
Debian 4.9.0-14-amd64 #1 SMP Debian 4.9.246-2 (2020-12-17) x86_64 GNU/Linux, ZFS |
||
| Attachments: |
prod-1-mariadb-prod-db-2-a-bkp-260122.prod.pfops.com-20230126-bt-all-threads-core-10462.txt
|
||||||||
| Issue Links: |
|
||||||||
| Description |
|
Our MariaDB server crashed with "[ERROR] [FATAL] InnoDB: Page old data size 7898 new data size 7922, page old max ins size 8288 new max ins size 8264" and subsequent automatic restarts also crashed soon after with the same error, presumably as soon as they saw the corrupt data. This seems very similar to
Most crashes mentioned this "wikivalorant_logging" table, which is a Mediawiki table:
Other potentially relavent system configuration:
We have upgraded through many older MariaDB versions, so whether this table was affected by the 10.4 corruption bug I cannot answer for certain. I restored the table from a backup before the first crash, and things seemed stable for a little while only to crash after 10 minutes. I then did an ANAYZLE TABLE FORCE on it and so far (~30 minutes) things have been stable. I have a copy of the wikivalorant_logging.ibd before I restored it that I am able to privately share if it would help. If there's any other information I can provide please let me know. |
| Comments |
| Comment by Marko Mäkelä [ 2022-11-03 ] |
|
The stack trace in the Description, above handler::ha_write_row(), is garbage. The message would be output by btr_page_reorganize_low(), which is basically collecting garbage within a B-tree page, to make an insert of a record succeed. The PAGE_FREE list is corrupted for some reason. A popular reason some years ago used to be With the fix of It would be useful to see how MariaDB Server 10.6.10 would behave on a copy of this corrupted data directory, when executing that INSERT statement. |
| Comment by Marko Mäkelä [ 2023-02-09 ] |
|
There is a stack trace of a crash that occurs deep inside row_ins_sec_index_entry(), due to running out of space when attempting to split a secondary index leaf page. Based on an analysis of a page dump that I obtained from the support customer, I determined that it was caused by |
| Comment by Marko Mäkelä [ 2023-02-09 ] |
|
R1CH, has the database been stable for you since the time you rebuilt the table with ALTER TABLE…FORCE? |
| Comment by Richard Stanway [ 2023-02-09 ] |
|
Yes, we've not had any similar incidents, though it took years before this issue manifested in the first place. |