[MDEV-25569] Restore crashed due to Data structure corruption Created: 2021-04-29  Updated: 2023-10-31  Resolved: 2021-04-30

Status: Closed
Project: MariaDB Server
Component/s: Backup, mariabackup
Affects Version/s: 10.5.9
Fix Version/s: 10.5.10

Type: Bug Priority: Major
Reporter: Nuno Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-25031 Not applying INSERT_REUSE_REDUNDANT d... Closed

 Description   

Hello,

Every day, I create a backup using MariaBackup, and restore the snapshot it on a Docker container.

The version in both the main server and the docker container is 10.5.9.

Usually, things go well, but recently I got two similar crashes due to "data structure corruption".
I confirm the version was 10.5.9 in both sides, on both incidents.

(many of these "Ignoring data file" logs)
2021-04-19 7:33:01 0 [Note] InnoDB: Ignoring data file 'database1/table1.ibd' with space ID 20668, since the redo log references database1/table1.ibd with space ID 20666.
2021-04-19 7:33:01 0 [Note] InnoDB: Ignoring data file './database1/table1_new.ibd' with space ID 20668. Another data file called database1/table1.ibd exists with the same space ID.
2021-04-19 7:33:01 0 [Note] InnoDB: Starting final batch to recover 153185 pages from redo log.
2021-04-19 7:33:01 0 [ERROR] InnoDB: Not applying INSERT_REUSE_REDUNDANT due to corruption on [page id: space=0, page number=7848]
2021-04-19 7:33:01 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
2021-04-19 7:33:01 0 [ERROR] InnoDB: Not applying INSERT_HEAP_REDUNDANT due to corruption on [page id: space=0, page number=7848]
2021-04-19 7:33:01 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
2021-04-19 7:33:08 0 [Note] InnoDB: To recover: 73635 pages from log
2021-04-19 7:33:20 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
[00] FATAL ERROR: 2021-04-19 07:33:20 mariabackup: innodb_init() returned 39 (Data structure corruption).

(many of these "Ignoring data file" logs)
2021-04-29 7:32:31 0 [Note] InnoDB: Ignoring data file 'database1/table1.ibd' with space ID 26720, since the redo log references database1/table1.ibd with space ID 26718.
2021-04-29 7:32:31 0 [Note] InnoDB: Ignoring data file 'database1/table2.ibd' with space ID 26721, since the redo log references database1/table2.ibd with space ID 26719.
2021-04-29 7:32:31 0 [Note] InnoDB: Starting final batch to recover 116118 pages from redo log.
2021-04-29 7:32:32 0 [ERROR] InnoDB: Not applying INSERT_REUSE_REDUNDANT due to corruption on [page id: space=0, page number=14132]
2021-04-29 7:32:32 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
2021-04-29 7:32:43 0 [Note] InnoDB: To recover: 28397 pages from log
2021-04-29 7:32:45 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
[00] FATAL ERROR: 2021-04-29 07:32:45 mariabackup: innodb_init() returned 39 (Data structure corruption).

Is there any way I can identify the problem here, or at least prevent this from happening?
I want to ensure all my backups are "clean" and "valid".

This may be related to the issue I reported about using "RENAME TABLE", MDEV-25568
It was in April 15 that I added a job that performs "RENAME TABLE" every 5 minutes on two tables, so I wonder if the fact that this incident happened twice after that day, may be the cause of this.

Thank you.



 Comments   
Comment by Marko Mäkelä [ 2021-04-30 ]

This looks like a duplicate of MDEV-25031, whose fix is present in 10.5.10, which will likely be released next week.

Comment by Marko Mäkelä [ 2021-04-30 ]

RENAME TABLE would update SYS_TABLES.NAME. All InnoDB data dictionary tables are stored in ROW_FORMAT=REDUNDANT (the original format). Also the change buffer uses that format. Because the default ROW_FORMAT has been something else since MySQL 5.0.3, we had failed to catch this bug in our internal testing.

Generated at Thu Feb 08 09:38:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.