[MDEV-29364] CRITICAL - MariaDB 10.8.4 creating corrupted backups - "InnoDB: Failed to read page 4 from file 'database/table.ibd': Page read from tablespace is corrupted." - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Duplicate
Affects Version/s: None
Fix Version/s: 10.10.1, 10.6.10, 10.7.6, 10.8.5, 10.9.3
Component/s: Backup
Labels:
None

Description

Hey

--prepare fails to read pretty much every( ? ) table, with many of these logs:

[Note] InnoDB: Read redo log up to LSN=10616627248656

[Note] InnoDB: Starting a batch to recover 404825 pages from redo log.

[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Failed to read page 4386 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Failed to read page 7975 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Failed to read page 6571 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Failed to read page 3647 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Failed to read page 5374 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Failed to read page 928 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Failed to read page 8182 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Failed to read page 5806 from file 'database/table.ibd': Page read from tablespace is corrupted.

[ERROR] InnoDB: Failed to read page 8319 from file 'database/table.ibd': Page read from tablespace is corrupted.

... (many of these "Page read from tablespace is corrupted.") ...

[Note] InnoDB: Starting final batch to recover 164530 pages from redo log.

[Note] InnoDB: To recover: 122658 pages from log

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=9536, page number=127]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=9536, page number=255]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]

[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

[ERROR] InnoDB: Failed to read page 31742 from file 'database/table.ibd': Page read from tablespace is corrupted.

... (many of these "Page read from tablespace is corrupted.") ...

and eventually crash with:

mariadb-backup: srv_start() returned 37 (Data structure corruption).

EDIT - Ignore the below bit. This isn't an issue with using HDD, as the same failures are happening on SSD.

From original message (no longer relevant):

I assume this might be related to this config I have in prod (which has NVMe):

innodb_page_size = 16384

Once I have a page size like this, I can no longer go back to HDD, is it?

Also, is it dangerous to use HDD as the "backup storage" - could it be corrupting the backups by storing them on these disks, or as long as I don't prepare them, the backup is Good & Correct, as long as I move them back to an NVMe before I prepare?

Attachments

Issue Links

relates to

MDEV-29374 Frequent "Data structure corruption" in InnoDB after OOM

Closed

MDEV-29383 Assertion mysql_mutex_assert_owner(&log_sys.flush_order_mutex) failed in mtr_t::commit()

Closed

MDEV-29511 "Page read from tablespace is corrupted" when starting mariadb service after crash (signal 6)

Closed

Activity

People

Assignee:: Marko Mäkelä

Reporter:: Nuno

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 2022-08-23 14:22

Updated:: 2023-01-12 12:17

Resolved:: 2022-10-07 07:40

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.