Details
Description
Hey
--prepare fails to read pretty much every( ? ) table, with many of these logs:
[Note] InnoDB: Read redo log up to LSN=10616627248656
|
[Note] InnoDB: Starting a batch to recover 404825 pages from redo log.
|
[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Failed to read page 4386 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Failed to read page 7975 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Failed to read page 6571 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Failed to read page 3647 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Failed to read page 5374 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Failed to read page 928 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Failed to read page 8182 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Failed to read page 5806 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
[ERROR] InnoDB: Failed to read page 8319 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
... (many of these "Page read from tablespace is corrupted.") ...
|
or
[Note] InnoDB: Starting final batch to recover 164530 pages from redo log.
|
[Note] InnoDB: To recover: 122658 pages from log
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=9536, page number=127]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=9536, page number=255]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
|
[ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
|
[ERROR] InnoDB: Failed to read page 31742 from file 'database/table.ibd': Page read from tablespace is corrupted.
|
... (many of these "Page read from tablespace is corrupted.") ...
|
and eventually crash with:
mariadb-backup: srv_start() returned 37 (Data structure corruption).
|
-
EDIT - Ignore the below bit. This isn't an issue with using HDD, as the same failures are happening on SSD.
-
From original message (no longer relevant):
I assume this might be related to this config I have in prod (which has NVMe):
innodb_page_size = 16384
Once I have a page size like this, I can no longer go back to HDD, is it?
Also, is it dangerous to use HDD as the "backup storage" - could it be corrupting the backups by storing them on these disks, or as long as I don't prepare them, the backup is Good & Correct, as long as I move them back to an NVMe before I prepare?
Attachments
Issue Links
- relates to
-
MDEV-29374 Frequent "Data structure corruption" in InnoDB after OOM
-
- Closed
-
-
MDEV-29383 Assertion mysql_mutex_assert_owner(&log_sys.flush_order_mutex) failed in mtr_t::commit()
-
- Closed
-
-
MDEV-29511 "Page read from tablespace is corrupted" when starting mariadb service after crash (signal 6)
-
- Closed
-
Hi marko
Thank you very much for checking.
I upgraded everything to 10.8.5 as soon as it came out!
So far, so good. The backups and restores seem to be working fine (with DDL changes, crash recovery from checkpoint, segfault, SELECT *, etc). I have only tested the restore on the SSD server.
Seems to be very stable.
MDEV-29097is still happening (but I know it wasn't expected to be fixed in 10.8.5).Have a great day. Thank you for all your support.