Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29364

CRITICAL - MariaDB 10.8.4 creating corrupted backups - "InnoDB: Failed to read page 4 from file 'database/table.ibd': Page read from tablespace is corrupted."

Details

    Description

      Hey

      --prepare fails to read pretty much every( ? ) table, with many of these logs:

      [Note] InnoDB: Read redo log up to LSN=10616627248656
      [Note] InnoDB: Starting a batch to recover 404825 pages from redo log.
      [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Failed to read page 4386 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Failed to read page 7975 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Failed to read page 6571 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Failed to read page 3647 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=4731, page number=5374]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Failed to read page 5374 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Failed to read page 928 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Failed to read page 8182 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Failed to read page 5806 from file 'database/table.ibd': Page read from tablespace is corrupted.
      [ERROR] InnoDB: Failed to read page 8319 from file 'database/table.ibd': Page read from tablespace is corrupted.
      ... (many of these "Page read from tablespace is corrupted.") ...
      

      or

      [Note] InnoDB: Starting final batch to recover 164530 pages from redo log.
      [Note] InnoDB: To recover: 122658 pages from log
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=9536, page number=127]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=9536, page number=255]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Not applying INSERT_HEAP_DYNAMIC due to corruption on [page id: space=8133, page number=31742]
      [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      [ERROR] InnoDB: Failed to read page 31742 from file 'database/table.ibd': Page read from tablespace is corrupted.
      ... (many of these "Page read from tablespace is corrupted.") ...
      

      and eventually crash with:

      mariadb-backup: srv_start() returned 37 (Data structure corruption).
      

      -

      EDIT - Ignore the below bit. This isn't an issue with using HDD, as the same failures are happening on SSD.

      -

      From original message (no longer relevant):


      I assume this might be related to this config I have in prod (which has NVMe):

      innodb_page_size = 16384

      Once I have a page size like this, I can no longer go back to HDD, is it?

      Also, is it dangerous to use HDD as the "backup storage" - could it be corrupting the backups by storing them on these disks, or as long as I don't prepare them, the backup is Good & Correct, as long as I move them back to an NVMe before I prepare?

      Attachments

        Issue Links

          Activity

            nunop Nuno added a comment -

            Hi marko
            Thank you very much for checking.

            I upgraded everything to 10.8.5 as soon as it came out!

            So far, so good. The backups and restores seem to be working fine (with DDL changes, crash recovery from checkpoint, segfault, SELECT *, etc). I have only tested the restore on the SSD server.
            Seems to be very stable.

            MDEV-29097 is still happening (but I know it wasn't expected to be fixed in 10.8.5).

            Have a great day. Thank you for all your support.

            nunop Nuno added a comment - Hi marko Thank you very much for checking. I upgraded everything to 10.8.5 as soon as it came out! So far, so good. The backups and restores seem to be working fine (with DDL changes, crash recovery from checkpoint, segfault, SELECT *, etc). I have only tested the restore on the SSD server. Seems to be very stable. MDEV-29097 is still happening (but I know it wasn't expected to be fixed in 10.8.5). Have a great day. Thank you for all your support.

            nunop, thank you, this is great news. To be on the safe side, I would wait for additional feedback from you in a week or two, before claiming that this was fixed in 10.8.5.

            marko Marko Mäkelä added a comment - nunop , thank you, this is great news. To be on the safe side, I would wait for additional feedback from you in a week or two, before claiming that this was fixed in 10.8.5.

            nunop, are your backups still doing fine in 10.8.5?

            marko Marko Mäkelä added a comment - nunop , are your backups still doing fine in 10.8.5?
            nunop Nuno added a comment -

            marko - working perfectly!!
            Thank you very much.
            Have a great week.

            nunop Nuno added a comment - marko - working perfectly!! Thank you very much. Have a great week.

            nunop, thank you. This was fixed by MDEV-29383 and MDEV-29374.

            marko Marko Mäkelä added a comment - nunop , thank you. This was fixed by MDEV-29383 and MDEV-29374 .

            People

              marko Marko Mäkelä
              nunop Nuno
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.