Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18193

Incremental prepare failed with InnoDB: Database page corruption on disk or a failed file read of tablespace

    XMLWordPrintable

    Details

      Description

      Incremental prepare failed with

      10.2 ab7e2b04

      2019-03-10 22:20:19 140578388577152 [Note] InnoDB: Ignoring data file './test/#sql-12d3_11.ibd' with space ID 448. Another data file called test/t8.ibd exists with the same space ID.
      2019-03-10 22:20:19 140578388577152 [Note] InnoDB: Starting final batch to recover 657 pages from redo log.
      2019-03-10 22:20:19 140578214340352 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace test/t8 page [page id: space=448, page number=1103]. You may have to recover from a backup.
      2019-03-10 22:20:19 140578214340352 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
      ...
      InnoDB: End of page dump
      2019-03-10 22:20:19 140578214340352 [Note] InnoDB: Uncompressed page, stored checksum in field1 3597174513, calculated checksums for field1: crc32 4144541337, innodb 1308896786,  page type 17855 == INDEX.none 3735928559, stored checksum in field2 3597174513, calculated checksums for field2: crc32 4144541337, innodb 2980011749, none 3735928559,  page LSN 0 167474941, low 4 bytes of LSN at page end 167474941, page number (if stored to page already) 1103, space id (if created with >= MySQL-4.1.1 and stored already) 448
      2019-03-10 22:20:19 140578214340352 [Note] InnoDB: Page may be an index page where index id is 16856757857625039547
      2019-03-10 22:20:19 140578214340352 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
      2019-03-10 22:20:19 140578214340352 [ERROR] InnoDB: Failed to read file 'test/t8.ibd' at offset 1103: Page read from tablespace is corrupted.
      

      after that, mariabackup hangs.
      Here is the complete log of the failed prepare.
      All backups and logs can be found here.

      To reproduce,

      • download and unpack initial backup, 1st incremental backup and 2nd incremental backup;
      • run

        bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0
        bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_1
        bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_2
        

        => the last one fails.

      It seems to be reproducible at the moment.

      Earlier a similar problem was observed on 10.4 30da40bb8c3. Logs and backups can be found here. However, this occurrence wasn't reproducible on the same backups at the time.


      To reproduce from scratch

      The command line below actually runs the same test several times, until it fails one or another way. It reproduces the problem quite reliably on current 10.2 and 10.3. On 10.4, I'm usually getting MDEV-18589 instead.
      While it can be reproduced with vardir (datadir) in shm, it happens more readily with vardir on a disk. I don't have statistics for HDD vs SSD, although it's been seen on both.

      git clone https://github.com/MariaDB/randgen --branch mdev18193 rqg-mdev18193
      cd rqg-mdev18193
      . ./mdev18193.cmd <your basedir> <your vardir>
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              thiru Thirunarayanan Balathandayuthapani
              Reporter:
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration