[MDEV-18193] Incremental prepare failed with InnoDB: Database page corruption on disk or a failed file read of tablespace Created: 2019-01-09  Updated: 2019-07-31  Resolved: 2019-07-31

Status: Closed
Project: MariaDB Server
Component/s: mariabackup, Storage Engine - InnoDB
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Thirunarayanan Balathandayuthapani
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-19348 MariaBackup prepare fails with InnoDB... Closed

 Description   

Incremental prepare failed with

10.2 ab7e2b04

2019-03-10 22:20:19 140578388577152 [Note] InnoDB: Ignoring data file './test/#sql-12d3_11.ibd' with space ID 448. Another data file called test/t8.ibd exists with the same space ID.
2019-03-10 22:20:19 140578388577152 [Note] InnoDB: Starting final batch to recover 657 pages from redo log.
2019-03-10 22:20:19 140578214340352 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace test/t8 page [page id: space=448, page number=1103]. You may have to recover from a backup.
2019-03-10 22:20:19 140578214340352 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
...
InnoDB: End of page dump
2019-03-10 22:20:19 140578214340352 [Note] InnoDB: Uncompressed page, stored checksum in field1 3597174513, calculated checksums for field1: crc32 4144541337, innodb 1308896786,  page type 17855 == INDEX.none 3735928559, stored checksum in field2 3597174513, calculated checksums for field2: crc32 4144541337, innodb 2980011749, none 3735928559,  page LSN 0 167474941, low 4 bytes of LSN at page end 167474941, page number (if stored to page already) 1103, space id (if created with >= MySQL-4.1.1 and stored already) 448
2019-03-10 22:20:19 140578214340352 [Note] InnoDB: Page may be an index page where index id is 16856757857625039547
2019-03-10 22:20:19 140578214340352 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
2019-03-10 22:20:19 140578214340352 [ERROR] InnoDB: Failed to read file 'test/t8.ibd' at offset 1103: Page read from tablespace is corrupted.

after that, mariabackup hangs.
Here is the complete log of the failed prepare.
All backups and logs can be found here.

To reproduce,

  • download and unpack initial backup, 1st incremental backup and 2nd incremental backup;
  • run

    bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0
    bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_1
    bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=`pwd`/backup_before_prepare_0 --incremental-dir=`pwd`/backup_before_prepare_2
    

    => the last one fails.

It seems to be reproducible at the moment.

Earlier a similar problem was observed on 10.4 30da40bb8c3. Logs and backups can be found here. However, this occurrence wasn't reproducible on the same backups at the time.


To reproduce from scratch

The command line below actually runs the same test several times, until it fails one or another way. It reproduces the problem quite reliably on current 10.2 and 10.3. On 10.4, I'm usually getting MDEV-18589 instead.
While it can be reproduced with vardir (datadir) in shm, it happens more readily with vardir on a disk. I don't have statistics for HDD vs SSD, although it's been seen on both.

git clone https://github.com/MariaDB/randgen --branch mdev18193 rqg-mdev18193
cd rqg-mdev18193
. ./mdev18193.cmd <your basedir> <your vardir>



 Comments   
Comment by Elena Stepanova [ 2019-07-04 ]

See also an alternative MTR test in MDEV-19348.

Comment by Elena Stepanova [ 2019-07-31 ]

Since thiru already looked into MDEV-19348 and commented on it, I'm closing this one as a duplicate, even though it's older.

Generated at Thu Feb 08 08:42:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.