[MDEV-12112] corruption in encrypted table may be overlooked Created: 2017-02-23  Updated: 2019-11-06  Resolved: 2018-12-14

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.1.22, 10.1.28, 10.2.9
Fix Version/s: 10.4.1, 10.1.38, 10.2.20, 10.3.12

Type: Bug Priority: Major
Reporter: Andrii Nikitin (Inactive) Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Duplicate
is duplicated by MDEV-20983 Page 38:4016 may be corrupted. Post e... Closed
Problem/Incident
causes MDEV-18105 Mariabackup fails to copy encrypted I... Closed
Relates
relates to MDEV-18025 Mariabackup fails to detect corrupted... Closed
relates to MDEV-18097 Indicate which InnoDB data files are ... Closed
relates to MDEV-18128 Simplify .ibd file creation Closed
relates to MDEV-18129 Backup fails for encrypted tables: ma... Closed
relates to MDEV-12711 backup may show corruption with custo... Closed
relates to MDEV-13542 Crashing on a corrupted page is unhel... Closed
relates to MDEV-17638 Improve error message about corruptio... Closed
relates to MDEV-18529 InnoDB wrongly skips decryption of en... Closed

 Description   

mariabackup may overlook file corruptions as shown in steps below:

1. Start MySQL Server with Encryption enabled
2. Create tables, insert data as below:

use test;
create table t(a varchar(128)) engine=innodb;
create table t1(a varchar(128)) engine=innodb;
insert into t select repeat('a',100);
insert into t1 select repeat('a',100);

3. Wait some time to make sure that all pages are flushed and query below shows 0 dirty pages:

SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_pages_dirty';

4. Corrupt t.ibd with commands below (replace data_directory with actual data directory):

cd data_directory
printf '\xAA\xAA\xAA\xAA' | dd of=test/t.ibd seek=16384 count=4 bs=1 conv=notrunc

5. Create backup , observe no errors :

170223 09:53:10 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
170223 09:53:10 [01]        ...done
..
170223 09:53:14 completed OK!

6. Check table with innochecksum to confirm that it is really corrupted:

> innochecksum dt/test/t.ibd 
InnoDB offline file checksum utility.
Table is uncompressed
Page size is 16384
Fail;  page 1 invalid (fails old style checksum)

7. (optional) make sure innochecksum doesn't complain about the other table t1:

innochecksum test/t1.ibd

Without rest encryption the same steps result in backup error:

170223 10:17:22 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
[01] xtrabackup: Database page corruption detected at page 1, retrying...
...
[01] xtrabackup: Error: failed to read page after 10 retries. File ./test/t.ibd seems to be corrupted.
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
xtrabackup based on MariaDB server 10.1.22-MariaDB Linux (x86_64) 
...
xtrabackup: Error: cannot open ./xtrabackup_checkpoints
xtrabackup: Error: failed to read metadata from './xtrabackup_checkpoints'



 Comments   
Comment by Andrii Nikitin (Inactive) [ 2017-10-28 ]

added 10.2 to version list, still relevant in current 10.1

Comment by Thirunarayanan Balathandayuthapani [ 2018-12-13 ]

The following test case repeats the reported scenario:

--source include/have_file_key_management.inc
 
CREATE TABLE t1(c VARCHAR(128)) ENGINE INNODB, encrypted=yes;
insert into t1 select repeat('a',100);
 
let $MYSQLD_DATADIR=`select @@datadir`;
let t1_IBD = $MYSQLD_DATADIR/test/t1.ibd;
 
--source include/shutdown_mysqld.inc
 
--echo # Corrupt the table
 
perl;
use strict;
use warnings;
use Fcntl qw(:DEFAULT :seek);
 
my $ibd_file = $ENV{'t1_IBD'};
 
my $chunk;
my $len;
 
sysopen IBD_FILE, $ibd_file, O_RDWR || die "Unable to open $ibd_file";
sysseek IBD_FILE, 16384 * 3, SEEK_CUR;
$chunk = '\xAA\xAA\xAA\xAA';
syswrite IBD_FILE, $chunk, 4;
 
close IBD_FILE;
EOF
 
#--exec $INNOCHECKSUM $MYSQLD_DATADIR/test/t1.ibd
 
--source include/start_mysqld.inc
 
echo # xtrabackup backup;
let $targetdir=$MYSQLTEST_VARDIR/tmp/backup;
let $backuplog=$MYSQLTEST_VARDIR/tmp/backup.log;
 
--disable_result_log
exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$targetdir > $backuplog;
--enable_result_log

Problem is that mariabackup always read the encrypted pages and checks the post-encryption checksum only and copies the page even though
the above case corrupted the pre-encryption checksum.

Comment by Thirunarayanan Balathandayuthapani [ 2018-12-14 ]

Patch is in bb-10.2-23578

Comment by Marko Mäkelä [ 2018-12-14 ]

I cleaned up thiru’s fix, backported to 10.1 and merged all the way up to 10.4.

Comment by Marko Mäkelä [ 2018-12-17 ]

Based on more work from thiru, I pushed a follow-up fix to 10.1.

It turns out that page_compressed=1 pages are not being checked for consistency. That was filed as MDEV-18025.

Comment by Marko Mäkelä [ 2018-12-19 ]

As part of a follow-up fix to MDEV-18025, I improved the test case so that it computes a valid checksum for the ‘encrypted’ page and writes it to all 3 locations. That is, by the old rules, this page would look like a valid encrypted page as well as a valid unencrypted page. Only by decrypting and checking the ‘before-encryption’ checksum we will detect the page as corrupted.

This test (and the extra decrypt step in mariabackup --backup) was motivated by a support case where a seemingly corrupted page has the same checksum in all 3 locations.

For page_compressed=1 pages (encrypted or not), the only way to validate the page checksum is to attempt to (optionally decrypt and) decompress the data and to compute the checksum.

Generated at Thu Feb 08 07:55:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.