Details
Description
mariabackup may overlook file corruptions as shown in steps below:
1. Start MySQL Server with Encryption enabled
2. Create tables, insert data as below:
use test; |
create table t(a varchar(128)) engine=innodb; |
create table t1(a varchar(128)) engine=innodb; |
insert into t select repeat('a',100); |
insert into t1 select repeat('a',100); |
3. Wait some time to make sure that all pages are flushed and query below shows 0 dirty pages:
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_pages_dirty'; |
4. Corrupt t.ibd with commands below (replace data_directory with actual data directory):
cd data_directory |
printf '\xAA\xAA\xAA\xAA' | dd of=test/t.ibd seek=16384 count=4 bs=1 conv=notrunc |
5. Create backup , observe no errors :
170223 09:53:10 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
|
170223 09:53:10 [01] ...done
|
..
|
170223 09:53:14 completed OK!
|
6. Check table with innochecksum to confirm that it is really corrupted:
> innochecksum dt/test/t.ibd |
InnoDB offline file checksum utility. |
Table is uncompressed
|
Page size is 16384
|
Fail; page 1 invalid (fails old style checksum)
|
7. (optional) make sure innochecksum doesn't complain about the other table t1:
innochecksum test/t1.ibd |
Without rest encryption the same steps result in backup error:
170223 10:17:22 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
|
[01] xtrabackup: Database page corruption detected at page 1, retrying...
|
...
|
[01] xtrabackup: Error: failed to read page after 10 retries. File ./test/t.ibd seems to be corrupted.
|
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
|
[01] xtrabackup: Error: failed to copy datafile.
|
xtrabackup based on MariaDB server 10.1.22-MariaDB Linux (x86_64)
|
...
|
xtrabackup: Error: cannot open ./xtrabackup_checkpoints
|
xtrabackup: Error: failed to read metadata from './xtrabackup_checkpoints'
|
Attachments
Issue Links
- causes
-
MDEV-18105 Mariabackup fails to copy encrypted InnoDB system tablespace if LSN>4G
-
- Closed
-
-
-
- Stalled
-
- is duplicated by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
Based on more work from thiru, I pushed a follow-up fix to 10.1.
It turns out that page_compressed=1 pages are not being checked for consistency. That was filed as MDEV-18025.
I cleaned up thiru’s fix, backported to 10.1 and merged all the way up to 10.4.
The following test case repeats the reported scenario:
--source include/have_file_key_management.inc
|
|
|
CREATE TABLE t1(c VARCHAR(128)) ENGINE INNODB, encrypted=yes;
|
insert into t1 select repeat('a',100);
|
|
|
let $MYSQLD_DATADIR=`select @@datadir`;
|
let t1_IBD = $MYSQLD_DATADIR/test/t1.ibd;
|
|
|
--source include/shutdown_mysqld.inc
|
|
|
--echo # Corrupt the table
|
|
|
perl;
|
use strict;
|
use warnings;
|
use Fcntl qw(:DEFAULT :seek);
|
|
|
my $ibd_file = $ENV{'t1_IBD'};
|
|
|
my $chunk;
|
my $len;
|
|
|
sysopen IBD_FILE, $ibd_file, O_RDWR || die "Unable to open $ibd_file";
|
sysseek IBD_FILE, 16384 * 3, SEEK_CUR;
|
$chunk = '\xAA\xAA\xAA\xAA';
|
syswrite IBD_FILE, $chunk, 4;
|
|
|
close IBD_FILE;
|
EOF
|
|
|
#--exec $INNOCHECKSUM $MYSQLD_DATADIR/test/t1.ibd
|
|
|
--source include/start_mysqld.inc
|
|
|
echo # xtrabackup backup;
|
let $targetdir=$MYSQLTEST_VARDIR/tmp/backup;
|
let $backuplog=$MYSQLTEST_VARDIR/tmp/backup.log;
|
|
|
--disable_result_log
|
exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$targetdir > $backuplog;
|
--enable_result_log
|
Problem is that mariabackup always read the encrypted pages and checks the post-encryption checksum only and copies the page even though
the above case corrupted the pre-encryption checksum.
added 10.2 to version list, still relevant in current 10.1
As part of a follow-up fix to MDEV-18025, I improved the test case so that it computes a valid checksum for the ‘encrypted’ page and writes it to all 3 locations. That is, by the old rules, this page would look like a valid encrypted page as well as a valid unencrypted page. Only by decrypting and checking the ‘before-encryption’ checksum we will detect the page as corrupted.
This test (and the extra decrypt step in mariabackup --backup) was motivated by a support case where a seemingly corrupted page has the same checksum in all 3 locations.
For page_compressed=1 pages (encrypted or not), the only way to validate the page checksum is to attempt to (optionally decrypt and) decompress the data and to compute the checksum.