Details
Description
mariabackup may overlook file corruptions as shown in steps below:
1. Start MySQL Server with Encryption enabled
2. Create tables, insert data as below:
use test; |
create table t(a varchar(128)) engine=innodb; |
create table t1(a varchar(128)) engine=innodb; |
insert into t select repeat('a',100); |
insert into t1 select repeat('a',100); |
3. Wait some time to make sure that all pages are flushed and query below shows 0 dirty pages:
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_pages_dirty'; |
4. Corrupt t.ibd with commands below (replace data_directory with actual data directory):
cd data_directory |
printf '\xAA\xAA\xAA\xAA' | dd of=test/t.ibd seek=16384 count=4 bs=1 conv=notrunc |
5. Create backup , observe no errors :
170223 09:53:10 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
|
170223 09:53:10 [01] ...done
|
..
|
170223 09:53:14 completed OK!
|
6. Check table with innochecksum to confirm that it is really corrupted:
> innochecksum dt/test/t.ibd |
InnoDB offline file checksum utility. |
Table is uncompressed
|
Page size is 16384
|
Fail; page 1 invalid (fails old style checksum)
|
7. (optional) make sure innochecksum doesn't complain about the other table t1:
innochecksum test/t1.ibd |
Without rest encryption the same steps result in backup error:
170223 10:17:22 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
|
[01] xtrabackup: Database page corruption detected at page 1, retrying...
|
...
|
[01] xtrabackup: Error: failed to read page after 10 retries. File ./test/t.ibd seems to be corrupted.
|
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
|
[01] xtrabackup: Error: failed to copy datafile.
|
xtrabackup based on MariaDB server 10.1.22-MariaDB Linux (x86_64)
|
...
|
xtrabackup: Error: cannot open ./xtrabackup_checkpoints
|
xtrabackup: Error: failed to read metadata from './xtrabackup_checkpoints'
|
Attachments
Issue Links
- causes
-
MDEV-18105 Mariabackup fails to copy encrypted InnoDB system tablespace if LSN>4G
-
- Closed
-
-
-
- Stalled
-
- is duplicated by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
The following test case repeats the reported scenario:
--source include/have_file_key_management.inc
|
|
|
CREATE TABLE t1(c VARCHAR(128)) ENGINE INNODB, encrypted=yes;
|
insert into t1 select repeat('a',100);
|
|
|
let $MYSQLD_DATADIR=`select @@datadir`;
|
let t1_IBD = $MYSQLD_DATADIR/test/t1.ibd;
|
|
|
--source include/shutdown_mysqld.inc
|
|
|
--echo # Corrupt the table
|
|
|
perl;
|
use strict;
|
use warnings;
|
use Fcntl qw(:DEFAULT :seek);
|
|
|
my $ibd_file = $ENV{'t1_IBD'};
|
|
|
my $chunk;
|
my $len;
|
|
|
sysopen IBD_FILE, $ibd_file, O_RDWR || die "Unable to open $ibd_file";
|
sysseek IBD_FILE, 16384 * 3, SEEK_CUR;
|
$chunk = '\xAA\xAA\xAA\xAA';
|
syswrite IBD_FILE, $chunk, 4;
|
|
|
close IBD_FILE;
|
EOF
|
|
|
#--exec $INNOCHECKSUM $MYSQLD_DATADIR/test/t1.ibd
|
|
|
--source include/start_mysqld.inc
|
|
|
echo # xtrabackup backup;
|
let $targetdir=$MYSQLTEST_VARDIR/tmp/backup;
|
let $backuplog=$MYSQLTEST_VARDIR/tmp/backup.log;
|
|
|
--disable_result_log
|
exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$targetdir > $backuplog;
|
--enable_result_log
|
Problem is that mariabackup always read the encrypted pages and checks the post-encryption checksum only and copies the page even though
the above case corrupted the pre-encryption checksum.
I cleaned up thiru’s fix, backported to 10.1 and merged all the way up to 10.4.
Based on more work from thiru, I pushed a follow-up fix to 10.1.
It turns out that page_compressed=1 pages are not being checked for consistency. That was filed as MDEV-18025.
As part of a follow-up fix to MDEV-18025, I improved the test case so that it computes a valid checksum for the ‘encrypted’ page and writes it to all 3 locations. That is, by the old rules, this page would look like a valid encrypted page as well as a valid unencrypted page. Only by decrypting and checking the ‘before-encryption’ checksum we will detect the page as corrupted.
This test (and the extra decrypt step in mariabackup --backup) was motivated by a support case where a seemingly corrupted page has the same checksum in all 3 locations.
For page_compressed=1 pages (encrypted or not), the only way to validate the page checksum is to attempt to (optionally decrypt and) decompress the data and to compute the checksum.
added 10.2 to version list, still relevant in current 10.1