[MDEV-11456] corruption in page 1 for big innodb tables in compressed format with mdb encryption Created: 2016-12-02  Updated: 2016-12-08  Resolved: 2016-12-08

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.1.20
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Andrii Nikitin (Inactive) Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Attachments: File xb_compressed_encrypted.opt     File xb_compressed_encrypted.test    

 Description   

Attached mtr test results in following errors during restore:
161202 11:41:53 [01] Copying ./test/t1.ibd to F:/git/mysql-test/var/tmp/backup/test/t1.ibd
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
161202 11:41:54 >> log scanned up to (21745372)
...
[01] xtrabackup: Error: failed to read page after 10 retries. File ./test/t1.ibd seems to be corrupted.
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.

When I reduce number of rows - test most likely to succeed.
If remove row_format=compressed from table definition - then always suceeds



 Comments   
Comment by Andrii Nikitin (Inactive) [ 2016-12-04 ]

I've tried the same with percona-xtrabackup 2.3.6 and the error is still there, so it is not specific to Backup component, thus closing at the moment

Comment by Vladislav Vaintroub [ 2016-12-04 ]

I'd rather have this open, we have to support our compression and encryption, Percona's xtrabackup does not have to.

Comment by Andrii Nikitin (Inactive) [ 2016-12-04 ]

yeah, my bad : without encryption no error is shown.

Comment by Vladislav Vaintroub [ 2016-12-06 ]

Jan, could you have a look?
It fails in xb_fil_cur_read(), around the place where buf_page_is_corrupted returns true.
possibly zip_size is wrongly calculated, I'm not sure.

/* check pages for corruption and re-read if necessary. i.e. in case of
partially written pages */
for (page = cursor->buf, i = 0; i < npages;
page = cursor->page_size, i+) {

if (buf_page_is_corrupted(TRUE, page, cursor->zip_size)) {

Comment by Vladislav Vaintroub [ 2016-12-06 ]

This would be fixed xb_compressed_encrypted.test , applicable to current bb-10.1-wlad-xtrabackup

CREATE TABLE t1(c1 INT, b VARCHAR(2400), index(b(100),c1)) ENGINE=INNODB ROW_FORMAT=compressed;
 
DELIMITER //;
CREATE PROCEDURE innodb_insert_proc (REPEAT_COUNT INT)
BEGIN
  DECLARE CURRENT_NUM INT;
  SET CURRENT_NUM = 0;
  WHILE CURRENT_NUM < REPEAT_COUNT DO
    INSERT INTO t1 VALUES(CURRENT_NUM, concat(uuid(), CURRENT_NUM, repeat('ab', floor(rand()*100) ), uuid()));
    SET CURRENT_NUM = CURRENT_NUM + 1;
  END WHILE;
END//
DELIMITER ;//
COMMIT;
 
SET AUTOCOMMIT=0;
CALL innodb_insert_proc(50000);
COMMIT;
 
SELECT plugin_name, plugin_library, @@plugin_dir FROM information_schema.plugins WHERE plugin_type='ENCRYPTION' AND plugin_status='ACTIVE';
                                                                                                                                           
echo # xtrabackup backup;
let $targetdir=$MYSQLTEST_VARDIR/tmp/backup;
# exec $XTRABACKUP --innobackupex --defaults-file=$MYSQLTEST_VARDIR/my.cnf --no-timestamp --compact $targetdir;
exec $XTRABACKUP --innobackupex --defaults-file=$MYSQLTEST_VARDIR/my.cnf --no-timestamp $targetdir;
 
drop table t1;
 
 
exec $XTRABACKUP --innobackupex --apply-log --rebuild-indexes --rebuild-threads=2 $targetdir;
# exec $XTRABACKUP --innobackupex --apply-log $targetdir;
-- source include/restart_and_restore.inc
 
select sum(c1) from t1;
DROP TABLE t1;
drop procedure innodb_insert_proc;
rmdir $targetdir;

Comment by Jan Lindström (Inactive) [ 2016-12-07 ]

Yes, you need to do buf_page_decrypt_after_read(bpage) before you try to compare checksums...

Comment by Jan Lindström (Inactive) [ 2016-12-08 ]

commit 0abfd68330293b8fa6c732e3128fbc44f5d41036
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Thu Dec 8 09:38:48 2016 +0200

MDEV-11456: corruption in page 1 for big innodb tables in compressed format with mdb encryption

Verify encrypted page checksum using the correct method.

Comment by Andrii Nikitin (Inactive) [ 2016-12-08 ]

For me current mtr test xb_compressed_encrypted.test is still crashing occasionally with stack below:

xtrabackup.exe!mach_read_from_4()[mach0data.ic:186]
xtrabackup.exe!fil_space_verify_crypt_checksum()[fil0crypt.cc:955]
xtrabackup.exe!xb_fil_cur_read()[fil_cur.cc:349]
xtrabackup.exe!xtrabackup_copy_datafile()[xtrabackup.cc:2418]
xtrabackup.exe!data_copy_thread_func()[xtrabackup.cc:2934]

It appears in debugger that memory passed to mach_read_from_4 above is either corrupted or garbage
Should I open new bug report or it may be the same issue?

Comment by Vladislav Vaintroub [ 2016-12-08 ]

I had this, too.
The immediate reason seem to be that fil_space_verify_crypt_checksum() expects a page of the size UNIV_PAGE_SIZE, 16K
Yes, the passed page had (from xtrabackup POV, which might or not held true) page size of 8K , and also zip_size ok 8K.
That page was the last one, the end of allocated region, this attempt to read 8K past allocated region crashed.

Generated at Thu Feb 08 07:50:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.