Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12112

corruption in encrypted table may be overlooked

Details

    Description

      mariabackup may overlook file corruptions as shown in steps below:

      1. Start MySQL Server with Encryption enabled
      2. Create tables, insert data as below:

      use test;
      create table t(a varchar(128)) engine=innodb;
      create table t1(a varchar(128)) engine=innodb;
      insert into t select repeat('a',100);
      insert into t1 select repeat('a',100);
      

      3. Wait some time to make sure that all pages are flushed and query below shows 0 dirty pages:

      SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_pages_dirty';

      4. Corrupt t.ibd with commands below (replace data_directory with actual data directory):

      cd data_directory
      printf '\xAA\xAA\xAA\xAA' | dd of=test/t.ibd seek=16384 count=4 bs=1 conv=notrunc
      

      5. Create backup , observe no errors :

      170223 09:53:10 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
      170223 09:53:10 [01]        ...done
      ..
      170223 09:53:14 completed OK!
      

      6. Check table with innochecksum to confirm that it is really corrupted:

      > innochecksum dt/test/t.ibd 
      InnoDB offline file checksum utility.
      Table is uncompressed
      Page size is 16384
      Fail;  page 1 invalid (fails old style checksum)
      

      7. (optional) make sure innochecksum doesn't complain about the other table t1:

      innochecksum test/t1.ibd
      

      Without rest encryption the same steps result in backup error:

      170223 10:17:22 [01] Copying ./test/t.ibd to /bkup/test/t.ibd
      [01] xtrabackup: Database page corruption detected at page 1, retrying...
      ...
      [01] xtrabackup: Error: failed to read page after 10 retries. File ./test/t.ibd seems to be corrupted.
      [01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
      [01] xtrabackup: Error: failed to copy datafile.
      xtrabackup based on MariaDB server 10.1.22-MariaDB Linux (x86_64) 
      ...
      xtrabackup: Error: cannot open ./xtrabackup_checkpoints
      xtrabackup: Error: failed to read metadata from './xtrabackup_checkpoints'
      

      Attachments

        Issue Links

          Activity

            anikitin Andrii Nikitin (Inactive) created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Fix Version/s 10.1 [ 16100 ]
            anikitin Andrii Nikitin (Inactive) made changes -
            Affects Version/s 10.2.9 [ 22611 ]
            Affects Version/s 10.1.28 [ 22610 ]
            anikitin Andrii Nikitin (Inactive) made changes -
            Fix Version/s 10.2 [ 14601 ]

            added 10.2 to version list, still relevant in current 10.1

            anikitin Andrii Nikitin (Inactive) added a comment - added 10.2 to version list, still relevant in current 10.1
            marko Marko Mäkelä made changes -
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Assignee Jan Lindström [ jplindst ] Thirunarayanan Balathandayuthapani [ thiru ]
            thiru Thirunarayanan Balathandayuthapani made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            thiru Thirunarayanan Balathandayuthapani made changes -
            Comment [ {code}

            266 xb_fil_cur_result_t
            267 xb_fil_cur_read(
            268 /*============*/
            269 xb_fil_cur_t* cursor) /*!< in/out: source file cursor */
            .................
            .............
            ................
            342 /* check pages for corruption and re-read if necessary. i.e. in case of
            343 partially written pages */
            344 for (page = cursor->buf, i = 0; i < npages;
            345 page += page_size, i++) {
            346 ulint page_no = cursor->buf_page_no + i;
            347
            348 if (cursor->space_id == TRX_SYS_SPACE &&
            349 page_no >= FSP_EXTENT_SIZE &&
            350 page_no < FSP_EXTENT_SIZE * 3) {
            351 /* We ignore the doublewrite buffer pages */
            352 } else if (!fil_space_verify_crypt_checksum(
            353 page, cursor->page_size, space->id, page_no)
            354 && buf_page_is_corrupted(true, page,
            355 cursor->page_size,
            356 space)) {

            {code}

            Basically mariabackup checks whether it is un-encrypted and corrupted. It blindly reads the encrypted page and doesn't check whether
            the page is corrupted. ]

            The following test case repeats the reported scenario:

            --source include/have_file_key_management.inc
             
            CREATE TABLE t1(c VARCHAR(128)) ENGINE INNODB, encrypted=yes;
            insert into t1 select repeat('a',100);
             
            let $MYSQLD_DATADIR=`select @@datadir`;
            let t1_IBD = $MYSQLD_DATADIR/test/t1.ibd;
             
            --source include/shutdown_mysqld.inc
             
            --echo # Corrupt the table
             
            perl;
            use strict;
            use warnings;
            use Fcntl qw(:DEFAULT :seek);
             
            my $ibd_file = $ENV{'t1_IBD'};
             
            my $chunk;
            my $len;
             
            sysopen IBD_FILE, $ibd_file, O_RDWR || die "Unable to open $ibd_file";
            sysseek IBD_FILE, 16384 * 3, SEEK_CUR;
            $chunk = '\xAA\xAA\xAA\xAA';
            syswrite IBD_FILE, $chunk, 4;
             
            close IBD_FILE;
            EOF
             
            #--exec $INNOCHECKSUM $MYSQLD_DATADIR/test/t1.ibd
             
            --source include/start_mysqld.inc
             
            echo # xtrabackup backup;
            let $targetdir=$MYSQLTEST_VARDIR/tmp/backup;
            let $backuplog=$MYSQLTEST_VARDIR/tmp/backup.log;
             
            --disable_result_log
            exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$targetdir > $backuplog;
            --enable_result_log
            

            Problem is that mariabackup always read the encrypted pages and checks the post-encryption checksum only and copies the page even though
            the above case corrupted the pre-encryption checksum.

            thiru Thirunarayanan Balathandayuthapani added a comment - The following test case repeats the reported scenario: --source include/have_file_key_management.inc   CREATE TABLE t1(c VARCHAR(128)) ENGINE INNODB, encrypted=yes; insert into t1 select repeat('a',100);   let $MYSQLD_DATADIR=`select @@datadir`; let t1_IBD = $MYSQLD_DATADIR/test/t1.ibd;   --source include/shutdown_mysqld.inc   --echo # Corrupt the table   perl; use strict; use warnings; use Fcntl qw(:DEFAULT :seek);   my $ibd_file = $ENV{'t1_IBD'};   my $chunk; my $len;   sysopen IBD_FILE, $ibd_file, O_RDWR || die "Unable to open $ibd_file"; sysseek IBD_FILE, 16384 * 3, SEEK_CUR; $chunk = '\xAA\xAA\xAA\xAA'; syswrite IBD_FILE, $chunk, 4;   close IBD_FILE; EOF   #--exec $INNOCHECKSUM $MYSQLD_DATADIR/test/t1.ibd   --source include/start_mysqld.inc   echo # xtrabackup backup; let $targetdir=$MYSQLTEST_VARDIR/tmp/backup; let $backuplog=$MYSQLTEST_VARDIR/tmp/backup.log;   --disable_result_log exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$targetdir > $backuplog; --enable_result_log Problem is that mariabackup always read the encrypted pages and checks the post-encryption checksum only and copies the page even though the above case corrupted the pre-encryption checksum.
            GeoffMontee Geoff Montee (Inactive) made changes -

            Patch is in bb-10.2-23578

            thiru Thirunarayanan Balathandayuthapani added a comment - Patch is in bb-10.2-23578
            thiru Thirunarayanan Balathandayuthapani made changes -
            Assignee Thirunarayanan Balathandayuthapani [ thiru ] Marko Mäkelä [ marko ]
            Status In Progress [ 3 ] In Review [ 10002 ]

            I cleaned up thiru’s fix, backported to 10.1 and merged all the way up to 10.4.

            marko Marko Mäkelä added a comment - I cleaned up thiru ’s fix, backported to 10.1 and merged all the way up to 10.4.
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2018-12-14 14:25:31.0 2018-12-14 14:25:31.121
            marko Marko Mäkelä made changes -
            Fix Version/s 10.4.1 [ 23228 ]
            Fix Version/s 10.1.38 [ 23209 ]
            Fix Version/s 10.2.20 [ 23212 ]
            Fix Version/s 10.3.12 [ 23214 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.1 [ 16100 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Fixed [ 1 ]
            Status In Review [ 10002 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -

            Based on more work from thiru, I pushed a follow-up fix to 10.1.

            It turns out that page_compressed=1 pages are not being checked for consistency. That was filed as MDEV-18025.

            marko Marko Mäkelä added a comment - Based on more work from thiru , I pushed a follow-up fix to 10.1 . It turns out that page_compressed=1 pages are not being checked for consistency. That was filed as MDEV-18025 .

            As part of a follow-up fix to MDEV-18025, I improved the test case so that it computes a valid checksum for the ‘encrypted’ page and writes it to all 3 locations. That is, by the old rules, this page would look like a valid encrypted page as well as a valid unencrypted page. Only by decrypting and checking the ‘before-encryption’ checksum we will detect the page as corrupted.

            This test (and the extra decrypt step in mariabackup --backup) was motivated by a support case where a seemingly corrupted page has the same checksum in all 3 locations.

            For page_compressed=1 pages (encrypted or not), the only way to validate the page checksum is to attempt to (optionally decrypt and) decompress the data and to compute the checksum.

            marko Marko Mäkelä added a comment - As part of a follow-up fix to MDEV-18025 , I improved the test case so that it computes a valid checksum for the ‘encrypted’ page and writes it to all 3 locations. That is, by the old rules, this page would look like a valid encrypted page as well as a valid unencrypted page. Only by decrypting and checking the ‘before-encryption’ checksum we will detect the page as corrupted. This test (and the extra decrypt step in mariabackup --backup ) was motivated by a support case where a seemingly corrupted page has the same checksum in all 3 locations. For page_compressed=1 pages (encrypted or not), the only way to validate the page checksum is to attempt to (optionally decrypt and) decompress the data and to compute the checksum.
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 79727 ] MariaDB v4 [ 151740 ]
            marko Marko Mäkelä made changes -

            People

              marko Marko Mäkelä
              anikitin Andrii Nikitin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.