Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20313

Transactional Aria table stays corrupt after crash-recovery: error : Found 2275 keys of 2274

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.5, 10.1, 10.2, 10.3, 10.4
    • Fix Version/s: 10.4
    • Component/s: Storage Engine - Aria
    • Labels:
      None

      Description

      The test creates several Aria tables, all with TRANSACTIONAL=1, populates them and runs concurrent DML on them: INSERT, UPDATE, DELETE and LOAD DATA INFILE. After ~1 minute while still running DML, it SIGKILLs the server, restarts the server on the same datadir and runs CHECK TABLE ... EXTENDED on all tables.

      What it often finds is this:

      # 2019-08-10T18:17:41 [7684] For table `test`.`oltp5` : error : Found 1144 keys of 1143
      # 2019-08-10T18:17:41 [7684] For table `test`.`oltp5` : error : Corrupt
      

      Error log doesn't reveal any problem:

      2019-08-10 18:30:37 0 [Note] mysqld: Aria engine: starting recovery
      recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (41.5 seconds); transactions to roll back: 3 2 1 0 (0.1 seconds); tables to flush: 8 7 6 5 4 3 2 1 0
       (0.0 seconds); 
      2019-08-10 18:31:19 0 [Note] mysqld: Aria engine: recovery done
      

      To reproduce:

      git clone https://github.com/MariaDB/randgen --branch mdev20313 rqg-mdev20313
      cd rqg-mdev20313
      perl ./runall-trials.pl --basedir=<your basedir> --vardir=<your vardir> --trials=5 --threads=6 --duration=100 --scenario=CrashUpgrade --mysqld=--loose-max-statement-time=10 --mysqld=--lock-wait-timeout=5 --grammar=mdev20313.yy --gendata=mdev20313.zz
      


      Attention:
      - Use the RQG branch specified above. It is tuned for the purpose of reproducing the problem.
      - Don't forget to set basedir and vardir on the command line.

      Other notes:

      • The failure happens both on disk and in shm, but it seems to have higher probability on disk (even on SSD).
      • The command line above will run the same test 5 times (controlled by --trials, unless it fails; it will exit after the first failure.
      • If you want to examine the tables or aria log after the recovery attempt, you'll find them in <vardir>/data . If you want them before the recovery attempt, after the server was SIGKILLed, they will be in <vardir>/data_orig.

      Example of a datadir with such a corrupt table (before any recovery attempts) can be found here: ftp://ftp.askmonty.org/public/mdev20313-data.tar.gz
      It was created on 10.4 server, all default options. Problematic tables there are test.oltp6 and test.oltp7:

      MariaDB [(none)]> CHECK TABLE test.oltp6 EXTENDED;
      +------------+-------+----------+-----------------------------------------------------------+
      | Table      | Op    | Msg_type | Msg_text                                                  |
      +------------+-------+----------+-----------------------------------------------------------+
      | test.oltp6 | check | warning  | Found row where the auto_increment column has the value 0 |
      | test.oltp6 | check | error    | Found 1138 keys of 1137                                   |
      | test.oltp6 | check | error    | Corrupt                                                   |
      +------------+-------+----------+-----------------------------------------------------------+
      3 rows in set (0.019 sec)
       
      MariaDB [(none)]> CHECK TABLE test.oltp7 EXTENDED;
      +------------+-------+----------+----------------------------+
      | Table      | Op    | Msg_type | Msg_text                   |
      +------------+-------+----------+----------------------------+
      | test.oltp7 | check | error    | Found 1088 keys of 1087    |
      | test.oltp7 | check | error    | Corrupt                    |
      +------------+-------+----------+----------------------------+
      2 rows in set (0.009 sec)
      

        Attachments

          Activity

            People

            • Assignee:
              monty Michael Widenius
              Reporter:
              elenst Elena Stepanova
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: