Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18395

InnoDB: "Database page corruption on disk or a failed file read of tablespace" and assertion failure in ha_innodb.cc line 22037 upon startup after minor crash-upgrade from the latest release

    XMLWordPrintable

Details

    Description

      Note: 10.2 and 10.3 crash the same way while trying to startup on this data, but I have no information whether it's the fault of the server which performs the recovery or of which initially created the data, and in the latter case, whether other versions are affected, so I'm filing it for 10.1 only for now.

      10.1 ef0b91e

      2019-01-27 10:50:33 140031557658496 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace ./ibdata1 page  [page id: space=0, page number=5]. You may have to recover from a backup.
      2019-01-27 10:50:33 7f5ba3406780 InnoDB: Page dump in ascii and hex (65536 bytes):
      ...
      InnoDB: End of page dump
      2019-01-27 10:50:33 7f5ba3406780 InnoDB: uncompressed page, stored checksum in field1 1815576519, calculated checksums for field1: crc32 3572139883, innodb 1815576519, none 3735928559, stored checksum in field2 2064831440, calculated checksums for field2: crc32 3572139883, innodb 866585586, none 3735928559, page LSN 0 24674042, low 4 bytes of LSN at page end 17346260, page number (if stored to page already) 5, space id (if created with >= MySQL-4.1.1 and stored already) 0
      InnoDB: page type 7 meaning TRX_SYS
      InnoDB: Page may be a transaction system page
      2019-01-27 10:50:33 140031557658496 [ERROR] InnoDB: Ending processing because of a corrupt database page.
      2019-01-27 10:50:33 7f5ba3406780  InnoDB: Assertion failure in thread 140031557658496 in file ha_innodb.cc line 22037
       
      #5  0x00007f5ba1382028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
      #6  0x0000563f471dbaed in ib_logf (level=IB_LOG_LEVEL_FATAL, format=0x563f4773b8e8 "Ending processing because of a corrupt database page.") at /home/travis/src/storage/xtradb/handler/ha_innodb.cc:22037
      #7  0x0000563f473c36e2 in buf_page_io_complete (bpage=0x7f5b7bff35f0) at /home/travis/src/storage/xtradb/buf/buf0buf.cc:5023
      #8  0x0000563f473e1c08 in buf_read_page_low (err=0x7fffaa2e5a1c, sync=true, mode=132, space=0, zip_size=0, unzip=0, tablespace_version=1, offset=5, trx=0x0, should_buffer=false) at /home/travis/src/storage/xtradb/buf/buf0rea.cc:279
      #9  0x0000563f473e2104 in buf_read_page (space_id=0, zip_size=0, offset=5, trx=0x0) at /home/travis/src/storage/xtradb/buf/buf0rea.cc:512
      #10 0x0000563f473bf57b in buf_page_get_gen (space=0, zip_size=0, offset=5, rw_latch=2, guess=0x0, mode=10, file=0x563f477266c8 "/home/travis/src/storage/xtradb/include/trx0sys.ic", line=103, mtr=0x7fffaa2e5fd0, err=0x0) at /home/travis/src/storage/xtradb/buf/buf0buf.cc:3211
      #11 0x0000563f47353947 in trx_sysf_get (mtr=0x7fffaa2e5fd0) at /home/travis/src/storage/xtradb/include/trx0sys.ic:103
      #12 0x0000563f47354953 in trx_rseg_get_n_undo_tablespaces (space_ids=0x7fffaa2e6520) at /home/travis/src/storage/xtradb/trx/trx0rseg.cc:383
      #13 0x0000563f4732ce22 in srv_undo_tablespaces_init (create_new_db=0, backup_mode=0, n_conf_tablespaces=0, n_opened=0x563f47d575c0 <srv_undo_tablespaces_open>) at /home/travis/src/storage/xtradb/srv/srv0start.cc:1494
      #14 0x0000563f4732e9ba in innobase_start_or_create_for_mysql () at /home/travis/src/storage/xtradb/srv/srv0start.cc:2503
      #15 0x0000563f471bd46f in innobase_init (p=0x563f49a01670) at /home/travis/src/storage/xtradb/handler/ha_innodb.cc:4525
      #16 0x0000563f46e7e865 in ha_initialize_handlerton (plugin=0x563f499f32c0) at /home/travis/src/sql/handler.cc:522
      #17 0x0000563f46c4e843 in plugin_initialize (tmp_root=0x7fffaa2eb5f0, plugin=0x563f499f32c0, argc=0x563f47f11968 <remaining_argc>, argv=0x563f499ae598, options_only=false) at /home/travis/src/sql/sql_plugin.cc:1409
      #18 0x0000563f46c4f471 in plugin_init (argc=0x563f47f11968 <remaining_argc>, argv=0x563f499ae598, flags=2) at /home/travis/src/sql/sql_plugin.cc:1687
      #19 0x0000563f46b6bba7 in init_server_components () at /home/travis/src/sql/mysqld.cc:5169
      #20 0x0000563f46b6cc7b in mysqld_main (argc=32, argv=0x563f499ae598) at /home/travis/src/sql/mysqld.cc:5760
      #21 0x0000563f46b62175 in main (argc=32, argv=0x7fffaa2ec3a8) at /home/travis/src/sql/main.cc:25
      

      Test scenario was a crash-upgrade from the previous 10.1 release (10.1.37) downloaded from here:
      https://downloads.mariadb.com/MariaDB/mariadb-10.1/bintar-linux-glibc_214-x86_64/mariadb-10.1.37-linux-glibc_214-x86_64.tar.gz

      The data directory after the enforced crash before any attempts to upgrade can be found here:
      ftp://ftp.askmonty.org/public/mdev18395-data.tar.gz

      To reproduce the failure, unpack it and start 10.1 server with

      --file-key-management --file-key-management-filename=/data/bld/keys.txt --plugin-load-add=file_key_management.so --innodb-encrypt-tables --innodb-encrypt-log --innodb-encryption-threads=4 --aria-encrypt-tables=1 --encrypt-tmp-disk-tables=1 --encrypt-binlog --innodb-page-size=64K --innodb-compression-algorithm=none --loose-innodb-file-format=Barracuda --loose-innodb-file-per-table=1 --loose-max-statement-time=20 --loose-lock-wait-timeout=20 --loose-innodb-lock-wait-timeout=10 --loose-innodb_log_compressed_pages=on
      

      (Modify the path to the key file as needed)


      Notes on the origin of the report

      Please note that it might be not reproducible easily the same way.
      https://travis-ci.org/elenst/travis-tests/jobs/484824862 [2602 9 7]

      elenst-jira-refs 56fe472 toolbox 572674d

      perl /home/travis/rqg//runall-new.pl --grammar=conf/mariadb/oltp-transactional.yy --grammar2=conf/mariadb/oltp_and_ddl.yy --gendata=conf/mariadb/innodb_upgrade.zz --gendata-advanced --mysqld=--server-id=111 --scenario=CrashUpgrade --duration=90 --basedir1=/home/travis/old --basedir2=/home/travis/server --mysqld=--file-key-management --mysqld=--file-key-management-filename=/home/travis/mariadb-toolbox/travis/../data/keys.txt --mysqld=--plugin-load-add=file_key_management.so --mysqld=--innodb-encrypt-tables --mysqld=--innodb-encrypt-log --mysqld=--innodb-encryption-threads=4 --mysqld=--aria-encrypt-tables=1 --mysqld=--encrypt-tmp-disk-tables=1 --mysqld=--encrypt-binlog --mysqld=--innodb-page-size=64K --mysqld=--innodb-compression-algorithm=none --mysqld=--loose-innodb-file-format=Barracuda --mysqld=--loose-innodb-file-per-table=1 --gendata=conf/mariadb/innodb_upgrade.zz --no-mask --seed=1548585422 --threads=4 --queries=100M --mysqld=--loose-max-statement-time=20 --mysqld=--loose-lock-wait-timeout=20 --mysqld=--loose-innodb-lock-wait-timeout=10 --mysqld=--loose-innodb_log_compressed_pages=on --mtr-build-thread=300 --vardir1=/home/travis/logs/current1_1
      

      Attachments

        Activity

          People

            marko Marko Mäkelä
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.