Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13591

InnoDB: Database page corruption on disk or a failed file read and assertion failure

Details

    Description

      $ perl ./mtr --mysqld=--innodb-undo-tablespaces=3 encryption.innodb_encryption
      

      10.1 06106c01481401

      2017-08-20  1:08:18 139892291375232 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace ./undo001 page  [page id: space=1, page number=3]. You may have to recover from a backup.
      2017-08-20 01:08:18 7f3b36559080 InnoDB: Page dump in ascii and hex (16384 bytes):
      ....
      ....
      InnoDB: End of page dump
      2017-08-20 01:08:19 7f3b36559080 InnoDB: uncompressed page, stored checksum in field1 545439436, calculated checksums for field1: crc32 858124927, innodb 1700049671, none 373592855
      9, stored checksum in field2 4071519962, calculated checksums for field2: crc32 858124927, innodb 4071519962, none 3735928559, page LSN 0 1485027, low 4 bytes of LSN at page end 14
      85027, page number (if stored to page already) 3, space id (if created with >= MySQL-4.1.1 and stored already) 1
      InnoDB: page type 6 meaning SYS
      InnoDB: Page may be a system page
      2017-08-20  1:08:19 139892291375232 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If
       the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for c
      orruption. Please refer to http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html for information about forcing recovery.
      2017-08-20  1:08:19 139892291375232 [ERROR] InnoDB: Unable to read tablespace 1 page no 3 into the buffer pool after 100 attempts. The most probable cause of this error may be that the table has been corrupted. You can try to fix this problem by using innodb_force_recovery. Please see http://dev.mysql.com/doc/refman/5.6/en/ for more details. Aborting...
      2017-08-20 01:08:19 7f3b36559080  InnoDB: Assertion failure in thread 139892291375232 in file ha_innodb.cc line 20652
      

      #5  0x00007f3b341d63fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
      #6  0x00007f3b2d70feda in ib_logf (level=IB_LOG_LEVEL_FATAL, format=0x7f3b2d8aa420 "Unable to read tablespace %lu page no %lu into the buffer pool after %lu attempts. The most probable cause of this error may be that the table has been corrupted. You can try to fix this problem by us"...) at /data/src/10.1/storage/innobase/handler/ha_innodb.cc:20652
      #7  0x00007f3b2d629137 in buf_page_get_gen (space=1, zip_size=0, offset=3, rw_latch=2, guess=0x0, mode=10, file=0x7f3b2d9300b0 "/data/src/10.1/storage/innobase/include/trx0rseg.ic", line=70, mtr=0x7ffc8ee951f0, err=0x0) at /data/src/10.1/storage/innobase/buf/buf0buf.cc:3236
      #8  0x00007f3b2d880c94 in trx_rsegf_get_new (space=1, zip_size=0, page_no=3, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/include/trx0rseg.ic:70
      #9  0x00007f3b2d8813d9 in trx_rseg_mem_create (id=1, space=1, zip_size=0, page_no=3, ib_bh=0x7f3b2ed8f400, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/trx/trx0rseg.cc:203
      #10 0x00007f3b2d8816e2 in trx_rseg_create_instance (sys_header=0x7f3b2b018026 "", ib_bh=0x7f3b2ed8f400, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/trx/trx0rseg.cc:288
      #11 0x00007f3b2d88197c in trx_rseg_array_init (sys_header=0x7f3b2b018026 "", ib_bh=0x7f3b2ed8f400, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/trx/trx0rseg.cc:357
      #12 0x00007f3b2d88462e in trx_sys_init_at_db_start () at /data/src/10.1/storage/innobase/trx/trx0sys.cc:647
      #13 0x00007f3b2d85e0a1 in innobase_start_or_create_for_mysql () at /data/src/10.1/storage/innobase/srv/srv0start.cc:2556
      #14 0x00007f3b2d6f1b5a in innobase_init (p=0x7f3b33837470) at /data/src/10.1/storage/innobase/handler/ha_innodb.cc:3970
      #15 0x0000559bcb880183 in ha_initialize_handlerton (plugin=0x7f3b2fbf6bf0) at /data/src/10.1/sql/handler.cc:513
      #16 0x0000559bcb65bcb0 in plugin_initialize (tmp_root=0x7ffc8ee99300, plugin=0x7f3b2fbf6bf0, argc=0x559bcc8dc7f0 <remaining_argc>, argv=0x7f3b3387dac0, options_only=false) at /data/src/10.1/sql/sql_plugin.cc:1406
      #17 0x0000559bcb65c899 in plugin_init (argc=0x559bcc8dc7f0 <remaining_argc>, argv=0x7f3b3387dac0, flags=2) at /data/src/10.1/sql/sql_plugin.cc:1684
      #18 0x0000559bcb57b1f7 in init_server_components () at /data/src/10.1/sql/mysqld.cc:5154
      #19 0x0000559bcb57c262 in mysqld_main (argc=156, argv=0x7f3b3387dac0) at /data/src/10.1/sql/mysqld.cc:5741
      #20 0x0000559bcb571930 in main (argc=32, argv=0x7ffc8ee9a098) at /data/src/10.1/sql/main.cc:25
      

      10.2 does not fail.

      Attachments

        Activity

          jplindst Jan Lindström (Inactive) added a comment - https://github.com/MariaDB/server/commit/ddf7e15dc0052a5da414a05cb27319b42e8147a4

          I would prefer a lower-level approach: amending the code where the first page of the undo tablespace is already being read for the very first time. We should have the encryption information available already during redo log apply. The submitted patch only reads the information after the redo log has been applied.

          marko Marko Mäkelä added a comment - I would prefer a lower-level approach: amending the code where the first page of the undo tablespace is already being read for the very first time. We should have the encryption information available already during redo log apply. The submitted patch only reads the information after the redo log has been applied.

          10.2 is not effected.

          jplindst Jan Lindström (Inactive) added a comment - 10.2 is not effected.

          https://github.com/MariaDB/server/commit/dc465f5af3674f4aa139d4fe496f242cc5947033

          My first patch was shorter. Note that no patch needed for 10.2 as crash is not repeatable there.

          jplindst Jan Lindström (Inactive) added a comment - https://github.com/MariaDB/server/commit/dc465f5af3674f4aa139d4fe496f242cc5947033 My first patch was shorter. Note that no patch needed for 10.2 as crash is not repeatable there.

          Thank you, it is better now. But, please split the function fil_read_first_page() into two, instead of adding a non-obvious parameter to it.

          marko Marko Mäkelä added a comment - Thank you, it is better now. But, please split the function fil_read_first_page() into two, instead of adding a non-obvious parameter to it.

          Can't use srv_is_undo_tablespace because space_id might not be correctly set at that point as it should be read from first page. Similarly, splitting the function would make the patch longer for GA-release. I will use new parameter called check_first_page that is by default true and from undo-tablespaces false.

          jplindst Jan Lindström (Inactive) added a comment - Can't use srv_is_undo_tablespace because space_id might not be correctly set at that point as it should be read from first page. Similarly, splitting the function would make the patch longer for GA-release. I will use new parameter called check_first_page that is by default true and from undo-tablespaces false.

          commit 61096ff214dec68a66efb811aa3a70ef49f06a11
          Author: Jan Lindström <jan.lindstrom@mariadb.com>
          Date: Mon Aug 28 09:45:54 2017 +0300

          MDEV-13591: InnoDB: Database page corruption on disk or a failed file read and assertion failure

          Problem is that page 0 and its possible enrryption information
          is not read for undo tablespaces.

          fil_crypt_get_latest_key_version(): Do not send event to
          encryption threads if event does not yet exists. Seen
          on regression testing.

          fil_read_first_page: Add new parameter does page belong to
          undo tablespace and if it does, we do not read FSP_HEADER.

          srv_undo_tablespace_open : Read first page of the tablespace
          to get crypt_data if it exists and pass it to fil_space_create.

          Tested using innodb_encryption with combinations with
          innodb-undo-tablespaces.

          commit 882f4566e55694b4f68f61ee723678b49d7e0d8f
          Author: Elena Stepanova <elenst@montyprogram.com>
          Date: Sat Aug 19 02:52:35 2017 +0300

          Combinations with innodb-undo-tablespaces to use in test files

          jplindst Jan Lindström (Inactive) added a comment - commit 61096ff214dec68a66efb811aa3a70ef49f06a11 Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Mon Aug 28 09:45:54 2017 +0300 MDEV-13591 : InnoDB: Database page corruption on disk or a failed file read and assertion failure Problem is that page 0 and its possible enrryption information is not read for undo tablespaces. fil_crypt_get_latest_key_version(): Do not send event to encryption threads if event does not yet exists. Seen on regression testing. fil_read_first_page: Add new parameter does page belong to undo tablespace and if it does, we do not read FSP_HEADER. srv_undo_tablespace_open : Read first page of the tablespace to get crypt_data if it exists and pass it to fil_space_create. Tested using innodb_encryption with combinations with innodb-undo-tablespaces. commit 882f4566e55694b4f68f61ee723678b49d7e0d8f Author: Elena Stepanova <elenst@montyprogram.com> Date: Sat Aug 19 02:52:35 2017 +0300 Combinations with innodb-undo-tablespaces to use in test files

          People

            jplindst Jan Lindström (Inactive)
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.