[MDEV-13591] InnoDB: Database page corruption on disk or a failed file read and assertion failure Created: 2017-08-19  Updated: 2017-08-28  Resolved: 2017-08-28

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.1
Fix Version/s: 10.1.27

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: None


 Description    

$ perl ./mtr --mysqld=--innodb-undo-tablespaces=3 encryption.innodb_encryption

10.1 06106c01481401

 
2017-08-20  1:08:18 139892291375232 [ERROR] InnoDB: Database page corruption on disk or a failed file read of tablespace ./undo001 page  [page id: space=1, page number=3]. You may have to recover from a backup.
 
2017-08-20 01:08:18 7f3b36559080 InnoDB: Page dump in ascii and hex (16384 bytes):
 
....
 
....
 
InnoDB: End of page dump
 
2017-08-20 01:08:19 7f3b36559080 InnoDB: uncompressed page, stored checksum in field1 545439436, calculated checksums for field1: crc32 858124927, innodb 1700049671, none 373592855
 
9, stored checksum in field2 4071519962, calculated checksums for field2: crc32 858124927, innodb 4071519962, none 3735928559, page LSN 0 1485027, low 4 bytes of LSN at page end 14
 
85027, page number (if stored to page already) 3, space id (if created with >= MySQL-4.1.1 and stored already) 1
 
InnoDB: page type 6 meaning SYS
 
InnoDB: Page may be a system page
 
2017-08-20  1:08:19 139892291375232 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If
 
 the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for c
 
orruption. Please refer to http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html for information about forcing recovery.
 
2017-08-20  1:08:19 139892291375232 [ERROR] InnoDB: Unable to read tablespace 1 page no 3 into the buffer pool after 100 attempts. The most probable cause of this error may be that the table has been corrupted. You can try to fix this problem by using innodb_force_recovery. Please see http://dev.mysql.com/doc/refman/5.6/en/ for more details. Aborting...
 
2017-08-20 01:08:19 7f3b36559080  InnoDB: Assertion failure in thread 139892291375232 in file ha_innodb.cc line 20652


#5  0x00007f3b341d63fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
 
#6  0x00007f3b2d70feda in ib_logf (level=IB_LOG_LEVEL_FATAL, format=0x7f3b2d8aa420 "Unable to read tablespace %lu page no %lu into the buffer pool after %lu attempts. The most probable cause of this error may be that the table has been corrupted. You can try to fix this problem by us"...) at /data/src/10.1/storage/innobase/handler/ha_innodb.cc:20652
 
#7  0x00007f3b2d629137 in buf_page_get_gen (space=1, zip_size=0, offset=3, rw_latch=2, guess=0x0, mode=10, file=0x7f3b2d9300b0 "/data/src/10.1/storage/innobase/include/trx0rseg.ic", line=70, mtr=0x7ffc8ee951f0, err=0x0) at /data/src/10.1/storage/innobase/buf/buf0buf.cc:3236
 
#8  0x00007f3b2d880c94 in trx_rsegf_get_new (space=1, zip_size=0, page_no=3, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/include/trx0rseg.ic:70
 
#9  0x00007f3b2d8813d9 in trx_rseg_mem_create (id=1, space=1, zip_size=0, page_no=3, ib_bh=0x7f3b2ed8f400, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/trx/trx0rseg.cc:203
 
#10 0x00007f3b2d8816e2 in trx_rseg_create_instance (sys_header=0x7f3b2b018026 "", ib_bh=0x7f3b2ed8f400, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/trx/trx0rseg.cc:288
 
#11 0x00007f3b2d88197c in trx_rseg_array_init (sys_header=0x7f3b2b018026 "", ib_bh=0x7f3b2ed8f400, mtr=0x7ffc8ee951f0) at /data/src/10.1/storage/innobase/trx/trx0rseg.cc:357
 
#12 0x00007f3b2d88462e in trx_sys_init_at_db_start () at /data/src/10.1/storage/innobase/trx/trx0sys.cc:647
 
#13 0x00007f3b2d85e0a1 in innobase_start_or_create_for_mysql () at /data/src/10.1/storage/innobase/srv/srv0start.cc:2556
 
#14 0x00007f3b2d6f1b5a in innobase_init (p=0x7f3b33837470) at /data/src/10.1/storage/innobase/handler/ha_innodb.cc:3970
 
#15 0x0000559bcb880183 in ha_initialize_handlerton (plugin=0x7f3b2fbf6bf0) at /data/src/10.1/sql/handler.cc:513
 
#16 0x0000559bcb65bcb0 in plugin_initialize (tmp_root=0x7ffc8ee99300, plugin=0x7f3b2fbf6bf0, argc=0x559bcc8dc7f0 <remaining_argc>, argv=0x7f3b3387dac0, options_only=false) at /data/src/10.1/sql/sql_plugin.cc:1406
 
#17 0x0000559bcb65c899 in plugin_init (argc=0x559bcc8dc7f0 <remaining_argc>, argv=0x7f3b3387dac0, flags=2) at /data/src/10.1/sql/sql_plugin.cc:1684
 
#18 0x0000559bcb57b1f7 in init_server_components () at /data/src/10.1/sql/mysqld.cc:5154
 
#19 0x0000559bcb57c262 in mysqld_main (argc=156, argv=0x7f3b3387dac0) at /data/src/10.1/sql/mysqld.cc:5741
 
#20 0x0000559bcb571930 in main (argc=32, argv=0x7ffc8ee9a098) at /data/src/10.1/sql/main.cc:25

10.2 does not fail.

 Comments   
Comment by Jan Lindström (Inactive) [ 2017-08-22 ]

https://github.com/MariaDB/server/commit/ddf7e15dc0052a5da414a05cb27319b42e8147a4

Comment by Marko Mäkelä [ 2017-08-22 ]

I would prefer a lower-level approach: amending the code where the first page of the undo tablespace is already being read for the very first time. We should have the encryption information available already during redo log apply. The submitted patch only reads the information after the redo log has been applied.

Comment by Jan Lindström (Inactive) [ 2017-08-23 ]

10.2 is not effected.

Comment by Jan Lindström (Inactive) [ 2017-08-23 ]

https://github.com/MariaDB/server/commit/dc465f5af3674f4aa139d4fe496f242cc5947033

My first patch was shorter. Note that no patch needed for 10.2 as crash is not repeatable there.

Comment by Marko Mäkelä [ 2017-08-23 ]

Thank you, it is better now. But, please split the function fil_read_first_page() into two, instead of adding a non-obvious parameter to it.

Comment by Jan Lindström (Inactive) [ 2017-08-28 ]

Can't use srv_is_undo_tablespace because space_id might not be correctly set at that point as it should be read from first page. Similarly, splitting the function would make the patch longer for GA-release. I will use new parameter called check_first_page that is by default true and from undo-tablespaces false.

Comment by Jan Lindström (Inactive) [ 2017-08-28 ]

commit 61096ff214dec68a66efb811aa3a70ef49f06a11
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Mon Aug 28 09:45:54 2017 +0300

MDEV-13591: InnoDB: Database page corruption on disk or a failed file read and assertion failure

Problem is that page 0 and its possible enrryption information
is not read for undo tablespaces.

fil_crypt_get_latest_key_version(): Do not send event to
encryption threads if event does not yet exists. Seen
on regression testing.

fil_read_first_page: Add new parameter does page belong to
undo tablespace and if it does, we do not read FSP_HEADER.

srv_undo_tablespace_open : Read first page of the tablespace
to get crypt_data if it exists and pass it to fil_space_create.

Tested using innodb_encryption with combinations with
innodb-undo-tablespaces.

commit 882f4566e55694b4f68f61ee723678b49d7e0d8f
Author: Elena Stepanova <elenst@montyprogram.com>
Date: Sat Aug 19 02:52:35 2017 +0300

Combinations with innodb-undo-tablespaces to use in test files

Generated at Thu Feb 08 08:06:48 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.