[MDEV-9774] Checksum errors on restart when killing busy instance that uses encrypted XtraDB tables with <10.1.13 Created: 2016-03-22  Updated: 2017-10-23  Resolved: 2017-10-23

Status: Closed
Project: MariaDB Server
Component/s: Encryption, Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.1.9, 10.1
Fix Version/s: 10.1.13

Type: Bug Priority: Major
Reporter: Jan Lindström (Inactive) Assignee: Jan Lindström (Inactive)
Resolution: Duplicate Votes: 1
Labels: None
Environment:

Linux

Issue Links:
PartOf
includes MDEV-9422 Encrypted redo log checksum errors on... Closed
includes MDEV-9793 getting mysqld crypto key from key ve... Closed
Sprint: 10.1.11, 10.1.12, 10.1.13

 Description   

When using encrypted XtraDB tables there is a "good" chance to run into checksum errors in recovery when killing mysqld hard after large inserts on a table (e.g. restoring a mysqldump or doing a large INSERT ... SELECT).

How to reproduce:

  • enable encryption by default
  • insert a lot of data
  • "killall -9 mysqld" right afterwards while changed tablespace data is still written back to disk
  • in about 1 of 3 cases I got a page checksum error when restarting the server

e.g.: 

2016-01-14 16:04:47 139797272356736 [Note] InnoDB: Starting crash recovery.
 
2016-01-14 16:04:47 139797272356736 [Note] InnoDB: Reading tablespace information from the .ibd files...
 
2016-01-14 16:04:47 139797272356736 [Note] InnoDB: Restoring possible half-written data pages
 
2016-01-14 16:04:47 139797272356736 [Note] InnoDB: from the doublewrite buffer...
 
InnoDB: Doing recovery: scanned up to log sequence number 34316812800
 
InnoDB: Doing recovery: scanned up to log sequence number 34322055680
 
InnoDB: Doing recovery: scanned up to log sequence number 34327298560
 
InnoDB: Doing recovery: scanned up to log sequence number 34332541440
 
InnoDB: Doing recovery: scanned up to log sequence number 34337784320
 
InnoDB: Doing recovery: scanned up to log sequence number 34343027200
 
InnoDB: Doing recovery: scanned up to log sequence number 34348270080
 
InnoDB: Doing recovery: scanned up to log sequence number 34353512960
 
BROKEN: block: 67102352 checkpoint: 1119 4d783007 94202545
 
BROKEN: block: 67102352 checkpoint: 1119 4d783007 94202545
 
InnoDB: Log block no 67102352 at lsn 34356403712 has
 
InnoDB: ok header, but checksum field contains 2485134661, should be 1299722247
 
2016-01-14 16:04:48 139797272356736 [ERROR] InnoDB: Redo log crypto: failed to decrypt log block. Reason could be that requested key version is not found, required encryption key management plugin is not found or configured encryption algorithm and/or method does not match.
 
2016-01-14 16:04:48 139797272356736 [ERROR] Plugin 'InnoDB' init function returned error.
 
2016-01-14 16:04:48 139797272356736 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
 
2016-01-14 16:04:48 139797272356736 [Note] Plugin 'FEEDBACK' is disabled.
 
2016-01-14 16:04:48 139797272356736 [ERROR] Unknown/unsupported storage engine: InnoDB
 
2016-01-14 16:04:48 139797272356736 [ERROR] Aborting

Full dump of crashed instance base- and datadir is in http://php-groupies.de/encryption-crash.tar.gz (~8GB)

 Comments   
Comment by Jan Lindström (Inactive) [ 2016-03-22 ]

Fix InnoDB crash recovery if we already have database created with MariaDB < 10.1.13 using encryption and server refuses to start because of checksum errors.

Comment by none now [ 2016-04-04 ]

how does one recover from this problem? I have not clue as how it happened, but... it's bad

BROKEN: block: 143666 checkpoint: 9716 69e8f3cb 2f3d2980
BROKEN: block: 143666 checkpoint: 9716 69e8f3cb 2f3d2980
InnoDB: Log block no 143666 at lsn 73556480 has
InnoDB: ok header, but checksum field contains 792537472, should be 1776874443
2016-04-04 15:12:54 140081694828672 [ERROR] InnoDB: Redo log crypto: failed to decrypt log block. Reason could be that requested key version is not found, required encryption key management plugin is not found or configured encryption algorithm and/or method does not match.

Comment by Jan Lindström (Inactive) [ 2016-04-04 ]

What did you do, did you shutdown server nicely or killed it? If this is normal shutdown with innodb_fast_shutdown=0 here is what you can do:

(1) Take backup from datadir
(2) Delete ib_log* files
(3) start your server again with MariaDB >= 10.1.13

Comment by none now [ 2016-04-06 ]

I don't know how/why it happened, but deleting ip_log* helped.

Comment by Jan Lindström (Inactive) [ 2017-10-23 ]

Duplicated by MDEV-9422.

Generated at Thu Feb 08 07:37:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.