[MDEV-20983] Page 38:4016 may be corrupted. Post encryption checksum 3690179534 stored Created: 2019-11-05  Updated: 2019-11-06  Resolved: 2019-11-06

Status: Closed
Project: MariaDB Server
Component/s: Encryption, Storage Engine - InnoDB
Affects Version/s: 10.2.8
Fix Version/s: 10.4.1, 10.1.38, 10.2.20, 10.3.12

Type: Bug Priority: Major
Reporter: Emmanuel Torre Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: corruption, encryption, innodb, need_feedback
Environment:

RH Linux 3.10.0-957.21.3.el7.x86_64, VMWare
4CPU 16GB RAM

Attachments: Text File before_innocrash_20191105.txt     Text File innocrash_20191105.txt    
Issue Links:
Duplicate
duplicates MDEV-12112 corruption in encrypted table may be ... Closed

 Description   

hi

In a Mariadb replication setup (1 master, 1 slave)

We got an
InnoDB: Page 38:4016 may be corrupted. Post encryption checksum 3690179534 stored [3690179534:3690179534] key_version

for no obvious reason, no HW failure or anything noticeable at OS level.

Then a bit later : Got error 192 when reading table

We had to switch role to let the application operate again. That went OK, although we lost the rows that could not be replicated for the corrupted table.

After a while, the old master that was promoted to slaved crashed an could not be restarted, besides using recovery level 6.

We ended resyncing with full backup but the issue is scary.

Asking for analysis and feedback.

Thank you
Emmanuel

 Comments   
Comment by Elena Stepanova [ 2019-11-05 ]

10.2.8 is over 2 years old and 20 releases behind the current one. There have been numerous bugfixes since then, specifically around encryption. Please upgrade to a current version and report a bug against it if the problem persists.

Comment by Marko Mäkelä [ 2019-11-06 ]

The first line of before_innocrash_20191105.txt is:

2019-11-04 12:55:17 140692352591616 [ERROR] InnoDB:  Page 38:4016 may be corrupted. Post encryption checksum 3690179534 stored [3690179534:3690179534] key_version 1

Note: it is showing the same checksum in all 3 places. This message is a telltale sign of severely broken logic. The code wrongly assumed that it would be impossible for the checksum of the data payload to be the same both before and after encryption. In fact, the probability of that is at least 1/4,294,967,296 (actually much more, due to checksum sloppiness issues that were addressed in MDEV-17957 and MDEV-17958).

The broken logic was fixed in a follow-up fix to MDEV-12112, in MariaDB 10.1.38, 10.2.20, 10.3.12.

Comment by Emmanuel Torre [ 2019-11-06 ]

thank you for the responses and explanations. We willl naturally upgrade.

Generated at Thu Feb 08 09:03:42 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.