[MDEV-17958] Make bug-endian innodb_checksum_algorithm=crc32 optional Created: 2018-12-10  Updated: 2021-03-11  Resolved: 2018-12-13

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.2.2, 10.3.0, 10.4.0
Fix Version/s: 10.4.1, 10.2.20, 10.3.12

Type: Bug Priority: Major
Reporter: Thirunarayanan Balathandayuthapani Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: checksum, compat56, compat57, corruption

Issue Links:
Relates
relates to MDEV-25105 Remove innodb_checksum_algorithm valu... Closed
relates to MDEV-13542 Crashing on a corrupted page is unhel... Closed
relates to MDEV-17638 Improve error message about corruptio... Closed
relates to MDEV-18529 InnoDB wrongly skips decryption of en... Closed

 Description   

In MySQL 5.7, it was noticed that files are not portable between big-endian and little-endian systems (such as SPARC and x86), because the original implementation of innodb_checksum_algorithm=crc32 was not byte order agnostic.

A byte order agnostic implementation of innodb_checksum_algorithm=crc32 was only added to MySQL 5.7, not backported to 5.6. Consequently, MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C implementation that works incorrectly on big-endian architectures, and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C implementation from MySQL 5.7.

MySQL 5.7 introduced a "legacy crc32" variant that is functionally equivalent to the big-endian version of the original crc32 implementation. Thanks to this variant, old data files can be transferred from big-endian systems to newer versions.

Introducing new variants of checksum algorithms (without introducing new names for them) generally is a bad idea, because each checksum algorithm is like a lottery ticket. The more algorithms you try, the more likely it will be for the checksum to match on a corrupted page.

So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32, and MariaDB 10.2.2 inherited this weakening.

We elect to remove the bug-compatible variant of innodb_checksum_algorithm=crc32 as follows:

  1. We assume that most users are on little-endian.
  2. Let us make the bug-compatible variant only present on big-endian systems (#ifdef WORDS_BIGENDIAN).
  3. If someone is upgrading from MariaDB 10.0 or 10.1 or MySQL 5.6 to MariaDB 10.2 or later, they will stay on the same architecture.
  4. Completely remove the bug-compatible variant from MariaDB 10.4.
  5. If someone is switching from big-endian to little-endian, they can do it with logical dump, or they can use innochecksum to recompute the checksums.


 Comments   
Comment by Marko Mäkelä [ 2018-12-13 ]

I introduced the build option

cmake -DWITH_INNODB_BUG_ENDIAN_CRC32=ON

It is OFF by default on little-endian systems, and ON by default on big-endian systems.

I plan to remove this option (and the code for supporting the buggy algorithm) in the upcoming MariaDB 10.4.1 beta release.

Generated at Thu Feb 08 08:40:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.