• 3 DB-Nodes running in virtualized environments (2 LXCs, 1 KVM)
• Debian 9.8 Stretch
• Galera Replication
• about 120 web servers access DBs using respective schemas and users
• ProxySQL as SQL-aware load balancer (only one DB-Node should Read/Write) in between the DB-Nodes and the Webservers

InnoDB is crashing because it detects corruption in a record header of a table that does not use ROW_FORMAT=REDUNDANT (which was the original and only format before MySQL 5.0.3). The function rec_get_offsets_func() is reading the 3 bits from rec_get_status(), and expects the most significant bit of it to be zero.

In MDEV-13542 we should make the error handling of InnoDB more robust, so that crashes like this would be avoided.

But, the cause of the corruption is unclear and cannot be found without a repeatable test case. I fixed some corruption bugs recently (MDEV-14126 just missed the 10.2.23 release), but I don’t think those fixes would affect this kind of corruption.

I would suggest using innodb_compression_algorithm=strict_crc32 to improve the chances of detecting this kind of corruption earlier, when reading pages into the buffer pool. If corrupted pages enter the buffer pool, InnoDB can crash in very many places.

There could be a problem in the snapshot transfer (SST) procedure, or the corruption could simply have been propagated by the SST. That is the downside of physical replication.

If you can provide more information, such as page dumps (which could be obtained by using a debugger), I can try to help further.

Firstly thanks a lot for the quick response.

After reading your comment i got some question which might help me to solve our situation.

1. We got some tables which use ROW_FORMAT=COMPRESSED. Is it possibly to alter the row-format for these tables and if yes would this bypass our problems.
2. You are suggesting to set innodb_compression_algorithm=strict_crc32 but in the official docs i can't find this algorithm as a supported option (https://mariadb.com/kb/en/library/compression/#choosing-compression-algorithm)
3. If this option leads to an earlier detection of corruptions, does that mean that they are also not loaded into buffer pool?
4. Assuming that the appropriate webservers to the tables (see 1) ) are less important, would it be helpful to change some table (or schema) specific options, forbid write access or re-import the tables from existing db dumps or even shut down the application?
5. If I'm correctly understanding there is currently no bugfix in any of the newer versions?

Sorry, I meant to write innodb_checksum_algorithm=strict_crc32. In MariaDB 10.4 there will be an even better option, full_crc32, but it will not affect the format of ROW_FORMAT=COMPRESSED pages.

I do not think that this particular form of corruption should be possible with ROW_FORMAT=COMPRESSED pages, because the status bits are generated when the page is decompressed. (I am the author of that format.) The only way how you could get this assertion failure for ROW_FORMAT=COMPRESSED is that something (software bug or hardware fault) corrupts the InnoDB buffer pool, which I think is rather unlikely.

Rebuilding tables can sometimes remove corruption, if that corruption is not in the pages that must be read in order to copy the table. SELECT * (possibly done by mysqldump) or a table rebuild will have to read all leaf pages of the clustered index tree, and all BLOB pages that are pointed to by leaf-page records. Additionally, all pages on the path from the root to the first leaf page will have to be read. If you are lucky, the corruption is in a secondary index page or in a non-leaf page that will not be accessed when copying the table.

I would have to see a dump of the corrupted page in order to determine the cause of the corruption. We have not seen this form of page corruption in our internal testing, except when some bug in mariabackup has been involved.

> Sorry, I meant to write innodb_checksum_algorithm=strict_crc32
Am i mistaken or is this exactly what you wrote in your first comment?
Ah ok, I see ...sorry it's friday
Unfortunately the servers are already configured with this algorithm

Rebuilding tables can sometimes remove corruption, if that corruption is not in the pages that must be read in order to copy the table. SELECT * (possibly done by mysqldump) or a table rebuild will have to read all leaf pages of the clustered index tree, and all BLOB pages that are pointed to by leaf-page records. Additionally, all pages on the path from the root to the first leaf page will have to be read. If you are lucky, the corruption is in a secondary index page or in a non-leaf page that will not be accessed when copying the table.
I will try this in the next days and until then hopefully nothing will happen.

I would have to see a dump of the corrupted page in order to determine the cause of the corruption. We have not seen this form of page corruption in our internal testing, except when some bug in mariabackup has been involved.
Sorry if I'm asking but how could I create this dump and within which time after a crash could I get it

Today we run into the same Error. I have added the backtrace and the memory-map as shown up
in the log file.

I hope this is what you mean with a dump of the corrupted page .
Otherwise I'm not really shure how to get such a dump.
db1b_backtrace-MemoryMap

You could get a dump of the corrupted page by attaching a debugger to the mysqld process before the crash, and running suitable commands. In GDB, you would use up or frame and then dump the buf_block_t::frame and also the buf_block_t::page metadata. In the stack frame of the failing assertion in db1b_backtrace-MemoryMap , that would be something like the following:

print latch_leaves.blocks[0]->page

dump binary memory /tmp/page.bin latch_leaves.blocks[0]->frame latch_leaves.blocks[0]->frame+srv_page_size

We’d also want to identify the index and the table. For our support customers, we would provide detailed instructions how to do that.