I don't know if it is related to this. MDEV-21726

Based on the comments there I applied these settings

SET GLOBAL innodb_change_buffering=inserts;

SET GLOBAL innodb_adaptive_hash_index=OFF;

But now still get a similar ERROR , this time the record was not found , I am not sure if related to this or not

2020-02-25 14:57:33 12 [ERROR] InnoDB: Record in index `index_name` of table `schema_name`.`table_name` was not found on update: TUPLE (info_bits=0, 4 fields):

at: COMPACT RECORD(info_bits=0, 4 fields):

and when I ran a mysqlcheck on the table it was flagged as corrupt

One note I forgot to add our architecture is a 5 node Galera cluster that we use for our most of our writes and some reads, we then have 10 slaves that replicate off of our galera nodes used for read-only. The slave servers that I upgrade to 10.3.21 that we are seeing the errors (we have not upgraded galera cluster yet so not sure if we will get it on galera as well). We are upgrading the slaves first , so will be replicating from 5.5 -> 10.3, once all the slaves are fully upgraded we then plan to upgrade the galera cluster from 5.5 to 10.3

The version of our galera cluster nodes which is also the master are:

MariaDB-devel-5.5.63-1.el6.x86_64
MariaDB-common-5.5.63-1.el6.x86_64
MariaDB-Galera-server-5.5.63-1.el6.x86_64
MariaDB-compat-5.5.63-1.el6.x86_64
MariaDB-client-5.5.63-1.el6.x86_64

The version for the slaves is all 10.3.21, we also have tried 10.3.22 but still have the same issue.

Here are some of the most common settings in the 10.3.21 slave in my.cnf just to give a background that we have either had to add in the upgrade or I think might be important to know:

binlog_format=ROW
default_storage_engine=innodb
innodb_change_buffering=inserts
innodb_adaptive_hash_index=OFF
innodb_strict_mode=OFF
optimizer_switch='extended_keys=off'
in_predicate_conversion_threshold = 0
sql_mode=""

Also just to add I ran mysqlcheck -C on our entire database while it was 5.5 on the slave server to confirm there was no corruption before the upgrade from 5.5 to 10.3. Once I upgraded these servers to 10.3, i then ran mysqlcheck -C again with no replication started yet showed no corruption, but once I start replication (start slave) and let it run from anywhere from a few hours to a week, the errors will then show up randomly just from the replication queries, without any other traffic hitting the server.

Wanted to give one more update, as a test, we launched a test cluster with just one galera node with 5.5 and a slave 5.5, then upgraded the slave first, did not start replication yet, then upgraded the galera node test cluster to 10.3. So now we had replication from 10.3 - > 10.3 , and once we started replication and let it catch up, we started to see the same errors and certain tables then became corrupt, we wanted to rule out that maybe it was a version issue from replicating from a very old version.

We changed the following settings and this seems to have fixed corruption issue, but we now we are taking performance hit in updates/inserts, the performance is worst then in 5.5:

innodb_change_buffering=none
innodb_adaptive_hash_index=OFF

As soon as we change these values back to default the performance is back to normal. Any ideas? is this normal?

In our internal stress testing with Random Query Generator (RQG), we rarely encounter something in the MDEV-9663 family of bugs (indexes of a table being inconsistent with each other). Over time, and with some luck, we may come up with another simplified test case, such as the one that helped us fix the MySQL 5.5 bug MDEV-18272 that can introduce such corruption. Recently, we have started employing https://rr-project.org/ which allows us to deterministically replay failures. That is our main hope for eventually catching such bugs.

The challenge is that we need the full history of the database server, from the creation of the table. If we lack that, we can only guess. Both the adaptive hash index and the change buffer are very nondeterministic in nature, which is adding some challenges. For the adaptive hash index, there are some open bugs which we have not tackled yet (noted in MDEV-20487). In 10.5, the change buffer is a little more deterministic thanks to MDEV-19514.

I am wondering whether newer versions are just better at catching corruption. If the table was connected to a FOREIGN KEY constraint, MDEV-18272 could have explained the corruption. But I think that it is a rather unlikely explanation.

The message that preceded the “tried to purge non-delete-marked record” (“entry not marked for deletion”) was introduced in MySQL 5.6.21 and MariaDB 10.0.15.

In MySQL 5.5 and MariaDB 5.5, there still was only one purge thread. Starting with 5.6 and 10.0, multiple innodb_purge_threads could be set. The default was changed from 1 to 4 in MySQL 5.7.8 and MariaDB 10.2.2.

Theoretically, it might be the case that multiple purge threads are causing this corruption. I have not carefully reviewed the code. But, we also have been unable to reproduce this internally. Maybe we should try more with non-debug executables?

Some progress on diagnosing this form of corruption has been made in MDEV-22373.