Hi,

I have been working on upgrading all of our MariaDB 5.5.63 servers since EOL is approaching.

So far I have upgraded 10 servers, and still have several more servers to go including our galera cluster. So far 3 of the servers after they were upgraded started with random corrupt secondary index on certain table not all just a few. I don't know or understand why it happening on some of the 10.3.21 servers that were upgrade but not all of them. I followed the same directions that is on the MariaDB site for each major version and made sure to run SET GLOBAL innodb_fast_shutdown=0 as per the documentation before each upgrade from 5.5. - > 10.0, 10.0 - > 10.1, 10.1 -> 10.2, 10.2 -> 10.3.21

After each upgrade I run mysql_upgrade and everything is clean. I started to see this error in the logs very soon after the upgrade. I know the fix is to recreate the indexes, but some of our tables are extremely large (for ex. 405 million rows ) and takes a very long time to create a new index.

Here is a the error that I see many times in the logs:

2020-02-20 14:28:09 3 [ERROR] InnoDB: tried to purge non-delete-marked record in index `index_name` of table `schemaname`.`tablename`: tuple: TUPLE (info_bits=0, 4 fields):

{NULL,[4] YCn(0xAD59436E),[4] (0x7FFFFFFF),[4] @(0x800CD040)}, record: COMPACT RECORD(info_bits=0, 4 fields): {NULL,[4] YCn(0xAD59436E),[4] (0x7FFFFFFF),[4] @(0x800CD040)}

2020-02-21 12:01:41 2 [ERROR] InnoDB: Unable to find a record to delete-mark
InnoDB: tuple DATA TUPLE: 2 fields;
0: len 4; hex 8108d277; asc w;;
1: len 4; hex 85cd439c; asc C ;;

InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
0: len 4; hex 8108d273; asc s;;
1: len 4; hex 85c057ef; asc W ;;
2020-02-21 12:01:41 2 [ERROR] InnoDB: page [page id: space=242, page number=96050] (941 records, index id 594).
2020-02-21 12:01:41 2 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

Is there anything I can do to prevent this from happening? Is this a bug? I am not concerned with small tables, but to get corrupted secondary indexes on large tables would cause a major issue in production, once we upgrade our Galera Cluster from MariaDB 5.5 to 10.3

Thanks