Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21790

secondary index corruption after 10.3.21 upgrade from 5.5

    XMLWordPrintable

    Details

      Description

      Hi,

      I have been working on upgrading all of our MariaDB 5.5.63 servers since EOL is approaching.

      So far I have upgraded 10 servers, and still have several more servers to go including our galera cluster. So far 3 of the servers after they were upgraded started with random corrupt secondary index on certain table not all just a few. I don't know or understand why it happening on some of the 10.3.21 servers that were upgrade but not all of them. I followed the same directions that is on the MariaDB site for each major version and made sure to run SET GLOBAL innodb_fast_shutdown=0 as per the documentation before each upgrade from 5.5. - > 10.0, 10.0 - > 10.1, 10.1 -> 10.2, 10.2 -> 10.3.21

      After each upgrade I run mysql_upgrade and everything is clean. I started to see this error in the logs very soon after the upgrade. I know the fix is to recreate the indexes, but some of our tables are extremely large (for ex. 405 million rows ) and takes a very long time to create a new index.

      Here is a the error that I see many times in the logs:

      2020-02-20 14:28:09 3 [ERROR] InnoDB: tried to purge non-delete-marked record in index `index_name` of table `schemaname`.`tablename`: tuple: TUPLE (info_bits=0, 4 fields):

      {NULL,[4] YCn(0xAD59436E),[4] (0x7FFFFFFF),[4] @(0x800CD040)}, record: COMPACT RECORD(info_bits=0, 4 fields): {NULL,[4] YCn(0xAD59436E),[4] (0x7FFFFFFF),[4] @(0x800CD040)}

      2020-02-21 12:01:41 2 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 2 fields;
      0: len 4; hex 8108d277; asc w;;
      1: len 4; hex 85cd439c; asc C ;;

      InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
      0: len 4; hex 8108d273; asc s;;
      1: len 4; hex 85c057ef; asc W ;;
      2020-02-21 12:01:41 2 [ERROR] InnoDB: page [page id: space=242, page number=96050] (941 records, index id 594).
      2020-02-21 12:01:41 2 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

      Is there anything I can do to prevent this from happening? Is this a bug? I am not concerned with small tables, but to get corrupted secondary indexes on large tables would cause a major issue in production, once we upgrade our Galera Cluster from MariaDB 5.5 to 10.3

      Thanks

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marko Marko Mäkelä
              Reporter:
              kmanero Kevin Manero
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated: