Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26746

InnoDB: Unable to find a record to delete-mark

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 10.3.31
    • Fix Version/s: None
    • Component/s: Galera
    • Labels:

      Description

      At a seemingly random time, I started seeing these errors on one of three Galera nodes:

      2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 3 fields;
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba5b5; asc      ;  ;;
       2: len 8; hex 00000000030f2f83; asc       / ;;
       
      InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba586; asc      ;  ;;
       2: len 8; hex 00000000030f2d76; asc       -v;;
      2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
      2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 3 fields;
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba5bd; asc      ;  ;;
       2: len 8; hex 00000000030f2fd7; asc       / ;;
       
      InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba586; asc      ;  ;;
       2: len 8; hex 00000000030f2d76; asc       -v;;
      2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
      2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 3 fields;
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba5c5; asc      ;  ;;
       2: len 8; hex 00000000030f302b; asc       0+;;
       
      InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba586; asc      ;  ;;
       2: len 8; hex 00000000030f2d76; asc       -v;;
      2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
      2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 3 fields;
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba5cd; asc      ;  ;;
       2: len 8; hex 00000000030f307f; asc       0 ;;
       
      InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
       0: len 8; hex 0000000000000357; asc        W;;
       1: len 8; hex 00000000003ba586; asc      ;  ;;
       2: len 8; hex 00000000030f2d76; asc       -v;;
      2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
      2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-10-01 16:06:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 2 fields;
       0: len 8; hex 0000000000000361; asc        a;;
       1: len 8; hex 00000000030f2f97; asc       / ;;
       
      InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 32
       0: len 8; hex 0000000000000361; asc        a;;
       1: len 8; hex 00000000030f2d94; asc       - ;;
      2021-10-01 16:06:04 0 [ERROR] InnoDB: page [page id: space=1983, page number=34598] (754 records, index id 4030).
      2021-10-01 16:06:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-10-01 16:06:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 2 fields;
       0: len 8; hex 0000000000000361; asc        a;;
       1: len 8; hex 00000000030f2feb; asc       / ;;
      

      This went on for a while.

      I found at least one database with a few corrupted tables:

      2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `activity_log_log_name_index` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
      2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `subject` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
      2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `causer` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
      

      The node seemed to work fine, aside for operations on these tables. The customer did report some slowness before this happened, so I'm thinking of InnoDB being put under too much pressure. I could not find any such signs though, like high I/O.

      I did find a recent report of - seemingly - the same issue in MDEV-22373 (albeit on a different MariaDB version). However, my MariaDB did not crash, hence the new issue.

      I let the node rejoin the cluster as new, and the situation seems to be remedied now.

      I updated from MariaDB 10.3.28 to 10.3.31 two days ago. "Unable to find a record to delete-mark" has not been logged before the upgrade.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              wedwards William Edwards
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration