[MDEV-26746] InnoDB: Unable to find a record to delete-mark Created: 2021-10-01  Updated: 2022-01-10  Resolved: 2022-01-10

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.3.31
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: William Edwards Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: crash

Issue Links:
Relates
relates to MDEV-22373 Unable to find a record to delete-mar... Closed

 Description   

At a seemingly random time, I started seeing these errors on one of three Galera nodes:

2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
 
InnoDB: tuple DATA TUPLE: 3 fields;
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba5b5; asc      ;  ;;
 
 2: len 8; hex 00000000030f2f83; asc       / ;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba586; asc      ;  ;;
 
 2: len 8; hex 00000000030f2d76; asc       -v;;
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
 
InnoDB: tuple DATA TUPLE: 3 fields;
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba5bd; asc      ;  ;;
 
 2: len 8; hex 00000000030f2fd7; asc       / ;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba586; asc      ;  ;;
 
 2: len 8; hex 00000000030f2d76; asc       -v;;
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
 
InnoDB: tuple DATA TUPLE: 3 fields;
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba5c5; asc      ;  ;;
 
 2: len 8; hex 00000000030f302b; asc       0+;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba586; asc      ;  ;;
 
 2: len 8; hex 00000000030f2d76; asc       -v;;
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
 
InnoDB: tuple DATA TUPLE: 3 fields;
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba5cd; asc      ;  ;;
 
 2: len 8; hex 00000000030f307f; asc       0 ;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
 
 0: len 8; hex 0000000000000357; asc        W;;
 
 1: len 8; hex 00000000003ba586; asc      ;  ;;
 
 2: len 8; hex 00000000030f2d76; asc       -v;;
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
 
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-10-01 16:06:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
 
InnoDB: tuple DATA TUPLE: 2 fields;
 
 0: len 8; hex 0000000000000361; asc        a;;
 
 1: len 8; hex 00000000030f2f97; asc       / ;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 32
 
 0: len 8; hex 0000000000000361; asc        a;;
 
 1: len 8; hex 00000000030f2d94; asc       - ;;
 
2021-10-01 16:06:04 0 [ERROR] InnoDB: page [page id: space=1983, page number=34598] (754 records, index id 4030).
 
2021-10-01 16:06:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-10-01 16:06:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
 
InnoDB: tuple DATA TUPLE: 2 fields;
 
 0: len 8; hex 0000000000000361; asc        a;;
 
 1: len 8; hex 00000000030f2feb; asc       / ;;

This went on for a while.

I found at least one database with a few corrupted tables:

2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `activity_log_log_name_index` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
 
2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `subject` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
 
2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `causer` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count

The node seemed to work fine, aside for operations on these tables. The customer did report some slowness before this happened, so I'm thinking of InnoDB being put under too much pressure. I could not find any such signs though, like high I/O.

I did find a recent report of - seemingly - the same issue in MDEV-22373 (albeit on a different MariaDB version). However, my MariaDB did not crash, hence the new issue.

I let the node rejoin the cluster as new, and the situation seems to be remedied now.

I updated from MariaDB 10.3.28 to 10.3.31 two days ago. "Unable to find a record to delete-mark" has not been logged before the upgrade.

 Comments   
Comment by Jan Lindström (Inactive) [ 2021-12-09 ]

wedwards When did this happen? Just after SST or in some other situation. Can you please provide us full error logs. Can you reproduce this?

Generated at Thu Feb 08 09:47:38 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.