Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Incomplete
-
10.3.31
Description
At a seemingly random time, I started seeing these errors on one of three Galera nodes:
2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 3 fields;
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba5b5; asc ; ;;
|
2: len 8; hex 00000000030f2f83; asc / ;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba586; asc ; ;;
|
2: len 8; hex 00000000030f2d76; asc -v;;
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 3 fields;
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba5bd; asc ; ;;
|
2: len 8; hex 00000000030f2fd7; asc / ;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba586; asc ; ;;
|
2: len 8; hex 00000000030f2d76; asc -v;;
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 3 fields;
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba5c5; asc ; ;;
|
2: len 8; hex 00000000030f302b; asc 0+;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba586; asc ; ;;
|
2: len 8; hex 00000000030f2d76; asc -v;;
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 3 fields;
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba5cd; asc ; ;;
|
2: len 8; hex 00000000030f307f; asc 0 ;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 3; compact format; info bits 32
|
0: len 8; hex 0000000000000357; asc W;;
|
1: len 8; hex 00000000003ba586; asc ; ;;
|
2: len 8; hex 00000000030f2d76; asc -v;;
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: page [page id: space=1983, page number=46360] (549 records, index id 10723).
|
2021-10-01 16:06:00 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2021-10-01 16:06:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 2 fields;
|
0: len 8; hex 0000000000000361; asc a;;
|
1: len 8; hex 00000000030f2f97; asc / ;;
|
|
|
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 32
|
0: len 8; hex 0000000000000361; asc a;;
|
1: len 8; hex 00000000030f2d94; asc - ;;
|
2021-10-01 16:06:04 0 [ERROR] InnoDB: page [page id: space=1983, page number=34598] (754 records, index id 4030).
|
2021-10-01 16:06:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
2021-10-01 16:06:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
|
InnoDB: tuple DATA TUPLE: 2 fields;
|
0: len 8; hex 0000000000000361; asc a;;
|
1: len 8; hex 00000000030f2feb; asc / ;;
|
This went on for a while.
I found at least one database with a few corrupted tables:
2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `activity_log_log_name_index` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
|
2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `subject` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
|
2021-10-01 16:30:19 201954 [ERROR] InnoDB: Flagged corruption of `causer` in table `mertis_master_d04ddb0d-adb4-4aba-814b-d22ed65f3c2c`.`activity_log` in CHECK TABLE; Wrong count
|
The node seemed to work fine, aside for operations on these tables. The customer did report some slowness before this happened, so I'm thinking of InnoDB being put under too much pressure. I could not find any such signs though, like high I/O.
I did find a recent report of - seemingly - the same issue in MDEV-22373 (albeit on a different MariaDB version). However, my MariaDB did not crash, hence the new issue.
I let the node rejoin the cluster as new, and the situation seems to be remedied now.
I updated from MariaDB 10.3.28 to 10.3.31 two days ago. "Unable to find a record to delete-mark" has not been logged before the upgrade.
Attachments
Issue Links
- relates to
-
-
- Closed
-