[MDEV-25658] Key tree N is empty, Can't find key for index and possible data loss on transactional Aria table after crash recovery Created: 2021-05-11  Updated: 2021-05-20

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - Aria
Affects Version/s: N/A
Fix Version/s: 10.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Michael Widenius
Resolution: Unresolved Votes: 0
Labels: rr-profile


 Description   

Note: I've set the affected version to N/A and fix version to 10.6 because the failure was encountered on bb-10.6-monty-innodb branch. Most likely however it is not specific to the branch, please adjust the versions as needed.

A test was running DML in one thread and queries from INFORMATION_SCHEMA in another when it was killed. The crash recovery passed without any errors:

bb-10.6-monty-innodb f179fdefd1

2021-05-11 18:57:00 0 [Note] mysqld: Aria engine: starting recovery
recovered pages: 0% 29% 51% 74% 96% 100% (0.0 seconds); tables to flush: 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
 (0.0 seconds); 
2021-05-11 18:57:00 0 [Note] mysqld: Aria engine: recovery done

However, further CHECK on a transactional Aria table returned a number of errors:

MariaDB [test]> check table table150_aria_compressed_int extended;
+-----------------------------------+-------+----------+---------------------------------------------------------+
| Table                             | Op    | Msg_type | Msg_text                                                |
+-----------------------------------+-------+----------+---------------------------------------------------------+
| test.table150_aria_compressed_int | check | error    | Key tree 1 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 2 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 3 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 4 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 5 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 6 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 7 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 8 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 9 is empty                                     |
| test.table150_aria_compressed_int | check | error    | Key tree 10 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 11 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 12 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 13 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 14 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 15 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 16 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 17 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 18 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 19 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 20 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 21 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 22 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 23 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 24 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 25 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 26 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 27 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 28 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 29 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 30 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 31 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 32 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 33 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 34 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 35 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 36 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 37 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 38 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 39 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 40 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 41 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 42 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Key tree 43 is empty                                    |
| test.table150_aria_compressed_int | check | error    | Record at:           65:0  Can't find key for index:  1 |
| test.table150_aria_compressed_int | check | error    | Corrupt                                                 |
+-----------------------------------+-------+----------+---------------------------------------------------------+
45 rows in set (0.015 sec)

REPAIR seemingly works, but after it the table only contains 1 row. It is hard to say if it's correct or not, but it seems suspicious.

MariaDB [test]> repair table table150_aria_compressed_int;
+-----------------------------------+--------+----------+----------+
| Table                             | Op     | Msg_type | Msg_text |
+-----------------------------------+--------+----------+----------+
| test.table150_aria_compressed_int | repair | status   | OK       |
+-----------------------------------+--------+----------+----------+
1 row in set (0.133 sec)
 
MariaDB [test]> check table table150_aria_compressed_int extended;
+-----------------------------------+-------+----------+----------+
| Table                             | Op    | Msg_type | Msg_text |
+-----------------------------------+-------+----------+----------+
| test.table150_aria_compressed_int | check | status   | OK       |
+-----------------------------------+-------+----------+----------+
1 row in set (0.005 sec)
 
MariaDB [test]> select count(*) from test.table150_aria_compressed_int;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.001 sec)

Logs, data directories and rr profiles are avialable.



 Comments   
Comment by Michael Widenius [ 2021-05-12 ]

The reason for this issue is that aria recovery ignores redo's for pages in the checkpoint dirty list.
If I remove that check, the recovery works.

I need to do some more investigation for how the dirty list is supposed to work (don't remember this from 14 years ago).
This has to wait until 10.6 is out: DDL recovery issues has to take precedence.

Here is the temporary patch (just to be used for a reference):

— a/storage/maria/ma_recovery.c
+++ b/storage/maria/ma_recovery.c
@@ -3202,9 +3202,13 @@ static MARIA_HA *get_MARIA_HA_from_REDO_record(const
REDO_INSERT_ROW_BLOBS will consult list by itself, as it covers several
pages.
*/
+ if (index_page_redo_entry)
+ my_debug_put_break_here();
+#ifdef QQ
if (_ma_redo_not_needed_for_page(sid, rec->lsn, page,
index_page_redo_entry))
return NULL;
+#endif
}
/*
So we are going to read the page, and if its LSN is older than the

Generated at Thu Feb 08 09:39:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.