Aria crash recovery failures (MDEV-19813)

[MDEV-19718] Assertion `rownr == 0 && new_page' failed upon Aria crash recovery Created: 2019-06-08  Updated: 2022-03-14  Resolved: 2022-03-14

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Aria
Affects Version/s: 10.1, 10.2, 10.3, 10.4
Fix Version/s: N/A

Type: Technical task Priority: Major
Reporter: Elena Stepanova Assignee: Elena Stepanova
Resolution: Incomplete Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-600 LP:982872 - Aria recovery crash, or a... Closed
relates to MDEV-18461 Aria crash recovery failures on the s... Stalled

 Description   

10.3 7060b032

2019-06-08 15:12:22 0 [Note] mysqld: Aria engine: starting recovery
recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%mysqld: /data/src/10.3/storage/maria/ma_blockrec.c:6358: _ma_apply_redo_insert_row_head_or_tail: Assertion `rownr == 0 && new_page' failed.
190608 15:12:28 [ERROR] mysqld got signal 6 ;
 
6  0x00007f0862a50e67 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x563229ac76f4 "rownr == 0 && new_page", file=file@entry=0x563229ac5ea0 "/data/src/10.3/storage/maria/ma_blockrec.c", line=line@entry=6358, function=function@entry=0x563229ac7ee0 <__PRETTY_FUNCTION__.16960> "_ma_apply_redo_insert_row_head_or_tail") at assert.c:92
#7  0x00007f0862a50f12 in __GI___assert_fail (assertion=0x563229ac76f4 "rownr == 0 && new_page", file=0x563229ac5ea0 "/data/src/10.3/storage/maria/ma_blockrec.c", line=6358, function=0x563229ac7ee0 <__PRETTY_FUNCTION__.16960> "_ma_apply_redo_insert_row_head_or_tail") at assert.c:101
#8  0x0000563229589835 in _ma_apply_redo_insert_row_head_or_tail (info=0x56322b818a10, lsn=4396194268, page_type=1, new_page=0 '\000', header=0x56322b8d8472 "\001", data=0x56322b8d8478 "", data_length=35) at /data/src/10.3/storage/maria/ma_blockrec.c:6358
#9  0x0000563229550544 in exec_REDO_LOGREC_REDO_INSERT_ROW_HEAD (rec=0x7ffcc5f93d80) at /data/src/10.3/storage/maria/ma_recovery.c:1558
#10 0x000056322954e180 in display_and_apply_record (log_desc=0x56322ac11108 <log_record_type_descriptor+72>, rec=0x7ffcc5f93d80) at /data/src/10.3/storage/maria/ma_recovery.c:595
#11 0x000056322955336e in run_redo_phase (lsn=4367068051, lsn_end=0, apply=MARIA_LOG_APPLY) at /data/src/10.3/storage/maria/ma_recovery.c:2715
#12 0x000056322954d990 in maria_apply_log (from_lsn=4367068051, end_lsn=0, apply=MARIA_LOG_APPLY, trace_file=0x0, should_run_undo_phase=1 '\001', skip_DDLs_arg=1 '\001', take_checkpoints=1 '\001', warnings_count=0x7ffcc5f98300) at /data/src/10.3/storage/maria/ma_recovery.c:350
#13 0x000056322954d5ff in maria_recovery_from_log () at /data/src/10.3/storage/maria/ma_recovery.c:240
#14 0x000056322951fe73 in ha_maria_init (p=0x56322b740e60) at /data/src/10.3/storage/maria/ha_maria.cc:3517
#15 0x0000563228f3d19a in ha_initialize_handlerton (plugin=0x56322b7372d0) at /data/src/10.3/sql/handler.cc:523
#16 0x0000563228c59e83 in plugin_initialize (tmp_root=0x7ffcc5f98720, plugin=0x56322b7372d0, argc=0x56322a37d7b0 <remaining_argc>, argv=0x56322b6cad68, options_only=false) at /data/src/10.3/sql/sql_plugin.cc:1432
#17 0x0000563228c5ab2c in plugin_init (argc=0x56322a37d7b0 <remaining_argc>, argv=0x56322b6cad68, flags=2) at /data/src/10.3/sql/sql_plugin.cc:1714
#18 0x0000563228b22ea6 in init_server_components () at /data/src/10.3/sql/mysqld.cc:5423
#19 0x0000563228b24016 in mysqld_main (argc=11, argv=0x56322b6cad68) at /data/src/10.3/sql/mysqld.cc:6030
#20 0x0000563228b18a00 in main (argc=11, argv=0x7ffcc5f98f58) at /data/src/10.3/sql/main.cc:25

To reproduce:

10.1, 10.2 and 10.4 also crash on the same datadir.



 Comments   
Comment by Michael Widenius [ 2019-06-15 ]

Unfortunately I was not able to repeat the failure.
I have talked with Elena about this and asked her to next time she tests this to compile MariaDB with -DEXTRA_DEBUG (should be default if using BUILD scripts). This will add more information into the aria_recover.trace file that may help me find the problem.

Comment by Elena Stepanova [ 2019-06-20 ]

I've added a test case and aria recovery trace to MDEV-18203. it causes the failure described in MDEV-18203

019-06-20 19:17:52 0 [ERROR] mysqld: Got error 126 when executing undo undo_key_insert

but possibly they share the same root cause.

Comment by Elena Stepanova [ 2022-03-14 ]

I don't suppose an 3-year-old pre-10.5 datadir is of interest to anyone. If we encounter the failure again, we'll try to get a test case and all the data.

Generated at Thu Feb 08 08:53:49 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.