[MDEV-16989] InnoDB hang on crash recovery: Waited for 10 seconds for 256 pending reads Created: 2018-08-15  Updated: 2022-01-14  Resolved: 2022-01-14

Status: Closed
Project: MariaDB Server
Component/s: Backup, Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 5.5, 10.0, 10.1, 10.2, 10.3
Fix Version/s: 10.5.0

Type: Bug Priority: Major
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: hang, not-10.5, recovery

Attachments: Zip Archive localtranprepare18604.zip    
Issue Links:
Relates
relates to MDEV-19514 Defer change buffer merge until pages... Closed

 Description   

wlad made me aware of a PBX-1467 fix for Percona Xtrabackup. I believe that the described hang scenario is possible in InnoDB and XtraDB crash recovery.

Quoting sergei-gl’s commit message:

Here is an example deadlock scenario:

Thread 1 in `recv_apply_hashed_log_recs' is waiting when
`buf_pool->n_pend_reads' become not too high to make a progress. It is
`apply_batch_on=TRUE' and will change it to be `FALSE' once apply batch
completed. Note that `buf_pool->n_pend_reads' is already high.

Now, one of the pending reads completes and `buf_page_io_complete'
invoked. It should decrement `buf_pool->n_pend_reads' and let current
apply batch to make progress.

But before decrementing `buf_pool->n_pend_reads', `buf_page_io_complete'
invoked `ibuf_merge_or_delete_for_page' which in turn triggered one more
`recv_apply_hashed_log_recs'. This new `recv_apply_hashed_log_recs'
cannot make progress because `apply_batch_on' is `TRUE', it is waiting
for thread 1. We are in the deadlock now.

Lets imagine that all IO handlers (`buf_page_io_complete') stuck in the
`recv_apply_hashed_log_recs', here is what we see in this case.

Proposed fix is to decrement `buf_pool->n_pend_reads' before invoking
`ibuf_merge_or_delete_for_page'.

This hang should only be possible if there were buffered changes to secondary index leaf pages, to pages which were read during the redo log processing, possibly by read-ahead.



 Comments   
Comment by YURII KANTONISTOV [ 2020-03-27 ]

Reproducible in stress tests with prepare of incremental backup - see attached log of mariabackup. localtranprepare18604.zip

Comment by Marko Mäkelä [ 2020-10-19 ]

MDEV-19514 in MariaDB Server 10.5 should have fixed this.

Comment by Marko Mäkelä [ 2021-05-11 ]

ykantoni, is this reproducible with MariaDB 10.5.10? The crash recovery was heavily refactored in 10.5 (among other things, in MDEV-12353, MDEV-21351, MDEV-23855.

Comment by Marko Mäkelä [ 2021-05-11 ]

Restoring an incremental backup should never invoke ibuf_merge_or_delete_for_page(), and in 10.5 that is not invoked at all during log-based recovery, thanks to MDEV-19514.

Comment by Marko Mäkelä [ 2022-01-14 ]

MDEV-19514 appeared in MariaDB Server 10.5.0.

Generated at Thu Feb 08 08:33:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.