Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24449

Corruption of system tablespace or last recovered page

    XMLWordPrintable

Details

    Description

      On Buildbot, we have a few cases of assertion failures in mariabackup --prepare, because buf_pool.n_pend_reads is not zero. mleich was able to reproduce the error under rr record, and I analyzed a case which is reported in MDEV-24448.

      In the analyzed trace, we are recovering 4 page reads. The read callback for 3 of the pages was finished (with no log to be applied, because the pages were up to date), but for the last page, the read callback was still in progress. Yet, srv_start() would concurrently initiate shutdown. We failed to wait for all asynchronous reads to complete. This omission could cause the failure to apply log to some data pages.

      While this error started to occur on Buildbot after we replaced the InnoDB mutexes with native ones (MDEV-21452), I think that it is possible in 10.5, and probably in older versions as well, even though the code was heavily refactored in 10.5, starting with MDEV-16264. The earliest known failure for the 10.6 main branch was as follows:

      10.6 cf2480dd77a45cf56389015a1eabf567

      mariabackup.xb_fulltext_encrypted        w2 [ fail ]
              Test ended at 2020-12-16 04:19:07
       
      CURRENT_TEST: mariabackup.xb_fulltext_encrypted
      mysqltest: At line 19: exec of '/mnt/buildbot/build/mariadb-10.6.0/extra/mariabackup/mariabackup --innobackupex --apply-log /mnt/buildbot/build/mariadb-10.6.0/mysql-test/var/2/tmp/backup 2>&1' failed, error: 34304, status: 134, errno: 11
      2020-12-16  4:19:05 0 [Note] InnoDB: Starting final batch to recover 7 pages from redo log.
      mariabackup: /home/buildbot/buildbot/build/mariadb-10.6.0/storage/innobase/srv/srv0start.cc:1582: dberr_t srv_start(bool): Assertion `!buf_pool.any_io_pending()' failed.
      

      I think that there are two causes to this bug in 10.5 and 10.6:

      • io_callback() was invoking read_slots->release() before invoking the callback function (for write callbacks the existing order is OK)
      • recv_sys_t::apply() is missing a call to read_slots->wait() before flushing the buffer pool. The recv_sys.apply(true) was called by srv_start(), so this should fix the observed race condition (MDEV-9663).

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.