Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24449

Corruption of system tablespace or last recovered page

    XMLWordPrintable

    Details

      Description

      On Buildbot, we have a few cases of assertion failures in mariabackup --prepare, because buf_pool.n_pend_reads is not zero. Matthias Leich was able to reproduce the error under rr record, and I analyzed a case which is reported in MDEV-24448.

      In the analyzed trace, we are recovering 4 page reads. The read callback for 3 of the pages was finished (with no log to be applied, because the pages were up to date), but for the last page, the read callback was still in progress. Yet, srv_start() would concurrently initiate shutdown. We failed to wait for all asynchronous reads to complete. This omission could cause the failure to apply log to some data pages.

      While this error started to occur on Buildbot after we replaced the InnoDB mutexes with native ones (MDEV-21452), I think that it is possible in 10.5, and probably in older versions as well, even though the code was heavily refactored in 10.5, starting with MDEV-16264. The earliest known failure for the 10.6 main branch was as follows:

      10.6 cf2480dd77a45cf56389015a1eabf567

      mariabackup.xb_fulltext_encrypted        w2 [ fail ]
              Test ended at 2020-12-16 04:19:07
       
      CURRENT_TEST: mariabackup.xb_fulltext_encrypted
      mysqltest: At line 19: exec of '/mnt/buildbot/build/mariadb-10.6.0/extra/mariabackup/mariabackup --innobackupex --apply-log /mnt/buildbot/build/mariadb-10.6.0/mysql-test/var/2/tmp/backup 2>&1' failed, error: 34304, status: 134, errno: 11
      2020-12-16  4:19:05 0 [Note] InnoDB: Starting final batch to recover 7 pages from redo log.
      mariabackup: /home/buildbot/buildbot/build/mariadb-10.6.0/storage/innobase/srv/srv0start.cc:1582: dberr_t srv_start(bool): Assertion `!buf_pool.any_io_pending()' failed.
      

      I think that there are two causes to this bug in 10.5 and 10.6:

      • io_callback() was invoking read_slots->release() before invoking the callback function (for write callbacks the existing order is OK)
      • recv_sys_t::apply() is missing a call to read_slots->wait() before flushing the buffer pool. The recv_sys.apply(true) was called by srv_start(), so this should fix the observed race condition (MDEV-9663).

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marko Marko Mäkelä
              Reporter:
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration