Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24449

Corruption of system tablespace or last recovered page

    XMLWordPrintable

Details

    Description

      On Buildbot, we have a few cases of assertion failures in mariabackup --prepare, because buf_pool.n_pend_reads is not zero. mleich was able to reproduce the error under rr record, and I analyzed a case which is reported in MDEV-24448.

      In the analyzed trace, we are recovering 4 page reads. The read callback for 3 of the pages was finished (with no log to be applied, because the pages were up to date), but for the last page, the read callback was still in progress. Yet, srv_start() would concurrently initiate shutdown. We failed to wait for all asynchronous reads to complete. This omission could cause the failure to apply log to some data pages.

      While this error started to occur on Buildbot after we replaced the InnoDB mutexes with native ones (MDEV-21452), I think that it is possible in 10.5, and probably in older versions as well, even though the code was heavily refactored in 10.5, starting with MDEV-16264. The earliest known failure for the 10.6 main branch was as follows:

      10.6 cf2480dd77a45cf56389015a1eabf567

      		 
      mariabackup.xb_fulltext_encrypted        w2 [ fail ]
      		 
              Test ended at 2020-12-16 04:19:07
      		 
       
      		 
      CURRENT_TEST: mariabackup.xb_fulltext_encrypted
      		 
      mysqltest: At line 19: exec of '/mnt/buildbot/build/mariadb-10.6.0/extra/mariabackup/mariabackup --innobackupex --apply-log /mnt/buildbot/build/mariadb-10.6.0/mysql-test/var/2/tmp/backup 2>&1' failed, error: 34304, status: 134, errno: 11
      		 
      …
      		 
      2020-12-16  4:19:05 0 [Note] InnoDB: Starting final batch to recover 7 pages from redo log.
      		 
      mariabackup: /home/buildbot/buildbot/build/mariadb-10.6.0/storage/innobase/srv/srv0start.cc:1582: dberr_t srv_start(bool): Assertion `!buf_pool.any_io_pending()' failed.

      I think that there are two causes to this bug in 10.5 and 10.6:

      • io_callback() was invoking read_slots->release() before invoking the callback function (for write callbacks the existing order is OK)
      • recv_sys_t::apply() is missing a call to read_slots->wait() before flushing the buffer pool. The recv_sys.apply(true) was called by srv_start(), so this should fix the observed race condition (MDEV-9663).

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22373 Unable to find a record to delete-mark ends up crashing mysqld process after upgrading from 10.1.43 to 10.4

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25198 mariadb backup fails with signal 6 and signal 11 during prepare

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25344 Two hosts crashed at the same time with: Failing assertion: !cursor->index->is_committed()

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-27734 Set innodb_change_buffering=none by default

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-29694 Remove the InnoDB change buffer

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-9663 InnoDB assertion failure: *cursor->index->name == TEMP_INDEX_PREFIX, or !cursor->index->is_committed()

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-14643 InnoDB: Failing assertion: !cursor->index->is_committed()

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15608 Crash during transaction rollback when using optimistic parallel replication, few threads, non-durable configuration.

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-16264 Implement a common work queue for InnoDB background tasks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20934 Infinite loop on innodb_fast_shutdown=0 with inconsistent change buffer

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21452 Use condition variables and normal mutexes instead of InnoDB os_event and mutex

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21811 MariaDB not starting

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23192 Crash in row_search_mvcc() probably related to secondary index corruption

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24141 Failing assertion

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24448 mariabackup: storage/innobase/srv/srv0start.cc:1582: dberr_t srv_start(bool): Assertion `!buf_pool.any_io_pending()' failed.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24709 Assertion !recv_no_ibuf_operations failed in ibuf_page_low()

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30422 Merge new release of InnoDB 5.7.41 to 10.3

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.