Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30009

InnoDB shutdown hangs when the change buffer is corrupted

    XMLWordPrintable

Details

    Description

      It looks like MDEV-20934 was reintroduced in MariaDB 10.5, possibly due to MDEV-19514 or some changes.

      It turns out that as part of fixing MDEV-23399 (MariaDB 10.5.7), I actually disarmed the regression test innodb.ibuf_not_empty by making the slow shutdown step run with innodb_force_recovery=2, which will prevent a change buffer merge. The following effectively reverts that change:

      diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test
      		 
      index 207427d9ab8..f5203a337bf 100644
      		 
      --- a/mysql-test/suite/innodb/t/ibuf_not_empty.test
      		 
      +++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test
      		 
      @@ -106,7 +106,7 @@ truncate(FILE, $ps * $pages);
      		 
       close(FILE) || die "Unable to close $file";
      		 
       EOF
      		 
       
      ---let $restart_parameters=
      		 
      +--let $restart_parameters=--innodb-force-recovery=0
      		 
       --source include/start_mysqld.inc
      		 
       SET GLOBAL innodb_fast_shutdown=0;
      		 
       --source include/restart_mysqld.inc

      With that, we will observe a hang. The following includes the progress reporting of MDEV-29984:

      10.6 ae6ebafd819d48c965d2615fc78f1f950e0fbf40

      		 
      ...
      		 
      2022-11-14 16:40:19 0 [ERROR] InnoDB: Trying to read 4096 bytes at 69632 outside the bounds of the file: ./test/t1.ibd
      		 
      2022-11-14 16:40:19 0 [Note] /dev/shm/10.6m/sql/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
      		 
      2022-11-14 16:40:19 0 [Note] InnoDB: FTS optimize thread exiting.
      		 
      2022-11-14 16:40:19 0 [Note] InnoDB: to purge 9 transactions
      		 
      2022-11-14 16:40:19 0 [Note] InnoDB: Starting shutdown...
      		 
      2022-11-14 16:40:19 0 [Note] InnoDB: Dumping buffer pool(s) to /dev/shm/10.6m/mysql-test/var/mysqld.1/data/ib_buffer_pool
      		 
      2022-11-14 16:40:19 0 [Note] InnoDB: Buffer pool(s) dump completed at 221114 16:40:19
      		 
      2022-11-14 16:40:34 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      		 
      2022-11-14 16:40:49 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      		 
      2022-11-14 16:41:04 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      		 
      2022-11-14 16:41:19 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      		 
      221114 16:41:20 [ERROR] mysqld got signal 6 ;

      On 10.5, this modified test will cause a crash when the purge of transaction history is attempting to access a corrupted page, which was intentionally corrupted by the test.

      I believe that the case that made me file MDEV-29905 is actually about this type of a shutdown hang.

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26917 InnoDB: Clustered record for sec rec not found index

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31065 MariaDB crashes with [ERROR] [FATAL] InnoDB: Page old data size 8483 new data size 7800, page old max ins size 7519 new max ins size 8202

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-29694 Remove the InnoDB change buffer

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32132 DROP INDEX followed by CREATE INDEX may corrupt data

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-9663 InnoDB assertion failure: *cursor->index->name == TEMP_INDEX_PREFIX, or !cursor->index->is_committed()

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20934 Infinite loop on innodb_fast_shutdown=0 with inconsistent change buffer

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22340 Server hangs on ibuf_merge_or_delete_for_page during shutdown with innodb_fast_shutdown=0

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22986 [ERROR] InnoDB: Record in index was not found on update in mysqld.log on slave

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-27734 Set innodb_change_buffering=none by default

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-29694 Remove the InnoDB change buffer

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29905 Change buffer operations fail to check for log file overflow

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29984 innodb_fast_shutdown=0 fails to report change buffer merge progress

          • Major - Major loss of function.
          • Closed

          Activity

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.