Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30009

InnoDB shutdown hangs when the change buffer is corrupted

    XMLWordPrintable

Details

    Description

      It looks like MDEV-20934 was reintroduced in MariaDB 10.5, possibly due to MDEV-19514 or some changes.

      It turns out that as part of fixing MDEV-23399 (MariaDB 10.5.7), I actually disarmed the regression test innodb.ibuf_not_empty by making the slow shutdown step run with innodb_force_recovery=2, which will prevent a change buffer merge. The following effectively reverts that change:

      diff --git a/mysql-test/suite/innodb/t/ibuf_not_empty.test b/mysql-test/suite/innodb/t/ibuf_not_empty.test
      index 207427d9ab8..f5203a337bf 100644
      --- a/mysql-test/suite/innodb/t/ibuf_not_empty.test
      +++ b/mysql-test/suite/innodb/t/ibuf_not_empty.test
      @@ -106,7 +106,7 @@ truncate(FILE, $ps * $pages);
       close(FILE) || die "Unable to close $file";
       EOF
       
      ---let $restart_parameters=
      +--let $restart_parameters=--innodb-force-recovery=0
       --source include/start_mysqld.inc
       SET GLOBAL innodb_fast_shutdown=0;
       --source include/restart_mysqld.inc
      

      With that, we will observe a hang. The following includes the progress reporting of MDEV-29984:

      10.6 ae6ebafd819d48c965d2615fc78f1f950e0fbf40

      ...
      2022-11-14 16:40:19 0 [ERROR] InnoDB: Trying to read 4096 bytes at 69632 outside the bounds of the file: ./test/t1.ibd
      2022-11-14 16:40:19 0 [Note] /dev/shm/10.6m/sql/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
      2022-11-14 16:40:19 0 [Note] InnoDB: FTS optimize thread exiting.
      2022-11-14 16:40:19 0 [Note] InnoDB: to purge 9 transactions
      2022-11-14 16:40:19 0 [Note] InnoDB: Starting shutdown...
      2022-11-14 16:40:19 0 [Note] InnoDB: Dumping buffer pool(s) to /dev/shm/10.6m/mysql-test/var/mysqld.1/data/ib_buffer_pool
      2022-11-14 16:40:19 0 [Note] InnoDB: Buffer pool(s) dump completed at 221114 16:40:19
      2022-11-14 16:40:34 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      2022-11-14 16:40:49 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      2022-11-14 16:41:04 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      2022-11-14 16:41:19 0 [Note] Completing change buffer merge; 409 page reads initiated; 1 change buffer pages remain
      221114 16:41:20 [ERROR] mysqld got signal 6 ;
      

      On 10.5, this modified test will cause a crash when the purge of transaction history is attempting to access a corrupted page, which was intentionally corrupted by the test.

      I believe that the case that made me file MDEV-29905 is actually about this type of a shutdown hang.

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.