Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25483

Shutdown crash during innodb.innodb_buffer_pool_resize_temporary

    XMLWordPrintable

    Details

      Description

      The test innodb.innodb_buffer_pool_resize_temporary triggered an InnoDB crash during shutdown, leading to a situation where log-based recovery would be needed:

      10.5 675c22c065110be03a5fab82442d2c3dc32aefff

      CURRENT_TEST: innodb.innodb_buffer_pool_resize_temporary
      Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.5.10/mysql-test/std_data/cacert.pem'
      Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.5.10/mysql-test/std_data/client-cert.pem'
      Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.5.10/mysql-test/std_data/client-key.pem'
      Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl'
      Got ERROR: "InnoDB: Operating system error number 2 in a file operation." errno: 2000
      Got ERROR: "InnoDB: The error means the system cannot find the path specified." errno: 2000
      Got ERROR: "InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them." errno: 2000
      Got ERROR: "InnoDB: Cannot open datafile for read-only: './mysql/innodb_index_stats.ibd' OS error: 71" errno: 2000
      Got ERROR: "InnoDB: Operating system error number 2 in a file operation." errno: 2000
      Got ERROR: "InnoDB: The error means the system cannot find the path specified." errno: 2000
      Got ERROR: "InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them." errno: 2000
      Got ERROR: "InnoDB: Cannot open datafile for read-only: './mysql/innodb_table_stats.ibd' OS error: 71" errno: 2000
      Got ERROR: "InnoDB: Operating system error number 2 in a file operation." errno: 2000
      Got ERROR: "InnoDB: The error means the system cannot find the path specified." errno: 2000
      Got ERROR: "InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them." errno: 2000
      Got ERROR: "InnoDB: Cannot open datafile for read-only: './mysql/transaction_registry.ibd' OS error: 71" errno: 2000
      2021-04-19 19:27:51 0xaf3feb40  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.5.10/storage/innobase/buf/buf0buf.cc line 2325
      InnoDB: Failing assertion: srv_shutdown_state == SRV_SHUTDOWN_NONE
      InnoDB: We intentionally generate a memory trap.
      

      The fix is simple:

      diff --git a/storage/innobase/buf/buf0buf.cc b/storage/innobase/buf/buf0buf.cc
      index 2923c6d4790..0079f6582ab 100644
      --- a/storage/innobase/buf/buf0buf.cc
      +++ b/storage/innobase/buf/buf0buf.cc
      @@ -2322,7 +2322,7 @@ inline void buf_pool_t::resize()
       static void buf_resize_callback(void *)
       {
         DBUG_ENTER("buf_resize_callback");
      -  ut_a(srv_shutdown_state == SRV_SHUTDOWN_NONE);
      +  ut_ad(srv_shutdown_state < SRV_SHUTDOWN_CLEANUP);
         mysql_mutex_lock(&buf_pool.mutex);
         const auto size= srv_buf_pool_size;
         const bool work= srv_buf_pool_old_size != size;
      

      It suffices to have an assertion in debug builds only. This test did fail in a debug build (IA-32 embedded server) after all. The shutdown state may have advanced a little already. The following in logs_empty_and_mark_files_at_shutdown() will make this work:

      	buf_resize_shutdown();
      	dict_stats_shutdown();
      	btr_defragment_shutdown();
       
      	srv_shutdown_state = SRV_SHUTDOWN_CLEANUP;
      

      There is another assignment to SRV_SHUTDOWN_CLEANUP in innodb_shutdown() as a special case for mariabackup --prepare that is not preceded by such task shutdown, but in that case, the tasks should never have been started by invoking buf_resize_start() et al:

      	case SRV_OPERATION_RESTORE:
      	case SRV_OPERATION_RESTORE_EXPORT:
      		srv_shutdown_state = SRV_SHUTDOWN_CLEANUP;
      

      As far as I understand, such crash may only occur if

      SET GLOBAL innodb_buffer_pool_size=…;
      

      was executed immediately before initiating server shutdown.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marko Marko Mäkelä
              Reporter:
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: