[MDEV-25483] Shutdown crash during innodb.innodb_buffer_pool_resize_temporary Created: 2021-04-22  Updated: 2021-04-22  Resolved: 2021-04-22

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.0, 10.5, 10.6
Fix Version/s: 10.5.10, 10.6.0

Type: Bug Priority: Major
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: race, regression-10.5

Issue Links:
Problem/Incident
is caused by MDEV-16264 Implement a common work queue for Inn... Closed

 Description   

The test innodb.innodb_buffer_pool_resize_temporary triggered an InnoDB crash during shutdown, leading to a situation where log-based recovery would be needed:

10.5 675c22c065110be03a5fab82442d2c3dc32aefff

CURRENT_TEST: innodb.innodb_buffer_pool_resize_temporary
Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.5.10/mysql-test/std_data/cacert.pem'
Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.5.10/mysql-test/std_data/client-cert.pem'
Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.5.10/mysql-test/std_data/client-key.pem'
Warning: /mnt/buildbot/build/mariadb-10.5.10/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl'
Got ERROR: "InnoDB: Operating system error number 2 in a file operation." errno: 2000
Got ERROR: "InnoDB: The error means the system cannot find the path specified." errno: 2000
Got ERROR: "InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them." errno: 2000
Got ERROR: "InnoDB: Cannot open datafile for read-only: './mysql/innodb_index_stats.ibd' OS error: 71" errno: 2000
Got ERROR: "InnoDB: Operating system error number 2 in a file operation." errno: 2000
Got ERROR: "InnoDB: The error means the system cannot find the path specified." errno: 2000
Got ERROR: "InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them." errno: 2000
Got ERROR: "InnoDB: Cannot open datafile for read-only: './mysql/innodb_table_stats.ibd' OS error: 71" errno: 2000
Got ERROR: "InnoDB: Operating system error number 2 in a file operation." errno: 2000
Got ERROR: "InnoDB: The error means the system cannot find the path specified." errno: 2000
Got ERROR: "InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them." errno: 2000
Got ERROR: "InnoDB: Cannot open datafile for read-only: './mysql/transaction_registry.ibd' OS error: 71" errno: 2000
2021-04-19 19:27:51 0xaf3feb40  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.5.10/storage/innobase/buf/buf0buf.cc line 2325
InnoDB: Failing assertion: srv_shutdown_state == SRV_SHUTDOWN_NONE
InnoDB: We intentionally generate a memory trap.

The fix is simple:

diff --git a/storage/innobase/buf/buf0buf.cc b/storage/innobase/buf/buf0buf.cc
index 2923c6d4790..0079f6582ab 100644
--- a/storage/innobase/buf/buf0buf.cc
+++ b/storage/innobase/buf/buf0buf.cc
@@ -2322,7 +2322,7 @@ inline void buf_pool_t::resize()
 static void buf_resize_callback(void *)
 {
   DBUG_ENTER("buf_resize_callback");
-  ut_a(srv_shutdown_state == SRV_SHUTDOWN_NONE);
+  ut_ad(srv_shutdown_state < SRV_SHUTDOWN_CLEANUP);
   mysql_mutex_lock(&buf_pool.mutex);
   const auto size= srv_buf_pool_size;
   const bool work= srv_buf_pool_old_size != size;

It suffices to have an assertion in debug builds only. This test did fail in a debug build (IA-32 embedded server) after all. The shutdown state may have advanced a little already. The following in logs_empty_and_mark_files_at_shutdown() will make this work:

	buf_resize_shutdown();
	dict_stats_shutdown();
	btr_defragment_shutdown();
 
	srv_shutdown_state = SRV_SHUTDOWN_CLEANUP;

There is another assignment to SRV_SHUTDOWN_CLEANUP in innodb_shutdown() as a special case for mariabackup --prepare that is not preceded by such task shutdown, but in that case, the tasks should never have been started by invoking buf_resize_start() et al:

	case SRV_OPERATION_RESTORE:
	case SRV_OPERATION_RESTORE_EXPORT:
		srv_shutdown_state = SRV_SHUTDOWN_CLEANUP;

As far as I understand, such crash may only occur if

SET GLOBAL innodb_buffer_pool_size=…;

was executed immediately before initiating server shutdown.


Generated at Thu Feb 08 09:38:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.