[MDEV-33023] Crash in mariadb-backup --prepare --export after --prepare Created: 2023-12-14  Updated: 2024-02-08  Resolved: 2024-02-08

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.5, 10.6
Fix Version/s: 10.5.25, 10.6.18

Type: Bug Priority: Major
Reporter: Marko Mäkelä Assignee: Debarun Banerjee
Resolution: Fixed Votes: 0
Labels: not-10.8, regression


 Description   

The following change to an existing regression test causes a crash:

diff --git a/mysql-test/suite/mariabackup/partial.test b/mysql-test/suite/mariabackup/partial.test
 
index d0d07daf2ea..035024aac3d 100644
 
--- a/mysql-test/suite/mariabackup/partial.test
 
+++ b/mysql-test/suite/mariabackup/partial.test
 
@@ -41,6 +41,7 @@ EOF
 
 
 echo # xtrabackup prepare;
 
 --disable_result_log
 
+exec $XTRABACKUP --defaults-file=$server_cnf --defaults-group-suffix=.1 --prepare --target-dir=$targetdir;
 
 exec $XTRABACKUP --defaults-file=$server_cnf --defaults-group-suffix=.1 --prepare --export --target-dir=$targetdir;
 
 --enable_result_log

10.5 e472b682e068319e2a27373903cd46fb93093286

 
mariabackup: /mariadb/10.5/storage/innobase/fil/fil0fil.cc:441: bool fil_node_open_file(fil_node_t*): Assertion `fil_is_user_tablespace_id(node->space->id) || srv_operation == SRV_OPERATION_BACKUP || srv_operation == SRV_OPERATION_RESTORE || srv_operation == SRV_OPERATION_RESTORE_DELTA' failed.

10.6 736a54f49c72d89fb82ef4165e96cddb506cf555

 
mariabackup: /mariadb/10.6/storage/innobase/fil/fil0fil.cc:426: bool fil_node_open_file(fil_node_t*): Assertion `fil_is_user_tablespace_id(node->space->id) || srv_operation == SRV_OPERATION_BACKUP || srv_operation == SRV_OPERATION_RESTORE || srv_operation == SRV_OPERATION_RESTORE_DELTA' failed.

The 10.4 branch does not crash.

 Comments   
Comment by Debarun Banerjee [ 2024-01-18 ]

mariadb-backup with --prepare option could result in empty redo log
file. When --prepare is followed by --prepare --export, we exit early
in srv_start function without opening the ibdata1 tablespace. Later
while trying to read rollback segment header page, we hit the debug
assert which claims that the system space should already have been
opened.

The assert is already marked not applicable for server operation backup
and restore. It seems appropriate to add SRV_OPERATION_RESTORE_EXPORT
in the assert condition.

https://github.com/MariaDB/server/pull/3009

Comment by Debarun Banerjee [ 2024-01-22 ]

The previous fix was good for 10.6 but 10.5 has another issue.
There are two assert cases here.

Issue-1: System tablespace object is not there in fil space hash i.e. srv_sys_space.open_or_create() is not called.
Fix: The system tablespace can be opened before checking the redo log. This is to be fixed only for 10.5 and 10.6 already does it.

Issue-2: The system tablespace data file ibdata1 is not opened i.e. fil_system.sys_space->open() is not called.
Fix: The assert is already marked not applicable for server operation backup and restore. It seems appropriate to add SRV_OPERATION_RESTORE_EXPORT in the assert condition.

Updated https://github.com/MariaDB/server/pull/3009

Comment by Debarun Banerjee [ 2024-01-23 ]

commit 7c170595d38938de516bfa0a9b7266a3a0edf001 (HEAD > 10.5MDEV-33023, origin/10.5-MDEV-33023)
Author: mariadb-DebarunBanerjee <debarun.banerjee@mariadb.com>
Date: Tue Jan 23 11:51:39 2024 +0530

MDEV-33023 Crash in mariadb-backup --prepare --export after --prepare

Address review comments from marko. Modified server_start to open
system and undo tablespaces before checking redo log.

Updated https://github.com/MariaDB/server/pull/3009

The only possible impact is if there is any case where mariabackup works without the system tablespace ibdata1

Comment by Marko Mäkelä [ 2024-01-24 ]

The problem is caused by the fact that no ib_logfile0 would exist, and therefore the system and undo tablespaces would not be opened. In MDEV-14425 and MDEV-27199 the logic had been changed in such a way that an ib_logfile0 must always exist. I tested that there is no crash (but a clear error message) in 10.11, when the ib_logfile0 is missing or empty.

Comment by Debarun Banerjee [ 2024-01-25 ]

The patch is already reviewed in github. Please mark the review complete.

Comment by Debarun Banerjee [ 2024-02-08 ]

Pushed to 10.5.

commit fb9da7f7518bc310aff4eac215e046d63acded62
Author: mariadb-DebarunBanerjee <debarun.banerjee@mariadb.com>
Date: Tue Jan 23 20:43:39 2024 +0530

MDEV-33023 Crash in mariadb-backup --prepare --export after --prepare

Generated at Thu Feb 08 10:35:48 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.