Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
10.2(EOL), 10.3(EOL), 10.4(EOL)
Description
10.4 301bd62b253 |
mariabackup: Generating a list of tablespaces
|
2019-01-10 3:51:49 0 [Warning] InnoDB: Allocated tablespace ID 3 for mysql/transaction_registry, old maximum was 0
|
2019-01-10 3:51:49 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
|
2019-01-10 3:51:49 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
|
2019-01-10 3:51:49 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
|
2019-01-10 3:51:49 0 [ERROR] InnoDB: File ./test/#sql-6041_11.ibd: 'open' returned OS error 71.
|
2019-01-10 03:51:49 0x7fa4fbbdb740 InnoDB: Assertion failure in file /home/travis/src/storage/innobase/fil/fil0fil.cc line 657
|
InnoDB: Failing assertion: success
|
InnoDB: We intentionally generate a memory trap.
|
|
/home/travis/server/bin/mariabackup(+0xfc703a)[0x55e5bbb9003a]
|
/home/travis/server/bin/mariabackup(+0xd38cdf)[0x55e5bb901cdf]
|
ut/ut0mem.cc:42(ut_strlcpy(char*, char const*, unsigned long))[0x55e5bb905105]
|
fil/fil0fil.cc:658(fil_node_open_file(fil_node_t*))[0x55e5bb1e3d2e]
|
fil/fil0fil.cc:1647(fil_space_t::open())[0x55e5bb1e4126]
|
mariabackup/xtrabackup.cc:3136(xb_load_single_table_tablespace(char const*, char const*, bool))[0x55e5bb1e46c1]
|
mariabackup/xtrabackup.cc:3251(enumerate_ibd_files(void (*)(char const*, char const*, bool)))[0x55e5bb1e6509]
|
mariabackup/xtrabackup.cc:3421(xb_load_tablespaces())[0x55e5bb1eda33]
|
mariabackup/xtrabackup.cc:4250(xtrabackup_backup_func())[0x55e5bb1ed283]
|
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fa4f9830830]
|
/home/travis/server/bin/mariabackup(_start+0x29)[0x55e5bb1dcef9]
|
Logs are attached.
Not reproducible right away.
Attachments
Issue Links
- is blocked by
-
MDEV-24044 mariabackup --backup: [ERROR] InnoDB: File <file>: 'opendir' returned OS error 71
- Closed
- is duplicated by
-
MDEV-18550 InnoDB: Failing assertion: success during mariabackup --backup ...
- Open
-
MDEV-24687 Mariabackup: Assertion `cursor->is_system() || srv_operation == SRV_OPERATION_RESTORE_DELTA || xb_close_files' failed
- Open
- relates to
-
MDEV-24626 Remove synchronous write of page0 and flushing file during file creation
- Closed
-
MDEV-26398 mariabackup failed on Windows 10
- Closed
- links to
User complained on [kb questions](https://mariadb.com/kb/en/mariabackup-gives-core-dump/)
It doesn't appear blocked by
MDEV-24044, this is just another case of innodb asserting of filesystem issues during mariabackup. However it would be good to handle it the same way.A simpler case is just a permission denied on ib_logfile0:
$ sudo -u tom gdb --args extra/mariabackup/mariabackup --user=root --target-dir=/tmp/b --socket=/tmp/build-mariadb-server-10.3.sock --backup
GNU gdb (GDB) Fedora 12.1-1.fc36
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from extra/mariabackup/mariabackup...
(gdb) run
Starting program: /home/dan/repos/build-mariadb-server-10.3/extra/mariabackup/mariabackup --user=root --target-dir=/tmp/b --socket=/tmp/build-mariadb-server-10.3.sock --backup
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[00] 2022-08-29 09:43:16 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: /tmp/build-mariadb-server-10.3.sock
[00] 2022-08-29 09:43:16 Using server version 10.3.37-MariaDB
[00] 2022-08-29 09:43:16 Warning: option 'datadir' has different values:
'/var/lib/mysql' in defaults file
'/tmp/build-mariadb-server-10.3-datadir/' in SHOW VARIABLES
/home/dan/repos/build-mariadb-server-10.3/extra/mariabackup/mariabackup based on MariaDB server 10.3.37-MariaDB Linux (x86_64)
[00] 2022-08-29 09:43:16 uses posix_fadvise().
[00] 2022-08-29 09:43:16 cd to /tmp/build-mariadb-server-10.3-datadir/
[00] 2022-08-29 09:43:16 open files limit requested 0, set to 1024
[00] 2022-08-29 09:43:16 mariabackup: using the following InnoDB configuration:
[00] 2022-08-29 09:43:16 innodb_data_home_dir =
[00] 2022-08-29 09:43:16 innodb_data_file_path = ibdata1:12M:autoextend
[00] 2022-08-29 09:43:16 innodb_log_group_home_dir = ./
[00] 2022-08-29 09:43:16 InnoDB: Using Linux native AIO
2022-08-29 9:43:16 0 [Note] InnoDB: Number of pools: 1
2022-08-29 9:43:16 0 [Warning] InnoDB: Unable to open "./ib_logfile0" to check native AIO read support.
2022-08-29 9:43:16 0 [Warning] InnoDB: Linux Native AIO disabled.
2022-08-29 9:43:16 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2022-08-29 9:43:16 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2022-08-29 09:43:16 0x7ffff7c447c0 InnoDB: Assertion failure in file /home/dan/repos/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc line 668
InnoDB: Failing assertion: success
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
Program received signal SIGABRT, Aborted.
0x00007ffff768ec4c in __pthread_kill_implementation () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.8-11.fc36.x86_64 glibc-2.35-15.fc36.x86_64 libaio-0.3.111-13.fc36.x86_64 libcap-2.48-4.fc36.x86_64 libgcc-12.2.1-1.fc36.x86_64 libgcrypt-1.10.1-3.fc36.x86_64 libgpg-error-1.45-1.fc36.x86_64 libstdc++-12.2.1-1.fc36.x86_64 libxcrypt-4.4.28-1.fc36.x86_64 libzstd-1.5.2-2.fc36.x86_64 lz4-libs-1.9.3-4.fc36.x86_64 pcre-8.45-1.fc36.1.x86_64 snappy-1.1.9-4.fc36.x86_64 systemd-libs-250.8-1.fc36.x86_64 xz-libs-5.2.5-9.fc36.x86_64 zlib-1.2.11-32.fc36.x86_64
(gdb) bt
#0 0x00007ffff768ec4c in __pthread_kill_implementation () from /lib64/libc.so.6
#1 0x00007ffff763e9c6 in raise () from /lib64/libc.so.6
#2 0x00007ffff76287f4 in abort () from /lib64/libc.so.6
#3 0x0000000000c4954b in ut_dbg_assertion_failed (expr=0xf3116f "success",
file=0xf161c0 "/home/dan/repos/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc", line=668)
at /home/dan/repos/mariadb-server-10.3/storage/innobase/ut/ut0dbg.cc:60
#4 0x0000000000ade1b4 in fil_node_open_file (node=0x1ed7160) at /home/dan/repos/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:668
#5 0x0000000000add6b4 in fil_node_prepare_for_io (node=0x1ed7160, space=0x1ed6f00)
at /home/dan/repos/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4068
#6 0x0000000000ae0b0d in fil_io (type=..., sync=false, page_id=..., page_size=..., byte_offset=0, len=512, buf=0x1675600 <log_sys+1024>,
message=0x0, ignore_missing_space=<optimized out>) at /home/dan/repos/mariadb-server-10.3/storage/innobase/fil/fil0fil.cc:4334
#7 0x0000000000b8537d in log_header_read (header=0) at /home/dan/repos/mariadb-server-10.3/storage/innobase/log/log0log.cc:1372
#8 recv_find_max_checkpoint (max_field=<optimized out>) at /home/dan/repos/mariadb-server-10.3/storage/innobase/log/log0recv.cc:1248
#9 0x00000000005a0708 in xtrabackup_backup_func () at /home/dan/repos/mariadb-server-10.3/extra/mariabackup/xtrabackup.cc:4409
#10 0x000000000059fe58 in main_low (argv=0x1e9f6f8) at /home/dan/repos/mariadb-server-10.3/extra/mariabackup/xtrabackup.cc:6745
#11 main (argc=<optimized out>, argv=0x7fffffffe468) at /home/dan/repos/mariadb-server-10.3/extra/mariabackup/xtrabackup.cc:6548
To avoid assertions on just filesystem permissions and other errors out of control of mariabackup a form of:
diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/fil0fil.cc
index a196303c39f..9ea9fa025f1 100644
--- a/storage/innobase/fil/fil0fil.cc
+++ b/storage/innobase/fil/fil0fil.cc
@@ -665,8 +665,14 @@ static bool fil_node_open_file(fil_node_t* node)
.physical())));
}
+#ifdef MYSQL_SERVER
ut_a(success);
ut_a(node->is_open());
+#else
+ if (!success || !node->is_open()) {
+ exit(1);
+ }
+#endif
fil_system.n_open++;