Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.4.12
-
None
-
Linux 4.19.121-gentoo x86_64 AMD Opteron(tm) 6376
Description
We run galera-based cluster with 3 nodes and considerably large (above 2TB) dataset with lots of small (less then 1GB) databases. Cluster is backed up using mariabackup, so the idea it to use same tool to backup individual databases and restore backup.
Tried to create a backup with
# mariabackup --backup --target-dir=/backup/test/ --innodb-doublewrite --databases='my_database_1' |
No issue so far, mariabackup copied files and reported 'Completed OK'.
Next step was attempt to prepare database and try restoring it. Accodring to https://mariadb.com/kb/en/partial-backup-and-restore-with-mariabackup/ tried backup preparation.
# mariabackup --prepare --target-dir=/backup/test/ --export
|
mariabackup based on MariaDB server 10.4.12-MariaDB Linux (x86_64) |
[00] 2020-05-21 13:36:30 mariabackup: auto-enabling --innodb-file-per-table due to the --export option |
[00] 2020-05-21 13:36:30 cd to /backup/test/ |
[00] 2020-05-21 13:36:30 This target seems to be not prepared yet. |
[00] 2020-05-21 13:36:30 mariabackup: using the following InnoDB configuration for recovery: |
[00] 2020-05-21 13:36:30 innodb_data_home_dir = . |
[00] 2020-05-21 13:36:30 innodb_data_file_path = ibdata1:12M:autoextend |
[00] 2020-05-21 13:36:30 innodb_log_group_home_dir = . |
[00] 2020-05-21 13:36:30 InnoDB: Using Linux native AIO |
[00] 2020-05-21 13:36:30 Starting InnoDB instance for recovery. |
[00] 2020-05-21 13:36:30 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter) |
2020-05-21 13:36:30 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins |
2020-05-21 13:36:30 0 [Note] InnoDB: Uses event mutexes |
2020-05-21 13:36:30 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 |
2020-05-21 13:36:30 0 [Note] InnoDB: Number of pools: 1 |
2020-05-21 13:36:30 0 [Note] InnoDB: Using SSE2 crc32 instructions |
mariabackup: O_TMPFILE is not supported on /tmp (disabling future attempts)
|
2020-05-21 13:36:30 0 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M |
2020-05-21 13:36:30 0 [Note] InnoDB: Completed initialization of buffer pool |
2020-05-21 13:36:30 0 [Note] InnoDB: page_cleaner coordinator priority: -20 |
2020-05-21 13:36:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=9770703608819 |
2020-05-21 13:36:31 0 [Note] InnoDB: Tablespace 11385 was not found at './yet_another_database1/query_parameter.ibd', but there were no modifications either. |
...
|
2020-05-21 13:36:31 0 [Note] InnoDB: Tablespace 13964 was not found at './yet_another_database2/trigger_log_event.ibd', but there were no modifications either. |
2020-05-21 13:36:31 0 [Note] InnoDB: Tablespace 13974 was not found at './yet_another_database3/webform_log.ibd', but there were no modifications either. |
...
|
2020-05-21 13:36:31 0 [Note] InnoDB: Starting final batch to recover 12050 pages from redo log. |
2020-05-21 13:36:31 0 [Note] InnoDB: Last binlog file './mariadb-bin.003700', position 3466 |
[00] 2020-05-21 13:36:31 Last binlog file ./mariadb-bin.003700, position 3466 |
[00] 2020-05-21 13:36:31 mariabackup: Recovered WSREP position: 0ca58cd7-****-****-****-23d8****4dfb:174447362 |
|
|
[00] 2020-05-21 13:36:33 Prepare export : executing "/usr/bin/mariabackup" --mysqld --defaults-file=./backup-my.cnf --defaults-group-suffix= --datadir=. --innodb --innodb-fast-shutdown=0 --loose-partition --innodb_purge_rseg_truncate_frequency=1 --innodb-buffer-pool-size=104857600 --console --log-error= --skip-log-bin --bootstrap < mariabackup_prepare_for_export.sql |
|
|
2020-05-21 13:36:33 0 [Note] mysqld (mysqld 10.4.12-MariaDB) starting as process 15052 ... |
However, this process never finish! Previous attempt was stopped after 3 days!
If backup preparation command is modified with omitting '--export', additonal mysql is not started, so backup preparation finishes quickly, but innodb .cfg files are not created.
Attempt to discard and import tablespaces (tried several databases, problem persists) without .cfg files is a lottery that on some tables finishes without any error (just prints a warning about misisng .cfg file) while on others renders into mysqld crash with following error logged:
2020-05-21 13:40:48 14 [ERROR] [FATAL] InnoDB: Trying to read page number 1285 in space 88165, space name my_database_1/contact, which is outside the tablespace bounds. Byte offset 0, len 16384 |
200521 13:40:48 [ERROR] mysqld got signal 6 ; |
This could be because you hit a bug.
|
...
|
We think the query pointer is invalid, but we will try to print it anyway. |
Query: ALTER TABLE my_database_1.contact IMPORT TABLESPACE
|
Writing a core file...
|
Manual creation with locking tables and copying files on intensively used database is a bad idea, so backup preparation procedure that is executed by mariabackup --prepare --export has an issue, it is normally expected that is finishes with preparing backup and creating .cfg files as described on documentation page.
Additional note. Attempt to run backup preparation on machine that is not configured to run galera and is plain mysql server behaves in same way.