[MDEV-22657] mariabackup prepare --export never finishes Created: 2020-05-21 Updated: 2023-06-25 Resolved: 2023-06-25 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | mariabackup, Storage Engine - InnoDB |
| Affects Version/s: | 10.4.12 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | Eugene | Assignee: | Marko Mäkelä |
| Resolution: | Incomplete | Votes: | 1 |
| Labels: | None | ||
| Environment: |
Linux 4.19.121-gentoo x86_64 AMD Opteron(tm) 6376 |
||
| Description |
|
We run galera-based cluster with 3 nodes and considerably large (above 2TB) dataset with lots of small (less then 1GB) databases. Cluster is backed up using mariabackup, so the idea it to use same tool to backup individual databases and restore backup.
No issue so far, mariabackup copied files and reported 'Completed OK'. Next step was attempt to prepare database and try restoring it. Accodring to https://mariadb.com/kb/en/partial-backup-and-restore-with-mariabackup/ tried backup preparation.
However, this process never finish! Previous attempt was stopped after 3 days! If backup preparation command is modified with omitting '--export', additonal mysql is not started, so backup preparation finishes quickly, but innodb .cfg files are not created. Attempt to discard and import tablespaces (tried several databases, problem persists) without .cfg files is a lottery that on some tables finishes without any error (just prints a warning about misisng .cfg file) while on others renders into mysqld crash with following error logged:
Manual creation with locking tables and copying files on intensively used database is a bad idea, so backup preparation procedure that is executed by mariabackup --prepare --export has an issue, it is normally expected that is finishes with preparing backup and creating .cfg files as described on documentation page. Additional note. Attempt to run backup preparation on machine that is not configured to run galera and is plain mysql server behaves in same way. |
| Comments |
| Comment by Marko Mäkelä [ 2022-01-20 ] |
|
euglorg, sorry, I had missed this report. If you start up the server on the prepared backup, will the table be accessible and would CHECK TABLE report any errors? If some DDL operation was run on the table while the backup had been created, the failure could be related to that. While Mariabackup already since version 10.2 should not have much trouble with concurrent DDL, DDL operations in the MariaDB Server itself are not crash-safe before version 10.6. |
| Comment by Eugene [ 2022-02-21 ] |
|
>>> Needs Feedback -> Closed After paying no attention on the issue for almost two years, you close it as you didn't get a feedback within a month. Brilliant! Now, some facts about the case. So, in fact mariabackup can't be used for restoring single database from the dataset, the procedure is extremely unreliable. This is practical side of this task resolution. For those who will find this task later: please consider the fact of necessity for restoring from mysqldump files (with long index creation that can take hours) or necessity of whole dataset replacement in case you would like to rely on mariabackup. For whole dataset mariabackup works perfectly, but for per-database backups this tool can't be used due to the reason mentioned in this task caption. |
| Comment by Marko Mäkelä [ 2022-02-21 ] |
|
euglorg, thank you for the feedback. Did you try to specify --use-memory? With a small buffer pool, applying the log will take a long time, and it might actually never finish due to some bugs, such as If you are already assigning a reasonable amount of buffer pool to have the logs applied and it still is slow, could you provide some output of sudo perf top -p $(pgrep mariabackup) to show where the time is being spent? Note: I expect |
| Comment by Eugene [ 2022-02-21 ] |
|
Hello Marko, We didn't use "--use-memory" option due to |
| Comment by Eugene [ 2022-03-29 ] |
|
elenst can I ask you to keep this task open (mean, not close it) for some more time? Due to the war that started recently by russia against Ukraine, all the plans are ruined. |
| Comment by Marko Mäkelä [ 2022-08-01 ] |
|
euglorg, please check the advice that I posted in |
| Comment by Sergei Golubchik [ 2023-06-25 ] |
|
euglorg, I'm going to close it again. Please, just add a comment whenever you want us to reopen it and we'll do that. |