[MDEV-25221] mariabackup deletes sourcefile on "no space left on device" Created: 2021-03-22  Updated: 2021-04-06  Resolved: 2021-03-31

Status: Closed
Project: MariaDB Server
Component/s: Backup, mariabackup
Affects Version/s: 10.2, 10.3, 10.4, 10.5
Fix Version/s: 10.2.38, 10.3.29, 10.4.19, 10.5.10

Type: Bug Priority: Major
Reporter: Oliver Friedrich Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None
Environment:

Linux, x86_64


 Description   

On executing of

mariabackup --move-back --target-dir=.

mariabackup deletes the current sourcefile after an "No space left on device" error before exiting. Only the incomplete target file is still available.

[01] 2021-03-22 16:33:28 Moving ./interactive@002dcreator_log/campaign__event#P#future.ibd to /var/lib/mysql/interactive@002dcreator_log/campaign__event#P#future.ibd
 
[01] 2021-03-22 16:33:28 Copying ./interactive@002dcreator_log/campaign__event#P#future.ibd to /var/lib/mysql/interactive@002dcreator_log/campaign__event#P#future.ibd
 
mariabackup: Error writing file '/var/lib/mysql/interactive@002dcreator_log/campaign__event#P#future.ibd' (errno: 28 "No space left on device")
 
[01] 2021-03-22 16:43:08 Error: copy_file() failed.
 
[01] 2021-03-22 16:43:08 Removing ./interactive@002dcreator_log/campaign__event#P#future.ibd

After this the prepared backup is useless as at least one file is missing/incomplete.

mariabackup should delete the incomplete target file instead and allow a re-run of the process to complete the restore.

Also mariabackup should try to use a native filesystem move if on same filesystem instead of a copy/delete cycle to reduce required disk space during the restore step and the option --move-back.



 Comments   
Comment by Vladislav Vaintroub [ 2021-03-31 ]

o.friedrich, I'm wondering what made you think that move-back is not optimized to move within the same filesystem.
According to my reading of source code it is. If you think it was not, can you provide strace output of mariabackup, apparently interesting is the rename() system call.

Comment by Oliver Friedrich [ 2021-04-06 ]

Hi @Vladislav Vaintroub, it was just a guess as mariabackup always writes a cycle of

  • Moving FileA to FileATarget
  • Copying FileA to FileATarget
  • Removing FileA

into the output. Also it requires several minutes in the copied example to copy the file. The prepared source file has approx 400GB and there were 120GB free on the device. So i have expected a real copy of the file as the source for the error (as written to the log). A rename() shouldn't require any additional space and also should happen in near to no time as just some directory informations are rewritten not the data itself. I don't know if it is relevant that i use BTRFS.

In the meantime i have replaced the disk in my development system so i have enough space to hold the prepared and the restored database for testing purposes.

Generated at Thu Feb 08 09:36:03 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.