[MDEV-14102] mariabackup --remove-original not working, when --decompress Created: 2017-10-22  Updated: 2017-10-25  Resolved: 2017-10-24

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.1.28
Fix Version/s: 10.1.29, 10.2.11, 10.3.3

Type: Bug Priority: Major
Reporter: Tech Magos Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: mariabackup


 Description   

When running this to decompress .qp files in-situ (in a successfull streamed full backup):
--decompress --parallel=8 --remove-original

--remove-original seems to not work.

Decompression works correctly, but all .qp files are left in the directory, not removed



 Comments   
Comment by Andrii Nikitin (Inactive) [ 2017-10-23 ]

wlad it is claimed that mariabackup is based on 2.3.8 and indeed the option is in upstream https://github.com/percona/percona-xtrabackup/blob/percona-xtrabackup-2.3.8/storage/innobase/xtrabackup/src/xtrabackup.cc#L936 , but not in our source https://github.com/MariaDB/server/blob/10.1/extra/mariabackup/xtrabackup.cc#L817 https://github.com/MariaDB/server/blob/10.2/extra/mariabackup/xtrabackup.cc#L817 , thus I mark this as verified

Comment by Vladislav Vaintroub [ 2017-10-23 ]

The option is gone in https://github.com/MariaDB/server/commit/cede2b6f0f78a29addd75ef4895404db69f96ba7 , which was some cleanup by marko . I do not claim it worked before, did not check

Comment by Marko Mäkelä [ 2017-10-23 ]

My intention was to remove the option, because it is documented as related to the .xbcrypt files. Perhaps at that time, I wrongly thought Mariabackup would also remove the support for qpress.

-  {"remove-original", OPT_REMOVE_ORIGINAL, "Remove .qp and .xbcrypt files "
-   "after decryption and decompression.",
-   (uchar *) &opt_remove_original,
-   (uchar *) &opt_remove_original,
-   0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},

I now see that I failed to remove the constant OPT_REMOVE_ORIGINAL and some references to the variable opt_remove_original (which is always FALSE).

It looks like Mariabackup does support .qp files, but by using an external program, qpress.
So, the proper fix would seem to reinstate the lines mentioned above, but remove the reference to .xbcrypt or decryption.
(MariaDB 10.1 supports encrypted data and log files. If you want encryption, you should enable the server options and let Mariabackup deal with the encrypted files directly.)

Comment by Tech Magos [ 2017-10-25 ]

This problem was identified after moving existing scripts from xtrabackup to mariabackup. It was working with xtrabackup. This is a very useful feature: db folders streamed and compressed on the fly, using xbcrypt,(and creating .qp files) are then decompresed in situ on the replica, whence the .qp files are incrementally removed (as decompressed); this also results in less disk space used on the replica.

How can removal of a useful feature be a resolution?

Is there an alternative to compress/decompress in a similarly efficient (and parallel) manner as xbcrypt?

Comment by Vladislav Vaintroub [ 2017-10-25 ]

Look, it is marked as "Fixed", which means feature is rei-nstantiated. If you're asking about "Any alternative", I personally I think a better alternative to Percona's builtin compression would be
backing up to stdout (--stream=xbstream parameter), and pipe through any 3rd party compression that supports parallel compression. 7zip, pigz, something like that

https://mariadb.com/kb/en/library/using-encryption-and-compression-tools-with-mariabackup/#compressing-and-encrypting-with-7zip gives 7Zip example, but you can adopt this easily to "pigz", "qpress" or whatever other parallel compression you need.

Comment by Vladislav Vaintroub [ 2017-10-25 ]

And, xbcrypt is not about compression, it is about Percona's homegrown encryption

Generated at Thu Feb 08 08:10:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.