[MDEV-17600] Mariabackup double compresses if qpress files are present Created: 2018-11-02 Updated: 2019-04-23 |
|
| Status: | Open |
| Project: | MariaDB Server |
| Component/s: | Backup |
| Affects Version/s: | 10.2.18 |
| Fix Version/s: | 10.2 |
| Type: | Bug | Priority: | Minor |
| Reporter: | Art van Scheppingen | Assignee: | Vladislav Lesin |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Environment: |
ubuntu 16.04 |
||
| Description |
|
Due to some reason uncompressing a previously made compressed backup the qpress files (.qp extension) were still left in the data directory. For the working of MariaDB this doesn't immediately do harm as these files aren't used by the database itself. So the data directory contains files like this: If mariabackup creates a backup now, it will simply include all MariaDB specific files with the .qp extension and include these in the backup stream. Once this backup is extracted from the xbstream the (temporary) directory looks like this: {{ Once mariabackup would get into extracting these files again, it might actually uncompress these files in the wrong order: for example first uncompress the aria_log.00000001.qp.qp and then the aria_log.00000001.qp, resulting in an older version of the aria_log file being extracted. Regardless of the the qpress files still left in the data directory, I personally would have expected mariabackup to ignore files with the .qp extension. Xtrabackup, upon which mariabackup has been based upon, does ignore the .qp extension and only backup files that actually are used. I think this issue is due to the addition of MariaDB specific files to xtrabackup to make it MariaDB compatible. My bug report is not about how these files got left there in the first place and why they had not been removed, but rather about the re-compression and inclusion of compressed files that actually should have been ignored in the first place. To reproduce the issue, create a backup with compression enabled:
Then extract the backup:
|
| Comments |
| Comment by Vladislav Vaintroub [ 2018-11-02 ] |
|
Out of curiosity, why do you use compress, at all? We do not encourage to use --compress option anymore. It is documented deprecated. And, there is a documentation on how to use 3rd party compression and or encryption tools, when backup up to stream, which is what you do https://mariadb.com/kb/en/library/using-encryption-and-compression-tools-with-mariabackup/ |
| Comment by Art van Scheppingen [ 2018-11-02 ] |
|
Hi Vladislav, Thanks for looking into this. To be honest, I didn't look up the options on the MariaDB website but in the binary it's not marked as deprecated. Can you elaborate a bit more why MariaDB decided to deprecate the compress option, while Percona did not deprecate it yet? |
| Comment by Vladislav Vaintroub [ 2018-11-02 ] |
|
I can elaborate on that. The individual file compression does not fit well into the architecture. There are only 2 possible backup types I can imagine
There is no need to reinvent a wheel and provide builtin support for functionality that is freely available and is actively developed by someone else. There are 3rd party multithreaded compressors, there are multithreaded encryptors, so that backup can actually concentrate on its core competencies, which is copying files |