Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14102

mariabackup --remove-original not working, when --decompress

Details

    Description

      When running this to decompress .qp files in-situ (in a successfull streamed full backup):
      --decompress --parallel=8 --remove-original

      --remove-original seems to not work.

      Decompression works correctly, but all .qp files are left in the directory, not removed

      Attachments

        Activity

          anikitin Andrii Nikitin (Inactive) added a comment - wlad it is claimed that mariabackup is based on 2.3.8 and indeed the option is in upstream https://github.com/percona/percona-xtrabackup/blob/percona-xtrabackup-2.3.8/storage/innobase/xtrabackup/src/xtrabackup.cc#L936 , but not in our source https://github.com/MariaDB/server/blob/10.1/extra/mariabackup/xtrabackup.cc#L817 https://github.com/MariaDB/server/blob/10.2/extra/mariabackup/xtrabackup.cc#L817 , thus I mark this as verified

          The option is gone in https://github.com/MariaDB/server/commit/cede2b6f0f78a29addd75ef4895404db69f96ba7 , which was some cleanup by marko . I do not claim it worked before, did not check

          wlad Vladislav Vaintroub added a comment - The option is gone in https://github.com/MariaDB/server/commit/cede2b6f0f78a29addd75ef4895404db69f96ba7 , which was some cleanup by marko . I do not claim it worked before, did not check

          My intention was to remove the option, because it is documented as related to the .xbcrypt files. Perhaps at that time, I wrongly thought Mariabackup would also remove the support for qpress.

          -  {"remove-original", OPT_REMOVE_ORIGINAL, "Remove .qp and .xbcrypt files "
          -   "after decryption and decompression.",
          -   (uchar *) &opt_remove_original,
          -   (uchar *) &opt_remove_original,
          -   0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
          

          I now see that I failed to remove the constant OPT_REMOVE_ORIGINAL and some references to the variable opt_remove_original (which is always FALSE).

          It looks like Mariabackup does support .qp files, but by using an external program, qpress.
          So, the proper fix would seem to reinstate the lines mentioned above, but remove the reference to .xbcrypt or decryption.
          (MariaDB 10.1 supports encrypted data and log files. If you want encryption, you should enable the server options and let Mariabackup deal with the encrypted files directly.)

          marko Marko Mäkelä added a comment - My intention was to remove the option, because it is documented as related to the .xbcrypt files. Perhaps at that time, I wrongly thought Mariabackup would also remove the support for qpress. - {"remove-original", OPT_REMOVE_ORIGINAL, "Remove .qp and .xbcrypt files " - "after decryption and decompression.", - (uchar *) &opt_remove_original, - (uchar *) &opt_remove_original, - 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, I now see that I failed to remove the constant OPT_REMOVE_ORIGINAL and some references to the variable opt_remove_original (which is always FALSE). It looks like Mariabackup does support .qp files, but by using an external program, qpress. So, the proper fix would seem to reinstate the lines mentioned above, but remove the reference to .xbcrypt or decryption. (MariaDB 10.1 supports encrypted data and log files. If you want encryption, you should enable the server options and let Mariabackup deal with the encrypted files directly.)
          technomagos Tech Magos added a comment -

          This problem was identified after moving existing scripts from xtrabackup to mariabackup. It was working with xtrabackup. This is a very useful feature: db folders streamed and compressed on the fly, using xbcrypt,(and creating .qp files) are then decompresed in situ on the replica, whence the .qp files are incrementally removed (as decompressed); this also results in less disk space used on the replica.

          How can removal of a useful feature be a resolution?

          Is there an alternative to compress/decompress in a similarly efficient (and parallel) manner as xbcrypt?

          technomagos Tech Magos added a comment - This problem was identified after moving existing scripts from xtrabackup to mariabackup. It was working with xtrabackup. This is a very useful feature: db folders streamed and compressed on the fly, using xbcrypt,(and creating .qp files) are then decompresed in situ on the replica, whence the .qp files are incrementally removed (as decompressed); this also results in less disk space used on the replica. How can removal of a useful feature be a resolution? Is there an alternative to compress/decompress in a similarly efficient (and parallel) manner as xbcrypt?
          wlad Vladislav Vaintroub added a comment - - edited

          Look, it is marked as "Fixed", which means feature is rei-nstantiated. If you're asking about "Any alternative", I personally I think a better alternative to Percona's builtin compression would be
          backing up to stdout (--stream=xbstream parameter), and pipe through any 3rd party compression that supports parallel compression. 7zip, pigz, something like that

          https://mariadb.com/kb/en/library/using-encryption-and-compression-tools-with-mariabackup/#compressing-and-encrypting-with-7zip gives 7Zip example, but you can adopt this easily to "pigz", "qpress" or whatever other parallel compression you need.

          wlad Vladislav Vaintroub added a comment - - edited Look, it is marked as "Fixed", which means feature is rei-nstantiated. If you're asking about "Any alternative", I personally I think a better alternative to Percona's builtin compression would be backing up to stdout (--stream=xbstream parameter), and pipe through any 3rd party compression that supports parallel compression. 7zip, pigz, something like that https://mariadb.com/kb/en/library/using-encryption-and-compression-tools-with-mariabackup/#compressing-and-encrypting-with-7zip gives 7Zip example, but you can adopt this easily to "pigz", "qpress" or whatever other parallel compression you need.

          And, xbcrypt is not about compression, it is about Percona's homegrown encryption

          wlad Vladislav Vaintroub added a comment - And, xbcrypt is not about compression, it is about Percona's homegrown encryption

          People

            wlad Vladislav Vaintroub
            technomagos Tech Magos
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.