Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16300

remove checkpoint generated with rocksdb_create_checkpoint

Details

    Description

      mariabackup will use rocksdb checkpoints, usually only for a short period (for create checkpoint, copy it to stdout in xbstream format, cleanup checkpoint).

      The "cleanup checkpoint" step - removing directory that was created previously by the server plugin, should also be done by the server plugin. the problem are OS permissions.

      • Usually, mysqld can only read and write within datadir
      • Ideally no other user should be able write within datadir.

      So far , backup OS user only needed read permissions to all files created by server. to cleanup checkpoint itself, backup and server would need a directory, where both of them can write. This is awkward to document and to setup.

      Ideally, server would provide mechanism to cleanup the temporary checkpoint.

      For example,
      ยด
      set global rocksdb_create_checkpoint=NULL
      would remove the last checkpoint dir made with SET rocksdb_create_checkpoint=<dir>

      Attachments

        Issue Links

          Activity

            Did some investigation around

            rocksdb_create_checkpoint variable

            It's a global variable.

            set global rocksdb_create_checkpoint='/path/to/checkpoint'
            

            will cause a checkpoint to be created at the specified path. The value of the variable will not change.

            Checkpoints inside RocksDB

            RocksDB provides a function to create checkpoints but then it doesn't remember that they were created.
            There is no function to remove the created checkpoint.

            psergei Sergei Petrunia added a comment - Did some investigation around rocksdb_create_checkpoint variable It's a global variable. set global rocksdb_create_checkpoint='/path/to/checkpoint' will cause a checkpoint to be created at the specified path. The value of the variable will not change. Checkpoints inside RocksDB RocksDB provides a function to create checkpoints but then it doesn't remember that they were created. There is no function to remove the created checkpoint.

            Issues with the suggested approach:

            Server crashes/shutdowns.

            Suppose the server is shut down after the snapshot creation has started but before mariabackup issues a command to release it. The new server will not remember that it created the snapshot and there will be no way to delete it.

            Multiple concurrent backups

            What happens in this scenario:

            • Client1 creates a snapshot S1 and starts backing it up
            • client2 creates a snapshot S2 ...
            • Client1 finishes work and instructs the server to delete the snapshot.

            Here, the server needs to know that S1 must be deleted while S2 must still be kept.

            psergei Sergei Petrunia added a comment - Issues with the suggested approach: Server crashes/shutdowns. Suppose the server is shut down after the snapshot creation has started but before mariabackup issues a command to release it. The new server will not remember that it created the snapshot and there will be no way to delete it. Multiple concurrent backups What happens in this scenario: Client1 creates a snapshot S1 and starts backing it up client2 creates a snapshot S2 ... Client1 finishes work and instructs the server to delete the snapshot. Here, the server needs to know that S1 must be deleted while S2 must still be kept.
            wlad Vladislav Vaintroub added a comment - - edited

            we can prevent multiple concurrent backups, with user level locks. So far, mariabackups uses hardcoded name for the temp-checkpoint directory, so second attempt to create checkpoint will fail anyway. Server (or backup) crashes /kills are not handled very well in this scenario, would require manual cleanup.

            wlad Vladislav Vaintroub added a comment - - edited we can prevent multiple concurrent backups, with user level locks. So far, mariabackups uses hardcoded name for the temp-checkpoint directory, so second attempt to create checkpoint will fail anyway. Server (or backup) crashes /kills are not handled very well in this scenario, would require manual cleanup.

            One possible way out:

            • Let the rocksdb_create_checkpoint remember paths to the checkpoints it has created (possibly even permanently, store them inside MyRocks).
            • Provide a command to remove the directories that were created with set rocksdb_create_checkpoint
            psergei Sergei Petrunia added a comment - One possible way out: Let the rocksdb_create_checkpoint remember paths to the checkpoints it has created (possibly even permanently, store them inside MyRocks). Provide a command to remove the directories that were created with set rocksdb_create_checkpoint

            Takeaways from Slack discussion:

            • Mariabackup always creates the snapshot at `$datadir/#rocksdb/mariabackup`.This is to make sure the volume is the same and hardlinking is possible.
            • There can be only one backup running at the same time.
            • If the server is restarted, the snapshot can be deleted

            We can meet mariabackup's requirements with this:

            • At server start, $datadir/#rocksdb/mariabackup is removed. It is a directory that only has regular files (i.e. hard links). If we find a soft link there, it's an error.
            • There is a global variable rocksdb_delete_mariabackup_checkpoint. Setting it to any value will cause $datadir/#rocksdb/mariabackup to be deleted.

            Setting a global variable requires a "super" privelege; one can only use this to delete a specific directory - this will not open any vulnerabilities.

            psergei Sergei Petrunia added a comment - Takeaways from Slack discussion: Mariabackup always creates the snapshot at `$datadir/#rocksdb/mariabackup`.This is to make sure the volume is the same and hardlinking is possible. There can be only one backup running at the same time. If the server is restarted, the snapshot can be deleted We can meet mariabackup's requirements with this: At server start, $datadir/#rocksdb/mariabackup is removed. It is a directory that only has regular files (i.e. hard links). If we find a soft link there, it's an error. There is a global variable rocksdb_delete_mariabackup_checkpoint . Setting it to any value will cause $datadir/#rocksdb/mariabackup to be deleted. Setting a global variable requires a "super" privelege; one can only use this to delete a specific directory - this will not open any vulnerabilities.

            People

              wlad Vladislav Vaintroub
              wlad Vladislav Vaintroub
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.