[MDEV-16300] remove checkpoint generated with rocksdb_create_checkpoint Created: 2018-05-26 Updated: 2018-06-07 Resolved: 2018-06-07 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Backup, Storage Engine - RocksDB |
| Fix Version/s: | 10.2.16, 10.3.8 |
| Type: | Task | Priority: | Critical |
| Reporter: | Vladislav Vaintroub | Assignee: | Vladislav Vaintroub |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Description |
|
mariabackup will use rocksdb checkpoints, usually only for a short period (for create checkpoint, copy it to stdout in xbstream format, cleanup checkpoint). The "cleanup checkpoint" step - removing directory that was created previously by the server plugin, should also be done by the server plugin. the problem are OS permissions.
So far , backup OS user only needed read permissions to all files created by server. to cleanup checkpoint itself, backup and server would need a directory, where both of them can write. This is awkward to document and to setup. Ideally, server would provide mechanism to cleanup the temporary checkpoint. For example, |
| Comments |
| Comment by Sergei Petrunia [ 2018-05-30 ] | |
|
Did some investigation around rocksdb_create_checkpoint variableIt's a global variable.
will cause a checkpoint to be created at the specified path. The value of the variable will not change. Checkpoints inside RocksDBRocksDB provides a function to create checkpoints but then it doesn't remember that they were created. | |
| Comment by Sergei Petrunia [ 2018-05-30 ] | |
|
Issues with the suggested approach: Server crashes/shutdowns.Suppose the server is shut down after the snapshot creation has started but before mariabackup issues a command to release it. The new server will not remember that it created the snapshot and there will be no way to delete it. Multiple concurrent backupsWhat happens in this scenario:
Here, the server needs to know that S1 must be deleted while S2 must still be kept. | |
| Comment by Vladislav Vaintroub [ 2018-05-30 ] | |
|
we can prevent multiple concurrent backups, with user level locks. So far, mariabackups uses hardcoded name for the temp-checkpoint directory, so second attempt to create checkpoint will fail anyway. Server (or backup) crashes /kills are not handled very well in this scenario, would require manual cleanup. | |
| Comment by Sergei Petrunia [ 2018-05-30 ] | |
|
One possible way out:
| |
| Comment by Sergei Petrunia [ 2018-05-30 ] | |
|
Takeaways from Slack discussion:
We can meet mariabackup's requirements with this:
Setting a global variable requires a "super" privelege; one can only use this to delete a specific directory - this will not open any vulnerabilities. |