Note to self: myrocks_hotbackup requires python's MySQLDB. (yum install MySQL-python on centos).
Note to self: If backup target directory doesn't exist, one gets a tottally confusing error message:
./scripts/myrocks_hotbackup --user=root --port=3311 --checkpoint_dir=/home/psergey/f2 > a2
|
2018-01-19 14:56:00.959 INFO Starting backup.
|
2018-01-19 14:56:00.996 INFO Set datadir: /home/psergey/data-10.2-rocksdb/
|
2018-01-19 14:56:00.997 INFO Creating checkpoint at /home/psergey/f2/1
|
2018-01-19 14:56:00.997 ERROR (-1, 'error totally whack')
|
2018-01-19 14:56:00.998 ERROR Traceback (most recent call last):
|
File "./scripts/myrocks_hotbackup", line 380, in start_backup_round
|
MySQLUtil.create_checkpoint(dbh, snapshot_dir)
|
File "./scripts/myrocks_hotbackup", line 334, in create_checkpoint
|
cur.execute(sql)
|
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
|
self.errorhandler(self, exc, value)
|
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
|
raise errorclass, errorvalue
|
InterfaceError: (-1, 'error totally whack')
|
The server stderr has:
2018-01-19 14:56:00 139739580237568 [Note] RocksDB: creating checkpoint in directory : /home/psergey/f2/1
|
which is NOT followed by a RocksDB: created checkpoint message.
Running myrocks_hotbackup on MariaDB currently fails like so:
2018-01-19 14:59:35.950 INFO Starting MySQL misc file traversal from database mtr..
|
2018-01-19 14:59:35.979 INFO Starting MySQL misc file traversal from database #rocksdb..
|
tar: Cowardly refusing to create an empty archive
|
Try 'tar --help' or 'tar --usage' for more information.
|
2018-01-19 14:59:35.982 ERROR Got error on stream write: 512 #rocksdb/LOG.old.1516362898906947
|
Modified myrocks_hotbackup so it now works with MariaDB:
diff --git a/scripts/myrocks_hotbackup b/scripts/myrocks_hotbackup
|
index cb10bb9..194ff53 100755
|
--- a/scripts/myrocks_hotbackup
|
+++ b/scripts/myrocks_hotbackup
|
@@ -124,7 +124,7 @@ class MiscFilesProcessor():
|
dirs = [ d for d in os.listdir(self.datadir) \
|
if not os.path.isfile(os.path.join(self.datadir,d))]
|
for db in dirs:
|
- if not db.startswith('.') and not self._is_socket(db):
|
+ if not db.startswith('.') and not self._is_socket(db) and not db.startswith("#rocksdb"):
|
dbs.append(db)
|
return dbs |
Backup command doesn't need any changes:
myrocks_hotbackup --user=root --port=3311 --checkpoint_dir=/home/psergey/backup-restore-2/backup-dir | ...
|
Restore command needs to take into account that in MariaDB RocksDB data is not
in '.rocksdb' but rather in '#rocksdb':
myrocks_hotbackup --move_back \
|
--datadir=/home/psergey/backup-restore-2/restore-dir \
|
--rocksdb_datadir=/home/psergey/backup-restore-2/restore-dir/\#rocksdb \
|
--rocksdb_waldir=/home/psergey/backup-restore-2/restore-dir/\#rocksdb \
|
--backup_dir=/home/psergey/backup-restore-2/tarball-unpack
|
Don't forget to escape the '#' character.
Looking at whether rocksdb_hotbackup testsuite has a testcase that is not using any FB-only features...
rocksdb_hotbackup.stream passes (run with ~/rocksdb-mtr), but it takes 2.5 minutes (of these, data loading takes 2 min 15 sec. The dataset is about 200M).
It also uses shell scripting heavily and I was getting pushback for including such tests in the default mix.
Current rocksdb_hotbackup script will not backup ibdata and ib_logfile, but will backup .ibd files.
Basically, it works for MyRocks-only instances, but it's difficult to guarantee anything if other storage engines are used.
Take aways from discussion with cvicentiu:
Python version
- Does myrocks_hotbackup need python2 or python3? Not clear.
Packaging options
- make a new package for myrocks_hotbackup, with proper dependencies.
- put myrocks_hotbackup into the same package as myrocks itself. Add dependencies as "recommended". it is acceptable some things in a package do not work when recommended dependencies are not met.
About MySQL dependency
- We (MariaDB) do not maintain a python connector.
- There are certain issues with the MySQLDB connector (didn't take note what they are exactly
). This connector is based on the C client library.
- There is also a pure python implementation, https://github.com/PyMySQL/PyMySQL (package python-pymysql or python3-pymysql)
- myrocks_hotbackup only needs two commands: select @@datadir and SET GLOBAL rocksdb_create_checkpoint=param. If dependency on mysql connector is too
hard, we could just invoke /usr/bin/mysql manually.
myrocks_hotbackup only works with python 2. It makes use of the commands module which is only available in python 2
Pushed cvicentiu's fix into main 10.2, and I see myrocks_hotbackup in the debian package:
mariadb-plugin-rocksdb_10.2.13+maria-artful_amd64.deb:
|
new debian package, version 2.0.
|
size 2236152 bytes: control archive= 836 bytes.
|
38 bytes, 1 lines conffiles
|
690 bytes, 15 lines control
|
397 bytes, 6 lines md5sums
|
Package: mariadb-plugin-rocksdb
|
Source: mariadb-10.2
|
Version: 10.2.13+maria~artful
|
Architecture: amd64
|
Maintainer: MariaDB Developers <maria-developers@lists.launchpad.net>
|
Installed-Size: 13223
|
Depends: mariadb-server-10.2 (= 10.2.13+maria~artful), libc6 (>= 2.17), liblz4-1 (>= 0.0~r130), libsnappy1v5, libstdc++6 (>= 7), zlib1g (>= 1:1.1.4)
|
Recommends: python-mysqldb
|
Section: database
|
Priority: optional
|
Homepage: http://mariadb.org/
|
Description: RocksDB storage engine for MariaDB
|
The RocksDB storage engine is a high performance storage engine, aimed
|
at maximising storage efficiency while maintaining InnoDB-like performance.
|
This package contains the RocksDB plugin for MariaDB.
|
|
|
*** Contents:
|
drwxr-xr-x root/root 0 2018-02-09 13:51 ./
|
drwxr-xr-x root/root 0 2018-02-09 13:51 ./etc/
|
drwxr-xr-x root/root 0 2018-02-09 13:51 ./etc/mysql/
|
drwxr-xr-x root/root 0 2018-02-09 13:51 ./etc/mysql/mariadb.conf.d/
|
-rw-r--r-- root/root 40 2018-02-09 13:51 ./etc/mysql/mariadb.conf.d/rocksdb.cnf
|
drwxr-xr-x root/root 0 2018-02-09 13:51 ./usr/
|
drwxr-xr-x root/root 0 2018-02-09 13:51 ./usr/bin/
|
-rwxr-xr-x root/root 24704 2018-02-09 13:47 ./usr/bin/myrocks_hotbackup
|
...
|
So, we include myrocks_backup script into at least some packages, but not a single test from rocksdb_hotbackup test suite works. This is quite bad. I suggest either to deprecate and remove the script if it's not supported, or add tests for it.