Does this mean it is coming to 10.2?

mdcallag, yes, that is the plan.

Exciting!

For anyone else that is new to this thread; Callaghan and Co. have some myrocks docs here:
https://github.com/facebook/mysql-5.6/wiki/Getting-Started-with-MyRocks,
and some deep-dive-slides from April.

MariaDB and TokuDB DBA's like myself are probably more interested in the hot backup tool!
https://github.com/facebook/mysql-5.6/wiki/Physical-backup-with-myrocks_hotbackup

Will rocks ever become compatible with the Online DDL implementation in MariaDB?

Not all these required server changes might be needed for MariaDB. For example (I just picked one commit at random) per-database uuids were obviously created for MySQL GTID implementation, and might be not needed for MariaDB GTID implementation.

Mail thread about support for optimistic parallel replication:

https://lists.launchpad.net/maria-developers/msg09989.html

Status update:

• the code in 10.2-mariarocks tree now compiles.
• a LOT of things related to replication and synchronization with binlog are still disabled (under #ifdef MARIAROCKS_NOT_YET) and are not expected to work yet.
• There are rocksdb MTR testcases that pass, but most testcases fail still

With the 'rocksdb' test suite I currently get:

Completed: Failed 134/200 tests, 33.00% were successful.

What is the magic to compile it? Using ubuntu 16.04

cmake was:
prefix=$1
cmake \
-DBUILD_CONFIG=mysql_release \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_INSTALL_PREFIX:PATH=$prefix \
-DWITH_ZLIB="bundled" \
-DMYSQL_DATADIR="${prefix}/data" \
-DMYSQL_UNIX_ADDR="${prefix}/var/mysql.sock" \
-DMYSQL_USER="mysql" \
-DENABLED_LOCAL_INFILE=1 \
-DMYSQL_MAINTAINER_MODE=1 \
.
Error is:
./mariadb/mysys/charset.c: In function ‘get_charsets_dir’:
./mariadb/mysys/charset.c:509:22: error: ‘DEFAULT_CHARSET_HOME’ undeclared (first use in this function)
is_prefix(sharedir, DEFAULT_CHARSET_HOME))

mdcallag, I wanted to note again that most of replication/binlog-related features are still disabled and do not work yet. Judging from the number of test failures, a number of other things are broken as well.

Having said that, here are the steps I just used on Ubuntu 16.04.1 to build:

sudo apt-get update

sudo apt-get install g++ cmake libbz2-dev libaio-dev bison zlib1g-dev libsnappy-dev 

sudo apt-get install libgflags-dev libreadline6-dev libncurses5-dev libssl-dev liblz4-dev gdb git

git clone https://github.com/MariaDB/server.git mariadb-10.2

cd  mariadb-10.2

git checkout 10.2-mariarocks

git submodule init

git submodule update

cmake . -DCMAKE_BUILD_TYPE=Debug -DWITHOUT_MROONGA:bool=1 -DWITHOUT_TOKUDB:bool=1

make -j16

(-DWITHOUT_TOKUDB and -DWITHOUT_MROONGA are just to make the build faster)
A test that works:

cd mysql-test

./mysql-test-run --mysqld=--default-storage-engine=rocksdb --mysqld=--skip-innodb --mysqld=--default-tmp-storage-engine=MyISAM --mysqld=--rocksdb rocksdb.type_binary

Also tried with -DCMAKE_BUILD_TYPE=RelWithDebInfo and it worked (I mean compilation finished and a few tests ran).

Still trying the release build on ubuntu 16.04. Must be more dependencies. I get this failure:

cd /home/mdcallag/git/mariadb/client && /usr/bin/cmake -E cmake_link_script CMakeFiles/mysqltest.dir/link.txt --verbose=1

/usr/bin/c++    -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -DWITH_INNODB_DISALLOW_WRITES -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF  -Wl,--export-dynamic CMakeFiles/mysqltest.dir/mysqltest.cc.o  -o mysqltest  -lpthread ../libmariadb/libmariadb/libmariadbclient.a ../mysys/libmysys.a ../pcre/libpcre.a ../pcre/libpcreposix.a -lpthread -lc -lgnutls ../dbug/libdbug.a ../mysys_ssl/libmysys_ssl.a ../mysys/libmysys.a ../dbug/libdbug.a ../mysys_ssl/libmysys_ssl.a ../zlib/libzlib.a -ldl -lm ../strings/libstrings.a ../extra/yassl/libyassl.a ../extra/yassl/taocrypt/libtaocrypt.a ../pcre/libpcre.a -lpthread 

/usr/bin/ld: ../mysys/libmysys.a(stacktrace.c.o): undefined reference to symbol '__bss_start'

//usr/lib/x86_64-linux-gnu/libgmp.so.10: error adding symbols: DSO missing from command line

collect2: error: ld returned 1 exit status

client/CMakeFiles/mysqltest.dir/build.make:110: recipe for target 'client/mysqltest' failed

There is a number of similar test failures in

• type_varchar
• type_fixed_indexes
• type_enum
• type_date_time_indexes

etc. They all look like this failure in rocksdb.type_varchar:

  create table t (id int primary key, email varchar(100), KEY email_i (email(30)));

  insert into t values (1, 'abcabcabcabcabcabcabcabcabcabcabc  ');

  explain select 'email_i' as index_name, count(*) AS count from t force index(email_i);

  id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra

-  1	SIMPLE	t	index	NULL	email_i	33	NULL	#	Using index

+  1	SIMPLE	t	ALL	NULL	NULL	NULL	NULL	#	

Debugging it, I can see that in MariaDB:

(gdb) p table->field[0]->part_of_key

  $67 = {map = 1}

while in MySQL

(gdb) p table->field[0]->part_of_key

  $20 = {map = 3}

and the cause of that is this commit in MariaDB: b6b5f9fabe4866a8753e81e1f80593b645f35d8e

In particular the chunk of code in table.cc that starts with:

        /* 

          Do not extend the key that contains a component

          defined over the beginning of a field.

	*/ 

I'll need to check with Igor what was the exact reason for this logic. MyRocks relies ALL keys having PK extension. (there is already one exception: SQL layer doesn't generate extensions for unique secondary indexes. MyRocks has code to produce index extension for its internal data structures).

rocksdb.type_char_indexes fails like this:

 rocksdb.type_char_indexes

--- /home/psergey/dev-git/10.2-mariarocks/storage/rocksdb/mysql-test/rocksdb/r/type_char_indexes.result 2016-10-06 17:30:25.595958776 +0000

+++ /home/psergey/dev-git/10.2-mariarocks/storage/rocksdb/mysql-test/rocksdb/r/type_char_indexes.reject 2016-10-29 13:16:57.672889620 +0000

@@ -45,7 +45,7 @@

 INSERT INTO t1 (c,c20,v16,v128,pk) VALUES ('a','char1','varchar1a','varchar1b','1'),('a','char2','varchar2a','varchar2b','2'),('b','char3','varchar1a','varchar1b','3'),('c','char4','varchar3a','varchar3b','4');

 EXPLAIN SELECT SUBSTRING(v16,0,3) FROM t1 WHERE v16 LIKE 'varchar%';

 id     select_type     table   type    possible_keys   key     key_len ref     rows    Extra

-1      SIMPLE  t1      index   v16     v16     19      NULL    #       Using where; Using index

+1      SIMPLE  t1      index   v16     v16     21      NULL    #       Using where; Using index

 SELECT SUBSTRING(v16,7,3) FROM t1 WHERE v16 LIKE 'varchar%';

 SUBSTRING(v16,7,3)

 r1a

.
The reason - it's a bug in MariaDB, MDEV-11172. It has no visible harm effects, so I'll just update the .result file.

rocksdb.rocksdb_range test failure and similar ones - see MDEV-11194

Got more tests to work, currently an attempt to run rocksdb test suite gives:

Completed: Failed 85/200 tests, 57.50% were successful.

Current status with tests is:

Completed: Failed 65/199 tests, 67.34% were successful.

Current status with tests is:

Completed: Failed 39/194 tests, 79.90% were successful.

which gives a speed of 26 tests/10 work days

Current status is

Completed: Failed 25/184 tests, 86.41% were successful.

Merged in the current MyRocks code.

This caused more tests to fail. Fixed some of these, but still have:

Completed: Failed 38/199 tests, 80.90% were successful.

Got the number of failures back down to

Completed: Failed 24/197 tests, 87.82% were successful.

Merged with the current MariaDB-10.2. After test result update, now have:

Completed: Failed 23/197 tests, 88.32% were successful.

Fixed or disabled a few more tests. Current status:

Completed: Failed 17/195 tests, 91.28% were successful.                                                                                                                          

From now on, will also run rocksdb_sys_vars test suite.

Current status is:

Completed: Failed 18/295 tests, 93.90% were successful.

Of 18 failing tests

• 17 failing are from rocksdb
• 1 test is from rocksdb_sys_vars

The tree is now in the buildbot:
https://github.com/MariaDB/server/commits/bb-10.2-mariarocks

A few builders: kvm-deb-trusty-amd64, work-amd64-valgrind build and run it.

Differences wrt stock MariaDB on the SQL layer: https://gist.github.com/spetrunia/e5d9fb8e94292ae7a7bc623a9c078111

Setting to 10.2-ga because of the server changes. The plugin itself can come later, if necessary.

Reduced the amount of changes wrt stock MariaDB outside of storage/rocksdb directory. The new diff: https://gist.github.com/spetrunia/c7665515f9c5e714dd8869701fdabcc0 .
serg, Please take a look.

Attempt to classify the new test failures after last merge:

Timeout

rocksdb.drop_table3 w4 [ fail ] timeout after 900 seconds
rocksdb.cardinality w3 [ fail ] timeout after 900 seconds
rocksdb.statistics w1 [ fail ] timeout after 900 seconds
rocksdb.truncate_table3 w4 [ fail ] timeout after 900 seconds
rocksdb.records_in_range w1 [ fail ] timeout after 900 seconds
rocksdb.rocksdb_checksums w3 [ fail ] timeout after 900 seconds
rocksdb.bloomfilter4 w4 [ fail ] timeout after 900 seconds
rocksdb.singledelete w2 [ fail ] timeout after 900 seconds
rocksdb.bloomfilter_skip w1 [ fail ] timeout after 900 seconds
rocksdb.drop_table w3 [ fail ] timeout after 900 seconds
rocksdb.optimize_table w4 [ fail ] timeout after 900 seconds
rocksdb.rocksdb_deadlock_stress_rc w2 [ fail ] timeout after 900 seconds
rocksdb.bloomfilter w1 [ fail ] timeout after 900 seconds
rocksdb.bloomfilter2 w3 [ fail ] timeout after 900 seconds
rocksdb.blind_delete_without_tx_api 'row' w2 [ fail ] timeout after 900 seconds
rocksdb.deadlock w4 [ fail ] timeout after 900 seconds
rocksdb.drop_table2 w2 [ fail ] timeout after 900 seconds
rocksdb.commit_in_the_middle_ddl w1 [ fail ] timeout after 900 seconds
rocksdb.rocksdb_deadlock_stress_rr w3 [ fail ] timeout after 900 seconds
rocksdb.rocksdb_table_stats_sampling_pct_change w2 [ fail ] timeout after 900 seconds

after rocksdb_deadlock_stress_rr on one machine and rocksdb_deadlock_stress_rc
on another, there are always loads of

2017-03-13 22:19:10 140500730528512 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction

2017-03-13 22:19:10 140500731134720 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction

2017-03-13 22:19:10 140500730831616 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction

errors in the log

Deadlock

rocksdb.unique_sec w2 [ fail ] Found warnings/errors in server log file!
rocksdb.rocksdb_locks w4 [ fail ] Found warnings/errors in server log file!
rocksdb.unique_sec_rev_cf w4 [ fail ] Found warnings/errors in server log file!
rocksdb.issue111 w4 [ fail ] Found warnings/errors in server log file!
rocksdb.rocksdb_deadlock_detect_rr w3 [ fail ] Found warnings/errors in server log file!
rocksdb.hermitage w1 [ fail ] Found warnings/errors in server log file!
rocksdb.level_repeatable_read w4 [ fail ] Found warnings/errors in server log file!

2017-03-14  0:19:17 139935554990848 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction

Crash in the client

rocksdb.rocksdb_range w3 [ fail ]
rocksdb.misc w3 [ fail ]

• Crash in the client?

Different failures affecting single tests

rocksdb.rocksdb w3 [ fail ] Found warnings/errors in server log file!

• "Sort aborted" after DELETE.

rocksdb.collation w2 [ fail ]

• Crash in sysvar update.

rocksdb.show_engine w2 [ fail ]

• SHOW ENGINE rocksdB TRANSACTION STATUS

rocksdb.blind_delete_without_tx_api 'mix' w2 [ fail ]
rocksdb.blind_delete_without_tx_api 'stmt' w4 [ fail ]
mysqltest: At line 23: query '$insert' failed: 1105: Can't execute updates on master with binlog_format != ROW.

Misc failures

rocksdb.rpl_row_not_found 'row' w4 [ fail ]

2017-03-14  0:20:26 140674317531008 [ERROR] /home/psergey/dev-git/10.2-mariarocks/sql/mysqld: Error while setting value 'SEMI_STRICT' to 'slave_exec_mode'

rocksdb.rpl_row_stats 'row' w4 [ fail ]
(result file difference)

rocksdb.rpl_row_triggers 'row' w4 [ fail ]
rocksdb.rpl_savepoint 'row' w4 [ fail ]

rocksdb.trx_info_rpl 'row' w4 [ fail ]

2017-03-14  0:21:04 140149053159296 [ERROR] /home/psergey/dev-git/10.2-mariarocks/sql/mysqld: unknown variable 'rpl_skip_tx_api=ON'

rocksdb.rocksdb_datadir w3 [ fail ]

2017-03-14  0:55:24 140347962312576 [ERROR] /home/psergey/dev-git/10.2-mariarocks/sql/mysqld: unknown variable 'rocksdb_datadir=/home/psergey/dev-git/10.2-mariarocks/mysql-test/var/tmp/3/.rocksdb_datadir.test'

rocksdb.tbl_opt_data_index_dir w1 [ fail ]

mysqltest: At line 14: query 'CREATE TABLE t1 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb DATA DIRECTORY = '/foo/bar/data'' failed with wrong errno 1005: 'Can't create table `test`.`t1` (errno: 198 "Unknown error 198")', instead of 1296...

error logs themselves:

• an aws instance https://gist.github.com/spetrunia/7bf36db2bf0ed4fbb8045ecd34a21358
• psergey-desktop: https://gist.github.com/spetrunia/0babf9e3a9f7b11487bb0ad78d641f9c

A lot of timeouts go away if I set @@rocksdb_flush_logs_at_trx_commit to 0 for the rocksdb test suite:

https://gist.github.com/spetrunia/d61bf3f92ca248cd57a3e87504daece0
https://gist.github.com/spetrunia/19eed7367eeb05b3500d062ca7e0e49e

Current status : we are down to 14 test failures, of which 7 are "something with deadlock detector" and another 7 are individual issues.

MTR log: https://gist.github.com/spetrunia/e58537b1efa5094f62ecf5b7b6f60573

Something with deadlock detector

rocksdb.rocksdb_locks
rocksdb.rocksdb_deadlock_stress_rc
rocksdb.unique_sec
rocksdb.unique_sec_rev_cf
rocksdb.locking_issues
rocksdb.rocksdb_deadlock_detect_rr
rocksdb.rocksdb_deadlock_stress_rr

Loads of errors in the log like this

2017-03-16  7:04:48 140240146146048 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction

Isolated failures

rocksdb.misc
Dumb .result difference

rocksdb.collation

mysqltest: At line 188: query 'SET GLOBAL rocksdb_strict_collation_exceptions=null' failed: 2013: Lost connection to MySQL server during query

rocksdb.tbl_opt_data_index_dir

  query 'CREATE TABLE t1 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb DATA DIRECTORY = '/foo/bar/data'' failed with wrong errno

rocksdb.rpl_row_stats
Counter increment difference

rocksdb.rpl_savepoint

  query 'rollback to savepoint a' succeeded - should have failed with errno 1105...

rocksdb.blind_delete_without_tx_api

• wait timeout, not always repeatable

rocksdb.rocksdb_datadir

/home/ubuntu/mariadb-10.2-dbg/sql/mysqld: unknown variable 'rocksdb_datadir=/home/ubuntu/mariadb-10.2-dbg/mysql-test/var/tmp/2/.rocksdb_datadir.test

Down to 7 4 test failures on the AWS instance: https://gist.github.com/spetrunia/8349449919264626a1f93dfb34be18a6

• rocksdb.collation
• rocksdb.misc
• rocksdb.rocksdb_datadir
• rocksdb.tbl_opt_data_index_dir
• rocksdb.rpl_row_stats
• rocksdb.rpl_savepoint
• rocksdb.blind_delete_without_tx_api
  These are the "Isolated failures" mentioned above.

On psergey-desktop, debug build, I get 10 failures: https://gist.github.com/spetrunia/d658ea3d195d420156feb8560a4d7b92 . Same as above plus three more.

Latest status: 4 3 test failures (6 in debug build):
https://gist.github.com/spetrunia/82a89d4bd35d1402145cef5ca11f1eda
https://gist.github.com/spetrunia/b19425d2bfd280fc29ee95151c9f2478

Latest status:
https://gist.github.com/spetrunia/ed35780f02121b5ee381c76fc6e765a3
https://gist.github.com/spetrunia/d3d7f715761026265f6d51ae8c65e188

3 2 1 in release build: rocksdb.rocksdb rocksdb.collation rocksdb.tbl_opt_data_index_dir
5 4 2 in debug build: rocksdb.rocksdb rocksdb.collation rocksdb.tbl_opt_data_index_dir rocksdb.persistent_cache (<- this is an upstream issue, see MDEV-1234) rocksdb.unique_check

There is a new failure - rocksdb.rocksdb. It is not easily reproducible. It doesn't look very bad - it might be a race condition in the test.

Current status:
(debug) https://gist.github.com/spetrunia/e9e08238479816c6c2e8bca9e4ec404e

Will Galera be supported?

obissick, currently there are no plans for Galera support. Galera integrates tightly with the storage engine, and at the moment the only engine it has integration with is InnoDB/XtraDB.

Current status is:

• Upstream is merged up to

  commit 184a4a2d82f4f6f3cbcb1015bcdb32bebe73315c

  Author: Abhinav Sharma <abhinavsharma@fb.com>

  Date: Thu Sep 14 11:40:08 2017 -0700

• Most of the features are in the 10.2 tree
• The candidate code for "Group Commit with Binlog" (MDEV-11934) feature is in bb-10.2-mariarocks tree.

Filed MDEV-14103, it is on elenst's plate now.

psergey,
The previous comment is ambiguous, could you please specify what exactly is on my plate – MDEV-14103 or "making MyRocks in MariaDB stable"?
I acknowledge receiving of MDEV-14103 (Testing for group commit in MyRocks), but it cannot possibly be the only obstacle between the current state and declaring MyRocks stable.
We have 7 open buildbot bugs for RocksDB which need to be closed before declaring MyRocks stable, and quite a few other open bugs/tasks, priority of which needs to be revised, because it was initially set based on the fact that MyRocks was alpha/beta.

elenst, it is only MDEV-14103 that is on your plate. Sorry for the confusion.