So, there is some slowdown (starting at 30% and going down as concurrency increases). I'm not sure about the cause.

Added current upstream FB/MySQL-5.6 to the chart.

Results with log-bin=0:

Percona-5.7

 20,  6574.14

 50, 15140.14

 80, 20138.38

100, 23784.30

150, 34138.96

MariaDB-10.3

 20,  665.07

 50,  641.08

 80,  626.83

100,  607.69

150,  654.11

FBMySQL-5.6

 20,  6340.32

 50, 14219.20

 80, 19054.88

100, 23190.26

150, 30586.72

MariaDB is much slower here:

This is obviously a bug and should not be happening.

Ok, the problem is in non-XA mode. In this mode, rocksdb_prepare and rocksdb_commit_ordered are not called. Only the rocksdb_commit() call is made.

In that call, MyRocks is expected to commit the transaction and make its changes persistent.

After last fixes (for XA mode), it does it like so:

  tx->set_sync(false);

  tx->commit();

  rdb->FlushWAL(true);

Looking at DBImpl::FlushWAL() code, I see that it doesn't participate RocksDB's GroupCommit (or, rather GroupFlush). It will attempt to make a flush call, followed by fsync/fdatasync call on its own.

This is consistent with the observed performance.

(RocksDB's PessimisticTransaction::Commit() will eventually call DBImpl::FlushWAL(). But it will do it inside its group commit implementation, only one call will be made for transactions that are committing concurrently).

Trying on a patched version:

MariaDB-10.3-patch1

 20,  6483.17

 50, 14635.17

 80, 19708.48

100, 23949.44

150, 32377.06

This is on par with other branches.

The code that is causing slowdown here was introduced in MDEV-15372.

That MDEV was fixing the performance of multi-threaded slave (non-XA variant of it). The slave wants to make commits in the same order as the master does, the idea was to let the transactions run, but then commit them (call rocksdb_commit) in their order on the master.

This caused them to be serialized. The way to un-serialize them was mimicking InnoDB, and it was:

  tx->set_sync(false);

  tx->commit(); // this establishes the commit order. It is serialized but it does not flush

  // this notifies the SQL layer that subsequent transactions can run:

  thd_wakeup_subsequent_commits(thd, 0);

  // this makes the changes persistent:

   rocksdb::Status s= rdb->FlushWAL(true);

I'm not sure why did this change fix the performance back then but is killing it now. Maybe, something has changed inside RocksDB? (looks like no)

Disable this for non-slave threads

An obvious thing to do is to disable the new code for non-slave threads (see THD::slave_thread).

Possible solutions for slave threads:

Hook in RocksDB

Add a hook inside RocksDB somewhere to call thd_wakeup_subsequent_commits().
roblems: there doesn't seem to be any hook for this, so adding it will
require A) finding the right place and B) convincing RocksDB to accept a PR
with a hook.

Non-durable mode for the slave commits

Transactions on the slave come from the binary log, so it is not an issue if
some of them are lost in a crash. They can be replayed from the relay log.

(We only need a guarantee that if transaction #N disappears, then all
subsequent transactions disappear as well. I think we have this property:
writes to RocksDB WAL are done sequentially. Failing to flush may truncate the
WAL, but will not create "gaps" in it)

One thing to check: when the slave thinks it has applied all events from
a relay log file, it may remove that relay log file. But what if the storage
engine has not persisted the transactions from that that log file yet
(assuming they can be replayed)? Can this situation happen and if yes can it
be prevented (e.g. have MyRocks flush its WAL before a relay log file is
removed)?

Use XA-mode for slave threads.

(TODO: this looked like a solution but now I'm trying to describe it and
it's not obvious how to achieve both performance and safety?)

Taking MariaDB 10.2 as a base, cset 0623cc7c16c3280d1f81b9049e1561d1b4b6c1d0.

Developed a patch to disable the MDEV-15372 code for non-slave threads. Trying it on a c5.2xlarge instance, with log-bin off, other settings being default:

n_threads	MariaDB-102-cur	MariaDB-10.2-patched

20	648.16	6195.19

50	630.57	13620.29

80	594.41	18855.85

100	599.48	22311.91

150	670.35	30937.94