Found while analyzing test failures for
To reproduce these problems compile MariaDB as described in
MDEV-5089 with -DUSE_MUTEX_INSTEAD_OF_RW_LOCKS. Run tokudb and tokudb_bugs test suites.
First, there are attempts to lock read-locked num_DBs_lock. It is locked in ha_tokudb::start_bulk_insert() and then in ha_tokudb::update_row() or ha_tokudb::acquire_table_lock(). It may be not a problem, but as this situation is handled in ha_tokudb::write_row() it should be checked at least.
Second, ha_tokudb::write_row() unlocks not locked num_DBs_lock (in case of error in the very beginning of the function). This should be a problem according to the pthread_rwlock_unlock manual: results are undefined if the read-write lock rwlock is not held by the calling thread.
A rough patching solving the above issues attached.