Mariadb 11.6.2-MariaDB-log MariaDB Server issue with RR isolation and TPCC

Mariadb 11.6.2-MariaDB-log MariaDB Server issue with RR isolation and TPCC
--------------------------------------------------------------------------------
Hello folks.
While I was doing some routine checks, I had the following issue with the MariaDB 11.6.2 version.

When running sysbench tpcc in Repeatable Read mode, when using more than 3 threads I got the following error:
FATAL: mysql_drv_query() returned error 1020 (Record has changed since last read in table 'district6') for query 'SELECT d_next_o_id, d_tax
FROM district6
WHERE d_w_id = 1
AND d_id = 4 FOR UPDATE'
FATAL: `thread_run' function failed: [string "sysbench.sql.lua"]:304: SQL error, errno = 1020, state = 'HY000': Record has changed since last read in table 'district6'

If I reduce the rate (--rate=50) the error (at least with 4 threads) is nto coming up.

Of course any higher number of threads is causing the same issue.

The error is not presenting itself when test is run in Read Committed mode.

Checking the slow log (attached)
I see that the last action on the table district6 is from thread 347, before the thread 345 (which raised the error) issue the BEGIN.

Given the same test did run successfully on previous MariaDB versions and also in latest Percona and MySQL Server version, I think it may be something you want to investigate.

Hot to reproduce it
------------------------
In a decently fast machine
1) install MariaDB
2) apply configuration as below.
3) clone and compile sysbench from given repo.
4) clone also TPCc
5) populate the db (see section prepare below)
6) run the tests (see commands below)

Server configuration:
-----------------------
[client]
port = 3307
socket = /opt/mysql_instances/test-tpcc-maria-116/mysql.sock
[mysqldump]
quick
quote-names
max_allowed_packet = 48M
[mysql]
prompt=(\\u@
h) \\d>
[isamchk]
key_buffer = 16M
[mysqld_safe]
log-error = /opt/mysql_instances/test-tpcc-maria-116/mysql-3307.err
open-files-limit= 65536
[mysqld]
basedir =/opt/mysql_templates/maria-116/
datadir = /opt/mysql_instances/test-tpcc-maria-116/data
lc_messages_dir =/opt/mysql_templates/maria-116/share/
local_infile = 1
log-error = /opt/mysql_instances/test-tpcc-maria-116/mysql-3307.err
pid-file = /opt/mysql_instances/test-tpcc-maria-116/mysql.pid
plugin-dir =/opt/mysql_templates/maria-116/lib/plugin/
port = 3307
secure-file-priv =""
server-id = 3307
socket = /opt/mysql_instances/test-tpcc-maria-116/mysql.sock
tmpdir = /opt/mysql_instances/test-tpcc-maria-116/temp
user =mysql
default_time_zone='+01:00'
max-connections =2500
table_open_cache = 4096
table-open-cache-instances =4
thread_cache_size = 1024
thread_stack = 256K
thread_handling = one-thread-per-connection
binlog_cache_size = 1M
binlog_stmt_cache_size = 1M
join_buffer_size = 32M
max_heap_table_size = 16M
read_buffer_size = 1M
read_rnd_buffer_size = 16M
sort_buffer_size = 1M
tmp_table_size = 16M
innodb-adaptive-hash-index =FALSE
innodb-buffer-pool-instances =3
innodb-buffer-pool-size =10G
innodb-data-file-path =ibdata1:100M:autoextend
innodb_data_home_dir = /opt/mysql_instances/test-tpcc-maria-116/data
innodb-flush-method =O_DIRECT
innodb-log-file-size =800M
innodb-log-files-in-group =4
innodb_monitor_enable =all
innodb_write_io_threads =4
innodb_read_io_threads =4
innodb_io_capacity = 1000
innodb_io_capacity_max = 4000
skip-log-bin
log_bin = /opt/mysql_instances/test-tpcc-maria-116/logs/binlog
binlog_expire_logs_seconds=10
slave_connections_needed_for_purge=0
sync-binlog =1
long_query_time=240
slow-query-log = 0
slow-query-log-file = /opt/mysql_instances/test-tpcc-maria-116/logs/slow.log
general-log-file = /opt/mysql_templates/maria-116/logs/general.log
general-log =0

Sysbench version
-------------------------------
sysbench --version
sysbench 1.1.0-75c2f63
Repo: https://github.com/Tusamarco/sysbench

Sysbench-Tpcc
--------------------------------
Repo: https://github.com/Tusamarco/sysbench-tpcc

Sysbench commands
--------------------------------
To Prepare:
-----------
sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307 --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1 --histogram --report_csv=yes --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=30 --threads=10 --mysql-ignore-errors=none --reconnect=0 prepare

Working:
sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307 --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1 --histogram --report_csv=yes --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=30 --threads=2 --mysql-ignore-errors=none --reconnect=0 run
OR
sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307 --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1 --histogram --report_csv=yes --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=30 --threads=4 --mysql-ignore-errors=none --reconnect=0 --rate=50 run

Not Working:
sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307 --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1 --histogram --report_csv=yes --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=30 --threads=4 --mysql-ignore-errors=none --reconnect=0 run
OR
sysbench /opt/tools/sysbench-tpcc/tpcc.lua --mysql-host=127.0.0.1 --mysql-port=3307 --mysql-user=app_test --mysql-password=test --mysql-db=tpcc --db-driver=mysql --tables=10 --scale=100 --rand-type=uniform --report-interval=1 --histogram --report_csv=yes --stats_format=csv --db-ps-mode=disable --trx_level=RR --enable_purge=yes --time=30 --threads=64 --mysql-ignore-errors=none --reconnect=0 run

Error
-------------------------------
1,4,651.49,19319.11,8846.45,8563.10,1909.56,12.08,2.00,0.00
FATAL: mysql_drv_query() returned error 1020 (Record has changed since last read in table 'district6') for query 'SELECT d_next_o_id, d_tax
FROM district6
WHERE d_w_id = 1
AND d_id = 4 FOR UPDATE'
FATAL: `thread_run' function failed: [string "sysbench.sql.lua"]:304: SQL error, errno = 1020, state = 'HY000': Record has changed since last read in table 'district6'

And similar.