Vladislav Vaintroub added a comment - 2020-02-11 22:26 So, I ran this benchmark, which I think resembles the axel "sweetspot" close enough my.cnf [mysqld]   #####non innodb options max_connections = 300 table_open_cache = 600 query_cache_type = 0   #####innodb options innodb_buffer_pool_size = 1G innodb_log_buffer_size = 32M innodb_log_file_size = 512M innodb_flush_log_at_trx_commit = 2 innodb_doublewrite = 0   loose-innodb_adaptive_hash_index_partitions = 32 loose-innodb_adaptive_hash_index_parts = 32   #####SSD innodb-flush-method = O_DIRECT innodb_io_capacity = 4000 loose-innodb_flush_neighbors = 0 innodb_write_io_threads = 8   #####the variables for this test innodb_buffer_pool_instances = 1   innodb_max_dirty_pages_pct = 99 skip-innodb_adaptive_hash_index skip-innodb-stats-persistent innodb-change-buffering=none innodb_file_per_table = 0 script to run with sysbench 1.0 sysbench --test=/usr/share/sysbench/oltp_update_index.lua --tables=32 --table-size=1250000 --rand-seed=42 --rand-type=uniform --num-threads=32 --report-interval=2 --mysql-socket=/tmp/mysql.sock --time=300 --max-requests=0 --mysql-user=root --percentile=95 $1 where $1 is either "prepare" or "run" (you need to have a database called sbtest) Note on benchmark itself - it uses very low bufferpool to data size ratio (I believe data would be around 8-10GB if it was in files, rather than ibdata1), and only 1GB buffer pool, so it is designed to be IO intensive. It uses whole 2 CPUs out of 56 on the benchmark machine (and the difference between 1 and 4 buffer pools was not obvious in "top") For the benchmarks, I ran the server with either innodb_buffer_pool_instances set to 1 or 4 4 buffer pools wins against 1 buffer pool with about 9000 tps against around 6000 tps, at least we can say that whatever Axel had found is reproducible for this use case. I attached pt-pmp output 1bp.txt (single buffer pool instance) and 4bp.txt (4 instances), made with 20 samples separated by 10 seconds delay (If someone knows a more modern tool to profile on contention , please tell) From that, I grepped TTAS to find out lines with innodb mutexes (but please, take a look also on anything else, maybe I missed something) I think the contention might be on the buffer pool mutex in buf_page_io_complete(buf0buf.cc:6019) , at least it appears rather oft in 1bp.txt in a couple of different callstacks . here is the code in question. https://github.com/MariaDB/server/blob/f3dac591747dfbd88bd8ae2855f9a0e64006ce75/storage/innobase/buf/buf0buf.cc#L6019

Marko Mäkelä added a comment - 2020-02-12 10:16 I ported the changes to 10.5 . Given that our benchmark was extreme and that normally the doublewrite buffer would be a scalability bottleneck for write-intensive workloads, I think that it should be an acceptable change. In most cases, a single buffer pool performed at least as well as multiple ones. Removing the code to handle multiple buffer pool instances could slightly improve the overall performance and open up opportunities to make more use of std::atomic (in MDEV-15053 ).

Marko Mäkelä added a comment - 2020-02-12 12:59 - edited I did some more testing, checked the buildbot results (no failures), and fixed a bug that was caught in the MDEV-12353 branch only. Pushed to 10.5.

Marko Mäkelä added a comment - 2020-02-12 18:59 Based on the feedback of wlad , I reverted changes to some INFORMATION_SCHEMA.INNODB_ tables . We will return a dummy buffer pool identifier 0, for compatibility.

Marko Mäkelä added a comment - 2020-03-24 08:27 The slightly increased contention in buf_page_io_complete() in the write-heavy workloads when moving from 4 buffer pool instances to 1 would likely not be helped by MDEV-15053 . That function would still acquire buf_pool_t::mutex (which was renamed to buf_pool_t::LRU_list_mutex ). MDEV-15053 did not show improved performance when wlad was testing it. To keep the latching rules more understandable and to avoid race conditions, it might be best to omit most of those changes.