LP:923772 - performance regression in mariadb-5.3 with many threads

sysbench OLTP benchmark with 64 threads yields 50% or fewer operations per second with MariaDB 5.3.3-rc compared to MariaDB 5.2.10.

I.e. on a machine with 24 cores we get 125K ops/sec with 5.2.10 but only 64K ops/sec with 5.3.3. With more cores the result gets worse. I.e. 32 cores give 175K vs. 39K.

sysbench parameters: --test=oltp --num-threads=64 --max-time=60 --max-requests=0 --oltp-table-size=10000000 --db-driver=mysql --mysql-table-engine=innodb

my.cnf:

skip-external-locking
bind-address = 127.0.0.1
key_buffer = 16M
max_allowed_packet = 16M
default-storage-engine = INNODB
innodb_buffer_pool_size = 8G
innodb_additional_mem_pool_size = 64M
innodb_log_buffer_size = 128M
innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 0
innodb_locks_unsafe_for_binlog = 1
innodb_data_file_path = ibdata1:11G;ibdata2:11G:autoextend
innodb_log_file_size = 1333M
innodb_log_files_in_group = 3
innodb_file_io_threads = 4
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
back_log = 50
max_connections = 100
table_open_cache = 2048
max_heap_table_size = 64M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 8
query_cache_size = 64M
query_cache_limit = 2M
thread_stack = 192K
transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M
long_query_time = 2

PMP shows significant mutex contention in the Query_cache::try_lock() method. If the query cache is disabled, 5.3.3 performance gets dramatically better. Still 5.2.10 is about 20% faster. There seems to be extra contention inside InnoDB, caused by make_join_statistics(). The same call stack can be seen in 5.2.10, but less often.