[MDEV-32065] Always check whether lock is free at first to optimize InnoDB mutexes Created: 2023-09-01  Updated: 2023-10-04

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Fix Version/s: None

Type: Task Priority: Major
Reporter: yuhanyang Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 0
Labels: innodb, performance

Attachments: File 2s-fix-mariadb103.svg     File 2s-ori-mariadb103.svg     PNG File buf_page_optimistic_get_asm.png     File mariadb106-debug.svg    
Issue Links:
Relates
relates to MDEV-21452 Use condition variables and normal mu... Closed
relates to MDEV-24142 rw_lock_t has unnecessarily complex w... Closed
relates to MDEV-24167 InnoDB unnecessarily uses complex rw-... Closed
relates to MDEV-26467 Unnecessary compare-and-swap loop in ... Closed

 Description   

`try_lock()` in InnoDB codes is an atomic operation. For example, it's compiled to "lock cmpxchg" in x86. It will force cacheline to be exclusive state and suffer significant performance degradation in heavy contended scenarios when many CPUs `try_lock()`.

I collected a perf report and it's assembly codes in my benchmark. Attach them in files.

Look at the TTASEventMutex implementation, for instance. We mark the codes, always check whether lock is free at first, then `try_lock()`.

old:		while (!try_lock()) {
new:		while (!(state() == MUTEX_STATE_UNLOCKED && try_lock())) {

Based on real real-world workload test on a 224 CPUs x86 server, this change
brings back 60%+ TPS/perf for MariaDB 10.3.



 Comments   
Comment by Marko Mäkelä [ 2023-09-01 ]

The last release of MariaDB Server 10.3 was 10.3.39 that was released on May 2, 2023.

In MariaDB Server 10.6, the InnoDB synchronization primitives were replaced or rewritten in MDEV-21452, MDEV-24142, and MDEV-24167.

In MDEV-26467, some code was rewritten so that the expensive loops around IA-32 or AMD64 lock cmpxchg can be avoided. We mostly try to use the 80486 lock xadd, sometimes the 80386 lock bts or lock btr.

Compiler support has evolved too, both in GCC and clang. https://github.com/llvm/llvm-project/issues/58685 is one example.

Comment by Marko Mäkelä [ 2023-09-01 ]

Can you suggest any improvement to MariaDB Server 10.6 or later?

The last significant architectural performance improvement in InnoDB was MDEV-27774, which allows multiple threads to copy their local log to the global log_sys.buf concurrently. Therefore, it might make sense to test both LTS branches 10.6 and 10.11.

Comment by yuhanyang [ 2023-09-01 ]

Thank you for you quick reponse and warm heart comments.
My benchmark is based on Ubuntu 20.04. So the default mariaDB vesion is 10.3.
Let me try to upgrade OS and continue to benchmark to see any performance degradation in MariaDB 10.6 or later.

Comment by Marko Mäkelä [ 2023-09-01 ]

Thank you for the explanation. MariaDB provides also its own packages for major GNU/Linux distributions. You can find a repository for Ubuntu 20.04 at https://mariadb.org/download/?t=repo-config.

We actually use Ubuntu 20.04 on some internal test systems. Its Linux kernel is just new enough to support io_uring (MDEV-24883), but there is no liburing package, so we use a self-built one in order to better test InnoDB on that system. The normal packages for Ubuntu 20.04 use the older libaio interface. I haven’t noticed much performance difference between the two, but io_uring has been claimed to be better.

Comment by yuhanyang [ 2023-09-04 ]

Hi, Marko
GCC has an option -mrelax-cmpxchg-loop added in GCC 12, to relax a spin loop, benefiting thread synchronization. In GCC 12, the option is off by default.
It does the same thing, like what I proposed in MariaDB10.3. Is it a good idea to support this option in MariaDB10.6?

Comment by yuhanyang [ 2023-09-04 ]

By the way, I used mariaDB10.6 to do the benchmark.
My workload is PHP-FPM + nginx + WordPress and mariaDB10.6. And my SUT OS is Ubuntu22.04. I collected some perf data. It shows, mariaDB10.6 has a heavy osq_lock. See the flamegraph.
Are you familiar with this? If so, do you have any software solution to mitigate the lock? If not, I would to look into it.

Generated at Thu Feb 08 10:28:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.