Details
-
Task
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
Description
`try_lock()` in InnoDB codes is an atomic operation. For example, it's compiled to "lock cmpxchg" in x86. It will force cacheline to be exclusive state and suffer significant performance degradation in heavy contended scenarios when many CPUs `try_lock()`.
I collected a perf report and it's assembly codes in my benchmark. Attach them in files.
Look at the TTASEventMutex implementation, for instance. We mark the codes, always check whether lock is free at first, then `try_lock()`.
old: while (!try_lock()) { |
new: while (!(state() == MUTEX_STATE_UNLOCKED && try_lock())) { |
Based on real real-world workload test on a 224 CPUs x86 server, this change
brings back 60%+ TPS/perf for MariaDB 10.3.
Attachments
Issue Links
- relates to
-
MDEV-21452 Use condition variables and normal mutexes instead of InnoDB os_event and mutex
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
Can you suggest any improvement to MariaDB Server 10.6 or later?
The last significant architectural performance improvement in InnoDB was MDEV-27774, which allows multiple threads to copy their local log to the global log_sys.buf concurrently. Therefore, it might make sense to test both LTS branches 10.6 and 10.11.
Thank you for you quick reponse and warm heart comments.
My benchmark is based on Ubuntu 20.04. So the default mariaDB vesion is 10.3.
Let me try to upgrade OS and continue to benchmark to see any performance degradation in MariaDB 10.6 or later.
Thank you for the explanation. MariaDB provides also its own packages for major GNU/Linux distributions. You can find a repository for Ubuntu 20.04 at https://mariadb.org/download/?t=repo-config.
We actually use Ubuntu 20.04 on some internal test systems. Its Linux kernel is just new enough to support io_uring (MDEV-24883), but there is no liburing package, so we use a self-built one in order to better test InnoDB on that system. The normal packages for Ubuntu 20.04 use the older libaio interface. I haven’t noticed much performance difference between the two, but io_uring has been claimed to be better.
Hi, Marko
GCC has an option -mrelax-cmpxchg-loop added in GCC 12, to relax a spin loop, benefiting thread synchronization. In GCC 12, the option is off by default.
It does the same thing, like what I proposed in MariaDB10.3. Is it a good idea to support this option in MariaDB10.6?
By the way, I used mariaDB10.6 to do the benchmark.
My workload is PHP-FPM + nginx + WordPress and mariaDB10.6. And my SUT OS is Ubuntu22.04. I collected some perf data. It shows, mariaDB10.6 has a heavy osq_lock. See the flamegraph.
Are you familiar with this? If so, do you have any software solution to mitigate the lock? If not, I would to look into it.
yuhanyang, sorry, it seems that I had forgotten about this.
Meanwhile, there have been some fixes to reduce contention, MDEV-34515 and MDEV-34178 to name the latest, and MDEV-34791 pending review. I see buf::Block_hint in mariadb106-debug.svg
; it was removed in MDEV-33588.
It would be great if you could repeat this experiment.
By the way, write-heavy workloads are known to hit contention on log_sys.mutex in MariaDB Server 10.6. For those, the 10.11 or 11.4 long-term releases would be more interesting to test.
The last release of MariaDB Server 10.3 was 10.3.39 that was released on May 2, 2023.
In MariaDB Server 10.6, the InnoDB synchronization primitives were replaced or rewritten in
MDEV-21452,MDEV-24142, andMDEV-24167.In
MDEV-26467, some code was rewritten so that the expensive loops around IA-32 or AMD64 lock cmpxchg can be avoided. We mostly try to use the 80486 lock xadd, sometimes the 80386 lock bts or lock btr.Compiler support has evolved too, both in GCC and clang. https://github.com/llvm/llvm-project/issues/58685 is one example.