Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.11.6
-
Debian 12 / Stable.
Description
Every ~2-7 days, around midnight, one of our SQL servers is experiencing an issue where it deadlocks near-completely. The log, as expected, just stops abruptly with no indication of what's wrong.
I can still connect using the 'root' account using a unix socket while this happens. Active queries (show processlist) seems independent.
Not the queries are deadlocking, the program itself is. No queries will process or complete as the program internally waits endlessly for mutexes.
I researched a possible cause; the most common appears to be calling unsafe functions in signal handlers.
I'm not too well versed in gdb. I don't know how to reproduce the problem (that's our entire issue). It can and does occur periodically. THe situation in [info threads] of gdb looks a bit like this:
- About 300 threads stuck at `0x7f7bf8fa86c0 (LWP 684969) "mariadbd" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38`
- 10-12 threads inside __futex_abstimed_wait_common64
- 5-6 threads in _GI__poll
- About twenty entries like this (unknown); `Thread 0x7f7bf838d6c0 (LWP 329146) "iou-wrk-298768" 0x0000000000000000 in ?? ()`
Attachments
Activity
npr, the condition variable buf_dblwr.cond is allocated statically. The issue could be that a pending write request is lost. In that case, write_slots->m_cache.m_pos should not be 0. What is it in your case?
Back in the Linux kernel 5.11 or thereabouts, we ran into a problem that io_uring would sometimes lose write requests. I wouldn’t expect that the 6.1 series would be affected by that.
(gdb) print write_slots->m_cache.m_pos
|
$6 = 125
|
(gdb) thread 5
|
[Switching to thread 5 (Thread 0x7f4cda75a6c0 (LWP 1294))]
|
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x564cfe1b1f0c <buf_dblwr+108>) at ./nptl/futex-internal.c:57
|
57 ./nptl/futex-internal.c: No such file or directory.
|
(gdb) print write_slots->m_cache.m_pos
|
$7 = 125
|
Looks like it is indeed nonzero.
For completeness, this machine is using a bog-standard, well tested filesystem; ext4 with 4kB blocks. There's no drive encryption, though there is VMware virturalization:
root@s05:~# blkid |
/dev/sda5: UUID="a28215bb-dc64-468e-9f04-b4c96f1ea552" TYPE="swap" PARTUUID="26c5a724-05" |
/dev/sda1: UUID="191e1da6-0127-4b30-b6c5-38e2a874e64d" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="26c5a724-01" |
npr, if you are using MariaDB Server 10.11.6, I think that this report should be a duplicate of MDEV-33669 or MDEV-32861, which were fixed in 10.11.8 and 10.11.7. I would recommend upgrading to 10.11.9 while you are at it.
Looking into things a bit further, the key thread you seem to identify is hanging on a condition value that never becomes true.
I've read that if a pthread condition is 'deleted' then undefined behaviour could result. The debugger reports the following info when dumping this variable;
(gdb) print *cond$5 = {__data = {__wseq = {__value64 = 8709, __value32 = {__low = 8709, __high = 0}}, __g1_start = {__value64 = 8705, __value32 = {__low = 8705, __high = 0}}, __g_refs = {0, 2},__g_size = {0, 0}, __g1_orig_size = 4, __wrefs = 8, __g_signals = {0, 0}},__size = "\005\"\000\000\000\000\000\000\001\"\000\000\000\000\000\000\000\000\000\000\002", '\000' <repeats 11 times>, "\004\000\000\000\b\000\000\000\000\000\000\000\000\000\000",__align = 8709}