[MDEV-30750] MariaDB 10.6.12 crash bug on CloudLinux 8.7 Created: 2023-02-28  Updated: 2023-04-01  Resolved: 2023-04-01

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: brmb Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: crash
Environment:

CloudLinux 8.7


Attachments: PNG File Screenshot 2023-02-28 at 13.04.56.png    

 Description   

mysqld crashed for unknown reasons.
Process had to be killed forcefully as it didn't respond to sigterm (systemd restart) in order to bring it back up.

mysqld logs at the time of crash:

Feb 28 09:00:30 server-name mysqld[201708]: double free or corruption (!prev)
Feb 28 09:00:30 server-name mysqld[201708]: 230228  9:00:30 [ERROR] mysqld got signal 6 ;
Feb 28 09:00:30 server-name mysqld[201708]: This could be because you hit a bug. It is also possible that this binary
Feb 28 09:00:30 server-name mysqld[201708]: or one of the libraries it was linked against is corrupt, improperly built,
Feb 28 09:00:30 server-name mysqld[201708]: or misconfigured. This error can also be caused by malfunctioning hardware.
Feb 28 09:00:30 server-name mysqld[201708]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Feb 28 09:00:30 server-name mysqld[201708]: We will try our best to scrape up some info that will hopefully help
Feb 28 09:00:30 server-name mysqld[201708]: diagnose the problem, but since we have already crashed,
Feb 28 09:00:30 server-name mysqld[201708]: something is definitely wrong and this may fail.
Feb 28 09:00:30 server-name mysqld[201708]: Server version: 10.6.12-MariaDB-cll-lve source revision: 4c79e15cc3716f69c044d4287ad2160da8101cdc
Feb 28 09:00:30 server-name mysqld[201708]: key_buffer_size=2147483648
Feb 28 09:00:30 server-name mysqld[201708]: read_buffer_size=4194304
Feb 28 09:00:30 server-name mysqld[201708]: max_used_connections=134
Feb 28 09:00:30 server-name mysqld[201708]: max_threads=1002
Feb 28 09:00:30 server-name mysqld[201708]: thread_count=78
Feb 28 09:00:30 server-name mysqld[201708]: It is possible that mysqld could use up to
Feb 28 09:00:30 server-name mysqld[201708]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10331757 K  bytes of memory
Feb 28 09:00:30 server-name mysqld[201708]: Hope that's ok; if not, decrease some variables in the equation.
Feb 28 09:00:30 server-name mysqld[201708]: Thread pointer: 0x7f55bc000c58
Feb 28 09:00:30 server-name mysqld[201708]: Attempting backtrace. You can use the following information to find out
Feb 28 09:00:30 server-name mysqld[201708]: where mysqld died. If you see no messages after this, something went
Feb 28 09:00:30 server-name mysqld[201708]: terribly wrong...
Feb 28 09:00:30 server-name mysqld[201708]: stack_bottom = 0x7f578f1d6bd8 thread_stack 0x49000

When straced mysqld process 201708, new futex output was still being generated:

[pid 1930306] futex(0x55fe8dd77ac8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 1930306] futex(0x55fe8dd7880c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1677577350, tv_nsec=807930442}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 201710] <... futex resumed>)       = -1 ETIMEDOUT (Connection timed out)
[pid 201710] futex(0x55fe8dd7880c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1930306] <... futex resumed>)      = 0
[pid 201710] <... futex resumed>)       = 1
[pid 1930306] futex(0x55fe8dd77ac8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 201710] futex(0x55fe8dd77ac8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1930306] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 201710] <... futex resumed>)       = 0
[pid 1930306] futex(0x55fe8dd77ac8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 201710] futex(0x55fe8c982f60, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1930306] <... futex resumed>)      = 0
[pid 201710] <... futex resumed>)       = 0
[pid 1930306] futex(0x55fe8dd78808, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1677577350, tv_nsec=833899302}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 201710] futex(0x55fe8c982f4c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1677577290, tv_nsec=956743000},



 Comments   
Comment by Daniel Black [ 2023-03-02 ]

Thank you for the bug report.

We need a bit more information to work out what is really going on here. If you can install the debuginfo package for CloudLinux's mariadb-server[-core] package, the one that contains mariadbd/mysqld, first. It might be with dnf debuginfo-install mariadb-server-core. I have found some info with https://repo.cloudlinux.com/cloudlinux/8.7/updates-testing/debug/x86_64/, but not 10.6, so I'm not sure how CloudLinux organise it. If you find the way, can you please share so I can be less vague with the next CloudLinux user that needs help.

When it next hangs, a stack trace of the running server would be most useful to resolve this:

https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/#getting-backtraces-from-a-running-mysqld-process-with-gdb-on-linux

Generated at Thu Feb 08 10:18:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.