Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.9.5
-
None
-
Amazon Linux 2
4.14.305-227.531.amzn2.x86_64
Description
We have had this crash 4 times in the last 3 weeks since we upgraded to 10.9.5. We were previously on 10.4.x and had not seen this or any other crashes recently. We run 12 production servers and we have seen this on 3 of them. All times were under pretty heavy load. We suspected that the heavy io load made have been an issue and increased the IO capacity of our EBS volumes so that we did not spend as much time at peak capacity. But the error happened today under the faster io config and the load was not reaching capacity this time, although the server was moderately busy at the time.
2023-03-16 11:35:32 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
|
230316 11:35:32 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.9.5-MariaDB-log source revision: 260f1fe7c394118cf034f1a48001e1aa06816b69
|
key_buffer_size=31457280
|
read_buffer_size=131072
|
max_used_connections=407
|
max_threads=4002
|
thread_count=284
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8844754 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
Can't start addr2line
|
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55cc6c00f65e]
|
/usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x55cc6ba80fb7]
|
/lib64/libpthread.so.0(+0x118e0)[0x7fbdc86df8e0]
|
/lib64/libc.so.6(gsignal+0x110)[0x7fbdc7a8eca0]
|
/lib64/libc.so.6(abort+0x148)[0x7fbdc7a90148]
|
/usr/sbin/mariadbd(+0xe54520)[0x55cc6be97520]
|
/usr/sbin/mariadbd(+0xe0c9d6)[0x55cc6be4f9d6]
|
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x40)[0x55cc6bf99540]
|
/usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2b)[0x55cc6bf9a44b]
|
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x61)[0x55cc6bf97881]
|
/lib64/libstdc++.so.6(+0xbae70)[0x7fbdc8202e70]
|
/lib64/libpthread.so.0(+0x744b)[0x7fbdc86d544b]
|
/lib64/libc.so.6(clone+0x3f)[0x7fbdc7b4a52f]
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql/data
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 254354 254354 processes
|
Max open files 32768 32768 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 254354 254354 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
|
|
Kernel version: Linux version 4.14.305-227.531.amzn2.x86_64 (mockbuild@ip-10-0-35-227) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-15) (GCC)) #1 SMP Tue Feb 14 09:55:28 UTC 2023
|
|
|
|
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
|
#1 0x000055cc6ba8102e in handle_fatal_signal (sig=6) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:357
|
#2 <signal handler called>
|
#3 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
|
#4 0x00007fbdc7a90148 in __GI_abort () at abort.c:79
|
#5 0x000055cc6be97520 in ib::fatal::~fatal (this=<optimized out>, __in_chrg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/innobase/ut/ut0ut.cc:527
|
#6 0x000055cc6be4f9d6 in srv_monitor_task () at /usr/src/debug/MariaDB-/src_0/storage/innobase/srv/srv0srv.cc:1250
|
#7 0x000055cc6bf99540 in tpool::thread_pool_generic::timer_generic::run (this=0x55cc6e4916c0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:343
|
#8 tpool::thread_pool_generic::timer_generic::execute (arg=0x55cc6e4916c0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:363
|
#9 0x000055cc6bf9a44b in tpool::task::execute (this=0x55cc6e491700) at /usr/src/debug/MariaDB-/src_0/tpool/task.cc:37
|
#10 0x000055cc6bf97881 in tpool::thread_pool_generic::worker_main (this=0x55cc6e1e6c60, thread_var=0x55cc6e2764a0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:580
|
#11 0x00007fbdc8202e70 in std::execute_native_thread_routine_compat (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:110
|
#12 0x00007fbdc86d544b in start_thread (arg=0x7fb1eaed7700) at pthread_create.c:465
|
#13 0x00007fbdc7b4a52f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Needs Feedback
-