[MDEV-30864] Crash on innodb_fatal_semaphore_wait_threshold Created: 2023-03-16  Updated: 2023-06-27  Resolved: 2023-06-27

Status: Closed
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB
Affects Version/s: 10.9.5
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Jacob Williams Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: None
Environment:

Amazon Linux 2
4.14.305-227.531.amzn2.x86_64

Attachments: Text File stack-trace-partial-MDEV-30864.txt    
Issue Links:
Relates
relates to MDEV-29835 Partial server freeze Closed

 Description   

We have had this crash 4 times in the last 3 weeks since we upgraded to 10.9.5. We were previously on 10.4.x and had not seen this or any other crashes recently. We run 12 production servers and we have seen this on 3 of them. All times were under pretty heavy load. We suspected that the heavy io load made have been an issue and increased the IO capacity of our EBS volumes so that we did not spend as much time at peak capacity. But the error happened today under the faster io config and the load was not reaching capacity this time, although the server was moderately busy at the time.

2023-03-16 11:35:32 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
 
230316 11:35:32 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.9.5-MariaDB-log source revision: 260f1fe7c394118cf034f1a48001e1aa06816b69
 
key_buffer_size=31457280
 
read_buffer_size=131072
 
max_used_connections=407
 
max_threads=4002
 
thread_count=284
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8844754 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
Can't start addr2line
 
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55cc6c00f65e]
 
/usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x55cc6ba80fb7]
 
/lib64/libpthread.so.0(+0x118e0)[0x7fbdc86df8e0]
 
/lib64/libc.so.6(gsignal+0x110)[0x7fbdc7a8eca0]
 
/lib64/libc.so.6(abort+0x148)[0x7fbdc7a90148]
 
/usr/sbin/mariadbd(+0xe54520)[0x55cc6be97520]
 
/usr/sbin/mariadbd(+0xe0c9d6)[0x55cc6be4f9d6]
 
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x40)[0x55cc6bf99540]
 
/usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2b)[0x55cc6bf9a44b]
 
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x61)[0x55cc6bf97881]
 
/lib64/libstdc++.so.6(+0xbae70)[0x7fbdc8202e70]
 
/lib64/libpthread.so.0(+0x744b)[0x7fbdc86d544b]
 
/lib64/libc.so.6(clone+0x3f)[0x7fbdc7b4a52f]
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /var/lib/mysql/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             254354               254354               processes 
Max open files            32768                32768                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       254354               254354               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
 
 
Kernel version: Linux version 4.14.305-227.531.amzn2.x86_64 (mockbuild@ip-10-0-35-227) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-15) (GCC)) #1 SMP Tue Feb 14 09:55:28 UTC 2023
 
 
 



#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
 
#1  0x000055cc6ba8102e in handle_fatal_signal (sig=6) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:357
 
#2  <signal handler called>
 
#3  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 
#4  0x00007fbdc7a90148 in __GI_abort () at abort.c:79
 
#5  0x000055cc6be97520 in ib::fatal::~fatal (this=<optimized out>, __in_chrg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/innobase/ut/ut0ut.cc:527
 
#6  0x000055cc6be4f9d6 in srv_monitor_task () at /usr/src/debug/MariaDB-/src_0/storage/innobase/srv/srv0srv.cc:1250
 
#7  0x000055cc6bf99540 in tpool::thread_pool_generic::timer_generic::run (this=0x55cc6e4916c0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:343
 
#8  tpool::thread_pool_generic::timer_generic::execute (arg=0x55cc6e4916c0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:363
 
#9  0x000055cc6bf9a44b in tpool::task::execute (this=0x55cc6e491700) at /usr/src/debug/MariaDB-/src_0/tpool/task.cc:37
 
#10 0x000055cc6bf97881 in tpool::thread_pool_generic::worker_main (this=0x55cc6e1e6c60, thread_var=0x55cc6e2764a0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:580
 
#11 0x00007fbdc8202e70 in std::execute_native_thread_routine_compat (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:110
 
#12 0x00007fbdc86d544b in start_thread (arg=0x7fb1eaed7700) at pthread_create.c:465
 
#13 0x00007fbdc7b4a52f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95



 Comments   
Comment by Marko Mäkelä [ 2023-03-16 ]

Can you please try to produce full stack traces of all threads of the hung process?

Without that information, it is impossible to diagnose hangs.

There are good chances that this would be a duplicate of MDEV-29835, but it is impossible to say that without having more information.

Comment by Jacob Williams [ 2023-03-16 ]

OK working on a full stack trace for all threads but it's slow going. Only at 33% after about 2hrs. And it will take us a little while to make sure we don't have any customer data in it.

In the mean time i picked out a few interesting threads in what has come out so far and cleaned all the sensitive strings from them. Maybe they can confirm that it is same as MDEV-29835

stack-trace-partial-MDEV-30864.txt

Comment by Jacob Williams [ 2023-03-16 ]

I uploaded full stacktrace of all threads to FTP as MDEV-30864-crash-full-stacktrace-all-threads.txt

Comment by Daniel Black [ 2023-04-03 ]

jacob.williams:

Thread 215 (Thread 0x7fb2efa6e700 (LWP 30276)) stack is like this stack in MDEV-29835 - recursive call on btr_page_split_and_insert

Comment by Jacob Williams [ 2023-04-07 ]

We reverted to 10.9.4 and have not seen this crash for the last few weeks. We will wait for 10.9.6. Unfortunately we have been seeing extraordinary memory usage in 10.9.4, but not completely sure that it is not present in 10.9.5. I'll create a ticket for that if I can get together enough information on it.

Comment by Daniel Black [ 2023-04-07 ]

> we have been seeing extraordinary memory usage in 10.9.4

I'm fairly sure that was known and fixed.

Comment by Marko Mäkelä [ 2023-05-26 ]

MariaDB Server 10.9.6 was released a couple of weeks ago. Would that fix the hang for you?
Unfortunately, that release is affected by MDEV-31234, and I can’t recommend using it unless you use innodb_undo_tablespaces=2 or more.

Generated at Thu Feb 08 10:19:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.