Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.6.8, 10.6.10
-
None
-
CentOS 7
Systems are replicas
Description
Since upgrading to MariaDB 10.6 our systems randomly abort due to InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_ sys.latch.
I could not tell whether it's the same bug as e.g. MDEV-29590, so here as new issue.
We could not find steps to reproduce it, the aborts seem to happen randomly, sometimes after a few days of uptime, sometimes after a few hours.
The affected systems are all replicas, but not sure that matters in this case.
error log:
2022-10-19 13:01:01 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
|
221019 13:01:01 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.10-MariaDB-log
|
key_buffer_size=67108864
|
read_buffer_size=131072
|
max_used_connections=320
|
max_threads=1002
|
thread_count=308
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 732888 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
??:0(my_print_stacktrace)[0x5655215226ae]
|
??:0(handle_fatal_signal)[0x565520f71527]
|
sigaction.c:0(__restore_rt)[0x7f3c7f19b630]
|
:0(__GI_raise)[0x7f3c7e5e6387]
|
:0(__GI_abort)[0x7f3c7e5e7a78]
|
??:0(std::pair<std::_Rb_tree_iterator<std::pair<unsigned long const, bool> >, bool> std::_Rb_tree<unsigned long, std::pair<unsigned long const, bool>, std::_Select1st<std::pair<unsigned long const, bool> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, bool> > >::_M_emplace_unique<unsigned long&, bool>(unsigned long&, bool&&))[0x5655213a5780]
|
??:0(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x56552135cd29]
|
??:0(tpool::thread_pool_generic::timer_generic::execute(void*))[0x5655214ac390]
|
??:0(tpool::task::execute())[0x5655214ad29b]
|
??:0(tpool::thread_pool_generic::worker_main(tpool::worker_data*))[0x5655214aa6d1]
|
??:0(std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >))[0x7f3c7ed35330]
|
pthread_create.c:0(start_thread)[0x7f3c7f193ea5]
|
??:0(__clone)[0x7f3c7e6aeb0d]
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /srv/mysql/pdns-customer-slave/data
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 2061349 2061349 processes
|
Max open files 50000 50000 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 2061349 2061349 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
|
|
Kernel version: Linux version 3.10.0-1160.76.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Aug 10 16:21:17 UTC 2022
|
gdb backtrace:
#0 0x00007f3c7f198aa1 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:59
|
#1 0x0000565520f7159e in handle_fatal_signal (sig=6) at /usr/src/debug/MariaDB-/src_0/sql/signal_handler.cc:355
|
#2 <signal handler called>
|
#3 0x00007f3c7e5e6387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
|
#4 0x00007f3c7e5e7a78 in __GI_abort () at abort.c:90
|
#5 0x00005655213a5780 in ib::fatal::~fatal (this=<optimized out>, __in_chrg=<optimized out>)
|
at /usr/src/debug/MariaDB-/src_0/storage/innobase/ut/ut0ut.cc:515
|
#6 0x000056552135cd29 in srv_monitor_task () at /usr/src/debug/MariaDB-/src_0/storage/innobase/srv/srv0srv.cc:1300
|
#7 0x00005655214ac390 in run (this=0x565522c552a0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:343
|
#8 tpool::thread_pool_generic::timer_generic::execute (arg=0x565522c552a0) at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:363
|
#9 0x00005655214ad29b in tpool::task::execute (this=0x565522c552e0) at /usr/src/debug/MariaDB-/src_0/tpool/task.cc:37
|
#10 0x00005655214aa6d1 in tpool::thread_pool_generic::worker_main (this=0x565522a65670, thread_var=0x565522a67fa0)
|
at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:580
|
#11 0x00007f3c7ed35330 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>)
|
at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
|
#12 0x00007f3c7f193ea5 in start_thread (arg=0x7f3ac3fff700) at pthread_create.c:307
|
#13 0x00007f3c7e6aeb0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
|
Attachments
Issue Links
- is caused by
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-