[MDEV-31249] sudden signal 11 after shutdown Created: 2023-05-12  Updated: 2023-06-09  Resolved: 2023-05-25

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6.9
Fix Version/s: 10.6.10, 10.9.3, 10.10.2

Type: Bug Priority: Critical
Reporter: Richard Stracke Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 0
Labels: None

Attachments: Text File stack_mariadb_core.txt    
Issue Links:
Duplicate
duplicates MDEV-27983 InnoDB hangs on multiple concurrent r... Closed

 Description    

2023-05-11 14:00:57 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
 
2023-05-11 14:00:57 1 [Note] Stopping ack receiver thread
 
2023-05-11 14:01:17 0 [Warning] /usr/sbin/mariadbd: Thread 118516 (user : 'xxxx') did not exit
 
 
 
2023-05-11 14:01:17 0 [Warning] /usr/sbin/mariadbd: Thread 118487 (user : 'xxxx') did not exit
 
 
 
230511 14:17:40 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.6.9-5-MariaDB-enterprise-log
 
key_buffer_size=67108864
 
read_buffer_size=131072
 
max_used_connections=76
 
max_threads=3002
 
thread_count=2
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6676289 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x40000
 
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5625f1296cce]
 
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5625f0d31705]
 
sigaction.c:0(__restore_rt)[0x7ff52e896630]
 
:0(__select_nocancel)[0x7ff52dda0a33]
 
/usr/sbin/mariadbd(my_sleep+0x3f)[0x5625f12949ef]
 
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x187a)[0x5625f0a2a61a]
 
??:0(__libc_start_main)[0x7ff52dccd555]
 
/usr/sbin/mariadbd(+0x6f0f43)[0x5625f0a1df43]
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /mariadb/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             1031060              1031060              processes 
Max open files            278528               278528               files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       1031060              1031060              signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
 
 
Kernel version: Linux version 3.10.0-1160.80.1.el7.x86_64 (mockbuild@x86-vm-41.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Oct 8 18:13:21 UTC 2022
 
 
 
2023-05-11 14:19:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.7

stacktrace is not very verbose, but a full stacktrace from core is attached,

 Comments   
Comment by Daniel Black [ 2023-05-12 ]

Looks like select query in thread 8 got all the way down to buf_page_get_low and after some time ~16mins, and externally applied SIGSEGV hit one of the few my_sleep functions in close_connections(). Alternately the SEGV may also be MDEV-30613 (now fixed) and it really was another signal happening (e.g. wait_for_signal_thread_to_end).

So this like a largely a failure to close off the innodb buf_page_get_low during shutdown (tpool appears still to be running).

Comment by Marko Mäkelä [ 2023-05-25 ]

It seems to me that an external process killed the mariadbd process with SIGSEGV (signal 11) due to some timeout being exceeded. In stack_mariadb_core.txt, the signal is received inside my_sleep(), invoked by mysqld_main().

I see that Thread 4 and Thread 8 are active in InnoDB code. Thread 8 is waiting for a page latch 0x7fd6523967c0 on the block descriptor 0x7fd6523967a8 (0x18 bytes less). Thread 4 seems to be be busy in a polling loop. It is difficult to say if it is operating on the same buffer block, but let me assume that it is. Let us look at the code corresponding to this:

mariadb-10.6.9

 
Thread 4 (Thread 0x7ff5300ec700 (LWP 30700)):
 
#0  0x00007ff52e895e9d in nanosleep () from /lib64/libpthread.so.0
 
No symbol table info available.
 
#1  0x00005625f0a19acb in sleep_for<long, std::ratio<1, 1000000> > (__rtime=..., __rtime=...)
 
    at /opt/rh/devtoolset-10/root/usr/include/c++/10/thread:401
 
        __ts = {tv_sec = 0, tv_nsec = 81712}
 
#2  buf_page_get_low(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, mtr_t*, dberr_t*, bool) ()
 
    at /usr/src/debug/MariaDB-/src_0/storage/innobase/buf/buf0buf.cc:2630


                        mysql_mutex_unlock(&buf_pool.mutex);
 
                        hash_lock.unlock();
 
                        std::this_thread::sleep_for(
 
                                std::chrono::microseconds(100));
 
                        goto wait_for_unfix;

This code may be executed when ROW_FORMAT=COMPRESSED tables are being accessed. This report is a duplicate of MDEV-27983, a bug that was introduced in MariaDB Server 10.6.6 and fixed in 10.6.10.

Generated at Thu Feb 08 10:22:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.