[MDEV-31249] sudden signal 11 after shutdown - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Duplicate
Affects Version/s: 10.6.9
Fix Version/s: 10.6.10, 10.9.3, 10.10.2
Component/s: Storage Engine - InnoDB
Labels:
None

Description

2023-05-11 14:00:57 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown

2023-05-11 14:00:57 1 [Note] Stopping ack receiver thread

2023-05-11 14:01:17 0 [Warning] /usr/sbin/mariadbd: Thread 118516 (user : 'xxxx') did not exit

2023-05-11 14:01:17 0 [Warning] /usr/sbin/mariadbd: Thread 118487 (user : 'xxxx') did not exit

230511 14:17:40 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.6.9-5-MariaDB-enterprise-log

key_buffer_size=67108864

read_buffer_size=131072

max_used_connections=76

max_threads=3002

thread_count=2

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6676289 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x0 thread_stack 0x40000

/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5625f1296cce]

/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5625f0d31705]

sigaction.c:0(__restore_rt)[0x7ff52e896630]

:0(__select_nocancel)[0x7ff52dda0a33]

/usr/sbin/mariadbd(my_sleep+0x3f)[0x5625f12949ef]

/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x187a)[0x5625f0a2a61a]

??:0(__libc_start_main)[0x7ff52dccd555]

/usr/sbin/mariadbd(+0x6f0f43)[0x5625f0a1df43]

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /mariadb/data

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        unlimited            unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             1031060              1031060              processes

Max open files            278528               278528               files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       1031060              1031060              signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Core pattern: core

Kernel version: Linux version 3.10.0-1160.80.1.el7.x86_64 (mockbuild@x86-vm-41.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Oct 8 18:13:21 UTC 2022

2023-05-11 14:19:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.7

stacktrace is not very verbose, but a full stacktrace from core is attached,

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

stack_mariadb_core.txt
941 kB
2023-05-12 06:48

Issue Links

duplicates

MDEV-27983 InnoDB hangs on multiple concurrent requests of a cold ROW_FORMAT=COMPRESSED page

Closed

Activity

Ascending order - Click to sort in descending order

Richard Stracke created issue - 2023-05-12 06:47

Richard Stracke made changes - 2023-05-12 06:48

Field	Original Value	New Value
Attachment		stack_mariadb_core.txt [ 70044 ]

Julien Fritsch made changes - 2023-05-12 06:53

Fix Version/s		10.4 [ 22408 ]
Fix Version/s		10.5 [ 23123 ]
Fix Version/s		10.6 [ 24028 ]

Julien Fritsch made changes - 2023-05-12 07:41

Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]

Julien Fritsch made changes - 2023-05-12 07:45

Assignee

Elena Stepanova [ elenst ]

Elena Stepanova made changes - 2023-05-12 08:20

Assignee

Elena Stepanova [ elenst ]

Daniel Black added a comment - 2023-05-12 08:24

Looks like select query in thread 8 got all the way down to buf_page_get_low and after some time ~16mins, and externally applied SIGSEGV hit one of the few my_sleep functions in close_connections(). Alternately the SEGV may also be ~~MDEV-30613~~ (now fixed) and it really was another signal happening (e.g. wait_for_signal_thread_to_end).

So this like a largely a failure to close off the innodb buf_page_get_low during shutdown (tpool appears still to be running).

Daniel Black added a comment - 2023-05-12 08:24 Looks like select query in thread 8 got all the way down to buf_page_get_low and after some time ~16mins, and externally applied SIGSEGV hit one of the few my_sleep functions in close_connections(). Alternately the SEGV may also be MDEV-30613 (now fixed) and it really was another signal happening (e.g. wait_for_signal_thread_to_end). So this like a largely a failure to close off the innodb buf_page_get_low during shutdown (tpool appears still to be running).

Elena Stepanova made changes - 2023-05-12 08:46

Comment

[ I'm not sure what's reported here.
There are two error logs attached to the support issue.
One demonstrates an obvious repeated crash upon execution of recursive CTE (but old one, in April).
Another one contains a single SIGSEGV on shutdown. At the first glance, it looks like the server was killed rather than crashed (possibly because of a real or perceived hang), a new one, of yesterday.
I don't see any relation between the two problems, but the stack trace seems to relate to the 2nd one.
]

Julien Fritsch made changes - 2023-05-12 09:13

Assignee

Marko Mäkelä [ marko ]

Julien Fritsch made changes - 2023-05-24 15:39

Priority

Major [ 3 ]

Critical [ 2 ]

Marko Mäkelä added a comment - 2023-05-25 08:43

It seems to me that an external process killed the mariadbd process with SIGSEGV (signal 11) due to some timeout being exceeded. In stack_mariadb_core.txt, the signal is received inside my_sleep(), invoked by mysqld_main().

I see that Thread 4 and Thread 8 are active in InnoDB code. Thread 8 is waiting for a page latch 0x7fd6523967c0 on the block descriptor 0x7fd6523967a8 (0x18 bytes less). Thread 4 seems to be be busy in a polling loop. It is difficult to say if it is operating on the same buffer block, but let me assume that it is. Let us look at the code corresponding to this:

mariadb-10.6.9
Thread 4 (Thread 0x7ff5300ec700 (LWP 30700)):
#0 0x00007ff52e895e9d in nanosleep () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00005625f0a19acb in sleep_for<long, std::ratio<1, 1000000> > (__rtime=..., __rtime=...)
at /opt/rh/devtoolset-10/root/usr/include/c++/10/thread:401
__ts = {tv_sec = 0, tv_nsec = 81712}
#2 buf_page_get_low(page_id_t, unsigned long, unsigned long, buf_block_t, unsigned long, mtr_t, dberr_t*, bool) ()
at /usr/src/debug/MariaDB-/src_0/storage/innobase/buf/buf0buf.cc:2630

                        mysql_mutex_unlock(&buf_pool.mutex);

                        hash_lock.unlock();

                        std::this_thread::sleep_for(

                                std::chrono::microseconds(100));

                        goto wait_for_unfix;

This code may be executed when ROW_FORMAT=COMPRESSED tables are being accessed. This report is a duplicate of ~~MDEV-27983~~, a bug that was introduced in MariaDB Server 10.6.6 and fixed in 10.6.10.

Marko Mäkelä added a comment - 2023-05-25 08:43 It seems to me that an external process killed the mariadbd process with SIGSEGV (signal 11) due to some timeout being exceeded. In stack_mariadb_core.txt , the signal is received inside my_sleep() , invoked by mysqld_main() . I see that Thread 4 and Thread 8 are active in InnoDB code. Thread 8 is waiting for a page latch 0x7fd6523967c0 on the block descriptor 0x7fd6523967a8 ( 0x18 bytes less). Thread 4 seems to be be busy in a polling loop. It is difficult to say if it is operating on the same buffer block, but let me assume that it is. Let us look at the code corresponding to this: mariadb-10.6.9 Thread 4 (Thread 0x7ff5300ec700 (LWP 30700)): #0 0x00007ff52e895e9d in nanosleep () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00005625f0a19acb in sleep_for<long, std::ratio<1, 1000000> > (__rtime=..., __rtime=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/thread:401 __ts = {tv_sec = 0, tv_nsec = 81712} #2 buf_page_get_low(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, mtr_t*, dberr_t*, bool) () at /usr/src/debug/MariaDB-/src_0/storage/innobase/buf/buf0buf.cc:2630 mysql_mutex_unlock(&buf_pool.mutex); hash_lock.unlock(); std::this_thread::sleep_for( std::chrono::microseconds(100)); goto wait_for_unfix; This code may be executed when ROW_FORMAT=COMPRESSED tables are being accessed. This report is a duplicate of MDEV-27983 , a bug that was introduced in MariaDB Server 10.6.6 and fixed in 10.6.10.

Marko Mäkelä made changes - 2023-05-25 08:43

Link

This issue duplicates ~~MDEV-27983~~ [ ~~MDEV-27983~~ ]

Marko Mäkelä made changes - 2023-05-25 08:43

Description

{code:java}
2023-05-11 14:00:57 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
2023-05-11 14:00:57 1 [Note] Stopping ack receiver thread
2023-05-11 14:01:17 0 [Warning] /usr/sbin/mariadbd: Thread 118516 (user : 'xxxx') did not exit

2023-05-11 14:01:17 0 [Warning] /usr/sbin/mariadbd: Thread 118487 (user : 'xxxx') did not exit

230511 14:17:40 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.6.9-5-MariaDB-enterprise-log
key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=76
max_threads=3002
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6676289 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5625f1296cce]
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5625f0d31705]
sigaction.c:0(__restore_rt)[0x7ff52e896630]
:0(__select_nocancel)[0x7ff52dda0a33]
/usr/sbin/mariadbd(my_sleep+0x3f)[0x5625f12949ef]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x187a)[0x5625f0a2a61a]
??:0(__libc_start_main)[0x7ff52dccd555]
/usr/sbin/mariadbd(+0x6f0f43)[0x5625f0a1df43]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /mariadb/data
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1031060 1031060 processes
Max open files 278528 278528 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 1031060 1031060 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core

Kernel version: Linux version 3.10.0-1160.80.1.el7.x86_64 (mockbuild@x86-vm-41.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Oct 8 18:13:21 UTC 2022

2023-05-11 14:19:13 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
{code}

stacktrace is not very verbose, but a full stacktrace from core is attached,

Marko Mäkelä made changes - 2023-05-25 08:44

issue.field.resolutiondate

2023-05-25 08:44:19.0

2023-05-25 08:44:19.36

Marko Mäkelä made changes - 2023-05-25 08:44

Component/s		Storage Engine - InnoDB [ 10129 ]
Fix Version/s		10.10.2 [ 28410 ]
Fix Version/s		10.9.3 [ 28409 ]
Fix Version/s		10.6.10 [ 28407 ]
Fix Version/s	10.6 [ 24028 ]
Resolution		Duplicate [ 3 ]
Status	Open [ 1 ]	Closed [ 6 ]

Jira Automation (IT) made changes - 2024-07-04 01:12

Zendesk Related Tickets

149720

People

Assignee:: Marko Mäkelä

Reporter:: Richard Stracke

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 2023-05-12 06:47

Updated:: 2024-07-07 19:36

Resolved:: 2023-05-25 08:44

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.