Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
10.3.27, 10.3(EOL), 10.4(EOL), 10.5
-
None
Description
Operating system: CentOS 7
Yum repository: http://yum.mariadb.org/10.3/centos7-amd64
CPU & mem: 2x Intel Gold 6128 CPU, 256G memory
After upgrading from 10.3.21 to 10.3.27 mariadb started crashing after around one day of uptime. After a few crashes a pattern emerged: it would crash after exactly 1 day and 5 minutes
2020-11-16 11:13:53 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-17 11:18:54 [ERROR] mysqld got signal 11 ;
|
|
|
2020-11-17 11:19:28 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-18 11:24:28 [ERROR] mysqld got signal 11 ;
|
|
|
2020-11-18 11:25:01 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-19 11:30:01 [ERROR] mysqld got signal 11 ;
|
|
|
2020-11-19 11:30:36 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-20 11:35:36 [ERROR] mysqld got signal 11 ;
|
|
|
2020-11-20 11:36:22 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-21 11:41:23 [ERROR] mysqld got signal 11 ;
|
|
|
2020-11-21 11:42:12 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-22 11:47:12 [ERROR] mysqld got signal 11 ;
|
|
|
2020-11-22 11:47:58 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-23 11:52:58 [ERROR] mysqld got signal 11 ;
|
|
|
2020-11-24 8:59:17 0 [Note] /usr/sbin/mysqld: ready for connections.
|
2020-11-25 9:04:17 [ERROR] mysqld got signal 11 ;
|
The stacktrace is the same for every segfault:
Thread pointer: 0x7f636c0eae58
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f6391b1a080 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e20502266e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e204ab9e7f]
|
/lib64/libpthread.so.0(+0xf630)[0x7f857b611630]
|
/usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x39)[0x55e2049421a9]
|
/usr/sbin/mysqld(+0x63c2fa)[0x55e2049462fa]
|
/usr/sbin/mysqld(_Z14fill_variablesP3THDP10TABLE_LISTP4Item+0x126)[0x55e204949706]
|
/usr/sbin/mysqld(+0xcd6fd8)[0x55e204fe0fd8]
|
/usr/sbin/mysqld(+0x4f4bc9)[0x55e2047febc9]
|
/usr/sbin/mysqld(+0xcd717a)[0x55e204fe117a]
|
pthread_create.c:0(start_thread)[0x7f857b609ea5]
|
/lib64/libc.so.6(clone+0x6d)[0x7f85799a996d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0): (null)
|
Connection ID (thread ID): 6
|
Status: NOT_KILLED
|
Seeing the low connection id it looks like a system thread.
The full latest crash log
201125 9:04:17 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.3.27-MariaDB
|
key_buffer_size=1610612736
|
read_buffer_size=4194304
|
max_used_connections=95
|
max_threads=452
|
thread_count=73
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7137201 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f636c0eae58
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f6391b1a080 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e20502266e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e204ab9e7f]
|
/lib64/libpthread.so.0(+0xf630)[0x7f857b611630]
|
/usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x39)[0x55e2049421a9]
|
/usr/sbin/mysqld(+0x63c2fa)[0x55e2049462fa]
|
/usr/sbin/mysqld(_Z14fill_variablesP3THDP10TABLE_LISTP4Item+0x126)[0x55e204949706]
|
/usr/sbin/mysqld(+0xcd6fd8)[0x55e204fe0fd8]
|
/usr/sbin/mysqld(+0x4f4bc9)[0x55e2047febc9]
|
/usr/sbin/mysqld(+0xcd717a)[0x55e204fe117a]
|
pthread_create.c:0(start_thread)[0x7f857b609ea5]
|
/lib64/libc.so.6(clone+0x6d)[0x7f85799a996d]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0): (null)
|
Connection ID (thread ID): 6
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
|
|
We think the query pointer is invalid, but we will try to print it anyway.
|
Query:
|
|
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 1028777 1028777 processes
|
Max open files 16384 16384 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 1028777 1028777 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-