Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.5.13
-
MariaDB 10.5.13 , galera provider 26.4.10 ,FreeBSD 12 and 13 . ZFS storage, 1500G RAM , 64 or 96 Cores
Description
Hello ,
one of our Galera Nodes dies hours ago with following state:
2021-12-02 21:34:38 41 [ERROR] [FATAL] InnoDB: Page old data size 15457 new data size 15959, page old max ins size 528 new max ins size 26
|
211202 21:34:38 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.5.13-MariaDB-log
|
key_buffer_size=5242880
|
read_buffer_size=131072
|
max_used_connections=6
|
max_threads=1502
|
thread_count=102
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3310938 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x129b99cb318
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7fffdce99f30 thread_stack 0xc0000
|
2021-12-02 21:34:41 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.54933S), skipping check
|
0x132bb7c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
|
0xc8b250 <handle_fatal_signal+0x290> at /usr/local/libexec/mariadbd
|
2021-12-02 21:34:42 0 [Note] WSREP: (7422f571-baaf, 'tcp://10.10.70.221:4567') turning message relay requesting on, nonlive peers: tcp://10.10.70.211:4567 tcp://10.10.70.231:4567 tcp://10.10.70.241:4567 tcp://10.10.70.251:4567
|
2021-12-02 21:34:42 0 [Note] WSREP: (7422f571-baaf, 'tcp://10.10.70.221:4567') connection established to 32fb230c-8d6c tcp://10.10.70.251:4567
|
2021-12-02 21:34:42 0 [Note] WSREP: (7422f571-baaf, 'tcp://10.10.70.221:4567') connection established to 98597714-a0b1 tcp://10.10.70.211:4567
|
2021-12-02 21:34:42 0 [Note] WSREP: (7422f571-baaf, 'tcp://10.10.70.221:4567') connection established to b764b1d2-b98b tcp://10.10.70.231:4567
|
2021-12-02 21:34:42 0 [Note] WSREP: (7422f571-baaf, 'tcp://10.10.70.221:4567') connection established to b82f6161-9027 tcp://10.10.70.241:4567
|
0x801924b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x1c9a48adc4): INSERT INTO QUERY SCRABLED
|
|
Connection ID (thread ID): 41
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Core pattern: %N.core
|
If we try to join the back to the cluster the same behavior repeat .
First we have the same situation with node which was GTID replica slave of this one.
What can we to supply more useful information? The core dump file is 150G