Details
-
New Feature
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Today, 2 out of 3 nodes of our production Galera cluster crashed simultaneously. It took about 20 minutes for the cluster to recover.
The cluster runs on a k8s cluster via mariadb-operator.
This was the log output of one node (the other node had almost identical output, at exactly the same time and with the same query):
[ERROR] mysqld got signal 11 ;
|
Sorry, we probably made a mistake, and this is a bug.
|
Your assistance in bug reporting will enable us to fix this for the next release.
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
Server version: 11.4.4-MariaDB-ubu2404 source revision: e9a502df08bad16aa8a354e854f3c014b1380e32
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=5
|
max_threads=153
|
thread_count=7
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468069 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
WSREP: Suppressing further logging
|
WSREP: Shutting down network communications
|
Thread pointer: 0x7ffb18000c68
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7ffb3c4ed000 thread_stack 0x49000
|
Printing to addr2line failed
|
mariadbd(my_print_stacktrace+0x32)[0x56064c644822]
|
mariadbd(handle_fatal_signal+0x2c3)[0x56064c0ef603]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x45320)[0x7ffb3e7ea320]
|
mariadbd(+0xefaaeb)[0x56064c4caaeb]
|
mariadbd(+0xe25a37)[0x56064c3f5a37]
|
mariadbd(+0xf0b8e6)[0x56064c4db8e6]
|
mariadbd(+0xf0cb37)[0x56064c4dcb37]
|
mariadbd(+0xf0e1f8)[0x56064c4de1f8]
|
mariadbd(+0xee6b9b)[0x56064c4b6b9b]
|
mariadbd(+0xecff8f)[0x56064c49ff8f]
|
mariadbd(+0xf08e1a)[0x56064c4d8e1a]
|
mariadbd(+0xf0c7c5)[0x56064c4dc7c5]
|
mariadbd(+0xf0e1f8)[0x56064c4de1f8]
|
mariadbd(+0xee5982)[0x56064c4b5982]
|
mariadbd(+0xe2bbe5)[0x56064c3fbbe5]
|
mariadbd(_ZN7handler13ha_update_rowEPKhS1_+0x2aa)[0x56064c1005aa]
|
mariadbd(_ZN21Update_rows_log_event11do_exec_rowEP14rpl_group_info+0x3d4)[0x56064c250a74]
|
mariadbd(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x38f)[0x56064c243a5f]
|
mariadbd(_ZN9Log_event11apply_eventEP14rpl_group_info+0x81)[0x56064c23ba61]
|
mariadbd(_Z18wsrep_apply_eventsP3THDP14Relay_log_infoPKvm+0xfe)[0x56064c3c771e]
|
mariadbd(_ZN21Wsrep_applier_service15apply_write_setERKN5wsrep7ws_metaERKNS0_12const_bufferERNS0_14mutable_bufferE+0xb5)[0x56064c3aa915]
|
mariadbd(+0x112fc30)[0x56064c6ffc30]
|
mariadbd(+0x1141493)[0x56064c711493]
|
/usr/lib/galera/libgalera_smm.so(+0x6194e)[0x7ffb3de6194e]
|
/usr/lib/galera/libgalera_smm.so(+0x6895a)[0x7ffb3de6895a]
|
/usr/lib/galera/libgalera_smm.so(+0x73c01)[0x7ffb3de73c01]
|
/usr/lib/galera/libgalera_smm.so(+0x97843)[0x7ffb3de97843]
|
/usr/lib/galera/libgalera_smm.so(+0x6a779)[0x7ffb3de6a779]
|
/usr/lib/galera/libgalera_smm.so(+0x51160)[0x7ffb3de51160]
|
mariadbd(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x56064c711a42]
|
mariadbd(+0xdf9711)[0x56064c3c9711]
|
mariadbd(_Z15start_wsrep_THDPv+0x254)[0x56064c3b7194]
|
mariadbd(+0xd696f8)[0x56064c3396f8]
|
/lib/x86_64-linux-gnu/libc.so.6(+0x9ca94)[0x7ffb3e841a94]
|
/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x7ffb3e8cea34]
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7ffb33bdd063): DELETE FROM `kind`
|
WHERE `id_kind` IN ('13')
|
Connection ID (thread ID): 2
|
Status: NOT_KILLED
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=on,cset_narrowing=off,sargable_casefold=on
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 256044 256044 processes
|
Max open files 1048576 1048576 files
|
Max locked memory 8388608 8388608 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 256044 256044 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
Kernel version: Linux version 6.6.60-talos (@buildkitsandbox) (gcc (GCC) 13.3.0, GNU ld (GNU Binutils) 2.43.1) #1 SMP Tue Nov 12 16:00:51 UTC 2024
|
Please let me know if you need more information.