Details
Description
We have two servers running MariaDB with galera for replication. Every few weeks we get alerts that MariaDB has crashed with a segfault. We were on older 10.X versions MariaDB ( https://serverfault.com/questions/1016977/mariadb-crashing ) and had the same issues. I am not sure if it is a specific query that is causing MariaDB to crash or an issue elsewhere. Below is what I am seeing with a back trace
[root@mon2 ccpp-2020-11-05-08:55:49-43159]# gdb /usr/libexec/mysqld coredump
|
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-119.el7
|
Copyright (C) 2013 Free Software Foundation, Inc.
|
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
This is free software: you are free to change and redistribute it.
|
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
|
and "show warranty" for details.
|
This GDB was configured as "x86_64-redhat-linux-gnu".
|
For bug reporting instructions, please see:
|
<http://www.gnu.org/software/gdb/bugs/>...
|
Reading symbols from /usr/libexec/mysqld...Reading symbols from /usr/libexec/mysqld...(no debugging symbols found)...done.
|
(no debugging symbols found)...done.
|
[New LWP 13153]
|
[New LWP 43164]
|
[New LWP 43163]
|
[New LWP 44190]
|
[New LWP 44193]
|
[New LWP 26180]
|
[New LWP 44196]
|
[New LWP 44051]
|
[New LWP 44186]
|
[New LWP 44194]
|
[New LWP 44192]
|
[New LWP 44189]
|
[New LWP 44188]
|
[New LWP 44208]
|
[New LWP 44195]
|
[New LWP 44191]
|
[New LWP 44200]
|
[New LWP 43159]
|
[New LWP 43166]
|
[New LWP 43162]
|
[New LWP 43161]
|
[New LWP 44187]
|
[New LWP 43165]
|
[New LWP 44209]
|
[New LWP 44210]
|
[New LWP 44211]
|
[New LWP 44197]
|
[New LWP 44216]
|
[New LWP 44215]
|
[New LWP 44198]
|
[New LWP 44214]
|
[New LWP 44217]
|
[New LWP 44218]
|
[New LWP 44212]
|
[New LWP 44224]
|
[New LWP 44261]
|
[New LWP 44213]
|
[New LWP 44207]
|
[New LWP 44223]
|
[New LWP 44222]
|
[New LWP 44220]
|
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib64/libthread_db.so.1".
|
Core was generated by `/usr/libexec/mysqld --basedir=/usr'.
|
Program terminated with signal 11, Segmentation fault.
|
#0 0x00007f4375bac0b8 in ?? () from /lib64/libgcc_s.so.1
|
Missing separate debuginfos, use: debuginfo-install mariadb103-server-10.3.21-2.el7.ius.x86_64
|
(gdb) bt
|
#0 0x00007f4375bac0b8 in ?? () from /lib64/libgcc_s.so.1
|
#1 0x00007f4375bacfb9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
|
#2 0x00007f4376fefaa6 in backtrace () from /lib64/libc.so.6
|
#3 0x000055d479977c3d in my_print_stacktrace ()
|
#4 0x000055d479458637 in handle_fatal_signal ()
|
#5 <signal handler called>
|
#6 0x0000000000000051 in ?? ()
|
#7 0x000055d4794c3fc5 in Item_func_release_lock::val_int() ()
|
#8 0x000055d4791d41fc in Item::update_null_value() ()
|
#9 0x000055d47923d215 in Item_func::is_null() ()
|
#10 0x000055d47959bde9 in mysql_do(THD*, List<Item>&) ()
|
#11 0x000055d47927ea66 in mysql_execute_command(THD*) ()
|
#12 0x000055d4791ead76 in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
|
#13 0x000055d4791f2949 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
|
#14 0x000055d4791f337c in sp_instr_stmt::execute(THD*, unsigned int*) ()
|
#15 0x000055d4791ee6c0 in sp_head::execute(THD*, bool) ()
|
#16 0x000055d4791ef91d in sp_head::execute_procedure(THD*, List<Item>*) ()
|
#17 0x000055d479270df2 in do_execute_sp(THD*, sp_head*) ()
|
#18 0x000055d4792722e6 in Sql_cmd_call::execute(THD*) [clone .part.293] ()
|
#19 0x000055d479272b60 in Sql_cmd_call::execute(THD*) ()
|
#20 0x000055d47927c2b8 in mysql_execute_command(THD*) ()
|
#21 0x000055d47928120b in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) ()
|
#22 0x000055d479281b81 in wsrep_mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) ()
|
#23 0x000055d479283306 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) ()
|
#24 0x000055d479284cae in do_command(THD*) ()
|
#25 0x000055d4793577d1 in do_handle_one_connection(CONNECT*) ()
|
#26 0x000055d47935789d in handle_one_connection ()
|
#27 0x00007f4378e4dea5 in start_thread () from /lib64/libpthread.so.0
|
#28 0x00007f4376fd98dd in clone () from /lib64/libc.so.6
|
Attached all the traces besides the backtrace (it's 4.3GB) and the hot name file. We do have all queries stored so we can pull them for the time of the crash if needed.
Attachments
Issue Links
- relates to
-
MDEV-17547 MariaDB 10.2 and Galera crashing with segfault
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
Even though there is no answer to your question from the reporter, I assume you still want to handle the crash, so I'm not closing it as incomplete.
Found similar crash with slightly different stack.
Test case
CREATE TABLE t1 (c1 BIGINT NOT NULL PRIMARY KEY, c2 BINARY (10), c3 DATETIME); |
SELECT get_lock ('test2', 0); |
DROP TABLE t1; |
CREATE TABLE t1 (c1 SMALLINT NOT NULL AUTO_INCREMENT PRIMARY KEY); |
INSERT INTO t1 VALUES (1); |
SET SESSION wsrep_trx_fragment_size=10; |
SET SESSION autocommit=0; |
SELECT * FROM t1 WHERE c1 <=0 ORDER BY c1 DESC; |
INSERT INTO t1 VALUES (4),(3),(1),(2); |
CREATE TABLE t1 (pk INT PRIMARY KEY, b INT) ENGINE=SEQUENCE; |
ALTER TABLE t1 DROP COLUMN c2; |
SELECT get_lock ('test', 1.5); |
|
10.5.14 fb40a2fabf8d8cf765c83a0b8e609dd893c75ec3 (Optimized) |
Core was generated by `/test/GAL_MD030222-mariadb-10.5.14-linux-x86_64-opt/bin/mysqld --defaults-file='.
|
Program terminated with signal SIGSEGV, Segmentation fault.
|
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=11)
|
at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
|
[Current thread is 1 (Thread 0x145bd0e49700 (LWP 4029129))]
|
(gdb) bt
|
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
|
#1 0x000055f1f65ed96f in my_write_core (sig=sig@entry=11) at /test/10.5_opt/mysys/stacktrace.c:424
|
#2 0x000055f1f600ad10 in handle_fatal_signal (sig=11) at /test/10.5_opt/sql/signal_handler.cc:344
|
#3 <signal handler called>
|
#4 ull_get_key (ptr=<optimized out>, length=0x145bd0e470d8, not_used=<optimized out>) at /test/10.5_opt/sql/mdl.h:398
|
#5 0x000055f1f65ccf25 in my_hash_key (first=1 '\001', length=0x145bd0e470d8, record=<optimized out>, hash=0x145b54003068) at /test/10.5_opt/mysys/hash.c:196
|
#6 hashcmp (pos=0x145b54021cf8, length=7, key=0x145bd0e47198 "\btest", hash=0x145b54003068) at /test/10.5_opt/mysys/hash.c:371
|
#7 my_hash_first_from_hash_value (hash=hash@entry=0x145b54003068, hash_value=<optimized out>, key=key@entry=0x145bd0e47198 "\btest", length=7, current_record=current_record@entry=0x145bd0e4712c) at /test/10.5_opt/mysys/hash.c:288
|
#8 0x000055f1f65cd01d in my_hash_first (hash=hash@entry=0x145b54003068, key=key@entry=0x145bd0e47198 "\btest", length=<optimized out>, current_record=current_record@entry=0x145bd0e4712c) at /test/10.5_opt/mysys/hash.c:262
|
#9 0x000055f1f65cd035 in my_hash_search (hash=hash@entry=0x145b54003068, key=key@entry=0x145bd0e47198 "\btest", length=<optimized out>) at /test/10.5_opt/mysys/hash.c:235
|
#10 0x000055f1f607622a in Item_func_get_lock::val_int (this=0x145b54010b10) at /test/10.5_opt/sql/mdl.h:398
|
#11 0x000055f1f5f7122d in Type_handler::Item_send_long (this=<optimized out>, item=0x145b54010b10, protocol=0x145b540011a8, buf=<optimized out>) at /test/10.5_opt/sql/sql_type.cc:7487
|
#12 0x000055f1f5d2e810 in Protocol::send_result_set_row (this=this@entry=0x145b540011a8, row_items=row_items@entry=0x145b540105e8) at /test/10.5_opt/sql/protocol.cc:1083
|
#13 0x000055f1f5da2367 in select_send::send_data (this=0x145b54011518, items=@0x145b540105e8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x145b54010bf8, last = 0x145b54010bf8, elements = 1}, <No data fields>}) at /test/10.5_opt/sql/sql_class.cc:3081
|
#14 0x000055f1f5e62518 in select_result_sink::send_data_with_check (u=<optimized out>, sent=0, items=<optimized out>, this=<optimized out>) at /test/10.5_opt/sql/sql_class.h:5342
|
#15 select_result_sink::send_data_with_check (sent=0, u=<optimized out>, items=<optimized out>, this=<optimized out>) at /test/10.5_opt/sql/sql_class.h:5332
|
#16 JOIN::exec_inner (this=0x145b54011540) at /test/10.5_opt/sql/sql_select.cc:4384
|
#17 0x000055f1f5e62919 in JOIN::exec (this=this@entry=0x145b54011540) at /test/10.5_opt/sql/sql_select.cc:4296
|
#18 0x000055f1f5e608da in mysql_select (thd=0x145b54000c58, tables=0x0, fields=@0x145b540105e8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x145b54010bf8, last = 0x145b54010bf8, elements = 1}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x145b54011518, unit=0x145b54004c40, select_lex=0x145b54010498) at /test/10.5_opt/sql/sql_select.cc:4773
|
#19 0x000055f1f5e612c7 in handle_select (thd=thd@entry=0x145b54000c58, lex=lex@entry=0x145b54004b78, result=result@entry=0x145b54011518, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.5_opt/sql/sql_select.cc:444
|
#20 0x000055f1f5deda31 in execute_sqlcom_select (thd=0x145b54000c58, all_tables=0x0) at /test/10.5_opt/sql/sql_parse.cc:6314
|
#21 0x000055f1f5dfc79b in mysql_execute_command (thd=0x145b54000c58) at /test/10.5_opt/sql/sql_parse.cc:4005
|
#22 0x000055f1f5de7fbf in mysql_parse (thd=thd@entry=0x145b54000c58, rawbuf=rawbuf@entry=0x145b54010400 "SELECT get_lock ('test', 1.5)", length=length@entry=29, parser_state=parser_state@entry=0x145bd0e48410, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /test/10.5_opt/sql/sql_parse.cc:8100
|
#23 0x000055f1f5de7729 in wsrep_mysql_parse (thd=0x145b54000c58, rawbuf=0x145b54010400 "SELECT get_lock ('test', 1.5)", length=29, parser_state=0x145bd0e48410, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_parse.cc:7903
|
#24 0x000055f1f5df684a in dispatch_command (command=COM_QUERY, thd=0x145b54000c58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_class.h:1290
|
#25 0x000055f1f5df772c in do_command (thd=0x145b54000c58) at /test/10.5_opt/sql/sql_parse.cc:1370
|
#26 0x000055f1f5eff631 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55f1f824b448, put_in_cache=put_in_cache@entry=true) at /test/10.5_opt/sql/sql_connect.cc:1418
|
#27 0x000055f1f5effaad in handle_one_connection (arg=arg@entry=0x55f1f824b448) at /test/10.5_opt/sql/sql_connect.cc:1312
|
#28 0x000055f1f6291cef in pfs_spawn_thread (arg=0x55f1f8262d98) at /test/10.5_opt/storage/perfschema/pfs.cc:2201
|
#29 0x0000145be14dc609 in start_thread (arg=<optimized out>) at pthread_create.c:477
|
#30 0x0000145be10ca293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
Bug confirmed present in:
MariaDB: 10.5.14 (dbg), 10.5.14 (opt), 10.6.6 (dbg), 10.6.6 (opt), 10.7.2 (dbg), 10.7.2 (opt), 10.8.1 (dbg)
Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.2.42 (dbg), 10.2.42 (opt), 10.3.33 (dbg), 10.3.33 (opt), 10.4.23 (dbg), 10.4.23 (opt)
It looks that root cause of this segfault is related to https://jira.mariadb.org/browse/MDEV-27713 , current fix is to handle properly clean thread ull structures after BF abort is triggered.
Please retest with fix found on related ticket.
I encountered the same issue.
Here is my case:
MariaDB 10.5.13
Debian 11
Galera 4.10
Steps to reproduce:
On node 1:
START TRANSACTION; |
SELECT GET_LOCK('name', 5); |
Exit.
|
The database crashes with signal 11.
This issue has been fixed since this commit.
The stored procedure has get_lock() / release_lock() function calls, and these are not supported in galera replication (also noted in KB limitations). However, although not safe, they probably should nevertheless work in topologies where all writes go to same dedicated node, i.e. cluster write conflict would not happen. Dovid does your application/load balancer direct all writes to same node?
A crash is not a good reaction to use of non supported feature, so some work remains for fixing this bug. Preferably by rejecting the use of get_lock() & release_lock() functions. But, this does not help this application's use case. Dovid is it possible to change the stored procedure definition to not use these functions?
It could be possible to support locking functions as new feature, e.g. streaming replication might be helpful technology for it.