[MDEV-20446] It can not re-joining Galera Cluster when the DB receives signal 11 and restart mysqld service Created: 2019-08-29  Updated: 2019-10-27  Resolved: 2019-10-27

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.1.24
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Hank Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: galera, need_feedback
Environment:

CentOS Linux release 7.3.1611 (Core) on AWS EC2
MariaDB 10.1.24


Attachments: File MariaDB02-mysql-bin.err.20190824     File server.cnf    
Issue Links:
Relates
relates to MDEV-7580 possible memory leak Closed

 Description   

in our prod environment
we have 3 nodes galera cluster (2 DB , 1 arbitrator grabd)
we meet a node receive signal 11 and auto restart mysqld service but It can not re-joinging Galera Cluster and hang at joining status



 Comments   
Comment by Elena Stepanova [ 2019-08-29 ]

From the error log:

/usr/sbin/mysqld(handle_fatal_signal+0x305) [0x7f775765c0a5]
/lib64/libpthread.so.0(+0xf370) [0x7f7756c75370]
/lib64/libc.so.6(fgets+0x1b) [0x7f775500034b]
/usr/lib64/mysql/plugin/lib_mysqludf_sys.so(sys_eval+0xe1) [0x7f7747d97377]
/usr/sbin/mysqld(udf_handler::val_str(String*, String*)+0x78) [0x7f77576c04b8]
/usr/sbin/mysqld(Item_func_udf_str::val_str(String*)+0x1c) [0x7f77576c057c]
/usr/sbin/mysqld(Item::save_in_field(Field*, bool)+0x6f) [0x7f775766c01f]
/usr/sbin/mysqld(sp_eval_expr(THD*, Field*, Item**)+0x72) [0x7f775776cb42]
/usr/sbin/mysqld(sp_rcontext::set_variable(THD*, Field*, Item**)+0x17) [0x7f77577751a7]
/usr/sbin/mysqld(sp_instr_set::exec_core(THD*, unsigned int*)+0x36) [0x7f775776ace6]
/usr/sbin/mysqld(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*)+0x2fa) [0x7f7757771aca]
/usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x786) [0x7f775776dd46]
/usr/sbin/mysqld(sp_head::execute_function(THD*, Item**, unsigned int, Field*)+0x2ea) [0x7f775776e66a]
/usr/sbin/mysqld(Item_func_sp::execute_impl(THD*)+0x109) [0x7f77576c4229]
/usr/sbin/mysqld(Item_func_sp::execute()+0x26) [0x7f77576c4306]
/usr/sbin/mysqld(Item_func_sp::val_str(String*)+0x57) [0x7f77576cacc7]
/usr/sbin/mysqld(Item::save_in_field(Field*, bool)+0x6f) [0x7f775766c01f]
/usr/sbin/mysqld(sp_eval_expr(THD*, Field*, Item**)+0x72) [0x7f775776cb42]
/usr/sbin/mysqld(sp_rcontext::set_variable(THD*, Field*, Item**)+0x17) [0x7f77577751a7]
/usr/sbin/mysqld(sp_instr_set::exec_core(THD*, unsigned int*)+0x36) [0x7f775776ace6]
/usr/sbin/mysqld(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*)+0x2fa) [0x7f7757771aca]
/usr/sbin/mysqld(sp_head::execute(THD*, bool)+0x786) [0x7f775776dd46]
/usr/sbin/mysqld(sp_head::execute_trigger(THD*, st_mysql_lex_string const*, st_mysql_lex_string const*, st_grant_info*)+0x195) [0x7f775776e225]
/usr/sbin/mysqld(Table_triggers_list::process_triggers(THD*, trg_event_type, trg_action_time_type, bool)+0x134) [0x7f775756a974]
/usr/sbin/mysqld(fill_record_n_invoke_before_triggers(THD*, TABLE*, List<Item>&, List<Item>&, bool, trg_event_type)+0x5b) [0x7f77574995db]
/usr/sbin/mysqld(mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, enum_duplicates, bool, unsigned long long*, unsigned long long*)+0x1257) [0x7f7757576407]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x2ada) [0x7f77574d7caa]
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x332) [0x7f77574de592]
/usr/sbin/mysqld(+0x43adb9) [0x7f77574dedb9]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1f85) [0x7f77574e13f5]
/usr/sbin/mysqld(do_command(THD*)+0x14a) [0x7f77574e22ca]
/usr/sbin/mysqld(do_handle_one_connection(THD*)+0x18a) [0x7f77575aa39a]
/usr/sbin/mysqld(handle_one_connection+0x40) [0x7f77575aa540]
/lib64/libpthread.so.0(+0x7dc5) [0x7f7756c6ddc5]
/lib64/libc.so.6(clone+0x6d) [0x7f775508c76d]

Comment by Elena Stepanova [ 2019-09-23 ]

So, according to the stack trace, it crashes in the UDF function.
I'll leave the "not rejoining" part to jplindst.

Comment by Jan Lindström (Inactive) [ 2019-09-25 ]

Can you please try with more recent version of MariaDB 10.1 e.g. 10.1.41.

Comment by Hank [ 2019-09-25 ]

i have added ram and runing fine currently but i still not understand why can not re-joining to galera cluster when mysqld restart .
this case can closed and thank your help .

Generated at Thu Feb 08 08:59:32 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.