Mysql segfault.

One of the mariadb processes in my 3 node galera cluster segfaulted whilst trying to apply the transaction from one of the other members. I can see from my application logs that this machine wasn't the originator of this transaction but it has segfaulted whilst applying it and the other machines are fine.

This node was already in a slightly odd state where any query involving "show status" from the mysql command line client just hangs forever and never returns to the CLI.

Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: 160711 13:26:33 [ERROR] mysqld got signal 11 ;
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: This could be because you hit a bug. It is also possible that this binary
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld:
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld:
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: We will try our best to scrape up some info that will hopefully help
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: diagnose the problem, but since we have already crashed,
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: something is definitely wrong and this may fail.
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld:
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Server version: 5.5.49-MariaDB-1~trusty-wsrep
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: key_buffer_size=134217728
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: read_buffer_size=2097152
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: max_used_connections=33
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: max_threads=502
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: thread_count=6766
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: It is possible that mysqld could use up to
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3224507 K bytes of memory
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld:
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Thread pointer: 0x0x7fd83fcd1000
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Attempting backtrace. You can use the following information to find out
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: where mysqld died. If you see no messages after this, something went
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: terribly wrong...
Jul 11 13:26:33 advancedportalnode-test2-20160705-10-144-95-218 mysqld: stack_bottom = 0x7fd83fadadd0 thread_stack 0x48000
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7fdade062bce]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7fdaddc478f7]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fdadc694340]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /lib/x86_64-linux-gnu/libc.so.6(memset+0x54)[0x7fdadbd41614]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(+0x92432b)[0x7fdade07c32b]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(get_charset_by_csname+0x14a)[0x7fdade03c86a]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN14Item_func_sha218fix_length_and_decEv+0x7b)[0x7fdaddcc4ccb]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x15c)[0x7fdaddca38cc]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN13Item_str_func10fix_fieldsEP3THDPP4Item+0x12)[0x7fdaddcc4262]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x1a8)[0x7fdaddca3918]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN13Item_str_func10fix_fieldsEP3THDPP4Item+0x12)[0x7fdaddcc4262]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x1a8)[0x7fdaddca3918]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN13Item_str_func10fix_fieldsEP3THDPP4Item+0x12)[0x7fdaddcc4262]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x1a8)[0x7fdaddca3918]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN13Item_str_func10fix_fieldsEP3THDPP4Item+0x12)[0x7fdaddcc4262]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x1a8)[0x7fdaddca3918]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_ZN13Item_str_func10fix_fieldsEP3THDPP4Item+0x12)[0x7fdaddcc4262]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_Z12setup_fieldsP3THDPP4ItemR4ListIS1_E17enum_mark_columnsPS5_b+0x17c)[0x7fdadda9a68c]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(+0x3d75b2)[0x7fdaddb2f5b2]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xb1f)[0x7fdaddb3fd0f]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(+0x385409)[0x7fdaddadd409]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5630)[0x7fdaddaebc10]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(+0x3973f3)[0x7fdaddaef3f3]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1f78)[0x7fdaddaf1a48]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x22f)[0x7fdaddaf257f]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x33e)[0x7fdaddbb244e]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /usr/sbin/mysqld(handle_one_connection+0x4a)[0x7fdaddbb253a]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fdadc68c182]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fdadbdaf47d]
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld:
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Trying to get some variables.
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Some pointers may be invalid and cause the dump to abort.
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Query (0x7fd83f495018): is an invalid pointer
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Connection ID (thread ID): 805049
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Status: NOT_KILLED
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld:
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld:
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Jul 11 13:26:34 advancedportalnode-test2-20160705-10-144-95-218 mysqld: information that should help you find out what is causing the crash.
Jul 11 13:26:35 advancedportalnode-test2-20160705-10-144-95-218 mysqld_safe: Number of processes running now: 0
Jul 11 13:26:35 advancedportalnode-test2-20160705-10-144-95-218 mysqld_safe: WSREP: not restarting wsrep node automatically
Jul 11 13:26:35 advancedportalnode-test2-20160705-10-144-95-218 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended