[MDEV-17547] MariaDB 10.2 and Galera crashing with segfault Created: 2018-10-25  Updated: 2021-12-02  Resolved: 2018-12-10

Status: Closed
Project: MariaDB Server
Component/s: Galera, Replication
Affects Version/s: 10.2.18
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: MichaƂ Parysek Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

CentOS 7
MariaDB-common-10.2.18-1.el7.centos.x86_64
MariaDB-server-10.2.18-1.el7.centos.x86_64
MariaDB-compat-10.2.18-1.el7.centos.x86_64
MariaDB-client-10.2.18-1.el7.centos.x86_64
MariaDB-backup-10.2.18-1.el7.centos.x86_64
MariaDB-shared-10.2.18-1.el7.centos.x86_64
galera-25.3.24-1.rhel7.el7.centos.x86_64


Attachments: Text File crash.log    
Issue Links:
Relates
relates to MDEV-24143 Galera nodes "randomly" crashing in I... Closed
relates to MDEV-27115 10.4.22 segfault at SELECT RELEASE_LO... Closed

 Description   

I've got very strange crash after upgrade to MariaDB 10.2 and running for several hours. Systemd restarted mysqld and everything was OK, but after few more hours mysqld crashed again.



 Comments   
Comment by Elena Stepanova [ 2018-11-11 ]

From the attached error log:

Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: 181025  8:03:35 [ERROR] mysqld got signal 11 ;
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: This could be because you hit a bug. It is also possible that this binary
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: We will try our best to scrape up some info that will hopefully help
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: diagnose the problem, but since we have already crashed,
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: something is definitely wrong and this may fail.
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Server version: 10.2.18-MariaDB
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: key_buffer_size=134217728
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: read_buffer_size=131072
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: max_used_connections=16
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: max_threads=1026
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: thread_count=71
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: It is possible that mysqld could use up to
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2385557 K  bytes of memory
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Thread pointer: 0x7fcd780009a8
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Attempting backtrace. You can use the following information to find out
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: where mysqld died. If you see no messages after this, something went
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: terribly wrong...
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: stack_bottom = 0x7fcedc347d00 thread_stack 0x49000
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: ======= Backtrace: =========
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(__fortify_fail+0x37)[0x7fd375bfe6e7]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(+0x116862)[0x7fd375bfc862]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(+0x118647)[0x7fd375bfe647]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_addr_resolve+0xda)[0x55e043f8a76a]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55e043f73e32]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x355)[0x55e0439f8c95]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libpthread.so.0(+0xf6d0)[0x7fd3778486d0]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z11ull_get_keyPKhPmc+0x14)[0x55e043a5e824]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_hash_first_from_hash_value+0x6b)[0x55e043f5124b]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_hash_search+0x11)[0x55e043f513b1]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN18Item_func_get_lock7val_intEv+0x1bc)[0x55e043a61f5c]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4Item4sendEP8ProtocolP6String+0x20c)[0x55e043a0972c]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN8Protocol19send_result_set_rowEP4ListI4ItemE+0xdb)[0x55e0437db75b]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN11select_send9send_dataER4ListI4ItemE+0x53)[0x55e0438339b3]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xaa4)[0x55e0438b7504]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55e0438b76b3]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x55e0438b780a]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x254)[0x55e0438b8364]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x4151d2)[0x55e04379a1d2]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1b71)[0x55e043860c21]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2de)[0x55e043868b9e]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x4e447b)[0x55e04386947b]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1786)[0x55e04386b5d6]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x55e04386d505]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x55e043934b3a]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e043934c5d]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x826e6d)[0x55e043babe6d]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libpthread.so.0(+0x7e25)[0x7fd377840e25]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7fd375be4bad]
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: ======= Memory map: ========
Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: 55e043385000-55e0444a9000 r-xp 00000000 fd:00 67153562                   /usr/sbin/mysqld
...

Comment by Elena Stepanova [ 2018-11-11 ]

Would you be able to enable the general log temporarily and try to find out which query causes the crash?

Generated at Thu Feb 08 08:37:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.