Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17547

MariaDB 10.2 and Galera crashing with segfault

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 10.2.18
    • N/A
    • Galera, Replication

    Description

      I've got very strange crash after upgrade to MariaDB 10.2 and running for several hours. Systemd restarted mysqld and everything was OK, but after few more hours mysqld crashed again.

      Attachments

        1. crash.log
          86 kB
          Michał Parysek

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24143 Galera nodes "randomly" crashing in Item_func_release_lock::val_int

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27115 10.4.22 segfault at SELECT RELEASE_LOCK() in ull_get_key (bad MDL_ticket)

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            elenst Elena Stepanova added a comment -

            From the attached error log:

            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: 181025  8:03:35 [ERROR] mysqld got signal 11 ;
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: This could be because you hit a bug. It is also possible that this binary
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: We will try our best to scrape up some info that will hopefully help
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: diagnose the problem, but since we have already crashed,
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: something is definitely wrong and this may fail.
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Server version: 10.2.18-MariaDB
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: key_buffer_size=134217728
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: read_buffer_size=131072
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: max_used_connections=16
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: max_threads=1026
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: thread_count=71
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: It is possible that mysqld could use up to
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2385557 K  bytes of memory
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Hope that's ok; if not, decrease some variables in the equation.
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Thread pointer: 0x7fcd780009a8
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Attempting backtrace. You can use the following information to find out
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: where mysqld died. If you see no messages after this, something went
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: terribly wrong...
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: stack_bottom = 0x7fcedc347d00 thread_stack 0x49000
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: ======= Backtrace: =========
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(__fortify_fail+0x37)[0x7fd375bfe6e7]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(+0x116862)[0x7fd375bfc862]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(+0x118647)[0x7fd375bfe647]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_addr_resolve+0xda)[0x55e043f8a76a]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55e043f73e32]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x355)[0x55e0439f8c95]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libpthread.so.0(+0xf6d0)[0x7fd3778486d0]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z11ull_get_keyPKhPmc+0x14)[0x55e043a5e824]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_hash_first_from_hash_value+0x6b)[0x55e043f5124b]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_hash_search+0x11)[0x55e043f513b1]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN18Item_func_get_lock7val_intEv+0x1bc)[0x55e043a61f5c]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4Item4sendEP8ProtocolP6String+0x20c)[0x55e043a0972c]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN8Protocol19send_result_set_rowEP4ListI4ItemE+0xdb)[0x55e0437db75b]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN11select_send9send_dataER4ListI4ItemE+0x53)[0x55e0438339b3]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xaa4)[0x55e0438b7504]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55e0438b76b3]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x55e0438b780a]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x254)[0x55e0438b8364]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x4151d2)[0x55e04379a1d2]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1b71)[0x55e043860c21]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2de)[0x55e043868b9e]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x4e447b)[0x55e04386947b]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1786)[0x55e04386b5d6]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x55e04386d505]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x55e043934b3a]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e043934c5d]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x826e6d)[0x55e043babe6d]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libpthread.so.0(+0x7e25)[0x7fd377840e25]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7fd375be4bad]
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: ======= Memory map: ========
            		 
            Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: 55e043385000-55e0444a9000 r-xp 00000000 fd:00 67153562                   /usr/sbin/mysqld
            		 
            ...

            elenst Elena Stepanova added a comment - From the attached error log: Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: 181025 8:03:35 [ERROR] mysqld got signal 11 ; Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: This could be because you hit a bug. It is also possible that this binary Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: We will try our best to scrape up some info that will hopefully help Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: diagnose the problem, but since we have already crashed, Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: something is definitely wrong and this may fail. Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Server version: 10.2.18-MariaDB Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: key_buffer_size=134217728 Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: read_buffer_size=131072 Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: max_used_connections=16 Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: max_threads=1026 Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: thread_count=71 Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: It is possible that mysqld could use up to Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2385557 K bytes of memory Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Hope that's ok; if not, decrease some variables in the equation. Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Thread pointer: 0x7fcd780009a8 Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: Attempting backtrace. You can use the following information to find out Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: where mysqld died. If you see no messages after this, something went Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: terribly wrong... Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: stack_bottom = 0x7fcedc347d00 thread_stack 0x49000 Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: *** buffer overflow detected ***: /usr/sbin/mysqld terminated Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: ======= Backtrace: ========= Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(__fortify_fail+0x37)[0x7fd375bfe6e7] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(+0x116862)[0x7fd375bfc862] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(+0x118647)[0x7fd375bfe647] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_addr_resolve+0xda)[0x55e043f8a76a] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55e043f73e32] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x355)[0x55e0439f8c95] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libpthread.so.0(+0xf6d0)[0x7fd3778486d0] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z11ull_get_keyPKhPmc+0x14)[0x55e043a5e824] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_hash_first_from_hash_value+0x6b)[0x55e043f5124b] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(my_hash_search+0x11)[0x55e043f513b1] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN18Item_func_get_lock7val_intEv+0x1bc)[0x55e043a61f5c] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4Item4sendEP8ProtocolP6String+0x20c)[0x55e043a0972c] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN8Protocol19send_result_set_rowEP4ListI4ItemE+0xdb)[0x55e0437db75b] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN11select_send9send_dataER4ListI4ItemE+0x53)[0x55e0438339b3] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xaa4)[0x55e0438b7504] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55e0438b76b3] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x55e0438b780a] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x254)[0x55e0438b8364] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x4151d2)[0x55e04379a1d2] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1b71)[0x55e043860c21] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x2de)[0x55e043868b9e] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x4e447b)[0x55e04386947b] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1786)[0x55e04386b5d6] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x165)[0x55e04386d505] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x55e043934b3a] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e043934c5d] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /usr/sbin/mysqld(+0x826e6d)[0x55e043babe6d] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libpthread.so.0(+0x7e25)[0x7fd377840e25] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7fd375be4bad] Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: ======= Memory map: ======== Oct 25 08:03:35 v-csb-p-galeradmzdb03 mysqld: 55e043385000-55e0444a9000 r-xp 00000000 fd:00 67153562 /usr/sbin/mysqld ...
            elenst Elena Stepanova added a comment -

            Would you be able to enable the general log temporarily and try to find out which query causes the crash?

            elenst Elena Stepanova added a comment - Would you be able to enable the general log temporarily and try to find out which query causes the crash?

            People

              Unassigned Unassigned
              GrimDemon Michał Parysek
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.