Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24143

Galera nodes "randomly" crashing in Item_func_release_lock::val_int

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 10.3.21, 10.2(EOL), 10.5, 10.6, 10.7(EOL), 10.8(EOL)
    • 10.4.25, 10.5.16, 10.6.8, 10.7.4
    • Galera
    • None
    • CentOS7 - 3.10.0-1127.8.2.el7.x86_64
      Packae: mariadb103-server-galera-10.3.21-2.el7.ius.x86_64

    Description

      We have two servers running MariaDB with galera for replication. Every few weeks we get alerts that MariaDB has crashed with a segfault. We were on older 10.X versions MariaDB ( https://serverfault.com/questions/1016977/mariadb-crashing ) and had the same issues. I am not sure if it is a specific query that is causing MariaDB to crash or an issue elsewhere. Below is what I am seeing with a back trace

      [root@mon2 ccpp-2020-11-05-08:55:49-43159]# gdb /usr/libexec/mysqld coredump
      		 
      GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-119.el7
      		 
      Copyright (C) 2013 Free Software Foundation, Inc.
      		 
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      		 
      This is free software: you are free to change and redistribute it.
      		 
      There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
      		 
      and "show warranty" for details.
      		 
      This GDB was configured as "x86_64-redhat-linux-gnu".
      		 
      For bug reporting instructions, please see:
      		 
      <http://www.gnu.org/software/gdb/bugs/>...
      		 
      Reading symbols from /usr/libexec/mysqld...Reading symbols from /usr/libexec/mysqld...(no debugging symbols found)...done.
      		 
      (no debugging symbols found)...done.
      		 
      [New LWP 13153]
      		 
      [New LWP 43164]
      		 
      [New LWP 43163]
      		 
      [New LWP 44190]
      		 
      [New LWP 44193]
      		 
      [New LWP 26180]
      		 
      [New LWP 44196]
      		 
      [New LWP 44051]
      		 
      [New LWP 44186]
      		 
      [New LWP 44194]
      		 
      [New LWP 44192]
      		 
      [New LWP 44189]
      		 
      [New LWP 44188]
      		 
      [New LWP 44208]
      		 
      [New LWP 44195]
      		 
      [New LWP 44191]
      		 
      [New LWP 44200]
      		 
      [New LWP 43159]
      		 
      [New LWP 43166]
      		 
      [New LWP 43162]
      		 
      [New LWP 43161]
      		 
      [New LWP 44187]
      		 
      [New LWP 43165]
      		 
      [New LWP 44209]
      		 
      [New LWP 44210]
      		 
      [New LWP 44211]
      		 
      [New LWP 44197]
      		 
      [New LWP 44216]
      		 
      [New LWP 44215]
      		 
      [New LWP 44198]
      		 
      [New LWP 44214]
      		 
      [New LWP 44217]
      		 
      [New LWP 44218]
      		 
      [New LWP 44212]
      		 
      [New LWP 44224]
      		 
      [New LWP 44261]
      		 
      [New LWP 44213]
      		 
      [New LWP 44207]
      		 
      [New LWP 44223]
      		 
      [New LWP 44222]
      		 
      [New LWP 44220]
      		 
      [Thread debugging using libthread_db enabled]
      		 
      Using host libthread_db library "/lib64/libthread_db.so.1".
      		 
      Core was generated by `/usr/libexec/mysqld --basedir=/usr'.
      		 
      Program terminated with signal 11, Segmentation fault.
      		 
      #0  0x00007f4375bac0b8 in ?? () from /lib64/libgcc_s.so.1
      		 
      Missing separate debuginfos, use: debuginfo-install mariadb103-server-10.3.21-2.el7.ius.x86_64
      		 
      (gdb) bt
      		 
      #0  0x00007f4375bac0b8 in ?? () from /lib64/libgcc_s.so.1
      		 
      #1  0x00007f4375bacfb9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
      		 
      #2  0x00007f4376fefaa6 in backtrace () from /lib64/libc.so.6
      		 
      #3  0x000055d479977c3d in my_print_stacktrace ()
      		 
      #4  0x000055d479458637 in handle_fatal_signal ()
      		 
      #5  <signal handler called>
      		 
      #6  0x0000000000000051 in ?? ()
      		 
      #7  0x000055d4794c3fc5 in Item_func_release_lock::val_int() ()
      		 
      #8  0x000055d4791d41fc in Item::update_null_value() ()
      		 
      #9  0x000055d47923d215 in Item_func::is_null() ()
      		 
      #10 0x000055d47959bde9 in mysql_do(THD*, List<Item>&) ()
      		 
      #11 0x000055d47927ea66 in mysql_execute_command(THD*) ()
      		 
      #12 0x000055d4791ead76 in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
      		 
      #13 0x000055d4791f2949 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
      		 
      #14 0x000055d4791f337c in sp_instr_stmt::execute(THD*, unsigned int*) ()
      		 
      #15 0x000055d4791ee6c0 in sp_head::execute(THD*, bool) ()
      		 
      #16 0x000055d4791ef91d in sp_head::execute_procedure(THD*, List<Item>*) ()
      		 
      #17 0x000055d479270df2 in do_execute_sp(THD*, sp_head*) ()
      		 
      #18 0x000055d4792722e6 in Sql_cmd_call::execute(THD*) [clone .part.293] ()
      		 
      #19 0x000055d479272b60 in Sql_cmd_call::execute(THD*) ()
      		 
      #20 0x000055d47927c2b8 in mysql_execute_command(THD*) ()
      		 
      #21 0x000055d47928120b in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) ()
      		 
      #22 0x000055d479281b81 in wsrep_mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) ()
      		 
      #23 0x000055d479283306 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) ()
      		 
      #24 0x000055d479284cae in do_command(THD*) ()
      		 
      #25 0x000055d4793577d1 in do_handle_one_connection(CONNECT*) ()
      		 
      #26 0x000055d47935789d in handle_one_connection ()
      		 
      #27 0x00007f4378e4dea5 in start_thread () from /lib64/libpthread.so.0
      		 
      #28 0x00007f4376fd98dd in clone () from /lib64/libc.so.6

      Attached all the traces besides the backtrace (it's 4.3GB) and the hot name file. We do have all queries stored so we can pull them for the time of the crash if needed.

      Attachments

        1. var_log_messages
          2 kB
        2. uuid
          0.0 kB
        3. username
          0.0 kB
        4. uid
          0.0 kB
        5. type
          0.0 kB
        6. time
          0.0 kB
        7. runlevel
          0.0 kB
        8. reason
          0.0 kB
        9. pwd
          0.0 kB
        10. proc_pid_status
          1 kB
        11. pkg_version
          0.0 kB
        12. pkg_vendor
          0.0 kB
        13. pkg_release
          0.0 kB
        14. pkg_name
          0.0 kB
        15. pkg_fingerprint
          0.0 kB
        16. pkg_epoch
          0.0 kB
        17. pkg_arch
          0.0 kB
        18. pid
          0.0 kB
        19. package
          0.0 kB
        20. os_release
          0.0 kB
        21. os_info
          0.4 kB
        22. open_fds
          9 kB
        23. maps
          24 kB
        24. machineid
          0.1 kB
        25. limits
          1 kB
        26. last_occurrence
          0.0 kB
        27. kernel
          0.0 kB
        28. global_pid
          0.0 kB
        29. exploitable
          0.1 kB
        30. executable
          0.0 kB
        31. environ
          0.2 kB
        32. dso_list
          3 kB
        33. count
          0.0 kB
        34. core_backtrace
          84 kB
        35. component
          0.0 kB
        36. cmdline
          0.0 kB
        37. cgroup
          0.2 kB
        38. architecture
          0.0 kB
        39. analyzer
          0.0 kB
        40. abrt_version
          0.0 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-17547 MariaDB 10.2 and Galera crashing with segfault

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27115 10.4.22 segfault at SELECT RELEASE_LOCK() in ull_get_key (bad MDL_ticket)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27713 Crash after a conflict of applier thread with stored procedure call by event scheduler

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jplindst Jan Lindström (Inactive)
              Dovid Dovid Bender
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.