XMLWordPrintable

    Details

      Description

      From time to time one of the nodes of our 3 node Galera Cluster crashes.
      After puppet restarts the crashed node, everything is working again until next crash.
      The problem exists since Mariadb 10.0.x.

      This is what the log says:

      Apr 12 06:53:57 mysql1 kernel: [11218669.570088] systemd-udevd[21154]: starting version 204
      Apr 12 06:54:19 mysql1 kernel: [11218690.738200] init: systemd-logind main process (6541) killed by TERM signal
      Apr 12 06:54:30 mysql1 mysqld: 160412  6:54:30 [ERROR] mysqld got signal 11 ;
      Apr 12 06:54:30 mysql1 mysqld: This could be because you hit a bug. It is also possible that this binary
      Apr 12 06:54:30 mysql1 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Apr 12 06:54:30 mysql1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Apr 12 06:54:30 mysql1 mysqld: 
      Apr 12 06:54:30 mysql1 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Apr 12 06:54:30 mysql1 mysqld: 
      Apr 12 06:54:30 mysql1 mysqld: We will try our best to scrape up some info that will hopefully help
      Apr 12 06:54:30 mysql1 mysqld: diagnose the problem, but since we have already crashed, 
      Apr 12 06:54:30 mysql1 mysqld: something is definitely wrong and this may fail.
      Apr 12 06:54:30 mysql1 mysqld: 
      Apr 12 06:54:30 mysql1 mysqld: Server version: 10.1.13-MariaDB-1~trusty
      Apr 12 06:54:30 mysql1 mysqld: key_buffer_size=134217728
      Apr 12 06:54:30 mysql1 mysqld: read_buffer_size=2097152
      Apr 12 06:54:30 mysql1 mysqld: max_used_connections=10
      Apr 12 06:54:30 mysql1 mysqld: max_threads=52
      Apr 12 06:54:30 mysql1 mysqld: thread_count=27
      Apr 12 06:54:30 mysql1 mysqld: It is possible that mysqld could use up to 
      Apr 12 06:54:30 mysql1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1942548 K  bytes of memory
      Apr 12 06:54:30 mysql1 mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Apr 12 06:54:30 mysql1 mysqld: 
      Apr 12 06:54:30 mysql1 mysqld: Thread pointer: 0x0x7ff0345a0008
      Apr 12 06:54:30 mysql1 mysqld: Attempting backtrace. You can use the following information to find out
      Apr 12 06:54:30 mysql1 mysqld: where mysqld died. If you see no messages after this, something went
      Apr 12 06:54:30 mysql1 mysqld: terribly wrong...
      Apr 12 06:54:30 mysql1 mysqld: stack_bottom = 0x7ff38d0851f0 thread_stack 0x48400
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7ff395f3fb6e]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7ff395a6c48d]
      Apr 12 06:54:30 mysql1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7ff393fbf340]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG21do_checkpoint_requestEm+0x9d)[0x7ff395b2dafd]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG20checkpoint_and_purgeEm+0x11)[0x7ff395b2db91]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG16rotate_and_purgeEb+0xc2)[0x7ff395b300b2]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x12c)[0x7ff3959d504c]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1334)[0x7ff3958eb994]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x7ff3958f3f4d]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(+0x438759)[0x7ff3958f4759]
      Apr 12 06:54:30 mysql1 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1e2b)[0x7ff3958f6c5b]
      Apr 12 06:54:31 mysql1 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7ff3958f7a49]
      Apr 12 06:54:31 mysql1 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7ff3959baf5a]
      Apr 12 06:54:31 mysql1 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7ff3959bb130]
      Apr 12 06:54:31 mysql1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7ff393fb7182]
      Apr 12 06:54:31 mysql1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ff3934d647d]
      Apr 12 06:54:31 mysql1 mysqld: 
      Apr 12 06:54:31 mysql1 mysqld: Trying to get some variables.
      Apr 12 06:54:31 mysql1 mysqld: Some pointers may be invalid and cause the dump to abort.
      Apr 12 06:54:31 mysql1 mysqld: Query (0x7ff02f0da020): is an invalid pointer
      Apr 12 06:54:31 mysql1 mysqld: Connection ID (thread ID): 918
      Apr 12 06:54:31 mysql1 mysqld: Status: NOT_KILLED
      Apr 12 06:54:31 mysql1 mysqld: 
      Apr 12 06:54:31 mysql1 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
      Apr 12 06:54:31 mysql1 mysqld: 
      Apr 12 06:54:31 mysql1 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Apr 12 06:54:31 mysql1 mysqld: information that should help you find out what is causing the crash.
      Apr 12 06:54:31 mysql1 mysqld_safe: Number of processes running now: 0
      Apr 12 06:54:31 mysql1 mysqld_safe: WSREP: not restarting wsrep node automatically
      Apr 12 06:54:31 mysql1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
      
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Elkin Andrei Elkin
              Reporter:
              monotek André Bauer
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: