Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28734

Galera node crashes with Signal 6 and no real info about the reason on FreeBSD

    XMLWordPrintable

Details

    Description

      Two nodes of a 5 node Galera cluster on FreeBSD 13 failed at about the same time, catching a SIGABRT signal. Nothing in the error log hits towards the reason, the most recent messages where peer timeout messages, but there were several of these earlier in the log already without causing this.

      No assertion check failure messages, no usable backtrace info:

      2022-05-15 20:57:50 0 [Note] WSREP: (4c0650f8-9562, 'tcp://10.10.70.231:4567') connection to peer 00000000-0000 with addr tcp://10.10.70.241:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1000 rttvar: 500 rto: 230000 lost: 0 last_data_recv: 3047000 cwnd: 14480 last_queued_since: 3047251488 last_delivered_since: 3047251488 send_queue_length: 0 send_queue_bytes: 0
      2022-05-15 20:57:54 0 [Note] WSREP: (4c0650f8-9562, 'tcp://10.10.70.231:4567') connection to peer 00000000-0000 with addr tcp://10.10.70.241:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1000 rttvar: 500 rto: 230000 lost: 0 last_data_recv: 3073000 cwnd: 14480 last_queued_since: 3072801757 last_delivered_since: 3072801757 send_queue_length: 0 send_queue_bytes: 0
      220515 20:57:54 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.6.8-4-MariaDB-enterprise-log
      key_buffer_size=5242880
      read_buffer_size=2097152
      max_used_connections=4
      max_threads=1502
      thread_count=100
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18500018 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x200000
      0x13ef71e <my_print_stacktrace+0x2e> at /usr/local/libexec/mariadbd
      0xce6460 <handle_fatal_signal+0x290> at /usr/local/libexec/mariadbd
      0x801935580 <pthread_sigmask+0x540> at /lib/libthr.so.3
      0x801934b3f <pthread_setschedparam+0x82f> at /lib/libthr.so.3
      0x7ffffffff2d3 <__gxx_personality_v0+0x7ffffeb6ae03> at ???
      0x801b9791a <thr_kill+0xa> at /lib/libc.so.7
      0x801b100f4 <_raise+0x34> at /lib/libc.so.7
      0x801bc16e9 <abort+0x49> at /lib/libc.so.7
      0x801a32de9 <__cxa_rethrow+0x1d9> at /lib/libcxxrt.so.1
      0x803a6bc18 <wsrep_ps_free_node_stat+0x159a98> at /rw_part/usr-local/lib/mysql/libgalera_enterprise_smm.so
      0x803a8777a <wsrep_ps_free_node_stat+0x1755fa> at /rw_part/usr-local/lib/mysql/libgalera_enterprise_smm.so
      0x803a81bbe <wsrep_ps_free_node_stat+0x16fa3e> at /rw_part/usr-local/lib/mysql/libgalera_enterprise_smm.so
      0x803a7f444 <wsrep_ps_free_node_stat+0x16d2c4> at /rw_part/usr-local/lib/mysql/libgalera_enterprise_smm.so
      0x803a8a5b9 <wsrep_ps_free_node_stat+0x178439> at /rw_part/usr-local/lib/mysql/libgalera_enterprise_smm.so
      0x803a6d9d8 <wsrep_ps_free_node_stat+0x15b858> at /rw_part/usr-local/lib/mysql/libgalera_enterprise_smm.so
      boost/throw_exception.hpp:0(void boost::throw_exception<std::__1::system_error>(std::__1::system_error const&))[0x803a6e181]
      0x803a68349 <wsrep_ps_free_node_stat+0x1561c9> at /rw_part/usr-local/lib/mysql/libgalera_enterprise_smm.so
      impl/throw_error.ipp:0(asio::detail::do_throw_error(std::__1::error_code const&, char const*))[0x80399711c]
      detail/endpoint.hpp:111(asio::ip::detail::endpoint::is_v4() const)[0x803988ca9]
      v1/memory:3720(std::__1::shared_ptr<gu::AsioStreamEngine>::operator->() const)[0x803988c19]
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Core pattern: %N.core
      

      Attachments

        Activity

          People

            hholzgra Hartmut Holzgraefe
            hholzgra Hartmut Holzgraefe
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.