Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12369

Crash on idle Galera node (libgalera_smm, libssl, libcrypto)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Won't Fix
    • 10.1.22
    • N/A
    • Galera, SSL
    • Gentoo, kernel 4.4.6, 8x Intel Xeon E5-2643 0 @ 3.30GHz, 8GB RAM, 3 Galera nodes on the same stable network (no geo-replication)

    Description

      Hi,

      today morning at 08:10:24 crashed one node from 3-node Galera cluster.

      All 3 nodes were very idle, in average about 10 selects/s and 1 insert or update/s. CPU, RAM, IO, all were idle.

      Below i attach crash log. When you need it, I can send my.cnf from all 3 nodes.

      Thank you for your support.

      170327  8:10:24 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.1.22-MariaDB
      key_buffer_size=33554432
      read_buffer_size=1048576
      max_used_connections=5
      max_threads=152
      thread_count=8
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 347165 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x80000
      /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x5598d88bd3bf]
      /usr/sbin/mysqld(handle_fatal_signal+0x327)[0x5598d84420b5]
      /lib64/libpthread.so.0(+0x104f0)[0x7f8309da54f0]
      /lib64/libc.so.6(+0x89a90)[0x7f830922aa90]
      /usr/lib64/libcrypto.so.1.0.0(+0x10b81a)[0x7f830a2be81a]
      /usr/lib64/libcrypto.so.1.0.0(BIO_write+0x6c)[0x7f830a2b7b19]
      /usr/lib64/libssl.so.1.0.0(ssl3_write_pending+0x68)[0x7f830a610aef]
      /usr/lib64/libssl.so.1.0.0(ssl3_dispatch_alert+0x3a)[0x7f830a612cf4]
      /usr/lib64/libssl.so.1.0.0(ssl3_shutdown+0xa2)[0x7f830a60e9aa]
      /usr/lib/galera/libgalera_smm.so(_ZN4asio3ssl6detail2ioINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS5_EEEENS1_11shutdown_opEEEmRT_RNS1_11stream_coreERKT0_RNS_10error_codeE+0x57)[0x7f82fe0f8737]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket12close_socketEv+0xc1)[0x7f82fe0e3719]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket5closeEv+0x15b)[0x7f82fe0e3e6b]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast11erase_protoESt17_Rb_tree_iteratorISt4pairIKPKvPNS_6gmcast5ProtoEEE+0xbe)[0x7f82fe098e72]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast13handle_failedEPNS_6gmcast5ProtoE+0x1ed)[0x7f82fe0a53d3]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast9handle_upEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x67b)[0x7f82fe0a7d63]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm10Protostack8dispatchEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x47)[0x7f82fe0d6921]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet8dispatchERKPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x41)[0x7f82fe106549]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket14failed_handlerERKN4asio10error_codeERKSsi+0x2c4)[0x7f82fe0e1ba2]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket25read_completion_conditionERKN4asio10error_codeEm+0x260)[0x7f82fe0e22be]
      /usr/lib/galera/libgalera_smm.so(_ZN4asio6detail7read_opINS_3ssl6streamINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS6_EEEEEEN5boost5arrayINS_14mutable_bufferELm1EEENSB_3_bi6bind_tImNSB_4_mfi3mf
      2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSF_5list3INSF_5valueINSB_10shared_ptrISK_EEEEPFNSB_3argILi1EEEvEPFNSU_ILi2EEEvEEEEENSG_IvNSI_IvSK_SN_mEES11_EEEclESN_mi+0x4aa)[0x7f82fe0fbdea]
      /usr/lib/galera/libgalera_smm.so(_ZN4asio3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS5_EEEENS1_7read_opINS_6detail17consuming_buffersINS_14mutable_bufferEN5boost5arrayISC_Lm1E
      EEEEEENSA_7read_opINS0_6streamIS8_EESF_NSD_3_bi6bind_tImNSD_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSL_5list3INSL_5valueINSD_10shared_ptrISQ_EEEEPFNSD_3argILi1EEEvEPFNS10_ILi2EEEvEEEEENSM_IvNSO_IvS
      Q_ST_mEES17_EEEEEclESR_mi+0x1d8)[0x7f82fe0fafd8]
      /usr/lib/galera/libgalera_smm.so(_ZN4asio6detail23reactive_socket_recv_opINS_17mutable_buffers_1ENS_3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS8_EEEENS4_7read_opINS0_17consum
      ing_buffersINS_14mutable_bufferEN5boost5arrayISE_Lm1EEEEEEENS0_7read_opINS3_6streamISB_EESH_NSF_3_bi6bind_tImNSF_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSN_5list3INSN_5valueINSF_10shared_ptrISS_EEE
      EPFNSF_3argILi1EEEvEPFNS12_ILi2EEEvEEEEENSO_IvNSQ_IvSS_SV_mEES19_EEEEEEE11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationESV_m+0xd5)[0x7f82fe0fc055]
      /usr/lib/galera/libgalera_smm.so(_ZN4asio6detail13epoll_reactor16descriptor_state11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationERKNS_10error_codeEm+0x115)[0x7f82fe0ee85d]
      /usr/lib/galera/libgalera_smm.so(_ZN4asio6detail15task_io_service3runERNS_10error_codeE+0x3d5)[0x7f82fe0ee125]
      /usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_loopERKN2gu8datetime6PeriodE+0x253)[0x7f82fe107681]
      /usr/lib/galera/libgalera_smm.so(_ZN9GCommConn3runEv+0xc0)[0x7f82fe11eaf0]
      /usr/lib/galera/libgalera_smm.so(_ZN9GCommConn6run_fnEPv+0x9)[0x7f82fe123540]
      /lib64/libpthread.so.0(+0x731c)[0x7f8309d9c31c]
      /lib64/libc.so.6(clone+0x6d)[0x7f8309280ced]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      

      Attachments

        Activity

          People

            jplindst Jan Lindström (Inactive)
            jan.reges Ján Regeš
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.