Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25068

Node crashes with Transport endpoint is not connected mysqld got signal 6 ;

Details

    Description

      3 Node Setup, all 3 nodes crashed. Bootstrap required.

      Node 1 crash output

      Mar  5 17:48:58  mysqld[116636]: terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::system_error> >'
      Mar  5 17:48:58  mysqld[116636]: what():  remote_endpoint: Transport endpoint is not connected
      Mar  5 17:48:58  mysqld[116636]: 210305 17:48:58 [ERROR] mysqld got signal 6 ;
      Mar  5 17:48:58  mysqld[116636]: This could be because you hit a bug. It is also possible that this binary
      Mar  5 17:48:58  mysqld[116636]: or one of the libraries it was linked against is corrupt, improperly built,
      Mar  5 17:48:58  mysqld[116636]: or misconfigured. This error can also be caused by malfunctioning hardware.
      Mar  5 17:48:58  mysqld[116636]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Mar  5 17:48:58  mysqld[116636]: We will try our best to scrape up some info that will hopefully help
      Mar  5 17:48:58  mysqld[116636]: diagnose the problem, but since we have already crashed,
      Mar  5 17:48:58  mysqld[116636]: something is definitely wrong and this may fail.
      Mar  5 17:48:58  mysqld[116636]: Server version: 10.4.18-MariaDB-log
      Mar  5 17:48:58  mysqld[116636]: key_buffer_size=134217728
      Mar  5 17:48:58  mysqld[116636]: read_buffer_size=131072
      Mar  5 17:48:58  mysqld[116636]: max_used_connections=76
      Mar  5 17:48:58  mysqld[116636]: max_threads=153
      Mar  5 17:48:58  mysqld[116636]: thread_count=75
      Mar  5 17:48:58  mysqld[116636]: It is possible that mysqld could use up to
      Mar  5 17:48:58  mysqld[116636]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 781104 K  bytes of memory
      Mar  5 17:48:58  mysqld[116636]: Hope that's ok; if not, decrease some variables in the equation.
      Mar  5 17:48:58  mysqld[116636]: Thread pointer: 0x0
      Mar  5 17:48:58  mysqld[116636]: Attempting backtrace. You can use the following information to find out
      Mar  5 17:48:58  mysqld[116636]: where mysqld died. If you see no messages after this, something went
      Mar  5 17:48:58  mysqld[116636]: terribly wrong...
      Mar  5 17:48:59  mysqld[116636]: stack_bottom = 0x0 thread_stack 0x49000
      Mar  5 17:48:59  mysqld[116636]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a14151e47e]
      Mar  5 17:48:59  mysqld[116636]: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55a140face8f]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libpthread.so.0(+0xf630)[0x7f5879976630]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libc.so.6(gsignal+0x37)[0x7f5877c463d7]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libc.so.6(abort+0x148)[0x7f5877c47ac8]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7f5878340a95]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(+0x5ea06)[0x7f587833ea06]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(+0x5ea33)[0x7f587833ea33]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libstdc++.so.6(+0x5ec53)[0x7f587833ec53]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1997a4)[0x7f586ddc07a4]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x19984c)[0x7f586ddc084c]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a57bf)[0x7f586ddcc7bf]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1a78b8)[0x7f586ddce8b8]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x1aaf13)[0x7f586ddd1f13]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x19897b)[0x7f586ddbf97b]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0x194b91)[0x7f586ddbbb91]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xc08c9)[0x7f586dce78c9]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xa6bcf)[0x7f586dccdbcf]
      Mar  5 17:48:59  mysqld[116636]: /usr/lib64/galera-4/libgalera_smm.so(+0xad859)[0x7f586dcd4859]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libpthread.so.0(+0x7ea5)[0x7f587996eea5]
      Mar  5 17:48:59  mysqld[116636]: /lib64/libc.so.6(clone+0x6d)[0x7f5877d0e9fd]
      
      

      Attachments

        Issue Links

          Activity

            stuart.green@doccentrics.com Stuart Green created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Fix Version/s 10.4 [ 22408 ]
            Assignee Jan Lindström [ jplindst ]
            mattwt Mathew Toms made changes -
            Attachment log-10.4.2 [ 59783 ]
            Attachment log-10.5.11 [ 59784 ]
            mattwt Mathew Toms made changes -
            Attachment gdb.txt [ 59785 ]
            mattwt Mathew Toms made changes -
            Comment [ I've also seen had this crash happen numerous times on 2 separate 3-node clusters.

            The 2 clusters are running different versions, one MariaDB 10.4.20 cluster & another 10.5.11 cluster.

            This has taken one of the clusters down - requiring a manual bootstrap (this happened when 2 nodes in the cluster both suffered from this crash within 20 seconds of each other). Otherwise, the clusters are able to recover when just one node suffers this crash.

            Included a backtrace for the cluster running 10.5.11

             [^log-10.5.11]
             [^gdb.txt]
             [^log-10.4.2] ]
            stuart.green@doccentrics.com Stuart Green made changes -
            Affects Version/s 10.4.22 [ 26031 ]
            stuart.green@doccentrics.com Stuart Green made changes -
            Summary 10.4.18 crashes with Transport endpoint is not connected mysqld got signal 6 ; Node crashes with Transport endpoint is not connected mysqld got signal 6 ;
            stuart.green@doccentrics.com Stuart Green made changes -
            Affects Version/s 10.4.21 [ 26030 ]
            Affects Version/s 10.4.20 [ 25733 ]
            Affects Version/s 10.4.19 [ 25205 ]
            stuart.green@doccentrics.com Stuart Green made changes -
            Affects Version/s 10.4 [ 22408 ]
            stuart.green@doccentrics.com Stuart Green made changes -
            Affects Version/s 10.4 [ 22408 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 119845 ] MariaDB v4 [ 142658 ]
            jplindst Jan Lindström (Inactive) made changes -
            Assignee Jan Lindström [ jplindst ] Seppo Jaakola [ seppo ]
            pramod.mahto@mariadb.com Pramod Mahto made changes -
            Affects Version/s 10.5.9 [ 25109 ]
            seppo Seppo Jaakola made changes -
            Assignee Seppo Jaakola [ seppo ] Teemu Ollakka [ teemu.ollakka ]
            seppo Seppo Jaakola made changes -
            Assignee Teemu Ollakka [ teemu.ollakka ] Mario Karuza [ mkaruza ]
            julien.fritsch Julien Fritsch made changes -
            Assignee Mario Karuza [ mkaruza ] Pramod Mahto [ pramod.mahto@mariadb.com ]
            Status Open [ 1 ] Needs Feedback [ 10501 ]
            pramod.mahto@mariadb.com Pramod Mahto made changes -
            Assignee Pramod Mahto [ pramod.mahto@mariadb.com ] Mario Karuza [ mkaruza ]
            ponsuresh.pandians Pon Suresh Pandian (Inactive) made changes -
            elenst Elena Stepanova made changes -
            Status Needs Feedback [ 10501 ] Open [ 1 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Assignee Mario Karuza [ mkaruza ] Jan Lindström [ jplindst ]
            julien.fritsch Julien Fritsch made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]
            julien.fritsch Julien Fritsch made changes -
            Status Confirmed [ 10101 ] In Progress [ 3 ]
            julien.fritsch Julien Fritsch made changes -
            Assignee Jan Lindström [ jplindst ] Julien Fritsch [ julien.fritsch ]
            julien.fritsch Julien Fritsch made changes -
            Assignee Julien Fritsch [ julien.fritsch ] Jan Lindström [ jplindst ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            alice Alice Sherepa made changes -
            alice Alice Sherepa made changes -
            julien.fritsch Julien Fritsch made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            jplindst Jan Lindström (Inactive) made changes -
            issue.field.resolutiondate 2022-06-03 05:31:13.0 2022-06-03 05:31:13.396
            jplindst Jan Lindström (Inactive) made changes -
            Component/s Galera [ 10124 ]
            Component/s wsrep [ 11500 ]
            Fix Version/s 10.4.26 [ 27511 ]
            Fix Version/s 10.5.17 [ 27509 ]
            Fix Version/s 10.8.4 [ 27503 ]
            Fix Version/s 10.6.9, 10.7.5 [ 28203 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Fixed [ 1 ]
            Status In Review [ 10002 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.6.9 [ 27507 ]
            Fix Version/s 10.7.5 [ 27505 ]
            Fix Version/s 10.6.9, 10.7.5 [ 28203 ]
            alice Alice Sherepa made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 10.3.36 [ 27513 ]
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 136631

            People

              jplindst Jan Lindström (Inactive)
              stuart.green@doccentrics.com Stuart Green
              Votes:
              4 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.