Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26861

Galera Crashing - what(): remote_endpoint: Transport endpoint is not connected

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 10.4.20, 10.5.11
    • Fix Version/s: 10.4, 10.5
    • Component/s: Galera
    • Labels:
    • Environment:
      Ubuntu 20.04.2 LTS, Dedicated hosts per node

      Description

      Been seeing Galera nodes crashing within a few minutes of each other with days between incidents.

      Running 2 clusters with 3 nodes each, one cluster running 10.5.11 and another cluster 10.4.20. From the logs, both clusters seem to be suffering crashes for the same reason:

      Oct 16 19:34:41 db1-core mysqld[3629505]: terminate called after throwing an instance of 'boost::wrapexcept<std::system_error>'
      Oct 16 19:34:41 db1-core mysqld[3629505]:   what():  remote_endpoint: Transport endpoint is not connected
      Oct 16 19:34:41 db1-core mysqld[3629505]: 211016 19:34:41 [ERROR] mysqld got signal 6 ;
      

      It appears that when the crash strikes one node, there is a high chance a second node will crash (with the same error) a few minutes after the 1st crash - causing the cluster to require a bootstrap. Other times, just one node will crash and automatically restart and rejoin the cluster 5-10 minutes later. Days between incidents overall.

      I've attached logs from both clusters and a stack trace from the 10.5.11 node.

        Attachments

        1. gdb.txt
          306 kB
        2. log-10.4.2
          7 kB
        3. log-10.5.11
          7 kB

          Issue Links

            Activity

              People

              Assignee:
              teemu.ollakka Teemu Ollakka
              Reporter:
              mattwt Mathew Toms
              Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.