Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6431

Galera - 3 node cluster breaks down after OOM and restart of one node

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • Kubernetes

    Description

      We are running mariadb 10.11.11 with Galera plugin 26.4.21 in Kubernetes as a statefulset with 3 replicas. I encountered the problem that one of the nodes reached its memory limit (cause is unclear) and was killed by Kubernetes. After the restart the node tried to join the cluster (the other 2 nodes were in primary state), but the cluster startes to log "WSREP: Action message in non-primary configuration from member X" and is not responsive anymore. This indicates that the quorum is lost. I cannot find any plausible reason why the quorum is lost.

      Would you be so nice to have a look into the attached logs to analyze what caused the breakdown of the quorum?

      Thank you very much for your help!

      I attached the logs, To better understand these, following explanations:

      • the IPs are randomly assigned by Kubernetes
      • the Galera node names are mariadb-0, mariadb-1 and mariadb-2
      • the mariadb-0 node is the one that OOMed and restarted
      • which node is logging is indicated right in front of the actual log message

      Attachments

        Activity

          People

            Unassigned Unassigned
            Malte Lehmann Malte Lehmann
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.