Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
-
Kubernetes
Description
We are running mariadb 10.11.11 with Galera plugin 26.4.21 in Kubernetes as a statefulset with 3 replicas. I encountered the problem that one of the nodes reached its memory limit (cause is unclear) and was killed by Kubernetes. After the restart the node tried to join the cluster (the other 2 nodes were in primary state), but the cluster startes to log "WSREP: Action message in non-primary configuration from member X" and is not responsive anymore. This indicates that the quorum is lost. I cannot find any plausible reason why the quorum is lost.
Would you be so nice to have a look into the attached logs to analyze what caused the breakdown of the quorum?
Thank you very much for your help!
I attached the logs, To better understand these, following explanations:
- the IPs are randomly assigned by Kubernetes
- the Galera node names are mariadb-0, mariadb-1 and mariadb-2
- the mariadb-0 node is the one that OOMed and restarted
- which node is logging is indicated right in front of the actual log message