[MDEV-21072] MariaDB Galera Services are down on all Nodes Created: 2019-11-18  Updated: 2019-12-02  Resolved: 2019-11-28

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.2.28
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Ras Tarun Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: None


 Description   

Hello ,

We have newly moved to Open Source MariaDB Galera 10.2.25 supporting 3 nodes.

During OS Upgrade activity , we brought down all the three nodes .

First we brought the MariaDB service down on least loaded node and then one by one brought the MariaDB Galera Cluster down

After completion of maintenance , I was only able to Bootstrap the mostly Advance node using command sudo galera_new_cluster and remaining nodes , I was not able to bring up using sudo systemctl start MariaDB.

I have only 1 node which is UP and MariaDB service is Active and running.

I changed the safe_to_bootstrap to 1 in grastate.dat file and bootstrapped all the 3 nodes using sudo galera_new_cluster command

I am able to bring MariaDB service UP on the all the nodes. But now every node is a local node and not part of Cluster.

Kindly suggest how to resolve this issue.

 Comments   
Comment by Ras Tarun [ 2019-11-18 ]

Now I am able to see wsrep_cluster_size = 1 on all the 3 Nodes of MariaDB Galera .

Previously , I could see 3 nodes part of Cluster and 3 incoming IP address

MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';
-------------------------+

Variable_name Value

-------------------------+

wsrep_cluster_size 3

-------------------------+
1 row in set (0.00 sec)

Comment by Ras Tarun [ 2019-11-18 ]

Here are the initial contents of grastate.dat file

[root@ Node1 ~]# cat grastate.dat

  1. GALERA saved state
    version: 2.1
    uuid: 124692a8-b12d-11e9-80a1
    seqno: -1
    safe_to_bootstrap: 0

[root@ Node2 ~]# cat grastate.dat

  1. GALERA saved state
    version: 2.1
    uuid: dwqj2db3-ffc9-11e9-9af9
    seqno: -1
    safe_to_bootstrap: 0

[root@ Node3 ~]# cat grastate.dat

  1. GALERA saved state
    version: 2.1
    uuid: fd36d0a3-ffc6-11e9-884f
    seqno: -1
    safe_to_bootstrap: 0

After editing safe_to_bootstrap to 1 , I have bootstrapped all the 3 nodes using sudo galera_new_cluster

Comment by Ras Tarun [ 2019-11-18 ]

Kindly advise how I can bring the system to normal state

Comment by Jan Lindström (Inactive) [ 2019-11-28 ]

You need to select one of the servers, boostrap that as a new master and then start node by node making sure that they will select master as donor. You should run galera_new_cluster only for the first node. What you have done is that every node is its own cluster.

Comment by Ras Tarun [ 2019-12-01 ]

Hello Jan Lindstrom ,

I am Tarun Pandit MariaDB Database Administrator , Thanks for your
cooperation and Anticipation.

You have closed the thread , but I had a question Would it be possible to
go other direction

1) Take Full L0 Backup of Database
2) Truncate Tables
3) Follow your comments as you have suggested in Jira ticket ( Regarding
Starting Cluster Nodes )
4) Once MariaDB Cluster nodes are joined , then Restore L0 backup

Merci ,
Tarun Pandit
Administrateur de base donnees

On Thu, Nov 28, 2019 at 6:27 PM Jan Lindström (Jira) <jira@mariadb.org>

Comment by Jan Lindström (Inactive) [ 2019-12-02 ]

Yes, it is possible. I suggest mariabackup.

Generated at Thu Feb 08 09:04:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.