[MDEV-19193] The --wsrep-new-cluster option does not reset wsrep_cluster_state_uuid and wsrep_last_committed Created: 2019-04-05  Updated: 2020-08-25  Resolved: 2019-05-23

Status: Closed
Project: MariaDB Server
Component/s: Galera, wsrep
Affects Version/s: 10.1.38, 10.2.23, 10.3.14
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Geoff Montee (Inactive) Assignee: Daniele Sciascia
Resolution: Not a Bug Votes: 3
Labels: None

Issue Links:
Relates
relates to MDEV-17458 Unable to Start Galera Node Closed

 Description   

galera_new_cluster bootstraps a new cluster. However, when this happens, the cluster keeps its old state. For example:

[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 8
 
Server version: 10.1.38-MariaDB MariaDB Server
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
 
+----------------------+----------------------------------------+
 
| Variable_name        | Value                                  |
 
+----------------------+----------------------------------------+
 
| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
 
+----------------------+----------------------------------------+
 
1 row in set (0.00 sec)
 
 
 
MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
 
+--------------------------+--------------------------------------+
 
| Variable_name            | Value                                |
 
+--------------------------+--------------------------------------+
 
| wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |
 
| wsrep_last_committed     | 6                                    |
 
+--------------------------+--------------------------------------+
 
2 rows in set (0.00 sec)
 
 
 
MariaDB [(none)]> \q
 
Bye
 
[ec2-user@ip-172-30-0-249 ~]$ sudo systemctl stop mariadb
 
[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 8
 
Server version: 10.1.38-MariaDB MariaDB Server
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
 
+----------------------+----------------------------------------+
 
| Variable_name        | Value                                  |
 
+----------------------+----------------------------------------+
 
| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
 
+----------------------+----------------------------------------+
 
1 row in set (0.00 sec)
 
 
 
MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
 
+--------------------------+--------------------------------------+
 
| Variable_name            | Value                                |
 
+--------------------------+--------------------------------------+
 
| wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |
 
| wsrep_last_committed     | 6                                    |
 
+--------------------------+--------------------------------------+
 
2 rows in set (0.00 sec)
 
 
 
MariaDB [(none)]> \q
 
Bye

If it's a "new" cluster, wouldn't it make sense to start with a new UUID and seqno?

galera_new_cluster starts "mysqld --wsrep-new-cluster" using systemd under the hood. I don't believe that this is a systemd or galera_new_cluster problem, because this still happens if you bypass systemd and start "mysqld --wsrep-new-cluster" directly. For example:

[ec2-user@ip-172-30-0-249 ~]$ sudo mysqld --user=mysql --wsrep-new-cluster &
 
[1] 4753
 
2019-05-02 17:16:42 140682328529088 [Note] mysqld (mysqld 10.1.39-MariaDB) starting as process 4754 ...
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 8
 
Server version: 10.1.39-MariaDB MariaDB Server
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
 
+----------------------+-----------------------------------------+
 
| Variable_name        | Value                                   |
 
+----------------------+-----------------------------------------+
 
| wsrep_start_position | 00000000-0000-0000-0000-000000000000:-1 |
 
+----------------------+-----------------------------------------+
 
1 row in set (0.00 sec)
 
 
 
MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
 
+--------------------------+--------------------------------------+
 
| Variable_name            | Value                                |
 
+--------------------------+--------------------------------------+
 
| wsrep_cluster_state_uuid | 22f991fd-6d1f-11e9-ac6a-b392983e8673 |
 
| wsrep_last_committed     | 0                                    |
 
+--------------------------+--------------------------------------+
 
2 rows in set (0.01 sec)
 
 
 
MariaDB [(none)]> \q
 
Bye
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysqladmin -u root shutdown
 
[1]+  Done                    sudo mysqld -u mysql --wsrep-new-cluster
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysqld --user=mysql --wsrep-new-cluster &
 
[1] 4799
 
2019-05-02 17:18:15 140331403352256 [Note] mysqld (mysqld 10.1.39-MariaDB) starting as process 4800 ...
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 8
 
Server version: 10.1.39-MariaDB MariaDB Server
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
 
+----------------------+-----------------------------------------+
 
| Variable_name        | Value                                   |
 
+----------------------+-----------------------------------------+
 
| wsrep_start_position | 00000000-0000-0000-0000-000000000000:-1 |
 
+----------------------+-----------------------------------------+
 
1 row in set (0.00 sec)
 
 
 
MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
 
+--------------------------+--------------------------------------+
 
| Variable_name            | Value                                |
 
+--------------------------+--------------------------------------+
 
| wsrep_cluster_state_uuid | 22f991fd-6d1f-11e9-ac6a-b392983e8673 |
 
| wsrep_last_committed     | 0                                    |
 
+--------------------------+--------------------------------------+
 
2 rows in set (0.01 sec)
 
 
 
MariaDB [(none)]> \q
 
Bye
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysqladmin -u root shutdown
 
[1]+  Done                    sudo mysqld -u mysql --wsrep-new-cluster

In order to reset 'wsrep_cluster_state_uuid' and 'wsrep_last_committed', you have to manually delete grastate.dat:

[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 9
 
Server version: 10.1.38-MariaDB MariaDB Server
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
 
+----------------------+----------------------------------------+
 
| Variable_name        | Value                                  |
 
+----------------------+----------------------------------------+
 
| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
 
+----------------------+----------------------------------------+
 
1 row in set (0.00 sec)
 
 
 
MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
 
+--------------------------+--------------------------------------+
 
| Variable_name            | Value                                |
 
+--------------------------+--------------------------------------+
 
| wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |
 
| wsrep_last_committed     | 6                                    |
 
+--------------------------+--------------------------------------+
 
2 rows in set (0.00 sec)
 
 
 
MariaDB [(none)]> \q
 
Bye
 
[ec2-user@ip-172-30-0-249 ~]$ sudo systemctl stop mariadb
 
[ec2-user@ip-172-30-0-249 ~]$ sudo rm /var/lib/mysql/grastate.dat
 
[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
 
[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
 
Your MariaDB connection id is 8
 
Server version: 10.1.38-MariaDB MariaDB Server
 
 
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
 
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
 
 
MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
 
+----------------------+----------------------------------------+
 
| Variable_name        | Value                                  |
 
+----------------------+----------------------------------------+
 
| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
 
+----------------------+----------------------------------------+
 
1 row in set (0.00 sec)
 
 
 
MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
 
+--------------------------+--------------------------------------+
 
| Variable_name            | Value                                |
 
+--------------------------+--------------------------------------+
 
| wsrep_cluster_state_uuid | 1ff475b4-57d2-11e9-80ac-077936e51cdd |
 
| wsrep_last_committed     | 0                                    |
 
+--------------------------+--------------------------------------+
 
2 rows in set (0.00 sec)
 
 
 
MariaDB [(none)]> \q
 
Bye

You can see from above that wsrep_start_position is still not reset when mysqld is started with systemd. This is because MariaDB's systemd unit file has an ExecStartPre option defined that recovers this position using the galera_recovery wrapper script:

https://github.com/MariaDB/server/blob/mariadb-10.1.39/support-files/mariadb.service.in#L72

https://github.com/MariaDB/server/blob/mariadb-10.1.39/scripts/galera_recovery.sh

 Comments   
Comment by Jan Lindström (Inactive) [ 2019-05-21 ]

https://github.com/MariaDB/server/pull/1305 should fix this also.

Comment by Marko Mäkelä [ 2019-05-21 ]

I do not think that MDEV-17458, which was a fix for a regression introduced by MDEV-15158, can possibly fix this. This bug is reported to affect the 10.1 and 10.2 series, and the MDEV-17458 fix only applies to 10.3 and later.

Comment by Jan Lindström (Inactive) [ 2019-05-21 ]

sciascid Can you please check 10.1 and 10.2 case and is there still something needed for 10.3->10.4

Comment by Daniele Sciascia [ 2019-05-23 ]

GeoffMontee This is not a bug.

Option --wsrep-new-cluster simply signals galera that the starting node is alone in the cluster, and will bootstrap.
This option is used in two cases:
1) When you create a cluster from scratch, and a new UUID is allocated for it
2) When restarting a cluster that already existed, which has a UUID already assigned

What you are doing in your examples above, falls in the second case: you are basically restarting the first node of your cluster. And because the cluster existed, its old state is preserved.

There currently is no option to reset galera state. As you mentioned, you can manually delete the old state in the datadir.
Let me know if that clarifies.

Comment by Geoff Montee (Inactive) [ 2019-05-23 ]

Thanks, sciascid. I expected this was the case. I think the new cluster in the --wsrep-new-cluster option name and the galera_new_cluster script name is a bit of a misnomer then. In hindsight, they probably should have been called something like --wsrep-bootstrap-cluster and galera_boostrap_cluster. But it probably doesn't matter.

Generated at Thu Feb 08 08:49:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.