[MDEV-19193] The --wsrep-new-cluster option does not reset wsrep_cluster_state_uuid and wsrep_last_committed - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Not a Bug
Affects Version/s: 10.1.38, 10.2.23, 10.3.14
Fix Version/s: N/A
Component/s: Galera, wsrep
Labels:
None

Description

galera_new_cluster bootstraps a new cluster. However, when this happens, the cluster keeps its old state. For example:

[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster

[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 8

Server version: 10.1.38-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');

+----------------------+----------------------------------------+

| Variable_name        | Value                                  |

+----------------------+----------------------------------------+

| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |

+----------------------+----------------------------------------+

1 row in set (0.00 sec)

MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');

+--------------------------+--------------------------------------+

| Variable_name            | Value                                |

+--------------------------+--------------------------------------+

| wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |

| wsrep_last_committed     | 6                                    |

+--------------------------+--------------------------------------+

2 rows in set (0.00 sec)

MariaDB [(none)]> \q

Bye

[ec2-user@ip-172-30-0-249 ~]$ sudo systemctl stop mariadb

[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster

[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 8

Server version: 10.1.38-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');

+----------------------+----------------------------------------+

| Variable_name        | Value                                  |

+----------------------+----------------------------------------+

| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |

+----------------------+----------------------------------------+

1 row in set (0.00 sec)

MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');

+--------------------------+--------------------------------------+

| Variable_name            | Value                                |

+--------------------------+--------------------------------------+

| wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |

| wsrep_last_committed     | 6                                    |

+--------------------------+--------------------------------------+

2 rows in set (0.00 sec)

MariaDB [(none)]> \q

Bye

If it's a "new" cluster, wouldn't it make sense to start with a new UUID and seqno?

galera_new_cluster starts "mysqld --wsrep-new-cluster" using systemd under the hood. I don't believe that this is a systemd or galera_new_cluster problem, because this still happens if you bypass systemd and start "mysqld --wsrep-new-cluster" directly. For example:

[ec2-user@ip-172-30-0-249 ~]$ sudo mysqld --user=mysql --wsrep-new-cluster &

[1] 4753

2019-05-02 17:16:42 140682328529088 [Note] mysqld (mysqld 10.1.39-MariaDB) starting as process 4754 ...

[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 8

Server version: 10.1.39-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');

+----------------------+-----------------------------------------+

| Variable_name        | Value                                   |

+----------------------+-----------------------------------------+

| wsrep_start_position | 00000000-0000-0000-0000-000000000000:-1 |

+----------------------+-----------------------------------------+

1 row in set (0.00 sec)

MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');

+--------------------------+--------------------------------------+

| Variable_name            | Value                                |

+--------------------------+--------------------------------------+

| wsrep_cluster_state_uuid | 22f991fd-6d1f-11e9-ac6a-b392983e8673 |

| wsrep_last_committed     | 0                                    |

+--------------------------+--------------------------------------+

2 rows in set (0.01 sec)

MariaDB [(none)]> \q

Bye

[ec2-user@ip-172-30-0-249 ~]$ sudo mysqladmin -u root shutdown

[1]+  Done                    sudo mysqld -u mysql --wsrep-new-cluster

[ec2-user@ip-172-30-0-249 ~]$ sudo mysqld --user=mysql --wsrep-new-cluster &

[1] 4799

2019-05-02 17:18:15 140331403352256 [Note] mysqld (mysqld 10.1.39-MariaDB) starting as process 4800 ...

[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 8

Server version: 10.1.39-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');

+----------------------+-----------------------------------------+

| Variable_name        | Value                                   |

+----------------------+-----------------------------------------+

| wsrep_start_position | 00000000-0000-0000-0000-000000000000:-1 |

+----------------------+-----------------------------------------+

1 row in set (0.00 sec)

MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');

+--------------------------+--------------------------------------+

| Variable_name            | Value                                |

+--------------------------+--------------------------------------+

| wsrep_cluster_state_uuid | 22f991fd-6d1f-11e9-ac6a-b392983e8673 |

| wsrep_last_committed     | 0                                    |

+--------------------------+--------------------------------------+

2 rows in set (0.01 sec)

MariaDB [(none)]> \q

Bye

[ec2-user@ip-172-30-0-249 ~]$ sudo mysqladmin -u root shutdown

[1]+  Done                    sudo mysqld -u mysql --wsrep-new-cluster

In order to reset 'wsrep_cluster_state_uuid' and 'wsrep_last_committed', you have to manually delete grastate.dat:

[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster

[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 9

Server version: 10.1.38-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');

+----------------------+----------------------------------------+

| Variable_name        | Value                                  |

+----------------------+----------------------------------------+

| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |

+----------------------+----------------------------------------+

1 row in set (0.00 sec)

MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');

+--------------------------+--------------------------------------+

| Variable_name            | Value                                |

+--------------------------+--------------------------------------+

| wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |

| wsrep_last_committed     | 6                                    |

+--------------------------+--------------------------------------+

2 rows in set (0.00 sec)

MariaDB [(none)]> \q

Bye

[ec2-user@ip-172-30-0-249 ~]$ sudo systemctl stop mariadb

[ec2-user@ip-172-30-0-249 ~]$ sudo rm /var/lib/mysql/grastate.dat

[ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster

[ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MariaDB connection id is 8

Server version: 10.1.38-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');

+----------------------+----------------------------------------+

| Variable_name        | Value                                  |

+----------------------+----------------------------------------+

| wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |

+----------------------+----------------------------------------+

1 row in set (0.00 sec)

MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');

+--------------------------+--------------------------------------+

| Variable_name            | Value                                |

+--------------------------+--------------------------------------+

| wsrep_cluster_state_uuid | 1ff475b4-57d2-11e9-80ac-077936e51cdd |

| wsrep_last_committed     | 0                                    |

+--------------------------+--------------------------------------+

2 rows in set (0.00 sec)

MariaDB [(none)]> \q

Bye

You can see from above that wsrep_start_position is still not reset when mysqld is started with systemd. This is because MariaDB's systemd unit file has an ExecStartPre option defined that recovers this position using the galera_recovery wrapper script:

https://github.com/MariaDB/server/blob/mariadb-10.1.39/support-files/mariadb.service.in#L72

https://github.com/MariaDB/server/blob/mariadb-10.1.39/scripts/galera_recovery.sh

Attachments

Issue Links

relates to

MDEV-17458 Unable to Start Galera Node

Closed

Activity

Transition	Time In Source Status	Execution Times

Geoff Montee (Inactive) made transition - 2019-05-05 21:24

Open

Confirmed

30d 2h 44m

Jan Lindström (Inactive) made transition - 2019-05-21 07:24

Confirmed

In Review

15d 10h

Marko Mäkelä made transition - 2019-05-21 12:13

In Review

Stalled

4h 48m

Daniele Sciascia made transition - 2019-05-23 07:25

Stalled

In Progress

1d 19h 12m

Daniele Sciascia made transition - 2019-05-23 12:10

In Progress

Stalled

4h 44m

Daniele Sciascia made transition - 2019-05-23 12:10

Stalled

Open

16s

Geoff Montee (Inactive) made transition - 2019-05-23 17:21

Open

Closed

5h 10m

People

Assignee:: Daniele Sciascia

Reporter:: Geoff Montee (Inactive)

Votes:: 3 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 2019-04-05 18:40

Updated:: 2024-07-07 23:21

Resolved:: 2019-05-23 17:21

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.