Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19193

The --wsrep-new-cluster option does not reset wsrep_cluster_state_uuid and wsrep_last_committed

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Not a Bug
    • Affects Version/s: 10.1.38, 10.2.23, 10.3.14
    • Fix Version/s: N/A
    • Component/s: Galera, wsrep
    • Labels:
      None

      Description

      galera_new_cluster bootstraps a new cluster. However, when this happens, the cluster keeps its old state. For example:

      [ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
      Welcome to the MariaDB monitor.  Commands end with ; or \g.
      Your MariaDB connection id is 8
      Server version: 10.1.38-MariaDB MariaDB Server
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
      +----------------------+----------------------------------------+
      | Variable_name        | Value                                  |
      +----------------------+----------------------------------------+
      | wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
      +----------------------+----------------------------------------+
      1 row in set (0.00 sec)
       
      MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
      +--------------------------+--------------------------------------+
      | Variable_name            | Value                                |
      +--------------------------+--------------------------------------+
      | wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |
      | wsrep_last_committed     | 6                                    |
      +--------------------------+--------------------------------------+
      2 rows in set (0.00 sec)
       
      MariaDB [(none)]> \q
      Bye
      [ec2-user@ip-172-30-0-249 ~]$ sudo systemctl stop mariadb
      [ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
      Welcome to the MariaDB monitor.  Commands end with ; or \g.
      Your MariaDB connection id is 8
      Server version: 10.1.38-MariaDB MariaDB Server
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
      +----------------------+----------------------------------------+
      | Variable_name        | Value                                  |
      +----------------------+----------------------------------------+
      | wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
      +----------------------+----------------------------------------+
      1 row in set (0.00 sec)
       
      MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
      +--------------------------+--------------------------------------+
      | Variable_name            | Value                                |
      +--------------------------+--------------------------------------+
      | wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |
      | wsrep_last_committed     | 6                                    |
      +--------------------------+--------------------------------------+
      2 rows in set (0.00 sec)
       
      MariaDB [(none)]> \q
      Bye
      

      If it's a "new" cluster, wouldn't it make sense to start with a new UUID and seqno?

      galera_new_cluster starts "mysqld --wsrep-new-cluster" using systemd under the hood. I don't believe that this is a systemd or galera_new_cluster problem, because this still happens if you bypass systemd and start "mysqld --wsrep-new-cluster" directly. For example:

      [ec2-user@ip-172-30-0-249 ~]$ sudo mysqld --user=mysql --wsrep-new-cluster &
      [1] 4753
      2019-05-02 17:16:42 140682328529088 [Note] mysqld (mysqld 10.1.39-MariaDB) starting as process 4754 ...
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
      Welcome to the MariaDB monitor.  Commands end with ; or \g.
      Your MariaDB connection id is 8
      Server version: 10.1.39-MariaDB MariaDB Server
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
      +----------------------+-----------------------------------------+
      | Variable_name        | Value                                   |
      +----------------------+-----------------------------------------+
      | wsrep_start_position | 00000000-0000-0000-0000-000000000000:-1 |
      +----------------------+-----------------------------------------+
      1 row in set (0.00 sec)
       
      MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
      +--------------------------+--------------------------------------+
      | Variable_name            | Value                                |
      +--------------------------+--------------------------------------+
      | wsrep_cluster_state_uuid | 22f991fd-6d1f-11e9-ac6a-b392983e8673 |
      | wsrep_last_committed     | 0                                    |
      +--------------------------+--------------------------------------+
      2 rows in set (0.01 sec)
       
      MariaDB [(none)]> \q
      Bye
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysqladmin -u root shutdown
      [1]+  Done                    sudo mysqld -u mysql --wsrep-new-cluster
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysqld --user=mysql --wsrep-new-cluster &
      [1] 4799
      2019-05-02 17:18:15 140331403352256 [Note] mysqld (mysqld 10.1.39-MariaDB) starting as process 4800 ...
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
      Welcome to the MariaDB monitor.  Commands end with ; or \g.
      Your MariaDB connection id is 8
      Server version: 10.1.39-MariaDB MariaDB Server
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
      +----------------------+-----------------------------------------+
      | Variable_name        | Value                                   |
      +----------------------+-----------------------------------------+
      | wsrep_start_position | 00000000-0000-0000-0000-000000000000:-1 |
      +----------------------+-----------------------------------------+
      1 row in set (0.00 sec)
       
      MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
      +--------------------------+--------------------------------------+
      | Variable_name            | Value                                |
      +--------------------------+--------------------------------------+
      | wsrep_cluster_state_uuid | 22f991fd-6d1f-11e9-ac6a-b392983e8673 |
      | wsrep_last_committed     | 0                                    |
      +--------------------------+--------------------------------------+
      2 rows in set (0.01 sec)
       
      MariaDB [(none)]> \q
      Bye
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysqladmin -u root shutdown
      [1]+  Done                    sudo mysqld -u mysql --wsrep-new-cluster
      

      In order to reset 'wsrep_cluster_state_uuid' and 'wsrep_last_committed', you have to manually delete grastate.dat:

      [ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
      Welcome to the MariaDB monitor.  Commands end with ; or \g.
      Your MariaDB connection id is 9
      Server version: 10.1.38-MariaDB MariaDB Server
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
      +----------------------+----------------------------------------+
      | Variable_name        | Value                                  |
      +----------------------+----------------------------------------+
      | wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
      +----------------------+----------------------------------------+
      1 row in set (0.00 sec)
       
      MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
      +--------------------------+--------------------------------------+
      | Variable_name            | Value                                |
      +--------------------------+--------------------------------------+
      | wsrep_cluster_state_uuid | 0d1774c1-2ce4-11e9-9071-fe16986f98a1 |
      | wsrep_last_committed     | 6                                    |
      +--------------------------+--------------------------------------+
      2 rows in set (0.00 sec)
       
      MariaDB [(none)]> \q
      Bye
      [ec2-user@ip-172-30-0-249 ~]$ sudo systemctl stop mariadb
      [ec2-user@ip-172-30-0-249 ~]$ sudo rm /var/lib/mysql/grastate.dat
      [ec2-user@ip-172-30-0-249 ~]$ sudo galera_new_cluster
      [ec2-user@ip-172-30-0-249 ~]$ sudo mysql -u root
      Welcome to the MariaDB monitor.  Commands end with ; or \g.
      Your MariaDB connection id is 8
      Server version: 10.1.38-MariaDB MariaDB Server
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [(none)]> SHOW GLOBAL VARIABLES WHERE Variable_name IN('wsrep_start_position');
      +----------------------+----------------------------------------+
      | Variable_name        | Value                                  |
      +----------------------+----------------------------------------+
      | wsrep_start_position | 0d1774c1-2ce4-11e9-9071-fe16986f98a1:6 |
      +----------------------+----------------------------------------+
      1 row in set (0.00 sec)
       
      MariaDB [(none)]> SHOW GLOBAL STATUS WHERE Variable_name IN('wsrep_cluster_state_uuid', 'wsrep_last_committed');
      +--------------------------+--------------------------------------+
      | Variable_name            | Value                                |
      +--------------------------+--------------------------------------+
      | wsrep_cluster_state_uuid | 1ff475b4-57d2-11e9-80ac-077936e51cdd |
      | wsrep_last_committed     | 0                                    |
      +--------------------------+--------------------------------------+
      2 rows in set (0.00 sec)
       
      MariaDB [(none)]> \q
      Bye
      

      You can see from above that wsrep_start_position is still not reset when mysqld is started with systemd. This is because MariaDB's systemd unit file has an ExecStartPre option defined that recovers this position using the galera_recovery wrapper script:

      https://github.com/MariaDB/server/blob/mariadb-10.1.39/support-files/mariadb.service.in#L72

      https://github.com/MariaDB/server/blob/mariadb-10.1.39/scripts/galera_recovery.sh

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              sciascid Daniele Sciascia
              Reporter:
              GeoffMontee Geoff Montee
              Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: