Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31905

WSREP GTID MODE is inconsistent

Details

    Description

      WSREP GTID MODE is inconsistent

      When you create a Galera cluster with the following configuration according to documentation:

      # NODE 1 (m1)
      		 
      wsrep_gtid_mode=ON
      		 
      wsrep_gtid_domain_id=1111
      		 
      gtid_domain_id=2
      		 
      server_id=10999
      		 
      log_slave_updates=ON
      		 
      log_bin = /var/log/mysql/mariadb-bin-log
      		 
       
      		 
      # NODE 2 (m2)
      		 
      wsrep_gtid_mode=ON
      		 
      wsrep_gtid_domain_id=1111
      		 
      gtid_domain_id=3
      		 
      server_id=10999
      		 
      log_slave_updates=ON
      		 
      log_bin = /var/log/mysql/mariadb-bin-log
      		 
       
      		 
      # NODE 3 (m3)
      		 
      wsrep_gtid_mode=ON
      		 
      wsrep_gtid_domain_id=1111
      		 
      gtid_domain_id=4
      		 
      server_id=10999
      		 
      log_slave_updates=ON
      		 
      log_bin = /var/log/mysql/mariadb-bin-log

      Sometimes, the nodes will stay in sync. However, if the following is done:
      1) transactions run on all three nodes
      2) restart mariadb on any node
      3) restart all three nodes, using galera_new_cluster on a different node from first startup

      Then, GTIDs will become inconsistent both at the domain level and at the seq_no level.

      This is not difficult to reproduce.

      Attachments

        1. after-restart-all-three-nodes-galera-new-cluster-was-on-m2.png
          after-restart-all-three-nodes-galera-new-cluster-was-on-m2.png
          87 kB
        2. n1.cnf
          1.0 kB
        3. n2.cnf
          1.0 kB
        4. n3.cnf
          1.0 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28015 Mariabackup | GTID value is missing, Galera Cluster , adding async slave to it

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34170 table gtid_slave_pos entries never been deleted with wsrep_gtid_mode = 0

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34924 gtid_slave_pos table rows newer been deleted on non replica nodes (wsrep_gtid_mode = 1)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32740 wsrep_gtid_domain_id is disregarded on any node other than bootstrapped node when the wsrep_gtid_mode is set to ON with wsrep_provider_version 26.4.14 (Galera 4)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33610 Circular replication breaks after upgrading

          • Major - Major loss of function.
          • Open

          Activity

            Ascending order - Click to sort in descending order
            edward Edward Stoever added a comment -

            This was tested on 10.6.14 Enterprise.

            edward Edward Stoever added a comment - This was tested on 10.6.14 Enterprise.
            janlindstrom Jan Lindström added a comment - - edited

            Test case
            1) Start the 3 node cluster
            2) OLTP load started on all nodes
            3) The second node is restarted
            4) After 1 minute all nodes are stopped one by one
            3) bootstrapped node3 and initialized remaining nodes

            Note: reproduces with mariabackup not with rsync

            janlindstrom Jan Lindström added a comment - - edited Test case 1) Start the 3 node cluster 2) OLTP load started on all nodes 3) The second node is restarted 4) After 1 minute all nodes are stopped one by one 3) bootstrapped node3 and initialized remaining nodes Note: reproduces with mariabackup not with rsync
            janlindstrom Jan Lindström added a comment -

            edward Does customer use mariabacup as SST method?

            With mariabackup I can reproduce it even at startup

            connection node_1;
            		 
            CREATE SCHEMA IF NOT EXISTS mysqlslap;
            		 
            USE mysqlslap;
            		 
            CREATE TABLE t1 (id bigint not null primary key auto_increment, name VARCHAR(64)) ENGINE=innodb;
            		 
            # node_1
            		 
            show variables like '%gtid_binlog_pos%';
            		 
            Variable_name	Value
            		 
            gtid_binlog_pos	1111-1-2
            		 
            connection node_2;
            		 
            # node_2
            		 
            show variables like '%gtid_binlog_pos%';
            		 
            Variable_name	Value
            		 
            gtid_binlog_pos	0-1-2
            		 
            connection node_3;
            		 
            # node_3
            		 
            show variables like '%gtid_binlog_pos%';
            		 
            Variable_name	Value
            		 
            gtid_binlog_pos	0-1-2

            janlindstrom Jan Lindström added a comment - edward Does customer use mariabacup as SST method? With mariabackup I can reproduce it even at startup connection node_1; CREATE SCHEMA IF NOT EXISTS mysqlslap; USE mysqlslap; CREATE TABLE t1 (id bigint not null primary key auto_increment, name VARCHAR(64)) ENGINE=innodb; # node_1 show variables like '%gtid_binlog_pos%'; Variable_name Value gtid_binlog_pos 1111-1-2 connection node_2; # node_2 show variables like '%gtid_binlog_pos%'; Variable_name Value gtid_binlog_pos 0-1-2 connection node_3; # node_3 show variables like '%gtid_binlog_pos%'; Variable_name Value gtid_binlog_pos 0-1-2
            Richard Richard Stracke added a comment - - edited

            I checked the sourcecode and I shortened the relevant parts

             
            		 
                ******************************************
            		 
                uint  wsrep_gtid_domain_id=0;                   // Domain id on above structure
            		 
                
                void wsrep_init_globals()
            		 
                      wsrep_init_gtid();
            		 
                      
                      wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
            		 
                       stored_gtid.domain_id= wsrep_gtid_server.domain_id;     
                           if (stored_gtid.server_id == 0)
            		 
                                 rpl_gtid wsrep_last_gtid;
            		 
                                 mysql_bin_log.lookup_domain_in_binlog_state(stored_gtid.domain_id,
            		 
                                                                &wsrep_last_gtid))
            		 
                       wsrep_gtid_server.gtid(stored_gtid);
            		 
             
            		 
             
            		 
            if (wsrep_new_cluster)
            		 
                new_gtid.domain_id= wsrep_gtid_domain_id;  #  wsrep_gtid_domain_id=0;       
                    
                        /* Try to search for domain_id and server_id combination in binlog if found continue from last seqno */
            		 
                            wsrep_get_binlog_gtid_seqno(new_gtid);
            		 
                wsrep_gtid_server.gtid(new_gtid);
            		 
                
                *********************************************

            So domain_id will be initialized with
            wsrep_gtid_domain_id=0

            wsrep_gtid_domain_id can be set from SST , otherwise it remain 0

            wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
            will set the grid from recovery process, otherwise it kept empty.

            If new cluster the domain_id will be set to
            new_gtid.domain_id= wsrep_gtid_domain_id;

            IF the recovery process gives no result and new cluster flag is set,
            wsrep_gtid_domain_id kept 0 and will be overwritten here

            if (wsrep_new_cluster)
            		 
                new_gtid.domain_id= wsrep_gtid_domain_id;  #  wsrep_gtid_domain_id=0;

            rsync copies maybe some more files for a successful recovery or whatever 

            wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();

            do .

            So I suggest to init

            uint wsrep_gtid_domain_id=0; // Domain id on above structure

            to real wsrep_gtid_domain_id defined in config file
            or sst_method = mariabackup have to copy the same files like rsync for a successul wsrep_get_SE_checkpoint<wsrep_server_gtid_t>();
            seppo and janlindstrom

            Can you check this, if this makes sense to you ?

            Richard Richard Stracke added a comment - - edited I checked the sourcecode and I shortened the relevant parts   ****************************************** uint wsrep_gtid_domain_id= 0 ; // Domain id on above structure void wsrep_init_globals() wsrep_init_gtid(); wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>(); stored_gtid.domain_id= wsrep_gtid_server.domain_id; if (stored_gtid.server_id == 0 ) rpl_gtid wsrep_last_gtid; mysql_bin_log.lookup_domain_in_binlog_state(stored_gtid.domain_id, &wsrep_last_gtid)) wsrep_gtid_server.gtid(stored_gtid);     if (wsrep_new_cluster) new_gtid.domain_id= wsrep_gtid_domain_id; # wsrep_gtid_domain_id= 0 ; /* Try to search for domain_id and server_id combination in binlog if found continue from last seqno */ wsrep_get_binlog_gtid_seqno(new_gtid); wsrep_gtid_server.gtid(new_gtid); ********************************************* So domain_id will be initialized with wsrep_gtid_domain_id=0 wsrep_gtid_domain_id can be set from SST , otherwise it remain 0 wsrep_server_gtid_t stored_gtid= wsrep_get_SE_checkpoint<wsrep_server_gtid_t>(); will set the grid from recovery process, otherwise it kept empty. If new cluster the domain_id will be set to new_gtid.domain_id= wsrep_gtid_domain_id; IF the recovery process gives no result and new cluster flag is set, wsrep_gtid_domain_id kept 0 and will be overwritten here if (wsrep_new_cluster) new_gtid.domain_id= wsrep_gtid_domain_id; # wsrep_gtid_domain_id= 0 ; rsync copies maybe some more files for a successful recovery or whatever wsrep_get_SE_checkpoint<wsrep_server_gtid_t>(); do . So I suggest to init uint wsrep_gtid_domain_id=0; // Domain id on above structure to real wsrep_gtid_domain_id defined in config file or sst_method = mariabackup have to copy the same files like rsync for a successul wsrep_get_SE_checkpoint<wsrep_server_gtid_t>(); seppo and janlindstrom Can you check this, if this makes sense to you ?
            seppo Seppo Jaakola added a comment -

            Pull request to handle domain-id inconsistency due to mariabackup SST was submitted yesterday

            seppo Seppo Jaakola added a comment - Pull request to handle domain-id inconsistency due to mariabackup SST was submitted yesterday
            seppo Seppo Jaakola added a comment -

            PR is submitted here: https://github.com/MariaDB/server/pull/2932

            seppo Seppo Jaakola added a comment - PR is submitted here: https://github.com/MariaDB/server/pull/2932
            sysprg Julius Goryavsky added a comment -

            Fix merged with head revision as https://github.com/MariaDB/server/commit/c89f769f2412202dd8d7e69b0942778c91602ec7

            sysprg Julius Goryavsky added a comment - Fix merged with head revision as https://github.com/MariaDB/server/commit/c89f769f2412202dd8d7e69b0942778c91602ec7

            People

              sysprg Julius Goryavsky
              edward Edward Stoever
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.