Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12012

gtid_domain_id doesn't work with multisource between 10.1 and 10.0 + GTID

    Details

    • Sprint:
      10.2.11

      Description

      Hello,

      I have the following scenario with a slave running multisource and no GTID and at some point I want to enable GTID on that existing slave. So I start setting the gtid_domain_id flags on both masters and it slaves as follows:

      master 1:
      gtid_domain_id=12
      version: 10.0.28

      master 2:
      gtid_domain_id=14
      version: 10.0.28

      slave1 which replicates from both masters with multi source
      gtid_domain_id=10
      version: 10.1.21
      gtid disabled

      Current status after setting gtid_domain_id on the three hosts:

      MariaDB [gtid_10]> show all slaves status\G
      *************************** 1. row ***************************
                    Connection_name: 12
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 192.168.56.12
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mariadb-bin.000061
                Read_Master_Log_Pos: 2510
                     Relay_Log_File: mysqld-relay-bin-12.000002
                      Relay_Log_Pos: 538
              Relay_Master_Log_File: mariadb-bin.000061
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 2510
                    Relay_Log_Space: 840
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: 0
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error:
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 12
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: No
                        Gtid_IO_Pos: 0-2-210669,12-12-24,14-14-1
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 29
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-2-23,14-14-1
      *************************** 2. row ***************************
                    Connection_name: 14
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 192.168.56.14
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mysqld-bin.000063
                Read_Master_Log_Pos: 500
                     Relay_Log_File: mysqld-relay-bin-14.000002
                      Relay_Log_Pos: 537
              Relay_Master_Log_File: mysqld-bin.000063
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 500
                    Relay_Log_Space: 839
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: 0
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error:
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 14
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: No
                        Gtid_IO_Pos: 0-2-210669,12-12-24,14-14-1
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 15
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-2-23,14-14-1
      2 rows in set (0.00 sec)
      

      I keep inserting data and at some point I want to enable GTID, (slave_pos) and they crash replication:

      MariaDB [gtid_10]>  stop all slaves;
      Query OK, 0 rows affected, 2 warnings (0.00 sec)
       
      MariaDB [gtid_10]> change master '12' to master_use_gtid=slave_pos;
      Query OK, 0 rows affected (0.01 sec)
       
      MariaDB [gtid_10]> change master '14' to master_use_gtid=slave_pos;
      Query OK, 0 rows affected (0.01 sec)
       
      MariaDB [gtid_10]> start all slaves;
      Query OK, 0 rows affected, 2 warnings (0.02 sec)
       
      MariaDB [gtid_10]> show all slaves status\G
      *************************** 1. row ***************************
                    Connection_name: 12
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 192.168.56.12
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mariadb-bin.000061
                Read_Master_Log_Pos: 2631
                     Relay_Log_File: mysqld-relay-bin-12.000002
                      Relay_Log_Pos: 762
              Relay_Master_Log_File: mariadb-bin.000061
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 2631
                    Relay_Log_Space: 1064
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: 0
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error:
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 12
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: Slave_Pos
                        Gtid_IO_Pos: 0-2-210669,12-12-25,14-14-2
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 40
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-12-25,14-14-2
      *************************** 2. row ***************************
                    Connection_name: 14
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State:
                        Master_Host: 192.168.56.14
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mysqld-bin.000063
                Read_Master_Log_Pos: 621
                     Relay_Log_File: mysqld-relay-bin-14.000001
                      Relay_Log_Pos: 4
              Relay_Master_Log_File: mysqld-bin.000063
                   Slave_IO_Running: No
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 621
                    Relay_Log_Space: 249
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: NULL
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 1236
                      Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID 12-12-25, which is not in the master's binlog'
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 14
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: Slave_Pos
                        Gtid_IO_Pos: 0-2-210669,12-12-25,14-14-2
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 18
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-12-25,14-14-2
      2 rows in set (0.00 sec)
      

      This happens with both, current_pos and slave_pos.
      If I go back to non GTID replication gets fixed and I can keep inserting data.

      MariaDB [gtid_10]> stop all slaves;
      Query OK, 0 rows affected, 2 warnings (0.01 sec)
       
      MariaDB [gtid_10]> change master '12' to master_use_gtid=no;
      Query OK, 0 rows affected (0.00 sec)
       
      MariaDB [gtid_10]> change master '14' to master_use_gtid=no;
      Query OK, 0 rows affected (0.01 sec)
       
      MariaDB [gtid_10]> start all slaves;
      Query OK, 0 rows affected, 2 warnings (0.00 sec)
       
      MariaDB [gtid_10]> show all slaves status\G
      *************************** 1. row ***************************
                    Connection_name: 12
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 192.168.56.12
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mariadb-bin.000061
                Read_Master_Log_Pos: 2631
                     Relay_Log_File: mysqld-relay-bin-12.000002
                      Relay_Log_Pos: 538
              Relay_Master_Log_File: mariadb-bin.000061
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 2631
                    Relay_Log_Space: 840
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: 0
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error:
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 12
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: No
                        Gtid_IO_Pos: 0-10-210671,12-2-23,14-14-2,10-10-17
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 51
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-2-23,14-14-2
      *************************** 2. row ***************************
                    Connection_name: 14
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 192.168.56.14
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mysqld-bin.000063
                Read_Master_Log_Pos: 621
                     Relay_Log_File: mysqld-relay-bin-14.000002
                      Relay_Log_Pos: 537
              Relay_Master_Log_File: mysqld-bin.000063
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 621
                    Relay_Log_Space: 839
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: 0
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error:
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 14
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: No
                        Gtid_IO_Pos: 0-10-210671,12-2-23,14-14-2,10-10-17
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 29
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-2-23,14-14-2
      2 rows in set (0.00 sec)
      

      And I insert data on the masters and they get replicated:

      MariaDB [gtid_10]> show all slaves status\G
      *************************** 1. row ***************************
                    Connection_name: 12
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 192.168.56.12
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mariadb-bin.000061
                Read_Master_Log_Pos: 2748
                     Relay_Log_File: mysqld-relay-bin-12.000002
                      Relay_Log_Pos: 655
              Relay_Master_Log_File: mariadb-bin.000061
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 2748
                    Relay_Log_Space: 957
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: 0
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error:
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 12
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: No
                        Gtid_IO_Pos: 0-10-210671,12-2-23,14-14-2,10-10-17
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 53
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-12-26,14-14-3
      *************************** 2. row ***************************
                    Connection_name: 14
                    Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 192.168.56.14
                        Master_User: slave_user
                        Master_Port: 3306
                      Connect_Retry: 60
                    Master_Log_File: mysqld-bin.000063
                Read_Master_Log_Pos: 748
                     Relay_Log_File: mysqld-relay-bin-14.000002
                      Relay_Log_Pos: 664
              Relay_Master_Log_File: mysqld-bin.000063
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB:
                Replicate_Ignore_DB:
                 Replicate_Do_Table:
             Replicate_Ignore_Table:
            Replicate_Wild_Do_Table:
        Replicate_Wild_Ignore_Table:
                         Last_Errno: 0
                         Last_Error:
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 748
                    Relay_Log_Space: 966
                    Until_Condition: None
                     Until_Log_File:
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File:
                 Master_SSL_CA_Path:
                    Master_SSL_Cert:
                  Master_SSL_Cipher:
                     Master_SSL_Key:
              Seconds_Behind_Master: 0
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error:
                     Last_SQL_Errno: 0
                     Last_SQL_Error:
        Replicate_Ignore_Server_Ids:
                   Master_Server_Id: 14
                     Master_SSL_Crl:
                 Master_SSL_Crlpath:
                         Using_Gtid: No
                        Gtid_IO_Pos: 0-10-210671,12-2-23,14-14-2,10-10-17
            Replicate_Do_Domain_Ids:
        Replicate_Ignore_Domain_Ids:
                      Parallel_Mode: conservative
               Retried_transactions: 0
                 Max_relay_log_size: 104857600
               Executed_log_entries: 31
          Slave_received_heartbeats: 0
             Slave_heartbeat_period: 1800.000
                     Gtid_Slave_Pos: 0-10-210671,10-10-17,12-12-26,14-14-3
      

      Can this be related?: https://mariadb.atlassian.net/browse/MDEV-4485

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Elkin Andrei Elkin
                Reporter:
                marostegui Manuel Arostegui
              • Votes:
                2 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: