[MXS-3304] binlogrouter can not manage GTID sequence number higher than 4294967295 Created: 2020-11-20  Updated: 2021-08-02  Resolved: 2021-08-02

Status: Closed
Project: MariaDB MaxScale
Component/s: binlogrouter
Affects Version/s: 2.4.13
Fix Version/s: 2.5.0

Type: Bug Priority: Major
Reporter: Yurii Assignee: markus makela
Resolution: Won't Fix Votes: 0
Labels: None
Environment:

Ubuntu 18

MariaDB server:

root@db-allc01 [(none)] > show global variables like "%gtid_binlog_pos%";
+-----------------+--------------------+
| Variable_name   | Value              |
+-----------------+--------------------+
| gtid_binlog_pos | 18-5141-6682454633 |
+-----------------+--------------------+
1 row in set (0.001 sec)
 
root@db-allc01 [(none)] > select @@version;
+--------------------------------------------+
| @@version                                  |
+--------------------------------------------+
| 10.4.17-MariaDB-1:10.4.17+maria~bionic-log |
+--------------------------------------------+
1 row in set (0.000 sec)

Maxscale (reproduces on version 2.4.13 as well):

# maxscale --version
MaxScale 2.2.16

Binlogrouter config:

[Binlog_Service]
type=service
router=binlogrouter
router_options=slave_hostname='maxscale02.adcloud-prod',server-id=22089,binlogdir=/mysql/maxscale_oltp/binlogs
user=maxscale
passwd=PASSWORD
mariadb10_master_gtid=on
 
[Binlog_Listener]
type=listener
service=Binlog_Service
protocol=MySQLClient
port=3389



 Description   

I think Maxscale binlogrouter can not manage GTID sequence number higher than 4294967295.

Steps to reproduce:
1. Connect to maxscale and init replica connection:

# mysql -umaxscale -p -h127.0.0.1 -P3389
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 10.4.17-MariaDB-1:10.4.17+maria~bionic-log
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]> SET @@global.gtid_slave_pos = '18-5141-6682454633';
Query OK, 0 rows affected (0.000 sec)
 
MariaDB [(none)]> CHANGE MASTER TO MASTER_HOST = '172.27.33.19', MASTER_PORT = 3306, MASTER_USER='maxscale', MASTER_PASSWORD='PASSWORD', MASTER_USE_GTID = slave_pos;
Query OK, 0 rows affected (0.000 sec)
 
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Slave stopped
                  Master_Host: 172.27.33.19
                  Master_User: maxscale
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File:
          Read_Master_Log_Pos: 4
               Relay_Log_File:
                Relay_Log_Pos: 4
        Relay_Master_Log_File:
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 4
              Relay_Log_Space: 4
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0
                  Master_UUID: 37195e8c-2ad5-11eb-81e3-122b63787327
             Master_Info_File: /mysql/maxscale_oltp/binlogs/master.ini
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave stopped
           Master_Retry_Count: 1000
                  Master_Bind:
      Last_IO_Error_TimeStamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: Slave_pos
                  Gtid_IO_Pos: 18-5141-6682454633
1 row in set (0.000 sec)
 
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (1.011 sec)
 
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Binlog Dump
                  Master_Host: 172.27.33.19
                  Master_User: maxscale
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 855
               Relay_Log_File: mysql-bin.000001
                Relay_Log_Pos: 342
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 342
              Relay_Log_Space: 342
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 5141
                  Master_UUID: 37195e8c-2ad5-11eb-81e3-122b63787327
             Master_Info_File: /mysql/maxscale_oltp/binlogs/master.ini
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave running
           Master_Retry_Count: 1000
                  Master_Bind:
      Last_IO_Error_TimeStamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: Slave_pos
                  Gtid_IO_Pos: 18-5141-6682454633
1 row in set (0.000 sec)

After starting replica position is correct and not moving.

2. running statement on db-allc01 (mariadb server):

root@db-allc01 [test] > create table t2 (c int);
Query OK, 0 rows affected (0.097 sec)
 
root@db-allc01 [test] > show global variables like "%gtid_binlog_pos%";
+-----------------+--------------------+
| Variable_name   | Value              |
+-----------------+--------------------+
| gtid_binlog_pos | 18-5141-6682454634 |
+-----------------+--------------------+
1 row in set (0.001 sec)

3. Gtid_IO_Pos possition on maxscale side has significantly changed to 18-5141-2387487339:

6682454634 - 4294967295 = 2387487339

MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Binlog Dump
                  Master_Host: 172.27.33.19
                  Master_User: maxscale
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 987
               Relay_Log_File: mysql-bin.000001
                Relay_Log_Pos: 987
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 987
              Relay_Log_Space: 987
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 5141
                  Master_UUID: 37195e8c-2ad5-11eb-81e3-122b63787327
             Master_Info_File: /mysql/maxscale_oltp/binlogs/master.ini
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave running
           Master_Retry_Count: 1000
                  Master_Bind:
      Last_IO_Error_TimeStamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: Slave_pos
                  Gtid_IO_Pos: 18-5141-2387487339
1 row in set (0.000 sec)

After stopping and starting replication stream with such position - we will get broken replication:

MariaDB [(none)]> stop slave;
Query OK, 0 rows affected (0.000 sec)
 
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Slave stopped
                  Master_Host: 172.27.33.19
                  Master_User: maxscale
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 987
               Relay_Log_File: mysql-bin.000001
                Relay_Log_Pos: 987
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 987
              Relay_Log_Space: 987
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 5141
                  Master_UUID: 37195e8c-2ad5-11eb-81e3-122b63787327
             Master_Info_File: /mysql/maxscale_oltp/binlogs/master.ini
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave stopped
           Master_Retry_Count: 1000
                  Master_Bind:
      Last_IO_Error_TimeStamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: Slave_pos
                  Gtid_IO_Pos: 18-5141-2387487339
1 row in set (0.000 sec)
 
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (1.010 sec)
 
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Slave stopped
                  Master_Host: 172.27.33.19
                  Master_User: maxscale
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 987
               Relay_Log_File: mysql-bin.000001
                Relay_Log_Pos: 987
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1236
                   Last_Error: Could not find GTID state requested by slave in any binlog files. Probably the slave state is too old and required binlog files have been purged.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 987
              Relay_Log_Space: 987
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 5141
                  Master_UUID: 37195e8c-2ad5-11eb-81e3-122b63787327
             Master_Info_File: /mysql/maxscale_oltp/binlogs/master.ini
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave stopped
           Master_Retry_Count: 1000
                  Master_Bind:
      Last_IO_Error_TimeStamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: Slave_pos
                  Gtid_IO_Pos: 18-5141-2387487339
1 row in set (0.000 sec)


I tested same thing in the same environment with original GTID on master MariaDB server = 18-5141-241358389 (sequence number less than 4294967295). In such a case everything works.

+Expected behaviour: + MaxScale binlog router service can process GTID sequence number higher than 4294967295 without any limitations.
In MariaDB server GTID sequence number has STRING type, looks like in MaxScale it is not.



 Comments   
Comment by markus makela [ 2020-11-23 ]

MaxScale 2.2 is EOL, can you reproduce this with MaxScale 2.5?

Comment by Yurii [ 2020-11-23 ]

It was reproduces with 2.4.13. Same steps, same issue.

Comment by markus makela [ 2020-11-23 ]

The binlogrouter was rewritten for 2.5 and it's possible the problem is already solved there.

Comment by Yurii [ 2020-11-23 ]

I do not have any information about 2.5. As I know, 2.4.13 version is supported version. Can we have a fix in 2.4 and other supported version of MaxScale?

Comment by markus makela [ 2020-11-24 ]

2.5 is the latest release of MaxScale and it is a supported version. Testing with 2.5 would help identify whether a workaround exists for this problem.

Comment by Yurii [ 2020-11-26 ]

I can not reproduce the issue in 2.5.5

root@maxscale03:/mysql/maxscale_ui# maxscale -V
MaxScale 2.5.5 - 91c3b76195d0057ddbe572bbb8d17f6ac6b09d5e
CMake flags:   -DBUILD_TESTS=N -DBUILD_MMMON=Y -DBUILD_CDC=Y -DBUILD_GUI=Y -DPACKAGE=Y -DDISTRIB_SUFFIX=ubuntu.bionic

Should we expect fix for 2.4 ?

Comment by markus makela [ 2020-12-08 ]

For now I'd recommend using 2.5 as a workaround to this problem until a fix for 2.4 is created.

Comment by markus makela [ 2021-08-02 ]

Since 2.5 is out and doesn't suffer from this, I don't think this can be easily fixed in 2.4. I'll close this as Won't Fix and I'd recommend using 2.5.

Generated at Thu Feb 08 04:20:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.