[MXS-2846] Binlogfilter: Add syntactic identifier replacement Created: 2020-01-24  Updated: 2022-09-08  Resolved: 2022-09-08

Status: Closed
Project: MariaDB MaxScale
Component/s: binlogfilter
Affects Version/s: None
Fix Version/s: N/A

Type: New Feature Priority: Major
Reporter: Zdravelina Sokolovska (Inactive) Assignee: Todd Stoffel (Inactive)
Resolution: Won't Do Votes: 0
Labels: externalDependency
Environment:

GKE
operator 0.4.3-3-skysql


Issue Links:
Relates
relates to MXS-2852 data dependency of the replicated dat... Closed
relates to MXS-2853 HTAP - Replication workflow dependenc... Closed

 Description   

The binlogfilter supports identifier replacement via regular expressions. Due to the complex nature of SQL, this is not a robust way of replacing identifiers. Parsing the SQL and doing syntactic replacement would solve this.


htap replication from innodb to columnstore table is not performed for object names overriding replication commands

deployed htab topology
created 3 source DBs and 1 target DB
created 3 inodb tables in the source DBs and 3 inodb tables in the first source DB
created 3 columnstore tables in the target DB
start replication between 3 tables in the first source DB to the target columnstore tables in the target DB.
insert values in the first innodb table. Data is not replicated to the columnstore table

[root@expmcsffooo2-mdb-cs-single-0 /]# mcsadmin getsystemi
getsysteminfo   Thu Jan 23 15:37:41 2020
 
System columnstore-1
 
System and Module statuses
 
Component     Status                       Last Status Change
------------  --------------------------   ------------------------
System        ACTIVE                       Thu Jan 23 15:29:16 2020
 
Module pm1    ACTIVE                       Thu Jan 23 15:29:13 2020
 
 
MariaDB ColumnStore Process statuses
 
Process             Module    Status            Last Status Change        Process ID
------------------  ------    ---------------   ------------------------  ----------
ProcessMonitor      pm1       ACTIVE            Thu Jan 23 15:27:12 2020         485
ProcessManager      pm1       ACTIVE            Thu Jan 23 15:27:20 2020         607
StorageManager      pm1       ACTIVE            Thu Jan 23 15:28:34 2020        3081
DBRMControllerNode  pm1       ACTIVE            Thu Jan 23 15:28:53 2020        3191
ServerMonitor       pm1       ACTIVE            Thu Jan 23 15:28:54 2020        3221
DBRMWorkerNode      pm1       ACTIVE            Thu Jan 23 15:28:55 2020        3251
PrimProc            pm1       ACTIVE            Thu Jan 23 15:28:59 2020        3406
ExeMgr              pm1       ACTIVE            Thu Jan 23 15:29:03 2020        3461
WriteEngineServer   pm1       ACTIVE            Thu Jan 23 15:29:07 2020        3522
DDLProc             pm1       ACTIVE            Thu Jan 23 15:29:11 2020        3579
DMLProc             pm1       ACTIVE            Thu Jan 23 15:29:16 2020        3626
mysqld              pm1       ACTIVE            Thu Jan 23 15:28:33 2020        3016
 
Active Alarm Counts: Critical = 0, Major = 0, Minor = 0, Warning = 0, Info = 0
[root@expmcsffooo2-mdb-cs-single-0 /]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 40
Server version: 10.4.11-5-MariaDB-enterprise-log MariaDB Enterprise Server
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]> create database b ;
Query OK, 1 row affected (0.004 sec)
 
MariaDB [(none)]> show databases ;
+---------------------+
| Database            |
+---------------------+
| b                   |
| calpontsys          |
| columnstore_info    |
| infinidb_querystats |
| information_schema  |
| mysql               |
| performance_schema  |
+---------------------+
7 rows in set (0.001 sec)
 
MariaDB [(none)]> create database a ;
Query OK, 1 row affected (0.005 sec)
 
MariaDB [(none)]> create database c ;
Query OK, 1 row affected (0.005 sec)
 
MariaDB [(none)]> create database d ;
Query OK, 1 row affected (0.005 sec)
 
MariaDB [(none)]>
MariaDB [(none)]> create table a.a ( a int) engine innidb ;
ERROR 1286 (42000): Unknown storage engine 'innidb'
MariaDB [(none)]> create table a.a ( a int) engine innodb ;
Query OK, 0 rows affected (0.017 sec)
 
MariaDB [(none)]> create table b.b ( a int) engine innodb ;
Query OK, 0 rows affected (0.021 sec)
 
MariaDB [(none)]> create table d.d ( a int) engine innodb ;
Query OK, 0 rows affected (0.020 sec)
 
MariaDB [(none)]> create table c.a ( a int) engine columnstore ;
Query OK, 0 rows affected (0.372 sec)
 
MariaDB [(none)]> create table c.b ( a int) engine columnstore ;
Query OK, 0 rows affected (1.066 sec)
 
MariaDB [(none)]> create table c.d ( a int) engine columnstore ;
Query OK, 0 rows affected (0.277 sec)
 
 
MariaDB [(none)]> create table a.a1 ( a int) engine innodb ;
Query OK, 0 rows affected (0.023 sec)
 
MariaDB [(none)]> create table a.a2 ( a int) engine innodb ;
Query OK, 0 rows affected (0.017 sec)
 
MariaDB [(none)]> create table c.a1 ( a int) engine columnstore ;
Query OK, 0 rows affected (0.312 sec)
 
MariaDB [(none)]> create table c.a2 ( a int) engine columnstore ;
Query OK, 0 rows affected (0.295 sec)
 
MariaDB [(none)]> select set_replications('a|a1|a2','a','c');
+-------------------------------------+
| set_replications('a|a1|a2','a','c') |
+-------------------------------------+
| Success.                            |
+-------------------------------------+
1 row in set (0.021 sec)
 
MariaDB [(none)]> insert into a.a values (1),(2),(3);
Query OK, 3 rows affected (0.008 sec)
Records: 3  Duplicates: 0  Warnings: 0
 
MariaDB [(none)]> select * from c.a ;
Empty set (0.087 sec)
 
MariaDB [(none)]> select list_replications();
+----------------------------------------------------------------------------------------+
| list_replications()                                                                    |
+----------------------------------------------------------------------------------------+
|
        === replication_filter ===
        table: a|a1|a2
        source database: a
        target database: c
 
 |
+----------------------------------------------------------------------------------------+
1 row in set (0.002 sec)
 
MariaDB [(none)]> show create table c.a;
+-------+-------------------------------------------------------------------------------------------+
| Table | Create Table                                                                              |
+-------+-------------------------------------------------------------------------------------------+
| a     | CREATE TABLE `a` (
  `a` int(11) DEFAULT NULL
) ENGINE=Columnstore DEFAULT CHARSET=latin1 |
+-------+-------------------------------------------------------------------------------------------+
1 row in set (0.000 sec)
 
MariaDB [(none)]> show create table a.a;
+-------+--------------------------------------------------------------------------------------+
| Table | Create Table                                                                         |
+-------+--------------------------------------------------------------------------------------+
| a     | CREATE TABLE `a` (
  `a` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+-------+--------------------------------------------------------------------------------------+
1 row in set (0.000 sec)
 
MariaDB [(none)]> select * from a.a ;
+------+
| a    |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.000 sec)
 
MariaDB [(none)]> select * from c.a ;
Empty set (0.010 sec)

MariaDB [c]> select set_replications('a','a','c');
+-------------------------------+
| set_replications('a','a','c') |
+-------------------------------+
| Success.                      |
+-------------------------------+
1 row in set (0.020 sec)
 
MariaDB [c]> insert into a.a values (1),(2),(3);
Query OK, 3 rows affected (0.005 sec)
Records: 3  Duplicates: 0  Warnings: 0
 
MariaDB [c]> select * from c.a ;
Empty set (0.054 sec)
 
MariaDB [c]> show slave status \G
*************************** 1. row ***************************
                Slave_IO_State: Waiting for master to send event
                   Master_Host: expmcsffooo2-mariadb-maxscale-replication
                   Master_User: skysql_replication
                   Master_Port: 3309
                 Connect_Retry: 60
               Master_Log_File: mysql-bin.000003
           Read_Master_Log_Pos: 19059
                Relay_Log_File: expmcsffooo2-mdb-cs-single-0-relay-bin.000002
                 Relay_Log_Pos: 684
         Relay_Master_Log_File: mysql-bin.000003
              Slave_IO_Running: Yes
             Slave_SQL_Running: No
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table:
                    Last_Errno: 1064
                    Last_Error: Error 'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'vclues (1),(2),(3)' at line 1' on query. Default database: 'c'. Query: 'insert into c.c vclues (1),(2),(3)'
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 18888
               Relay_Log_Space: 1187
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: No
            Master_SSL_CA_File:
            Master_SSL_CA_Path:
               Master_SSL_Cert:
             Master_SSL_Cipher:
                Master_SSL_Key:
         Seconds_Behind_Master: NULL
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno: 0
                 Last_IO_Error:
                Last_SQL_Errno: 1064
                Last_SQL_Error: Error 'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'vclues (1),(2),(3)' at line 1' on query. Default database: 'c'. Query: 'insert into c.c vclues (1),(2),(3)'
   Replicate_Ignore_Server_Ids:
              Master_Server_Id: 1
                Master_SSL_Crl:
            Master_SSL_Crlpath:
                    Using_Gtid: Current_Pos
                   Gtid_IO_Pos: 1-1-145
       Replicate_Do_Domain_Ids:
   Replicate_Ignore_Domain_Ids:
                 Parallel_Mode: conservative
                     SQL_Delay: 0
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State:
              Slave_DDL_Groups: 10
Slave_Non_Transactional_Groups: 0
    Slave_Transactional_Groups: 5
1 row in set (0.000 sec)



 Comments   
Comment by markus makela [ 2020-01-24 ]

This is a documented limitation: https://mariadb.com/kb/en/mariadb-maxscale-24-binlog-filter/#rewrite_src-and-rewrite_dest

The replacement is done both on the default database as well as the SQL statement in the query event. This means that great care must be taken when defining the rewriting rules. To prevent accidental modification of the SQL into a form that is no longer valid, use database and table names that never occur in the inserted data and is never used as a constant value.

Comment by Todd Stoffel (Inactive) [ 2020-01-24 ]

markus makela, is this a bug or a new feature request?

Comment by markus makela [ 2020-01-24 ]

Given the fact that it was never planned to do syntactic replacement, it'd be a feature request.

Comment by Nedyalko Petrov (Inactive) [ 2020-01-27 ]

toddstoffel - linking two more issues to the same functionality
1)Replication issues may have impact on the user data consistenccy ( not covered in the above limitation)
2)replication fails on CS side without clear user notification

Generated at Thu Feb 08 04:17:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.