[MXS-3806] Provide filtering for the KafkaCDC Router Created: 2021-10-07  Updated: 2022-12-09  Resolved: 2022-03-09

Status: Closed
Project: MariaDB MaxScale
Component/s: cdc
Affects Version/s: 6.1.3
Fix Version/s: 6.3.0

Type: New Feature Priority: Major
Reporter: Kyle Joiner (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 1
Labels: None

Sprint: MXS-SPRINT-152

 Description   

Provide a means to filter on database and/or table the replication stream of the KafkaCDC Router from a primary or Galera cluster.



 Comments   
Comment by Naresh Chandra [ 2021-11-05 ]

Provide a filter like Exclude a table/schema and Include a table/schema from the Galera cluster or Primary node to the KafkaCDC router. So that we will be replicate which table or schema need to replicate from the Galera cluster or Primary node to KafkaCDC router.

Comment by Naresh Chandra [ 2021-11-19 ]

Can we fix this issue in the next release?

Comment by Naresh Chandra [ 2021-11-22 ]

Provide a Schema/Table filtering option directly from Galera Cluster/Primary Node to KafkaCDC router without binlog router.

Comment by Naresh Chandra [ 2021-12-08 ]

Provide a Schema/Table filtering option directly from Galera Cluster/Primary Node to KafkaCDC router without binlog router.
Please add an exclude and include options for Schema or Table would be good.

Comment by Naresh Chandra [ 2022-02-17 ]

Hi Todd Stoffel,

Can you please take it up in the coming release? We need filtering for KafkaCDC router as we are planning for Production environment.

Comment by Naresh Chandra [ 2022-02-17 ]

We tried with binlog router but we are facing crash issues with binlog filter. and the main thing its not converting the table structures if use the binlog router.

So instead of that, we need own filter for KafkaCDC router would be good.

Comment by Naresh Chandra [ 2022-02-21 ]

Hi Todd Stoffel,

Can you please take it up in the coming release? We need filtering for Kafka CDC router as we are planning for Production environment.

We tried with binlog router but we are facing crash issues with binlog filter. and the main thing its not converting the table structures if use the binlog router.

So instead of that, we need own filter for Kafka CDC router would be good.

Comment by Naresh Chandra [ 2022-02-25 ]

Hi Todd,

Thanks for the update.

Comment by markus makela [ 2022-03-04 ]

Would two regex parameters, exclude and match, be enough to solve the problem for you? This would be the same that the binlogfilter provides.

Comment by Naresh Chandra [ 2022-03-04 ]

Yes Markus, We need Table/Schema level filtering which is exclude and match cases like binlog router.

Comment by Naresh Chandra [ 2022-03-07 ]

Hi Markus,

We can filter entire database right?

ex1: db1.* - filter all the tables.
ex2: db2.tbl1 - we can filter only one table right?
ex3: db3.tb1, db4.tbl8, db9.tbl20 - we can filter specific tables from different db right?

The above combination should work for both the match and exclude parameters right?

Comment by markus makela [ 2022-03-07 ]

If this is implemented using regular expressions then all the cases are possible as long as the correct regular expression is used. The three patterns for those cases would be:

db1[.].*
db2[.]tbl1
db3[.]tbl1|db4[.]tbl8|db9[.]tbl20

The benefit of using a regular expression is that it is very versatile and allows for a great degree of freedom on what to match against but the syntax is not as convenient as a simple list of table or database names.

Comment by Naresh Chandra [ 2022-03-07 ]

Hi Markus,

Looks good, thanks for the quick fix.

Generated at Thu Feb 08 04:24:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.