[MXS-3806] Provide filtering for the KafkaCDC Router Created: 2021-10-07 Updated: 2022-12-09 Resolved: 2022-03-09 |
|
| Status: | Closed |
| Project: | MariaDB MaxScale |
| Component/s: | cdc |
| Affects Version/s: | 6.1.3 |
| Fix Version/s: | 6.3.0 |
| Type: | New Feature | Priority: | Major |
| Reporter: | Kyle Joiner (Inactive) | Assignee: | markus makela |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Sprint: | MXS-SPRINT-152 |
| Description |
|
Provide a means to filter on database and/or table the replication stream of the KafkaCDC Router from a primary or Galera cluster. |
| Comments |
| Comment by Naresh Chandra [ 2021-11-05 ] | |||
|
Provide a filter like Exclude a table/schema and Include a table/schema from the Galera cluster or Primary node to the KafkaCDC router. So that we will be replicate which table or schema need to replicate from the Galera cluster or Primary node to KafkaCDC router. | |||
| Comment by Naresh Chandra [ 2021-11-19 ] | |||
|
Can we fix this issue in the next release? | |||
| Comment by Naresh Chandra [ 2021-11-22 ] | |||
|
Provide a Schema/Table filtering option directly from Galera Cluster/Primary Node to KafkaCDC router without binlog router. | |||
| Comment by Naresh Chandra [ 2021-12-08 ] | |||
|
Provide a Schema/Table filtering option directly from Galera Cluster/Primary Node to KafkaCDC router without binlog router. | |||
| Comment by Naresh Chandra [ 2022-02-17 ] | |||
|
Hi Todd Stoffel, Can you please take it up in the coming release? We need filtering for KafkaCDC router as we are planning for Production environment. | |||
| Comment by Naresh Chandra [ 2022-02-17 ] | |||
|
We tried with binlog router but we are facing crash issues with binlog filter. and the main thing its not converting the table structures if use the binlog router. So instead of that, we need own filter for KafkaCDC router would be good. | |||
| Comment by Naresh Chandra [ 2022-02-21 ] | |||
|
Hi Todd Stoffel, Can you please take it up in the coming release? We need filtering for Kafka CDC router as we are planning for Production environment. We tried with binlog router but we are facing crash issues with binlog filter. and the main thing its not converting the table structures if use the binlog router. So instead of that, we need own filter for Kafka CDC router would be good. | |||
| Comment by Naresh Chandra [ 2022-02-25 ] | |||
|
Hi Todd, Thanks for the update. | |||
| Comment by markus makela [ 2022-03-04 ] | |||
|
Would two regex parameters, exclude and match, be enough to solve the problem for you? This would be the same that the binlogfilter provides. | |||
| Comment by Naresh Chandra [ 2022-03-04 ] | |||
|
Yes Markus, We need Table/Schema level filtering which is exclude and match cases like binlog router. | |||
| Comment by Naresh Chandra [ 2022-03-07 ] | |||
|
Hi Markus, We can filter entire database right? ex1: db1.* - filter all the tables. The above combination should work for both the match and exclude parameters right? | |||
| Comment by markus makela [ 2022-03-07 ] | |||
|
If this is implemented using regular expressions then all the cases are possible as long as the correct regular expression is used. The three patterns for those cases would be:
The benefit of using a regular expression is that it is very versatile and allows for a great degree of freedom on what to match against but the syntax is not as convenient as a simple list of table or database names. | |||
| Comment by Naresh Chandra [ 2022-03-07 ] | |||
|
Hi Markus, Looks good, thanks for the quick fix. |