[MXS-1686] Direct Kafka Integration Created: 2018-02-26 Updated: 2024-01-04 Resolved: 2020-02-03 |
|
| Status: | Closed |
| Project: | MariaDB MaxScale |
| Component/s: | avrorouter, failover |
| Affects Version/s: | None |
| Fix Version/s: | N/A |
| Type: | New Feature | Priority: | Major |
| Reporter: | Drew Schatt | Assignee: | markus makela |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Epic Link: | Stream To Kafka (Milestone 1) | ||||||||
| Sprint: | MXS-SPRINT-98 | ||||||||
| Description |
|
Currently, the CDC events are presented to the user via a particular channel, and a custom script has to pull those events, and then send them on to kafka (if using the kafka integration). It would be ideal if maxscale could directly send these events directly to kafka. Necessary information would be the kafka cluster address, the zookeeper address(es), and what tables (via regex, ideally both white and black lists) that should be sent. In theory, this could happen on the fly, and completely skip writing these events to disk. It would be nice if maxscale could use zookeeper to keep the current position of the last event sent, in order to not send duplicates after a crash. |
| Comments |
| Comment by markus makela [ 2020-02-03 ] |
|
The new kafkacdc router module streams data from MariaDB into a Kafka broker. |
| Comment by wilsonlau [ 2020-06-28 ] |
|
how to set kafkacdc GTID? |
| Comment by markus makela [ 2020-06-29 ] |
|
The kafkacdc module starts replication from the beginning. This is an oversight and the GTID should be configurable like it is for other modules that rely on it. |