Details
-
New Feature
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
As we look to scaling migrations, one of the needs is the ability to migrate data from oracle and other data sources with minimal downtimes.
The model that we can envision to achieve that is as follows :
*Oracle DB → Debezium (Oracle Kafka Connector) → Kafka Topic → MaxScale (KafkaImporter) → MariaDB
*
Debezium is an open source distributed platform for change data capture that supports a number of commercial and opensource databases. In the case of Oracle, it uses native LogMiner to capture the changes (https://debezium.io/documentation/reference/3.1/connectors/oracle.html)
The output from debezium can be a JSON payload or a kafkatopic for each table.
With maxscale's capability to support KafkaCDC and KafkaImporter a natural extension to this would be to read CDC JSON / Avro data and replicate them directly into the table.
Currently we support the ability where the payload of the message is inserted into the *data field *of the table and this is a column that it expects.
The feature request is to directly apply the table changes to a supported table format instead of just dumping the payload into MariaDB
We need to be able to read the CDC operation type and apply the appropriate changes into the database for a
"op": Operation type:
"c" = create (INSERT)
"u" = update
"d" = delete
"r" = snapshot read
The relevant section of the document is https://debezium.io/documentation/reference/3.1/connectors/oracle.html#oracle-events