[MCOL-1661] Transform CDC events into UPDATE and DELETE statements Created: 2018-08-20  Updated: 2023-10-26  Resolved: 2018-09-12

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1.6

Type: New Feature Priority: Major
Reporter: markus makela Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MCOL-1718 Clarify configuration requirements fo... Closed
Sprint: 2018-17

 Description   

Being able to transform the CDC events into events that update the ColumnStore instance allows replication of data from a remote MariaDB server to a ColumnStore setup.

As the ColumnStore API does not support updates or deletes, the adapter has to transform them into SQL statements to provide a current data stream instead of an append-only historical data stream.

 Comments   
Comment by markus makela [ 2018-08-20 ]

Implemented a mode that replicates changes to CS. It's slow since it uses DMLs but MCOL-1662 should improve it slightly.

Comment by Elena Kotsinova (Inactive) [ 2018-08-22 ]

markus makela What is the Fixed version? In which version fix will be included?

Comment by markus makela [ 2018-08-22 ]

Probably the next release made from develop-1.1. I'll update the fixVersion field.

Comment by Elena Kotsinova (Inactive) [ 2018-09-11 ]

Markus, I need more information on this.
Does this mean that the record updated in TX appears updated in CS instead of inserting 2 new additional rows with event_type - "update before" and "update after"?
Because I continue to see additional records for update and delete in 1.1.6

Comment by markus makela [ 2018-09-11 ]

The new `-z` flag was added that enables the transformation mode. Was it enabled? This is required to have the adapter "replicate" changes to ColumnStore.

Comment by Elena Kotsinova (Inactive) [ 2018-09-12 ]

I tried the -z option and I received the following failure message:

[root@maxscale-cdc QA]# mxs_adapter -s /root/QA/ -c /etc/Columnstore.xml -u cdcuser -p cdcpassword -h localhost -P 4001 -f /root/QA/2DBConfig.tsv -z
2018-09-12 14:06:08 [cdc_eli.edk] Continuing from GTID: 0-1-38:2
2018-09-12 14:06:08 [cdc_eli.edk_delta] Continuing from GTID: 0-1-33:1
2018-09-12 14:06:08 [cdc_eli.edk] Failed to connect to ColumnStore SQL interface: Access denied for user 'root'@'172.20.2.240' (using password: NO)
2018-09-12 14:06:08 [cdc_eli.edk] Failed to connect to ColumnStore SQL interface: Access denied for user 'root'@'172.20.2.240' (using password: NO)
2018-09-12 14:06:08 [cdc_eli.edk] Failed to connect to ColumnStore SQL interface: Access denied for user 'root'@'172.20.2.240' (using password: NO)
2018-09-12 14:06:08 [cdc_eli.edk] Failed to connect to ColumnStore SQL interface: Access denied for user 'root'@'172.20.2.240' (using password: NO)

Do I need to setup something in addition?
The 172.20.2.240 is the server where the maxscale and adapter run. ColumnStore is on other server.

Comment by markus makela [ 2018-09-12 ]

The transformation mode uses the user used for cross-engine queries. The user defined in Columnstore.xml would require access to create and modify tables.

It might be that this is not a proper user to use for this and a separate one should be defined in Columnstore.xml.

Comment by Elena Kotsinova (Inactive) [ 2018-09-12 ]

I think that this should be explicitly mentioned in the user guide.
User should prepare in addition cross engine configuration (i.e. to create proper user and grant permissions).
I will create a separate issue about this.

Generated at Thu Feb 08 02:30:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.