[MCOL-4221] HTAP Deployment - poor performance on the ColumnStore replication Created: 2020-07-30 Updated: 2021-05-03 Resolved: 2020-09-12 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | N/A |
| Affects Version/s: | 1.0.0 |
| Fix Version/s: | 5.4.1 |
| Type: | Bug | Priority: | Major |
| Reporter: | David Hill (Inactive) | Assignee: | Todd Stoffel (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Environment: |
Maxscale and Columnstore using HTAP Deployment |
||
| Issue Links: |
|
||||||||
| Description |
|
Reported by customer We configured HTAP using "Deploy an Enterprise HTAP Server with MariaDB Enterprise ColumnStore 1.5 and MariaDB Enterprise Server 10.5." documentation on URL https://mariadb.com/docs/deploy/enterprise-htap/#deploy-target-enterprise-htap. We have tested replication from innoDB to ColumnStore using this configuration and are experiencing poor performance on the ColumnStore replication. We are trying to insert 10,000 rows into the innoDB table and seeing very poor performance replicating to the ColumnStore table. Our main question is the utility cpimport being used in the replication process, and if not is there a way the replicate process can utilize cpimport to improve performance? Are we using the right deployment which would invoke cpimport in the replication process? |
| Comments |
| Comment by David Hill (Inactive) [ 2020-07-30 ] |
|
From customer Can you please escalate the ticket to S1 since it is affecting our production implementation date? Cant change issue to S1, but we need some quick feedback to them if there are a y thoughts that |
| Comment by Todd Stoffel (Inactive) [ 2020-07-30 ] |
|
Replicating from a transactional engine into an analytics engine has limitations. Columnstore for instance cannot perform TPS at the same level as InnoDB. This feature is good for light to medium duty. If a higher TPS is required, then xpand should be considered. |
| Comment by David Hill (Inactive) [ 2020-07-31 ] |
|
This part of the initial comment wasn't answered. Customer again is asking: We are asking if the replication can invoke cpimport when replicating to columnstore. There was a documented API which was used in replication referred to here (see the diagram also) https://mariadb.com/kb/en/columnstore-streaming-data-adapters/ and here https://mariadb.com/kb/en/columnstore-bulk-write-sdk/ The link states the following: Starting with MariaDB ColumnStore 1.1 a C++ SDK is available which supports bulk write into ColumnStore. Conceptually this is an API version of cpimport. The SDK is intended to be integrated by custom code and adapters to enable easier publishing of data into ColumnStore. Since this was available in ColumnStore 1.1 we are surprised we are having these performance issues with Columnstore 1.5 |
| Comment by David Hill (Inactive) [ 2020-07-31 ] |
|
Need additional questions answer from customer |