[MXS-4653] Integrate LDI filter with ColumnStore bulk loading API Created: 2023-06-22 Updated: 2023-08-01 Resolved: 2023-08-01 |
|
| Status: | Closed |
| Project: | MariaDB MaxScale |
| Component/s: | N/A |
| Affects Version/s: | None |
| Fix Version/s: | N/A |
| Type: | New Feature | Priority: | Major |
| Reporter: | markus makela | Assignee: | Todd Stoffel (Inactive) |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||||||||||
| Description |
|
The ldi filter supports xpand_import as a data loading tool. Adding support for cpimport would allow potentially faster data loads from S3 into ColumnStore with a LOAD DATA INFILE. |
| Comments |
| Comment by markus makela [ 2023-07-13 ] |
|
Split off from |
| Comment by Roman [ 2023-07-21 ] |
|
There are couple notes on the original case, namely how to speed up uploads from S3 into MCS. JFYI there is a way to use cpimport to ingest data from S3 into MCS using cpimport. There is a dedicated UDF that can be called from MDB. Check this issue for the details. This UDF internally triggers cpimport that knows how to download CSV(other formats are WIP) from S3. |