[MCOL-4888] improve cpimport operating on wide tables Created: 2021-10-04 Updated: 2023-12-19 Resolved: 2023-12-19 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | PrimProc |
| Affects Version/s: | None |
| Fix Version/s: | 23.10 |
| Type: | New Feature | Priority: | Major |
| Reporter: | Gregory Dorman (Inactive) | Assignee: | Roman |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | None | ||
| Sprint: | 2021-11, 2021-12, 2021-13, 2021-14, 2021-15 |
| Description |
|
drrtuy speaking: make a cpimport optimization that will benefit bulk insertion for tables with many columns. In short cpimport saves some extra data to be able to rollback. It is about 200kB per column file so for 200 column table this is 40MB that is not so big for the modern disks but slows down the operation. I recall a prospect who tolld that ClickHouse ingests data faster then MCS and this metadata was the reason why MCS was slow ingesting. |
| Comments |
| Comment by Roman [ 2023-12-19 ] |
|
CH ingests faster from CSV and it does this b/c it leverages parallel CSV read. |