[MCOL-5269] LOAD DATA INFILE poor performance Created: 2022-10-18  Updated: 2023-11-17  Resolved: 2023-11-10

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: 6.4.4-dompe
Fix Version/s: 23.10

Type: Bug Priority: Critical
Reporter: Bryan Bancroft (Inactive) Assignee: Gagan Goel (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Environment:

56cores, 528gb, 5 node cluster


Attachments: Text File directCall.txt     Text File infileCall.txt    
Issue Links:
Relates
relates to MCOL-4853 allow cpimport to work when run with ... Closed

 Description   

mariadb documentation https://mariadb.com/docs/reference/col6/cli/cpimport/ ,says

"If the LOAD DATA [ LOCAL ] INFILE or INSERT INTO .. SELECT FROM .. statements are used with a ColumnStore table, MariaDB Enterprise ColumnStore 6 transforms the operation into a bulk load that uses cpimport by default"

The performance between LOAD DATA INFILE and cpimport by loading from the same file, there's a huge difference. Below is the log.

root@multi-unmanaged-db-p09:~# cpimport -s ',' -E '"' test tab1 /snfs1/temp/.db_backups/multi/test.csv
2022-10-17 04:43:07 (462877) INFO : Running distributed import (mode 1) on all PMs...
2022-10-17 04:44:16 (462877) INFO : For table test.tab1: 109593561 rows processed and 109593561 rows inserted.
2022-10-17 04:44:16 (462877) INFO : Bulk load completed, total run time : 69.135 seconds

MariaDB [test]> LOAD DATA INFILE
-> '/snfs1/temp/.db_backups/multi/test.csv'
-> INTO TABLE tab2
-> FIELDS TERMINATED BY ','
-> OPTIONALLY ENCLOSED BY '"'
-> LINES TERMINATED BY '\n'
-> ;
Query OK, 109593561 rows affected (6 min 59.174 sec)
Records: 109593561 Deleted: 0 Skipped: 0 Warnings: 0



 Comments   
Comment by Bryan Bancroft (Inactive) [ 2022-10-26 ]

https://jira.mariadb.org/browse/MCOL-4853 works around this issue

Comment by Chris Calender (Inactive) [ 2023-01-18 ]

regarding above comment, see the last comment in MCOL-4853 for the actual work-around

Generated at Thu Feb 08 02:56:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.