[MCOL-659] cpimport failure for > 1k col wide table = 300k csv lines Created: 2017-04-10  Updated: 2020-08-25  Resolved: 2017-08-09

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: 1.0.7
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: David Hill (Inactive) Assignee: David Thompson (Inactive)
Resolution: Not a Bug Votes: 0
Labels: None


 Description   

200k or less csv lines works ok

300k or more always fails for me and leaves the draws table locked ... i restarted the server after that and it was ok until i locked it up again

root@columnstore-testing-d01:/home/techteamdbdev/draws# cpimport -s ',' -E '"' draws draws draws.csv
Locale is : C
Column delimiter : ,
Enclosed by Character : "

Using table OID 20879 as the default JOB ID
Input file(s) will be read from : /home/techteamdbdev/draws
Job description file : /usr/local/mariadb/columnstore/data/bulk/tmpjob/20879_D20170410_T181326_S307847_Job_20879.xml
Log file for this job: /usr/local/mariadb/columnstore/data/bulk/log/Job_20879.log
2017-04-10 18:13:26 (55594) INFO : successfully loaded job file /usr/local/mariadb/columnstore/data/bulk/tmpjob/20879_D20170410_T181326_S307847_Job_20879.xml
2017-04-10 18:13:26 (55594) INFO : Job file loaded, run time for this step : 0.165293 seconds
2017-04-10 18:13:26 (55594) INFO : PreProcessing check starts
2017-04-10 18:13:26 (55594) INFO : input data file /home/techteamdbdev/draws/draws.csv
2017-04-10 18:13:34 (55594) INFO : PreProcessing check completed
2017-04-10 18:13:34 (55594) INFO : preProcess completed, run time for this step : 8.17341 seconds
2017-04-10 18:13:34 (55594) INFO : No of Read Threads Spawned = 1
2017-04-10 18:13:34 (55594) INFO : No of Parse Threads Spawned = 3
Killed

-----------------

root@columnstore-testing-d01:/home/techteamdbdev/draws# cpimport -s ',' -E '"' draws draws draws.csv
Locale is : C
Column delimiter : ,
Enclosed by Character : "

Using table OID 20879 as the default JOB ID
Input file(s) will be read from : /home/techteamdbdev/draws
Job description file : /usr/local/mariadb/columnstore/data/bulk/tmpjob/20879_D20170410_T181526_S254449_Job_20879.xml
Log file for this job: /usr/local/mariadb/columnstore/data/bulk/log/Job_20879.log
2017-04-10 18:15:26 (56061) INFO : successfully loaded job file /usr/local/mariadb/columnstore/data/bulk/tmpjob/20879_D20170410_T181526_S254449_Job_20879.xml
2017-04-10 18:15:26 (56061) INFO : Job file loaded, run time for this step : 0.174996 seconds
2017-04-10 18:15:26 (56061) INFO : PreProcessing check starts
2017-04-10 18:15:26 (56061) INFO : input data file /home/techteamdbdev/draws/draws.csv
2017-04-10 18:15:36 (56061) ERR : Unable to acquire lock for table draws.draws; OID-20879; table currently locked by process-cpimport.bin (pm1); pid-55594; session-1; txn-1 [1203]

Error in loading job data

 Comments   
Comment by David Thompson (Inactive) [ 2017-08-09 ]

An earlier failure had left the table logged:
Apr 10 16:45:58 columnstore-testing-d01 cpimport.bin[33253]: 58.845411 |0|0|0| I 34 CAL0086: Initiating BulkLoad: -s , -E " -P pm1-33253 -u9d939fbb-ca32-41b8-96fc-4116af1d644c draws draws draws.csv
Apr 10 16:45:59 columnstore-testing-d01 cpimport.bin[33253]: 59.034441 |0|0|0| I 34 CAL0081: Start BulkLoad: JobId-20879; db-draws
Apr 10 16:45:59 columnstore-testing-d01 cpimport.bin[33253]: 59.037743 |0|0|0| E 34 CAL0087: BulkLoad Error: Error accessing DBRoot[0] /usr/local/mariadb/columnstore/data1; Permission denied

Not sure why there was no access to the directory but that is the likely cause. In this case clearing the table lock and retrying was the right approach.

Also saw evidence of memory limits being hit.

Generated at Thu Feb 08 02:22:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.