[MCOL-3683] Add optional errordir for .bad and .err to cpimport (like in cpimport.bin) Created: 2019-12-18  Updated: 2021-05-02  Resolved: 2021-03-22

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: None
Fix Version/s: 5.5.2

Type: New Feature Priority: Major
Reporter: Richard Stracke Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MCOL-4483 Consolidate Log Location Closed

 Description   

cpimport.bin include the possibility of custom directory for .bad and .err files.

If cpimport will be called manually, .bad and .err files will be saved in the directory like the import files.

If for some reason (policy etc.) only read access is possible, .bad and .err files can't be written.

So a custom directory for writing the logs would be good.

This option already exists in cpimport.bin

cpimport.bin -h

  -L send *.err and *.bad (reject) files here

And already declared in we_bulkload.h

 std::string  fErrorDir;             // Opt dir for *.err and *.bad files
 
 
 inline const std::string& BulkLoad::getErrorDir( ) const
 
{
 
    return fErrorDir;
 
}

So maybe an easy solution would be to add this option also in cpimport and set the errordir like cpimport.cpp

case 'L':                                // -L: Error log directory
 
            {
 
                curJob.setErrorDir( optarg );
 
                break;
 
            }



 Comments   
Comment by Daniel Lee (Inactive) [ 2021-03-22 ]

Build verified: 5.5.2-1

cpimport -h

.
.
-L Directory for the output .err and .bad files.
Default is /var/log/mariadb/columnstore
.
.

[centos8:root~]# cpimport -L /data/qa/source mytest t1 t.txt
Locale = en_US.UTF-8Error gaining r/w access to import path .: Permission denied

[centos8:root~]# cpimport mytest t1 t.txt
Locale = en_US.UTF-8
Using table OID 3037 as the default JOB ID
Input file(s) will be read from : /root
Job description file : /var/log/mariadb/columnstore/data/bulk/tmpjob/3037_D20210322_T153344_S843665_Job_3037.xml
Log file for this job: /var/log/mariadb/columnstore/cpimport/Job_3037.log
2021-03-22 15:33:44 (5561) INFO : successfully loaded job file /var/log/mariadb/columnstore/data/bulk/tmpjob/3037_D20210322_T153344_S843665_Job_3037.xml
2021-03-22 15:33:44 (5561) INFO : Job file loaded, run time for this step : 0.0591531 seconds
2021-03-22 15:33:44 (5561) INFO : PreProcessing check starts
2021-03-22 15:33:44 (5561) INFO : input data file /root/t.txt
2021-03-22 15:33:44 (5561) INFO : PreProcessing check completed
2021-03-22 15:33:44 (5561) INFO : preProcess completed, run time for this step : 0.00701499 seconds
2021-03-22 15:33:44 (5561) INFO : No of Read Threads Spawned = 1
2021-03-22 15:33:44 (5561) INFO : No of Parse Threads Spawned = 3
2021-03-22 15:33:44 (5561) INFO : Number of rows with errors = 4. Exact rows are listed in file located here: /var/log/mariadb/columnstore/cpimport/
2021-03-22 15:33:44 (5561) INFO : Number of rows with bad data = 4. Exact rows are listed in file located here: /var/log/mariadb/columnstore/cpimport/
2021-03-22 15:33:44 (5561) INFO : For table mytest.t1: 5 rows processed and 1 rows inserted.
2021-03-22 15:33:45 (5561) INFO : Bulk load completed, total run time : 1.10763 seconds

[centos8:root~]# cpimport -L /tmp mytest t1 t.txt
Locale = en_US.UTF-8
Using table OID 3037 as the default JOB ID
Input file(s) will be read from : /root
Job description file : /var/log/mariadb/columnstore/data/bulk/tmpjob/3037_D20210322_T153540_S423567_Job_3037.xml
Log file for this job: /var/log/mariadb/columnstore/cpimport/Job_3037.log
2021-03-22 15:35:40 (5646) INFO : successfully loaded job file /var/log/mariadb/columnstore/data/bulk/tmpjob/3037_D20210322_T153540_S423567_Job_3037.xml
2021-03-22 15:35:40 (5646) INFO : Job file loaded, run time for this step : 0.054472 seconds
2021-03-22 15:35:40 (5646) INFO : PreProcessing check starts
2021-03-22 15:35:40 (5646) INFO : input data file /root/t.txt
2021-03-22 15:35:40 (5646) INFO : PreProcessing check completed
2021-03-22 15:35:40 (5646) INFO : preProcess completed, run time for this step : 0.0082109 seconds
2021-03-22 15:35:40 (5646) INFO : No of Read Threads Spawned = 1
2021-03-22 15:35:40 (5646) INFO : No of Parse Threads Spawned = 3
2021-03-22 15:35:40 (5646) INFO : Number of rows with errors = 4. Exact rows are listed in file located here: /tmp/
2021-03-22 15:35:40 (5646) INFO : Number of rows with bad data = 4. Exact rows are listed in file located here: /tmp/
2021-03-22 15:35:40 (5646) INFO : For table mytest.t1: 5 rows processed and 1 rows inserted.
2021-03-22 15:35:41 (5646) INFO : Bulk load completed, total run time : 1.0719 seconds

Generated at Thu Feb 08 02:44:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.