cpimport not generating bad file

We can't see the rejected records as the bad file is not being created. This is a blocking condition for us going live in production.

I changed the directory to /var/log/mariadb/columnstore/data/bulk/log (as per https://jira.mariadb.org/browse/MCOL-4483) but still seeing the same behavior....

$ mcsGetConfig -a |grep -i writeengine
pm1_WriteEngineServer.IPAddr = ******
pm1_WriteEngineServer.Port = 8630
WriteEngine.BulkRoot = /var/log/mariadb/columnstore/data/bulk
WriteEngine.BulkRollbackDir = /var/lib/columnstore/data1/systemFiles/bulkRollback
WriteEngine.MaxFileSystemDiskUsagePct = 98
WriteEngine.CompressedPaddingBlocks = 1
pm2_WriteEngineServer.IPAddr = ******
pm2_WriteEngineServer.Port = 8630
pm3_WriteEngineServer.IPAddr = ******
pm3_WriteEngineServer.Port = 8630

$ rm /var/lib/columnstore/data/bulk/log/*.log
$ rm /var/lib/columnstore/data/bulk/log/*.err
$
$ mcsSetConfig WriteEngine BulkRoot /var/lib/columnstore/data/bulk
$
$ mcsGetConfig -a |grep -i writeengine
pm1_WriteEngineServer.IPAddr = ******
pm1_WriteEngineServer.Port = 8630
WriteEngine.BulkRoot = /var/lib/columnstore/data/bulk
WriteEngine.BulkRollbackDir = /var/lib/columnstore/data1/systemFiles/bulkRollback
WriteEngine.MaxFileSystemDiskUsagePct = 98
WriteEngine.CompressedPaddingBlocks = 1
pm2_WriteEngineServer.IPAddr = ******
pm2_WriteEngineServer.Port = 8630
pm3_WriteEngineServer.IPAddr = ******
pm3_WriteEngineServer.Port = 8630
$
$ cpimport dwhs cs1 cs1.csv -s ',' -E '"' -e 0
2021-01-11 11:05:40 (624518) INFO : Running distributed import (mode 1) on all PMs...
2021-01-11 11:05:40 (624518) ERR : Received a Cpimport Failure from PM2
2021-01-11 11:05:40 (624518) INFO : Please verify error log files in PM2
2021-01-11 11:05:40 (624518) INFO : Canceling outstanding cpimports
2021-01-11 11:05:40 (624518) ERR : PM2 : Bulkload Read (thread 0) Failed for Table dwhs.cs1. Terminating this job.
2021-01-11 11:05:41 (624518) INFO : Table dwhs.cs1: (OID-18071) was NOT successfully loaded.
2021-01-11 11:05:41 (624518) INFO : Bulk load completed, total run time : 0.458629 seconds
$
$ ls -ltr /var/lib/columnstore/data/bulk/log/
total 12
rw-rr- 1 mysql mysql 0 Jan 11 11:05 Job_18071.err
rw-rr- 1 mysql mysql 3334 Jan 11 11:05 Job_18071.log
rw-rw-r- 1 mysql mysql 201 Jan 11 11:05 cpimport_0111110540_624518.err
rw-rw-r- 1 mysql mysql 1509 Jan 11 11:05 cpimport_0111110540_624518.log
$
$ cat /var/lib/columnstore/data/bulk/log/cpimport_0111110540_624518.err
2021-01-11 11:05:40 (624518) ERR : Received a Cpimport Failure from PM2
2021-01-11 11:05:40 (624518) ERR : PM2 : Bulkload Read (thread 0) Failed for Table dwhs.cs1. Terminating this job.
$
$ cat /var/lib/columnstore/data/bulk/log/cpimport_0111110540_624518.log
2021-01-11 11:05:40 (624518) INFO : Running distributed import (mode 1) on all PMs...
2021-01-11 11:05:40 (624518) INFO2 : /usr/bin/cpimport.bin -e 0 -s , -E " -R /tmp/columnstore_tmp_files/BrmRpt0011110540624518.rpt -m 1 -P pm1-624518 -T SYSTEM -uf7d4ce55-7ed0-4972-ac1f-2cc7b6acf03b dwhs cs1
2021-01-11 11:05:40 (624518) INFO2 : Received a BRM-Report from 3
2021-01-11 11:05:40 (624518) INFO2 : Received a Cpimport Pass from PM3
2021-01-11 11:05:40 (624518) INFO2 : Received a Cpimport Failure from PM2
2021-01-11 11:05:40 (624518) ERR : Received a Cpimport Failure from PM2
2021-01-11 11:05:40 (624518) INFO : Please verify error log files in PM2
2021-01-11 11:05:40 (624518) INFO : Canceling outstanding cpimports
2021-01-11 11:05:40 (624518) INFO2 : Received a BRM-Report from 2
2021-01-11 11:05:40 (624518) ERR : PM2 : Bulkload Read (thread 0) Failed for Table dwhs.cs1. Terminating this job.
2021-01-11 11:05:40 (624518) INFO2 : Received a BRM-Report from 1
2021-01-11 11:05:40 (624518) INFO2 : Received a Cpimport Pass from PM1
2021-01-11 11:05:40 (624518) INFO2 : Rollback succeed on all PMs
2021-01-11 11:05:41 (624518) INFO2 : Released Table Lock
2021-01-11 11:05:41 (624518) INFO2 : Cleanup succeed on all PMs
2021-01-11 11:05:41 (624518) INFO : Table dwhs.cs1: (OID-18071) was NOT successfully loaded.
2021-01-11 11:05:41 (624518) INFO : Bulk load completed, total run time : 0.458629 seconds
2021-01-11 11:05:41 (624518) INFO2 : Shutdown of all child threads Finished!!

— I created line 3 and 5 with 4 columns and the table expects 3
$ cat cs1.csv
1,"ajksadkada","2021-01-11"
2,"ajksadkada","2021-01-11"
3,"cksnks","ajksadkada","2021-01-11"
4,"ajksadkada","2021-01-11"
5,"cksnks","ajksadkada","2021-01-11"

These are the contents of Job_18071.log
$ cat /var/lib/columnstore/data/bulk/log/Job_18071.log
2021-01-11 11:05:40 (624547) INFO : successfully loaded job file /var/lib/columnstore/data/bulk/tmpjob/18071_D20210111_T110540_S642034_Job_18071.xml
2021-01-11 11:05:40 (624547) INFO2 : Command line options: -e 0 -s , -E " -R /tmp/columnstore_tmp_files/BrmRpt0011110540624518.rpt -m 1 -P pm1-624518 -T SYSTEM -uf7d4ce55-7ed0-4972-ac1f-2cc7b6acf03b dwhs cs1
2021-01-11 11:05:40 (624547) INFO : Job file loaded, run time for this step : 0.035315 seconds
2021-01-11 11:05:40 (624547) INFO2 : XMLJobFile: Delim(|); EnclosedBy(n/a); EscapeChar(); ReadBufs(5); ReadBufSize(1048576); setvbufSize(10485760)
2021-01-11 11:05:40 (624547) INFO2 : Table(dwhs.cs1); OID(18071); MaxErrNum(10)
2021-01-11 11:05:40 (624547) INFO2 : Column(xx); OID(18072); Type(integer); Width(4); Comp(2)
2021-01-11 11:05:40 (624547) INFO2 : Column(yy); OID(18073); Type(varchar); Width(20); Comp(2); DctnryOid(18075)
2021-01-11 11:05:40 (624547) INFO2 : Column(zz); OID(18074); Type(date); Width(4); Comp(2)
2021-01-11 11:05:40 (624547) INFO : PreProcessing check starts
2021-01-11 11:05:40 (624547) INFO2 : Using STDIN for input data
2021-01-11 11:05:40 (624547) INFO2 : Starting DBRoot info for OID 18072
DBRoot-1, part/seg/hwm/LBID/totBlks/state: 0/0/0/0/0/EmptyDbRoot
2021-01-11 11:05:40 (624547) INFO2 : No active extents; will add partition to start adding rows for oid-18072; DBRoot-1
2021-01-11 11:05:40 (624547) INFO2 : Starting DBRoot info for OID 18073
DBRoot-1, part/seg/hwm/LBID/totBlks/state: 0/0/0/0/0/EmptyDbRoot
2021-01-11 11:05:40 (624547) INFO2 : No active extents; will add partition to start adding rows for oid-18073; DBRoot-1
2021-01-11 11:05:40 (624547) INFO2 : Starting DBRoot info for OID 18074
DBRoot-1, part/seg/hwm/LBID/totBlks/state: 0/0/0/0/0/EmptyDbRoot
2021-01-11 11:05:40 (624547) INFO2 : No active extents; will add partition to start adding rows for oid-18074; DBRoot-1
2021-01-11 11:05:40 (624547) INFO2 : Initializing import: Table-dwhs.cs1...
2021-01-11 11:05:40 (624547) INFO2 : Currently no extents on dbroot1 for column OID 18072; will create starting extent
2021-01-11 11:05:40 (624547) INFO2 : Currently no extents on dbroot1 for column OID 18073; will create starting extent
2021-01-11 11:05:40 (624547) INFO2 : Currently no extents on dbroot1 for column OID 18074; will create starting extent
2021-01-11 11:05:40 (624547) INFO : PreProcessing check completed
2021-01-11 11:05:40 (624547) INFO : preProcess completed, run time for this step : 0.00965285 seconds
2021-01-11 11:05:40 (624547) INFO : No of Read Threads Spawned = 1
2021-01-11 11:05:40 (624547) INFO : No of Parse Threads Spawned = 3
2021-01-11 11:05:40 (624547) INFO : Reading input from STDIN to import into table dwhs.cs1...
2021-01-11 11:05:40 (624547) INFO2 : Start reading and loading table dwhs.cs1
2021-01-11 11:05:40 (624547) INFO2 : Finished loading dwhs.cs1 from STDIN, Time taken = 0 seconds
2021-01-11 11:05:40 (624547) INFO2 : BulkLoad::ReadOperation No more tables available for processing. Read thread 0 exiting...
2021-01-11 11:05:40 (624547) INFO : For table dwhs.cs1: 0 rows processed and 0 rows inserted.
2021-01-11 11:05:40 (624547) INFO2 : For table dwhs.cs1: Elapsed time to load this table: 0.0614159 secs
2021-01-11 11:05:40 (624547) INFO : Bulk load completed, total run time : 0.107072 seconds