Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4491

cpimport not generating bad file

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 5.4.3
    • Fix Version/s: N/A
    • Component/s: cpimport
    • Labels:
      None

      Description

      We can't see the rejected records as the bad file is not being created. This is a blocking condition for us going live in production.

      I changed the directory to /var/log/mariadb/columnstore/data/bulk/log (as per https://jira.mariadb.org/browse/MCOL-4483) but still seeing the same behavior....

      $ mcsGetConfig -a |grep -i writeengine
      pm1_WriteEngineServer.IPAddr = ******
      pm1_WriteEngineServer.Port = 8630
      WriteEngine.BulkRoot = /var/log/mariadb/columnstore/data/bulk
      WriteEngine.BulkRollbackDir = /var/lib/columnstore/data1/systemFiles/bulkRollback
      WriteEngine.MaxFileSystemDiskUsagePct = 98
      WriteEngine.CompressedPaddingBlocks = 1
      pm2_WriteEngineServer.IPAddr = ******
      pm2_WriteEngineServer.Port = 8630
      pm3_WriteEngineServer.IPAddr = ******
      pm3_WriteEngineServer.Port = 8630

      $ rm /var/lib/columnstore/data/bulk/log/*.log
      $ rm /var/lib/columnstore/data/bulk/log/*.err
      $
      $ mcsSetConfig WriteEngine BulkRoot /var/lib/columnstore/data/bulk
      $
      $ mcsGetConfig -a |grep -i writeengine
      pm1_WriteEngineServer.IPAddr = ******
      pm1_WriteEngineServer.Port = 8630
      WriteEngine.BulkRoot = /var/lib/columnstore/data/bulk
      WriteEngine.BulkRollbackDir = /var/lib/columnstore/data1/systemFiles/bulkRollback
      WriteEngine.MaxFileSystemDiskUsagePct = 98
      WriteEngine.CompressedPaddingBlocks = 1
      pm2_WriteEngineServer.IPAddr = ******
      pm2_WriteEngineServer.Port = 8630
      pm3_WriteEngineServer.IPAddr = ******
      pm3_WriteEngineServer.Port = 8630
      $
      $ cpimport dwhs cs1 cs1.csv -s ',' -E '"' -e 0
      2021-01-11 11:05:40 (624518) INFO : Running distributed import (mode 1) on all PMs...
      2021-01-11 11:05:40 (624518) ERR : Received a Cpimport Failure from PM2
      2021-01-11 11:05:40 (624518) INFO : Please verify error log files in PM2
      2021-01-11 11:05:40 (624518) INFO : Canceling outstanding cpimports
      2021-01-11 11:05:40 (624518) ERR : PM2 : Bulkload Read (thread 0) Failed for Table dwhs.cs1. Terminating this job.
      2021-01-11 11:05:41 (624518) INFO : Table dwhs.cs1: (OID-18071) was NOT successfully loaded.
      2021-01-11 11:05:41 (624518) INFO : Bulk load completed, total run time : 0.458629 seconds
      $
      $ ls -ltr /var/lib/columnstore/data/bulk/log/
      total 12
      rw-rr- 1 mysql mysql 0 Jan 11 11:05 Job_18071.err
      rw-rr- 1 mysql mysql 3334 Jan 11 11:05 Job_18071.log
      rw-rw-r- 1 mysql mysql 201 Jan 11 11:05 cpimport_0111110540_624518.err
      rw-rw-r- 1 mysql mysql 1509 Jan 11 11:05 cpimport_0111110540_624518.log
      $
      $ cat /var/lib/columnstore/data/bulk/log/cpimport_0111110540_624518.err
      2021-01-11 11:05:40 (624518) ERR : Received a Cpimport Failure from PM2
      2021-01-11 11:05:40 (624518) ERR : PM2 : Bulkload Read (thread 0) Failed for Table dwhs.cs1. Terminating this job.
      $
      $ cat /var/lib/columnstore/data/bulk/log/cpimport_0111110540_624518.log
      2021-01-11 11:05:40 (624518) INFO : Running distributed import (mode 1) on all PMs...
      2021-01-11 11:05:40 (624518) INFO2 : /usr/bin/cpimport.bin -e 0 -s , -E " -R /tmp/columnstore_tmp_files/BrmRpt0011110540624518.rpt -m 1 -P pm1-624518 -T SYSTEM -uf7d4ce55-7ed0-4972-ac1f-2cc7b6acf03b dwhs cs1
      2021-01-11 11:05:40 (624518) INFO2 : Received a BRM-Report from 3
      2021-01-11 11:05:40 (624518) INFO2 : Received a Cpimport Pass from PM3
      2021-01-11 11:05:40 (624518) INFO2 : Received a Cpimport Failure from PM2
      2021-01-11 11:05:40 (624518) ERR : Received a Cpimport Failure from PM2
      2021-01-11 11:05:40 (624518) INFO : Please verify error log files in PM2
      2021-01-11 11:05:40 (624518) INFO : Canceling outstanding cpimports
      2021-01-11 11:05:40 (624518) INFO2 : Received a BRM-Report from 2
      2021-01-11 11:05:40 (624518) ERR : PM2 : Bulkload Read (thread 0) Failed for Table dwhs.cs1. Terminating this job.
      2021-01-11 11:05:40 (624518) INFO2 : Received a BRM-Report from 1
      2021-01-11 11:05:40 (624518) INFO2 : Received a Cpimport Pass from PM1
      2021-01-11 11:05:40 (624518) INFO2 : Rollback succeed on all PMs
      2021-01-11 11:05:41 (624518) INFO2 : Released Table Lock
      2021-01-11 11:05:41 (624518) INFO2 : Cleanup succeed on all PMs
      2021-01-11 11:05:41 (624518) INFO : Table dwhs.cs1: (OID-18071) was NOT successfully loaded.
      2021-01-11 11:05:41 (624518) INFO : Bulk load completed, total run time : 0.458629 seconds
      2021-01-11 11:05:41 (624518) INFO2 : Shutdown of all child threads Finished!!

      — I created line 3 and 5 with 4 columns and the table expects 3
      $ cat cs1.csv
      1,"ajksadkada","2021-01-11"
      2,"ajksadkada","2021-01-11"
      3,"cksnks","ajksadkada","2021-01-11"
      4,"ajksadkada","2021-01-11"
      5,"cksnks","ajksadkada","2021-01-11"

      These are the contents of Job_18071.log
      $ cat /var/lib/columnstore/data/bulk/log/Job_18071.log
      2021-01-11 11:05:40 (624547) INFO : successfully loaded job file /var/lib/columnstore/data/bulk/tmpjob/18071_D20210111_T110540_S642034_Job_18071.xml
      2021-01-11 11:05:40 (624547) INFO2 : Command line options: -e 0 -s , -E " -R /tmp/columnstore_tmp_files/BrmRpt0011110540624518.rpt -m 1 -P pm1-624518 -T SYSTEM -uf7d4ce55-7ed0-4972-ac1f-2cc7b6acf03b dwhs cs1
      2021-01-11 11:05:40 (624547) INFO : Job file loaded, run time for this step : 0.035315 seconds
      2021-01-11 11:05:40 (624547) INFO2 : XMLJobFile: Delim(|); EnclosedBy(n/a); EscapeChar(); ReadBufs(5); ReadBufSize(1048576); setvbufSize(10485760)
      2021-01-11 11:05:40 (624547) INFO2 : Table(dwhs.cs1); OID(18071); MaxErrNum(10)
      2021-01-11 11:05:40 (624547) INFO2 : Column(xx); OID(18072); Type(integer); Width(4); Comp(2)
      2021-01-11 11:05:40 (624547) INFO2 : Column(yy); OID(18073); Type(varchar); Width(20); Comp(2); DctnryOid(18075)
      2021-01-11 11:05:40 (624547) INFO2 : Column(zz); OID(18074); Type(date); Width(4); Comp(2)
      2021-01-11 11:05:40 (624547) INFO : PreProcessing check starts
      2021-01-11 11:05:40 (624547) INFO2 : Using STDIN for input data
      2021-01-11 11:05:40 (624547) INFO2 : Starting DBRoot info for OID 18072
      DBRoot-1, part/seg/hwm/LBID/totBlks/state: 0/0/0/0/0/EmptyDbRoot
      2021-01-11 11:05:40 (624547) INFO2 : No active extents; will add partition to start adding rows for oid-18072; DBRoot-1
      2021-01-11 11:05:40 (624547) INFO2 : Starting DBRoot info for OID 18073
      DBRoot-1, part/seg/hwm/LBID/totBlks/state: 0/0/0/0/0/EmptyDbRoot
      2021-01-11 11:05:40 (624547) INFO2 : No active extents; will add partition to start adding rows for oid-18073; DBRoot-1
      2021-01-11 11:05:40 (624547) INFO2 : Starting DBRoot info for OID 18074
      DBRoot-1, part/seg/hwm/LBID/totBlks/state: 0/0/0/0/0/EmptyDbRoot
      2021-01-11 11:05:40 (624547) INFO2 : No active extents; will add partition to start adding rows for oid-18074; DBRoot-1
      2021-01-11 11:05:40 (624547) INFO2 : Initializing import: Table-dwhs.cs1...
      2021-01-11 11:05:40 (624547) INFO2 : Currently no extents on dbroot1 for column OID 18072; will create starting extent
      2021-01-11 11:05:40 (624547) INFO2 : Currently no extents on dbroot1 for column OID 18073; will create starting extent
      2021-01-11 11:05:40 (624547) INFO2 : Currently no extents on dbroot1 for column OID 18074; will create starting extent
      2021-01-11 11:05:40 (624547) INFO : PreProcessing check completed
      2021-01-11 11:05:40 (624547) INFO : preProcess completed, run time for this step : 0.00965285 seconds
      2021-01-11 11:05:40 (624547) INFO : No of Read Threads Spawned = 1
      2021-01-11 11:05:40 (624547) INFO : No of Parse Threads Spawned = 3
      2021-01-11 11:05:40 (624547) INFO : Reading input from STDIN to import into table dwhs.cs1...
      2021-01-11 11:05:40 (624547) INFO2 : Start reading and loading table dwhs.cs1
      2021-01-11 11:05:40 (624547) INFO2 : Finished loading dwhs.cs1 from STDIN, Time taken = 0 seconds
      2021-01-11 11:05:40 (624547) INFO2 : BulkLoad::ReadOperation No more tables available for processing. Read thread 0 exiting...
      2021-01-11 11:05:40 (624547) INFO : For table dwhs.cs1: 0 rows processed and 0 rows inserted.
      2021-01-11 11:05:40 (624547) INFO2 : For table dwhs.cs1: Elapsed time to load this table: 0.0614159 secs
      2021-01-11 11:05:40 (624547) INFO : Bulk load completed, total run time : 0.107072 seconds

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            ravidhingra18 Ravi Dhingra
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.