Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-1858

An `invalid` records indication when loading the table unsing mcsimport

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: mcsimport
    • Labels:
      None
    • Sprint:
      2018-20

      Description

      While loading data from CSV files into a ColumnStore tables using mcsimport utility most of the files ended successfully loading as many records as there were rows in the CSV file but indicated a various amount of invalid records.
      I was not able to track down what invalid records they are and what was causing the invalid state of each of them as I did not find any log (was looking for something similar to cpimport error logs).
      Here is the list of all files loading mcsimport responses. But I am only attaching two smaller files that indicated an issue together with the create table command for each one of them.

      /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert calendar calendar-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert groups groups-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert onelots onelots-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert open_interest open_interest-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert optimal_early_exercise optimal_ex-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert rates rates-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert securities securities-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert underlyings underlyings-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert usec_hist usec_hist-datadock_2018_Q1.csv

      Execution time: 0.817828s
      Rows inserted: 10064
      Truncation count: 0
      Saturated count: 0
      Invalid count: 0
      cat calendar-datadock_2018_Q1.csv | wc -l
      10064

      Execution time: 0.161977s
      Rows inserted: 2077
      Truncation count: 0
      Saturated count: 0
      Invalid count: 1
      cat groups-datadock_2018_Q1.csv | wc -l
      2077

      Execution time: 4281.18s
      Rows inserted: 87483118
      Truncation count: 0
      Saturated count: 0
      Invalid count: 32809561
      cat onelots-datadock_2018_Q1.csv | wc -l
      87483118

      Execution time: 1043.61s
      Rows inserted: 53818130
      Truncation count: 0
      Saturated count: 0
      Invalid count: 38551020
      cat open_interest-datadock_2018_Q1.csv | wc -l
      53818130

      Execution time: 0.481345s
      Rows inserted: 17387
      Truncation count: 0
      Saturated count: 0
      Invalid count: 159
      cat optimal_ex-datadock_2018_Q1.csv | wc -l
      17387

      Execution time: 2718.69s
      Rows inserted: 101144576
      Truncation count: 0
      Saturated count: 0
      Invalid count: 146538555
      cat rates-datadock_2018_Q1.csv | wc -l
      101144576

      Execution time: 91.0303s
      Rows inserted: 4278121
      Truncation count: 0
      Saturated count: 0
      Invalid count: 34791
      cat securities-datadock_2018_Q1.csv | wc -l
      4278121

      Execution time: 38.2951s
      Rows inserted: 264387
      Truncation count: 0
      Saturated count: 0
      Invalid count: 17
      cat underlyings-datadock_2018_Q1.csv | wc -l
      264387

      Execution time: 26.1987s
      Rows inserted: 494710
      Truncation count: 61
      Saturated count: 0
      Invalid count: 1940652
      cat usec_hist-datadock_2018_Q1.csv | wc -l
      494710

      CREATE TABLE `optimal_early_exercise` (
      `secid` int(11) DEFAULT NULL,
      `date` date DEFAULT NULL,
      `net_div` float DEFAULT NULL,
      `exdiv` date DEFAULT NULL,
      `open_int` int(11) DEFAULT NULL,
      `hedged_gains` float DEFAULT NULL,
      `put_mid` float DEFAULT NULL,
      `call_mid` float DEFAULT NULL,
      `funding` float DEFAULT NULL,
      `stock_px` float DEFAULT NULL,
      `unexercised` int(11) DEFAULT NULL,
      `value_lost` float DEFAULT NULL,
      `theo_gains` float DEFAULT NULL,
      `put_ask` float DEFAULT NULL,
      `stamp` char(12) DEFAULT NULL,
      `latest` tinyint(1) DEFAULT NULL
      ) ENGINE=Columnstore DEFAULT CHARSET=latin1;

      CREATE TABLE `usec_hist` (
      `date` date NOT NULL DEFAULT '0000-00-00',
      `usymbol` varchar(8) DEFAULT '',
      `sectype` varchar(4) DEFAULT NULL,
      `description` varchar(128) DEFAULT NULL,
      `sector` varchar(8) DEFAULT NULL,
      `subsector` varchar(8) DEFAULT NULL,
      `cap` varchar(10) DEFAULT NULL,
      `shares_outstanding` float DEFAULT NULL,
      `next_earnings` date DEFAULT NULL,
      `next_exdiv` date DEFAULT NULL,
      `price` float DEFAULT NULL,
      `close` float DEFAULT NULL,
      `chg` float DEFAULT NULL,
      `high` float DEFAULT NULL,
      `low` float DEFAULT NULL,
      `high_date` date DEFAULT NULL,
      `low_date` date DEFAULT NULL,
      `htb` int(11) DEFAULT NULL,
      `riskarb` tinyint(4) DEFAULT NULL,
      `primary_exch` char(3) DEFAULT NULL,
      `has_options` int(11) DEFAULT NULL,
      `new_issue` int(11) DEFAULT NULL,
      `volatility20day` float DEFAULT NULL,
      `volatility60day` float DEFAULT NULL,
      `volatility120day` float DEFAULT NULL,
      `avg_size` int(11) DEFAULT NULL,
      `bats_price` float DEFAULT NULL,
      `open` float DEFAULT NULL
      ) ENGINE=Columnstore DEFAULT CHARSET=latin1;

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              winstone Zdravelina Sokolovska (Inactive)
              Reporter:
              madamec Martin Adamec
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.