Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
1.2.0
-
None
-
None
-
2018-20
Description
While loading data from CSV files into a ColumnStore tables using mcsimport utility most of the files ended successfully loading as many records as there were rows in the CSV file but indicated a various amount of invalid records.
I was not able to track down what invalid records they are and what was causing the invalid state of each of them as I did not find any log (was looking for something similar to cpimport error logs).
Here is the list of all files loading mcsimport responses. But I am only attaching two smaller files that indicated an issue together with the create table command for each one of them.
/usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert calendar calendar-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert groups groups-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert onelots onelots-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert open_interest open_interest-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert optimal_early_exercise optimal_ex-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert rates rates-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert securities securities-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert underlyings underlyings-datadock_2018_Q1.csv && /usr/local/mariadb/columnstore/tools/mcsimport/mcsimport tradealert usec_hist usec_hist-datadock_2018_Q1.csv
Execution time: 0.817828s
Rows inserted: 10064
Truncation count: 0
Saturated count: 0
Invalid count: 0
cat calendar-datadock_2018_Q1.csv | wc -l
10064
Execution time: 0.161977s
Rows inserted: 2077
Truncation count: 0
Saturated count: 0
Invalid count: 1
cat groups-datadock_2018_Q1.csv | wc -l
2077
Execution time: 4281.18s
Rows inserted: 87483118
Truncation count: 0
Saturated count: 0
Invalid count: 32809561
cat onelots-datadock_2018_Q1.csv | wc -l
87483118
Execution time: 1043.61s
Rows inserted: 53818130
Truncation count: 0
Saturated count: 0
Invalid count: 38551020
cat open_interest-datadock_2018_Q1.csv | wc -l
53818130
Execution time: 0.481345s
Rows inserted: 17387
Truncation count: 0
Saturated count: 0
Invalid count: 159
cat optimal_ex-datadock_2018_Q1.csv | wc -l
17387
Execution time: 2718.69s
Rows inserted: 101144576
Truncation count: 0
Saturated count: 0
Invalid count: 146538555
cat rates-datadock_2018_Q1.csv | wc -l
101144576
Execution time: 91.0303s
Rows inserted: 4278121
Truncation count: 0
Saturated count: 0
Invalid count: 34791
cat securities-datadock_2018_Q1.csv | wc -l
4278121
Execution time: 38.2951s
Rows inserted: 264387
Truncation count: 0
Saturated count: 0
Invalid count: 17
cat underlyings-datadock_2018_Q1.csv | wc -l
264387
Execution time: 26.1987s
Rows inserted: 494710
Truncation count: 61
Saturated count: 0
Invalid count: 1940652
cat usec_hist-datadock_2018_Q1.csv | wc -l
494710
CREATE TABLE `optimal_early_exercise` (
`secid` int(11) DEFAULT NULL,
`date` date DEFAULT NULL,
`net_div` float DEFAULT NULL,
`exdiv` date DEFAULT NULL,
`open_int` int(11) DEFAULT NULL,
`hedged_gains` float DEFAULT NULL,
`put_mid` float DEFAULT NULL,
`call_mid` float DEFAULT NULL,
`funding` float DEFAULT NULL,
`stock_px` float DEFAULT NULL,
`unexercised` int(11) DEFAULT NULL,
`value_lost` float DEFAULT NULL,
`theo_gains` float DEFAULT NULL,
`put_ask` float DEFAULT NULL,
`stamp` char(12) DEFAULT NULL,
`latest` tinyint(1) DEFAULT NULL
) ENGINE=Columnstore DEFAULT CHARSET=latin1;
CREATE TABLE `usec_hist` (
`date` date NOT NULL DEFAULT '0000-00-00',
`usymbol` varchar(8) DEFAULT '',
`sectype` varchar(4) DEFAULT NULL,
`description` varchar(128) DEFAULT NULL,
`sector` varchar(8) DEFAULT NULL,
`subsector` varchar(8) DEFAULT NULL,
`cap` varchar(10) DEFAULT NULL,
`shares_outstanding` float DEFAULT NULL,
`next_earnings` date DEFAULT NULL,
`next_exdiv` date DEFAULT NULL,
`price` float DEFAULT NULL,
`close` float DEFAULT NULL,
`chg` float DEFAULT NULL,
`high` float DEFAULT NULL,
`low` float DEFAULT NULL,
`high_date` date DEFAULT NULL,
`low_date` date DEFAULT NULL,
`htb` int(11) DEFAULT NULL,
`riskarb` tinyint(4) DEFAULT NULL,
`primary_exch` char(3) DEFAULT NULL,
`has_options` int(11) DEFAULT NULL,
`new_issue` int(11) DEFAULT NULL,
`volatility20day` float DEFAULT NULL,
`volatility60day` float DEFAULT NULL,
`volatility120day` float DEFAULT NULL,
`avg_size` int(11) DEFAULT NULL,
`bats_price` float DEFAULT NULL,
`open` float DEFAULT NULL
) ENGINE=Columnstore DEFAULT CHARSET=latin1;
Attachments
Issue Links
- includes
-
MCOL-1853 ColumnStore breaks on S3 NFS mounts
- Closed