[MCOL-16] LDI into datetime columns, data got saturated. Created: 2016-05-03  Updated: 2016-06-22  Resolved: 2016-06-22

Status: Closed
Project: MariaDB ColumnStore
Component/s: ExeMgr
Affects Version/s: None
Fix Version/s: 1.0.1

Type: Bug Priority: Minor
Reporter: Dipti Joshi (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Sprint: 1.0.2-1

 Description   

LDI into datetime columns, data got saturated. insert works correctly

value used for testing is " 1400-01-01 00:00:00"
"char data type only is ok
+--------+--------+--------+--------+--------+--------+---------+----------+-----------+------------+
| CCHAR1 | CCHAR2 | CCHAR3 | CCHAR4 | CCHAR5 | CCHAR6 | CCHAR7  | CCHAR8   | CCHAR9    | CCHAR255   |
+--------+--------+--------+--------+--------+--------+---------+----------+-----------+------------+
| z      | zz     | zzz    | zzzz   | zzzzz  | zzzzzz | zzzzzzz | zzzzzzzz | zzzzzzzzz | zzzzzzzzzz |
+--------+--------+--------+--------+--------+--------+---------+----------+-----------+------------+
1 row in set (0.26 sec)
 
Adding a datetime datatype in front and use a value of ""1400-01-01 00:00:00""
Both c1 and cchar255 got saturated
 
MariaDB [mytest]> select * from t1;
+------+--------+--------+--------+--------+--------+--------+---------+----------+-----------+----------+
| c1   | CCHAR1 | CCHAR2 | CCHAR3 | CCHAR4 | CCHAR5 | CCHAR6 | CCHAR7  | CCHAR8   | CCHAR9    | CCHAR255 |
+------+--------+--------+--------+--------+--------+--------+---------+----------+-----------+----------+
| NULL | z      | zz     | zzz    | zzzz   | zzzzz  | zzzzzz | zzzzzzz | zzzzzzzz | zzzzzzzzz | zzzzzzz  |
 
 
adding datetime to the end, then only c1 got saturated
+--------+--------+--------+--------+--------+--------+---------+----------+-----------+------------+------+
| CCHAR1 | CCHAR2 | CCHAR3 | CCHAR4 | CCHAR5 | CCHAR6 | CCHAR7  | CCHAR8   | CCHAR9    | CCHAR255   | c1   |
+--------+--------+--------+--------+--------+--------+---------+----------+-----------+------------+------+
| z      | zz     | zzz    | zzzz   | zzzzz  | zzzzzz | zzzzzzz | zzzzzzzz | zzzzzzzzz | zzzzzzzzzz | NULL |
+--------+--------+--------+--------+--------+--------+---------+----------+-----------+------------+------+
1 row in set (0.12 sec)

"



 Comments   
Comment by Dipti Joshi (Inactive) [ 2016-05-10 ]

dleeyh If you do the same test using cpimport what result do you get ?

Comment by Daniel Lee (Inactive) [ 2016-05-10 ]

In this test, LDI is not matching to cpimport. cpimport has the correct result.

Comment by Daniel Lee (Inactive) [ 2016-06-22 ]

Build verified: 1.0.1 alpha

getsoftwareinfo Wed Jun 22 11:01:15 2016

Name : mariadb-columnstore-platform Relocations: (not relocatable)
Version : 1.0 Vendor: MariaDB Corporation Ab
Release : 1 Build Date: Mon 13 Jun 2016 06:50:26 PM CDT
Install Date: Fri 17 Jun 2016 10:53:22 AM CDT Build Host: srvbuilder
Group : Applications Source RPM: mariadb-columnstore-1.0-1.src.rpm

Verified that the reported issue is no longer exist in this build.

The original issue was that when during LDI operation, if the leftmost datetime columns gets saturation, the rightmost column also got truncated:

MariaDB [mytest]> select * from t1;
--------------------------------------------------------------------------

c1 CCHAR1 CCHAR2 CCHAR3 CCHAR4 CCHAR5 CCHAR6 CCHAR7 CCHAR8 CCHAR9 CCHAR255

--------------------------------------------------------------------------

NULL z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzz

The rightmost column should be have 10 characters, not 7, as shown above.

MariaDB [mytest]> create table t1 (c1 datetime, cchar1 char(1), cchar2 char(2),cchar3 char(3),cchar4 char(4),cchar5 char(5),cchar6 char(6),cchar7 char(7),cchar8 char(8),cchar9 char(9),cchar255 char(255)) engine=columnstore;
Query OK, 0 rows affected (0.20 sec)

MariaDB [mytest]> insert into t1 values ('1400-01-01 00:00:00','z','zz','zzz','zzzz','zzzzz','zzzzzz','zzzzzzz','zzzzzzzz','zzzzzzzzz','zzzzzzzzzzzzzzz');
Query OK, 1 row affected (0.36 sec)

MariaDB [mytest]> select * from t1;
------------------------------------------------------------------------------------------------

c1 cchar1 cchar2 cchar3 cchar4 cchar5 cchar6 cchar7 cchar8 cchar9 cchar255

------------------------------------------------------------------------------------------------

1400-01-01 00:00:00 z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz

------------------------------------------------------------------------------------------------
1 row in set (0.13 sec)

MariaDB [mytest]> insert into t1 values ('"1400-01-01 00:00:00"','z','zz','zzz','zzzz','zzzzz','zzzzzz','zzzzzzz','zzzzzzzz','zzzzzzzzz','zzzzzzzzzzzzzzz');
Query OK, 1 row affected, 1 warning (0.08 sec)

MariaDB [mytest]> show warnings;
----------------------------------------------------

Level Code Message

----------------------------------------------------

Warning 1265 Data truncated for column 'c1' at row 1

----------------------------------------------------
1 row in set (0.00 sec)

MariaDB [mytest]> select * from t1;
------------------------------------------------------------------------------------------------

c1 cchar1 cchar2 cchar3 cchar4 cchar5 cchar6 cchar7 cchar8 cchar9 cchar255

------------------------------------------------------------------------------------------------

1400-01-01 00:00:00 z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz
NULL z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz

------------------------------------------------------------------------------------------------
2 rows in set (0.04 sec)

MariaDB [mytest]> load data infile "/tmp/t1.txt" into table t1 fields terminated by "|";
Query OK, 1 row affected, 2 warnings (1.61 sec)
Records: 1 Deleted: 0 Skipped: 0 Warnings: 2

MariaDB [mytest]> show warnings;
----------------------------------------------------

Level Code Message

----------------------------------------------------

Warning 1265 Data truncated for column 'c1' at row 1
Warning 9999 Values saturated

----------------------------------------------------
2 rows in set (0.00 sec)

MariaDB [mytest]> select * from t1;
------------------------------------------------------------------------------------------------

c1 cchar1 cchar2 cchar3 cchar4 cchar5 cchar6 cchar7 cchar8 cchar9 cchar255

------------------------------------------------------------------------------------------------

1400-01-01 00:00:00 z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz
NULL z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz
NULL z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz

------------------------------------------------------------------------------------------------
3 rows in set (0.04 sec)

MariaDB [mytest]> quit
Bye
[root@columnStore ~]# /usr/local/mariadb/columnstore/bin/cpimport mytest t1 /tmp/t1.txt
Locale is : C

Using table OID 31350 as the default JOB ID
Input file(s) will be read from : /root
Job description file : /usr/local/mariadb/columnstore/data/bulk/tmpjob/31350_D20160622_T105604_S712847_Job_31350.xml
Log file for this job: /usr/local/mariadb/columnstore/data/bulk/log/Job_31350.log
2016-06-22 10:56:04 (54485) INFO : successfully loaded job file /usr/local/mariadb/columnstore/data/bulk/tmpjob/31350_D20160622_T105604_S712847_Job_31350.xml
2016-06-22 10:56:04 (54485) INFO : Job file loaded, run time for this step : 0.153057 seconds
2016-06-22 10:56:04 (54485) INFO : PreProcessing check starts
2016-06-22 10:56:04 (54485) INFO : input data file /tmp/t1.txt
2016-06-22 10:56:04 (54485) INFO : PreProcessing check completed
2016-06-22 10:56:04 (54485) INFO : preProcess completed, run time for this step : 0.059808 seconds
2016-06-22 10:56:04 (54485) INFO : No of Read Threads Spawned = 1
2016-06-22 10:56:04 (54485) INFO : No of Parse Threads Spawned = 3
2016-06-22 10:56:04 (54485) INFO : For table mytest.t1: 1 rows processed and 1 rows inserted.
2016-06-22 10:56:04 (54485) WARN : Column mytest.t1.c1; Number of invalid date/times replaced with null: 1
2016-06-22 10:56:05 (54485) INFO : Bulk load completed, total run time : 1.22252 seconds

[root@columnStore ~]# mcsmysql mytest
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 60233
Server version: 10.1.14-MariaDB Columnstore 1.0-1

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [mytest]> select * from t1;
------------------------------------------------------------------------------------------------

c1 cchar1 cchar2 cchar3 cchar4 cchar5 cchar6 cchar7 cchar8 cchar9 cchar255

------------------------------------------------------------------------------------------------

1400-01-01 00:00:00 z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz
NULL z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz
NULL z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz
NULL z zz zzz zzzz zzzzz zzzzzz zzzzzzz zzzzzzzz zzzzzzzzz zzzzzzzzzzzzzzz

------------------------------------------------------------------------------------------------
4 rows in set (0.09 sec)

cat /tmp/t1.txt
[root@columnStore tmp]# cat /tmp/t1.txt
"1400-01-01 00:00:00"|z|zz|zzz|zzzz|zzzzz|zzzzzz|zzzzzzz|zzzzzzzz|zzzzzzzzz|zzzzzzzzzzzzzzz|[root@columnStore tmp]#
[

Generated at Thu Feb 08 02:17:53 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.