[MCOL-298] Inconsistent value saturation behavior for DATE and DATETIME between INSERT and UPDATE, cpimport Created: 2016-09-14  Updated: 2016-09-16  Resolved: 2016-09-16

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: 1.0.3
Fix Version/s: 1.0.3

Type: Bug Priority: Major
Reporter: Daniel Lee (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Sprint: 1.0.3

 Description   

Build tested:

mscadmin> getsoft
getsoftwareinfo Wed Sep 14 12:59:32 2016

Name : mariadb-columnstore-platform
Version : 1.0.3
Release : 1
Architecture: x86_64
Install Date: Mon 12 Sep 2016 09:48:35 AM CDT

There are couple tickets in this release make changes related to DATE and DATETIME

MCOL-171 Added support for 0000-00-00, and it also changed value saturation behavior for DATE and DATETIME datatypes.

MCOL-274 Change support value range

Columnstore 1.0.2 value saturation behavior
Both DML statements and cpimport saturate out-of-range values to NULL

Columnstore 1.0.3 value saturation behavior

Note: The mininum supported year value for DATE and DATETIME has been changed from 1400 to 1000, matching standard MySQL

INSERT saturates to 0000-00-00
LDI saturates to 0000-00-00
UPDATE saturates to NULL
cpimport saturates to NULL

Standard MySQL DML statements saturates to NULL so I suggest we fix both UPDATE and cpimport to do the same.

Changes also needed in the Syntax Guide.

Page 8

Need to update the supported value range for DATE and DATETIME datatypes.

Page 10

For the date and datetime datatypes, a value outside of the supported range (1400-01-01 to 9999-12-31) will be stored as NULL in InfiniDB

needs to be updated to

For the date and datetime datatypes, a value outside of the supported range (1000-01-01 to 9999-12-31) will be stored as 0000-00-00 in ColumnStore



 Comments   
Comment by Andrew Hutchings (Inactive) [ 2016-09-14 ]

Pull request for this available at:

https://github.com/mariadb-corporation/mariadb-columnstore-engine/pull/10

Note: this should probably go into 1.0.3 since this is sort-of a regression.

Comment by Daniel Lee (Inactive) [ 2016-09-15 ]

Build tested:

mscadmin> getsoft
getsoftwareinfo Thu Sep 15 12:06:24 2016

Name : mariadb-columnstore-platform
Version : 1.0.3
Release : 1
Architecture: x86_64
Install Date: Thu 15 Sep 2016 11:39:01 AM CDT
Group : Applications
Size : 25431329
License : Copyright (c) 2016 MariaDB Corporation Ab., all rights reserved; redistributable under the terms of the GPL, see the file COPYING for details.
Signature : (none)
Source RPM : mariadb-columnstore-1.0.3-1.src.rpm
Build Date : Thu 15 Sep 2016 09:56:11 AM CDT

When updated DATETIME, out of range values are not being saturated and column values are not updated.

MariaDB [mytest]> select * from t1;
-------------------------------+

c1 c2

-------------------------------+

0000-00-00 0000-00-00 00:00:00
0000-00-00 1300-01-01 00:00:00
0000-00-00 0000-00-00 00:00:00

-------------------------------+
3 rows in set (0.02 sec)

[1:09]
it looks good for c1

[1:09]
MariaDB [mytest]> update t1 set c2 = '-1-12-01 00:00:00';
Query OK, 3 rows affected, 1 warning (0.19 sec)
Rows matched: 0 Changed: 0 Warnings: 1

MariaDB [mytest]> select * from t1;
-------------------------------+

c1 c2

-------------------------------+

0000-00-00 0000-00-00 00:00:00
0000-00-00 1300-01-01 00:00:00
0000-00-00 0000-00-00 00:00:00

-------------------------------+
3 rows in set (0.03 sec)

Comment by Daniel Lee (Inactive) [ 2016-09-15 ]

Build tested:

mscadmin> getsoft
getsoftwareinfo Thu Sep 15 16:52:58 2016

Name : mariadb-columnstore-platform
Version : 1.0.3
Release : 1
Architecture: x86_64
Install Date: Thu 15 Sep 2016 04:25:09 PM CDT
Group : Applications
Size : 25431329
License : Copyright (c) 2016 MariaDB Corporation Ab., all rights reserved; redistributable under the terms of the GPL, see the file COPYING for details.
Signature : (none)
Source RPM : mariadb-columnstore-1.0.3-1.src.rpm
Build Date : Thu 15 Sep 2016 03:41:46 PM CDT

We still have an issue with cpimport. When source file contains empty values (""), DATE and DATETIME saturated to NULL.

Source file has 3 rows.

[root@mcsce7 datatypes]# cat /tmp/t1.txt
1300-01-01|1300-01-01 00:00:00|
-1300-01-01|-1300-01-01 00:00:00|

MariaDB [mytest]> truncate t1;
Query OK, 0 rows affected (0.11 sec)

MariaDB [mytest]> load data infile '/tmp/t1.txt' into table t1 fields terminated by "|";
Query OK, 3 rows affected, 4 warnings (3.58 sec)
Records: 3 Deleted: 0 Skipped: 0 Warnings: 4

MariaDB [mytest]> show warnings;
----------------------------------------------------

Level Code Message

----------------------------------------------------

Warning 1265 Data truncated for column 'c1' at row 2
Warning 1265 Data truncated for column 'c2' at row 2
Warning 1265 Data truncated for column 'c1' at row 3
Warning 1265 Data truncated for column 'c2' at row 3

----------------------------------------------------
4 rows in set (0.00 sec)

MariaDB [mytest]> select * from t1;
-------------------------------+

c1 c2

-------------------------------+

1300-01-01 1300-01-01 00:00:00
0000-00-00 0000-00-00 00:00:00
0000-00-00 0000-00-00 00:00:00

-------------------------------+
3 rows in set (0.08 sec)

MariaDB [mytest]> quit
Bye
[root@mcsce7 datatypes]# /usr/local/mariadb/columnstore/bin/cpimport mytest t1 /tmp/t1.txt
No support for any checked transport domain
snmpd: create_trap_session:
Locale is : C

Using table OID 62932 as the default JOB ID
Input file(s) will be read from : /root/tests/systemTest/datatypes
Job description file : /usr/local/mariadb/columnstore/data/bulk/tmpjob/62932_D20160915_T164817_S753751_Job_62932.xml
Log file for this job: /usr/local/mariadb/columnstore/data/bulk/log/Job_62932.log
2016-09-15 16:48:17 (16063) INFO : successfully loaded job file /usr/local/mariadb/columnstore/data/bulk/tmpjob/62932_D20160915_T164817_S753751_Job_62932.xml
2016-09-15 16:48:17 (16063) INFO : Job file loaded, run time for this step : 0.170738 seconds
2016-09-15 16:48:17 (16063) INFO : PreProcessing check starts
2016-09-15 16:48:17 (16063) INFO : input data file /tmp/t1.txt
2016-09-15 16:48:18 (16063) INFO : PreProcessing check completed
2016-09-15 16:48:18 (16063) INFO : preProcess completed, run time for this step : 1.1447 seconds
2016-09-15 16:48:18 (16063) INFO : No of Read Threads Spawned = 1
2016-09-15 16:48:18 (16063) INFO : No of Parse Threads Spawned = 3
2016-09-15 16:48:18 (16063) INFO : For table mytest.t1: 3 rows processed and 3 rows inserted.
2016-09-15 16:48:18 (16063) WARN : Column mytest.t1.c1; Number of invalid dates replaced with zero value : 1
2016-09-15 16:48:18 (16063) WARN : Column mytest.t1.c2; Number of invalid date/times replaced with zero value : 1
2016-09-15 16:48:19 (16063) INFO : Bulk load completed, total run time : 2.32625 seconds

[root@mcsce7 datatypes]# mcsmysql mytest
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 795
Server version: 10.1.17-MariaDB Columnstore 1.0.3-1

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [mytest]> select * from t1;
-------------------------------+

c1 c2

-------------------------------+

1300-01-01 1300-01-01 00:00:00
0000-00-00 0000-00-00 00:00:00
0000-00-00 0000-00-00 00:00:00
1300-01-01 1300-01-01 00:00:00
0000-00-00 0000-00-00 00:00:00
NULL NULL

-------------------------------+
6 rows in set (0.10 sec)

MariaDB [mytest]>

LDI ran ok, but cpimport did not. The last row should be | 0000-00-00 | 0000-00-00 00:00:00 |.

Comment by Daniel Lee (Inactive) [ 2016-09-15 ]

BTW, the update issue was fixed.

Comment by Andrew Hutchings (Inactive) [ 2016-09-16 ]

cpimport treats all empty values as NULL, even integers. For non-char/varchar that is probably correct behaviour.

Comment by Andrew Hutchings (Inactive) [ 2016-09-16 ]

No changes, as discussed cpimport empty string handling will be in MCOL-300. Assign back to David Hall for review.

Comment by Daniel Lee (Inactive) [ 2016-09-16 ]

The remaining inconsistency will be covered by MCOL-300. Closing this ticket now.

Generated at Thu Feb 08 02:20:00 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.