[MCOL-2062] cpimport scientific notation conversion problem Created: 2019-01-04  Updated: 2020-08-25  Resolved: 2019-01-14

Status: Closed
Project: MariaDB ColumnStore
Component/s: cpimport
Affects Version/s: 1.0.15, 1.1.6, 1.2.2
Fix Version/s: 1.0.16, 1.1.7, 1.2.3

Type: Bug Priority: Major
Reporter: Juan Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Problem/Incident
causes MCOL-2149 Regression in decimal saturation hand... Closed
Relates

 Description   

After loading data (using cpimport), noticed some issues with it. When I load the data using cpimport, the field value which is stored as in scientific form (8e-05), it gets loaded into the database as 0.000008 instead of 0.000080

When I use the regular load infile command, the data is loaded just fine.

[root@localhost ~]# cat t1.dat
 
8e-5|8e-5|8e-5|
 
after cpimport, I got
 
MariaDB [mytest]> select * from t1;
 
+------------+---------+---------+
 
| c1         | c2      | c3      |
 
+------------+---------+---------+
 
| 0.00000008 | 0.00008 | 0.00008 |
 
+------------+---------+---------+
 
1 row in set (0.155 sec)
 
so issue is with cpimport on decimal column
 
insert looks good
 
MariaDB [mytest]> insert t1 values (8e-5,8e-5,8e-5);
 
Query OK, 1 row affected (0.439 sec)
 
 
 
MariaDB [mytest]> select * from t1;
 
+------------+---------+---------+
 
| c1         | c2      | c3      |
 
+------------+---------+---------+
 
| 0.00000008 | 0.00008 | 0.00008 |
 
| 0.00008000 | 0.00008 | 0.00008 |
 
+------------+---------+---------+
 
2 rows in set (0.194 sec)
 
first row is cpimport, 2nd row is insert.  Insert matched that on Innodb



 Comments   
Comment by Patrick LeBlanc (Inactive) [ 2019-01-09 ]

IIRC we had to fix this at Tune, and the fix was relatively easy. I think there is one fcn in cpimport that does the conversion, and I think I just made it use a conversion function from the standard library instead of doing it 'by hand'. So, if memory serves, this should be an easy fix once we get to it.

https://en.cppreference.com/w/c/string/byte/strtof

Comment by Patrick LeBlanc (Inactive) [ 2019-01-09 ]

Since folks are waiting on this I went ahead and did it while I was thinking about it. It's probably similar to what I did at Tune b/c there aren't a lot of other ways to do it unless I made it more complicated than it needs to be. Added a test to our regression suite, so there are 2 PRs for this.

https://github.com/mariadb-corporation/mariadb-columnstore-regression-test/pull/96
https://github.com/mariadb-corporation/mariadb-columnstore-engine/pull/672

Comment by Daniel Lee (Inactive) [ 2019-01-14 ]

Test table:

create table t1 (c1 decimal(18,8), c2 double, c3 float ) engine=columnstore;

Build verified:

1.2.3-1
server commit:
15816d7
engine commit:
db10662

1-1-7-1
server commit:
88da1b6
engine commit:
63824fa

1.0.16-1
Builedbot server does not have the git commit info. The packages files are dated as:

rw-rr-. 1 10103 2000 376111920 Jan 11 04:11 mariadb-columnstore-1.0.16-1-centos7.x86_64.bin.tar.gz
rw-rr-. 1 10103 2000 155606401 Jan 11 04:11 mariadb-columnstore-1.0.16-1-centos7.x86_64.rpm.tar.gz

Generated at Thu Feb 08 02:33:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.