[MCOL-4650] TreeNode::getIntVal() looses precision for narrow decimal Created: 2021-03-30  Updated: 2021-03-30  Resolved: 2021-03-30

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 5.6.1, 6.1.1
Fix Version/s: 6.1.1

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Alexander Barkov
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MCOL-4361 Replace pow(10.0, (double)scale) expr... Closed
Relates
relates to MCOL-4609 TreeNode::getIntVal() does not round:... Closed
relates to MCOL-4610 TreeNode::getUintVal() looses precisi... Closed
relates to MCOL-4619 TreeNode::getUintVal() does not round... Closed

 Description   

The code in TreeNode::getIntVal() performs DECIMAL-to-INT conversion using pow(), so the calculation is done in double format. This leads to precision loss and therefore to MariaDB incompatible results.

DROP TABLE IF EXISTS t1;
 
CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
 
INSERT INTO t1 VALUES (9999999999999999);
 
INSERT INTO t1 VALUES (99999999999999998);
 
INSERT INTO t1 VALUES (999999999999999997);
 
SELECT RAND(a) FROM t1;


+-----------------------+
 
| RAND(a)               |
 
+-----------------------+
 
|   0.13983957482486925 |
 
| 0.0014118393896257872 |
 
|    0.6171344999383525 |
 
+-----------------------+

The above result is wrong. The expected result is:

DROP TABLE IF EXISTS t1;
 
CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=InnoDB;
 
INSERT INTO t1 VALUES (9999999999999999);
 
INSERT INTO t1 VALUES (99999999999999998);
 
INSERT INTO t1 VALUES (999999999999999997);
 
SELECT RAND(a) FROM t1;


+--------------------+
 
| RAND(a)            |
 
+--------------------+
 
| 0.8896564653978277 |
 
| 0.5010456205355428 |
 
|  0.866585171657228 |
 
+--------------------+


Generated at Thu Feb 08 02:51:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.