[MCOL-4609] TreeNode::getIntVal() does not round: implicit DECIMAL->INT cast is not MariaDB compatible Created: 2021-03-15  Updated: 2021-03-30  Resolved: 2021-03-30

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 5.5.1, 5.6.1, 6.1.1
Fix Version/s: 6.1.1

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Gagan Goel (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MCOL-4361 Replace pow(10.0, (double)scale) expr... Closed
Relates
relates to MCOL-4650 TreeNode::getIntVal() looses precisio... Closed

 Description   

The problem described below is also with narrow and wide DECIMAL.

The method TreeNode::getIntVal() performs truncation instead of rounding when converting from DECIMAL to signed integer:

        case CalpontSystemCatalog::DECIMAL:
        case CalpontSystemCatalog::UDECIMAL:
        {
            if (fResultType.colWidth == datatypes::MAXDECIMALWIDTH)
            {
                return static_cast<int64_t>(fResult.decimalVal.getIntegralPart());
            }
            else
            {
                return (int64_t)(fResult.decimalVal.value / pow((double)10, fResult.decimalVal.scale));
            }
        }

This makes ColumnStore return MariaDB incompatible values:

RAND()

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(10,1)) ENGINE=ColumnStore;
INSERT INTO t1 VALUES (1.4),(1.5);
SELECT a, RAND(a), RAND(CAST(a AS DECIMAL(10,1))) FROM t1;

+------+---------------------+--------------------------------+
| a    | RAND(a)             | RAND(CAST(a AS DECIMAL(10,1))) |
+------+---------------------+--------------------------------+
|  1.4 | 0.40540353712197724 |            0.40540353712197724 |
|  1.5 | 0.40540353712197724 |            0.40540353712197724 |
+------+---------------------+--------------------------------+

Looks wrong. RAND() did not round the seed value from 1.5 to 2.

The expected result is:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(10,1)) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1.4),(1.5);
SELECT a, RAND(a), RAND(CAST(a AS DECIMAL(10,1))) FROM t1;

+------+---------------------+--------------------------------+
| a    | RAND(a)             | RAND(CAST(a AS DECIMAL(10,1))) |
+------+---------------------+--------------------------------+
|  1.4 | 0.40540353712197724 |            0.40540353712197724 |
|  1.5 |  0.6555866465490187 |             0.6555866465490187 |
+------+---------------------+--------------------------------+

TRUNCATE()

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(10,1)) ENGINE=ColumnStore;
INSERT INTO t1 VALUES (1.4),(1.5);
SELECT TRUNCATE(1.2345, a) FROM t1;

+---------------------+
| TRUNCATE(1.2345, a) |
+---------------------+
|              1.2000 |
|              1.2000 |
+---------------------+

The expected result is:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(10,1)) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1.4),(1.5);
SELECT TRUNCATE(1.2345, a) FROM t1;

+---------------------+
| TRUNCATE(1.2345, a) |
+---------------------+
|              1.2000 |
|              1.2300 |
+---------------------+

CONV()

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(10,1)) ENGINE=ColumnStore;
INSERT INTO t1 VALUES (15.4),(15.5),(16.0);
SELECT a, CONV(16,a,10) FROM t1;

+------+---------------+
| a    | CONV(16,a,10) |
+------+---------------+
| 15.4 | 21            |
| 15.5 | 21            | <--- this is wrong
| 16.0 | 22            |
+------+---------------+

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(10,1)) ENGINE=InnoDB;
INSERT INTO t1 VALUES (15.4),(15.5),(16.0);
SELECT a, CONV(16,a,10) FROM t1;

+------+---------------+
| a    | CONV(16,a,10) |
+------+---------------+
| 15.4 | 21            |
| 15.5 | 22            |
| 16.0 | 22            |
+------+---------------+



 Comments   
Comment by Alexander Barkov [ 2021-03-20 ]

gdorman yes, I think so.

Generated at Thu Feb 08 02:51:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.