[MCOL-4600] CAST(decimal AS SIGNED/UNSIGNED) returns a wrong result Created: 2021-03-11  Updated: 2021-03-30  Resolved: 2021-03-30

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 5.5.1, 5.6.1, 6.1.1
Fix Version/s: 6.1.1

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Alexander Barkov
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MCOL-4361 Replace pow(10.0, (double)scale) expr... Closed
Sprint: 2021-5

 Description    

DROP TABLE IF EXISTS t1;
 
CREATE TABLE t1 (a DECIMAL(18,17)) ENGINE=ColumnStore;
 
INSERT INTO t1 VALUES (9.49999999999999999);
 
SELECT a, CAST(a AS UNSIGNED), CAST(a AS SIGNED) FROM t1;


+---------------------+---------------------+-------------------+
 
| a                   | CAST(a AS UNSIGNED) | CAST(a AS SIGNED) |
 
+---------------------+---------------------+-------------------+
 
| 9.49999999999999999 |                  10 |                10 |
 
+---------------------+---------------------+-------------------+

The above looks wrong. If I change to ENGINE=InnoDB, it correctly returns:

+---------------------+---------------------+-------------------+
 
| a                   | CAST(a AS UNSIGNED) | CAST(a AS SIGNED) |
 
+---------------------+---------------------+-------------------+
 
| 9.49999999999999999 |                   9 |                 9 |
 
+---------------------+---------------------+-------------------+

The problem happens due to the precision loss in this code in Func_cast_unsigned::getUintVal():

            }
 
            else
 
            {
 
                if (d.value < 0)
 
                {
 
                    return 0;
 
                }
 
 
 
                double dscale = d.scale;
 
 
 
                uint64_t value = d.value / pow(10.0, dscale);
 
                int lefto = (d.value - value * pow(10.0, dscale)) / pow(10.0, dscale - 1);
 
 
 
            if ( utils::is_nonnegative(value) && lefto > 4 )
 
            {
 
                value++;
 
            }
 
 
 
                return value;
 
             }


Generated at Thu Feb 08 02:51:35 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.