[MCOL-4649] CAST(double AS UNSIGNED) returns 0 Created: 2021-03-30  Updated: 2021-04-05  Resolved: 2021-04-05

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 5.6.1, 6.1.1
Fix Version/s: 6.1.1

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Alexander Barkov
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MCOL-4361 Replace pow(10.0, (double)scale) expr... Closed
Relates
relates to MCOL-4631 CAST(double AS SIGNED) returns 0 or NULL Closed
Sprint: 2021-5

 Description   

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (d1 DOUBLE, d2 DOUBLE NOT NULL) ENGINE=ColumnStore;
INSERT INTO t1 VALUES (18446744073709551614,18446744073709551614);
SELECT d1, CAST(d1 AS UNSIGNED), CAST(d2 AS UNSIGNED) FROM t1;

+-----------------------+----------------------+----------------------+
| d1                    | CAST(d1 AS UNSIGNED) | CAST(d2 AS UNSIGNED) |
+-----------------------+----------------------+----------------------+
| 1.8446744073709552e19 |                    0 |                    0 |
+-----------------------+----------------------+----------------------+

Looks wrong.

The expected result is:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (d1 DOUBLE, d2 DOUBLE NOT NULL) ENGINE=InnoDB;
INSERT INTO t1 VALUES (18446744073709551614,18446744073709551614);
SELECT d1, CAST(d1 AS UNSIGNED), CAST(d2 AS UNSIGNED) FROM t1;

+-----------------------+----------------------+----------------------+
| d1                    | CAST(d1 AS UNSIGNED) | CAST(d2 AS UNSIGNED) |
+-----------------------+----------------------+----------------------+
| 1.8446744073709552e19 | 18446744073709551615 | 18446744073709551615 |
+-----------------------+----------------------+----------------------+

The problem happens in this piece of the code in Func_cast_unsigned::getUintVal():

289                 case execplan::CalpontSystemCatalog::FLOAT:
290                 case execplan::CalpontSystemCatalog::UFLOAT:
291                 case execplan::CalpontSystemCatalog::DOUBLE:
292                 case execplan::CalpontSystemCatalog::UDOUBLE:
293                 {
294                     double value = parm[0]->data()->getDoubleVal(row, isNull);
295
296                     if (value > 0)
297                         value += 0.5;
298                     else if (value < 0)
299                         value -= 0.5;
300
301                     uint64_t ret = (uint64_t) value;    ///   THIS LINE IS WRONG
302
303                     if (value > (double) numeric_limits<uint64_t>::max() - 2)
304                         ret = numeric_limits<int64_t>::max();
305                     else if (value < 0)
306                         ret = 0;
307
308                     return ret;  

GDB session screenshot:

(gdb) p	value
$4 = 1.8446744073709552e+19
(gdb) p	ret
$5 = 0


Generated at Thu Feb 08 02:51:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.