[MCOL-4611] MOD loses precision on huge narrow decimal Created: 2021-03-16  Updated: 2023-07-02

Status: Open
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: 5.5.1, 5.6.1, 6.1.1
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: beginner-friendly

Issue Links:
Relates
relates to MCOL-641 Full DECIMAL support in ColumnStore Closed
relates to MCOL-4361 Replace pow(10.0, (double)scale) expr... Closed

 Description   

Because of using double as a temporary storage for decimal in Func_mod::getDecimalVal, the MOD operator looses precision on huge narrow decimal values:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
INSERT INTO t1 VALUES (999999999999999999);
SELECT a MOD 10000 FROM t1;

+-------------+
| a MOD 10000 |
+-------------+
|           0 |
+-------------+

Looks wrong. The expected result is:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=InnoDB;
INSERT INTO t1 VALUES (999999999999999999);
SELECT a MOD 10000 FROM t1;

+-------------+
| a MOD 10000 |
+-------------+
|        9999 |
+-------------+

The same problem presents in the template Func_mod::doDecimal:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
INSERT INTO t1 VALUES (999999999999999999);
SELECT RAND(9999), RAND(a MOD 10000) FROM t1;

+--------------------+---------------------+
| RAND(9999)         | RAND(a MOD 10000)   |
+--------------------+---------------------+
| 0.7361292641015065 | 0.15522042769493574 |
+--------------------+---------------------+

The result for the second column looks wrong.

The expected result is:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=InnoDB;
INSERT INTO t1 VALUES (999999999999999999);
SELECT RAND(9999), RAND(a MOD 10000) FROM t1;

+--------------------+--------------------+
| RAND(9999)         | RAND(a MOD 10000)  |
+--------------------+--------------------+
| 0.7361292641015065 | 0.7361292641015065 |
+--------------------+--------------------+


Generated at Thu Feb 08 02:51:40 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.