[MCOL-4667] Precision loss on AVG(decimal) Created: 2021-04-06  Updated: 2023-07-02

Status: Open
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 5.6.1, 6.1.1
Fix Version/s: Icebox

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Duplicate
is duplicated by MCOL-4603 Replace long double with wide/narrow-... Closed
Relates
relates to MCOL-641 Full DECIMAL support in ColumnStore Closed
relates to MCOL-4361 Replace pow(10.0, (double)scale) expr... Closed

 Description   

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(18,17)) ENGINE=ColumnStore;
INSERT INTO t1 VALUES (-8.99999999999999999);
SELECT AVG(a) FROM t1;

+--------------------------+
| AVG(a)                   |
+--------------------------+
| -8.999999999999999989600 |
+--------------------------+

Looks wrong. The expected result is:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a DECIMAL(18,17)) ENGINE=InnoDB;
INSERT INTO t1 VALUES (-8.99999999999999999);
SELECT AVG(a) FROM t1;

+--------------------------+
| AVG(a)                   |
+--------------------------+
| -8.999999999999999990000 |
+--------------------------+


Generated at Thu Feb 08 02:52:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.