[MDEV-19050] Item_sum_std: change internal temporary variables to long double Created: 2019-03-26  Updated: 2021-05-11

Status: Stalled
Project: MariaDB Server
Component/s: Data types, OTHER
Fix Version/s: None

Type: Task Priority: Minor
Reporter: Alexander Barkov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: functions

Issue Links:
Blocks
is blocked by MDEV-19062 Item_sum_variance: move recurrence_m,... Closed
Relates
relates to MDEV-17643 Assertion `nr >= 0.0' failed in Item_... Closed

 Description   

The function variance_fp_recurrence_next() which is used for STDDEV calculation, has this code:

    volatile double diff= nr - m_kminusone;
    *m= m_kminusone + diff / (double) *count;

When nr and m_kminuseone are huge numbers, the result of the "minus" operation, stored in the diff variable, can overflow to -inf or +inf.
However, the result of division would normally fit into the supported double range.

We'll change this code to use the long double data type for temporary variables such as diff, to improve the precision.



 Comments   
Comment by Alexander Barkov [ 2019-04-02 ]

Windows does not have a 16-byte double implementation. Let's postpone this task.

Generated at Thu Feb 08 08:48:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.