Details
-
Task
-
Status: Stalled (View Workflow)
-
Minor
-
Resolution: Unresolved
-
None
Description
The function variance_fp_recurrence_next() which is used for STDDEV calculation, has this code:
volatile double diff= nr - m_kminusone; |
*m= m_kminusone + diff / (double) *count; |
When nr and m_kminuseone are huge numbers, the result of the "minus" operation, stored in the diff variable, can overflow to -inf or +inf.
However, the result of division would normally fit into the supported double range.
We'll change this code to use the long double data type for temporary variables such as diff, to improve the precision.
Attachments
Issue Links
- is blocked by
-
MDEV-19062 Item_sum_variance: move recurrence_m, recurrence_s, count to a separate class
- Closed
- relates to
-
MDEV-17643 Assertion `nr >= 0.0' failed in Item_sum_std::val_real()
- Closed