[MCOL-5188] regr funtions have numeric error when numers are very small Created: 2022-08-11  Updated: 2022-10-25  Resolved: 2022-10-12

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 6.4.2
Fix Version/s: 22.08.2

Type: Task Priority: Major
Reporter: David Hall (Inactive) Assignee: Andrey Piskunov (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
PartOf
includes MCOL-3655 regr_sxy() returns different results ... Closed
Sprint: 2021-17
Assigned for Review: David Hall David Hall (Inactive)
Assigned for Testing: Daniel Lee Daniel Lee (Inactive)

 Description   

Specifically regr_r2 and regr_slope, but there could be a similar behavior in others.

When we calculate the final result, often the amount of rounding error in the accumulators is such that we can get different results depending on the order of the data read. This makes our test suite spuriously fail.

There are techniques found in the literature that can help compensate for these numerical issues.
See
https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
https://www.johndcook.com/blog/standard_deviation/
For ideas. I'm sure there are other references that can be found.


Generated at Thu Feb 08 02:56:00 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.