Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-5188

regr funtions have numeric error when numers are very small

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.4.2
    • Fix Version/s: 22.08.2
    • Component/s: PrimProc
    • Labels:
      None
    • Sprint:
      2021-17

      Description

      Specifically regr_r2 and regr_slope, but there could be a similar behavior in others.

      When we calculate the final result, often the amount of rounding error in the accumulators is such that we can get different results depending on the order of the data read. This makes our test suite spuriously fail.

      There are techniques found in the literature that can help compensate for these numerical issues.
      See
      https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
      https://www.johndcook.com/blog/standard_deviation/
      For ideas. I'm sure there are other references that can be found.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Andrey Andrey Piskunov
              Reporter:
              David.Hall David Hall
              Assigned for Review:
              David Hall David Hall
              Assigned for Testing:
              Daniel Lee Daniel Lee
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.