Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-5188

regr funtions have numeric error when numers are very small

    XMLWordPrintable

Details

    • Task
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 6.4.2
    • 22.08.2
    • PrimProc
    • None
    • 2021-17

    Description

      Specifically regr_r2 and regr_slope, but there could be a similar behavior in others.

      When we calculate the final result, often the amount of rounding error in the accumulators is such that we can get different results depending on the order of the data read. This makes our test suite spuriously fail.

      There are techniques found in the literature that can help compensate for these numerical issues.
      See
      https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
      https://www.johndcook.com/blog/standard_deviation/
      For ideas. I'm sure there are other references that can be found.

      Attachments

        Issue Links

          Activity

            People

              Andrey Andrey Piskunov (Inactive)
              David.Hall David Hall (Inactive)
              David Hall David Hall (Inactive)
              Daniel Lee Daniel Lee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.