Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4516

regr_r2 and potentially other statistical functions can't handle tiny negative float values as variance

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 5.4.1
    • 5.6.1
    • ExeMgr
    • None
    • 2021-2, 2021-3, 2021-4, 2021-5

    Description

      The current statistical functions code doesn't take into account a minuscule negative deviation value that is actually represents a float 0.
      According with the basic properties of Variance [1] and the regr_r2 calculation algo [2] the covar_popy must be non-negative so that sqrt(covar_popy) in regr_r2::evaluate() doesn't return NaN.

      4QA This query must return 0.0000000000 according with the reference

      { select round(regr_r2(l_tax, l_extendedprice),10) from lineitem where l_tax = 0.05 group by l_tax order by l_tax; }

      1. https://en.wikipedia.org/wiki/Variance
      2. https://docs.oracle.com/cd/B28359_01/server.111/b28286/functions139.htm#SQLRF00696

      Attachments

        Issue Links

          Activity

            People

              dleeyh Daniel Lee (Inactive)
              drrtuy Roman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.