Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4516

regr_r2 and potentially other statistical functions can't handle tiny negative float values as variance

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 5.4.1
    • Fix Version/s: 5.6.1
    • Component/s: ExeMgr
    • Labels:
      None
    • Sprint:
      2021-2, 2021-3, 2021-4, 2021-5

      Description

      The current statistical functions code doesn't take into account a minuscule negative deviation value that is actually represents a float 0.
      According with the basic properties of Variance [1] and the regr_r2 calculation algo [2] the covar_popy must be non-negative so that sqrt(covar_popy) in regr_r2::evaluate() doesn't return NaN.

      4QA This query must return 0.0000000000 according with the reference

      { select round(regr_r2(l_tax, l_extendedprice),10) from lineitem where l_tax = 0.05 group by l_tax order by l_tax; }

      1. https://en.wikipedia.org/wiki/Variance
      2. https://docs.oracle.com/cd/B28359_01/server.111/b28286/functions139.htm#SQLRF00696

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dleeyh Daniel Lee
              Reporter:
              drrtuy Roman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration