Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19050

Item_sum_std: change internal temporary variables to long double

    XMLWordPrintable

Details

    Description

      The function variance_fp_recurrence_next() which is used for STDDEV calculation, has this code:

          volatile double diff= nr - m_kminusone;
          *m= m_kminusone + diff / (double) *count;
      

      When nr and m_kminuseone are huge numbers, the result of the "minus" operation, stored in the diff variable, can overflow to -inf or +inf.
      However, the result of division would normally fit into the supported double range.

      We'll change this code to use the long double data type for temporary variables such as diff, to improve the precision.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.