Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19050

Item_sum_std: change internal temporary variables to long double

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Stalled (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Fix Version/s: 10.4
    • Component/s: OTHER
    • Labels:
      None

      Description

      The function variance_fp_recurrence_next() which is used for STDDEV calculation, has this code:

          volatile double diff= nr - m_kminusone;
          *m= m_kminusone + diff / (double) *count;
      

      When nr and m_kminuseone are huge numbers, the result of the "minus" operation, stored in the diff variable, can overflow to -inf or +inf.
      However, the result of division would normally fit into the supported double range.

      We'll change this code to use the long double data type for temporary variables such as diff, to improve the precision.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              bar Alexander Barkov
              Reporter:
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: