Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4667

Precision loss on AVG(decimal)

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 5.6.1, 6.1.1
    • Icebox
    • PrimProc
    • None

    Description

      DROP TABLE IF EXISTS t1;
      		 
      CREATE TABLE t1 (a DECIMAL(18,17)) ENGINE=ColumnStore;
      		 
      INSERT INTO t1 VALUES (-8.99999999999999999);
      		 
      SELECT AVG(a) FROM t1;


      +--------------------------+
      		 
      | AVG(a)                   |
      		 
      +--------------------------+
      		 
      | -8.999999999999999989600 |
      		 
      +--------------------------+

      Looks wrong. The expected result is:

      DROP TABLE IF EXISTS t1;
      		 
      CREATE TABLE t1 (a DECIMAL(18,17)) ENGINE=InnoDB;
      		 
      INSERT INTO t1 VALUES (-8.99999999999999999);
      		 
      SELECT AVG(a) FROM t1;


      +--------------------------+
      		 
      | AVG(a)                   |
      		 
      +--------------------------+
      		 
      | -8.999999999999999990000 |
      		 
      +--------------------------+

      Attachments

        Issue Links

          is duplicated by

          New Feature - A new feature of the product, which has yet to be developed. MCOL-4603 Replace long double with wide/narrow-decimal for avg() and sum() result type for all numerical datatypesdecimal result

          • Major - Major loss of function.
          • Closed
          relates to

          New Feature - A new feature of the product, which has yet to be developed. MCOL-641 Full DECIMAL support in ColumnStore

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MCOL-4361 Replace pow(10.0, (double)scale) expressions with a static dictionary lookup.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              Unassigned Unassigned
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.