Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4650

TreeNode::getIntVal() looses precision for narrow decimal

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 5.6.1, 6.1.1
    • 6.1.1
    • PrimProc
    • None

    Description

      The code in TreeNode::getIntVal() performs DECIMAL-to-INT conversion using pow(), so the calculation is done in double format. This leads to precision loss and therefore to MariaDB incompatible results.

      DROP TABLE IF EXISTS t1;
      		 
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
      		 
      INSERT INTO t1 VALUES (9999999999999999);
      		 
      INSERT INTO t1 VALUES (99999999999999998);
      		 
      INSERT INTO t1 VALUES (999999999999999997);
      		 
      SELECT RAND(a) FROM t1;


      +-----------------------+
      		 
      | RAND(a)               |
      		 
      +-----------------------+
      		 
      |   0.13983957482486925 |
      		 
      | 0.0014118393896257872 |
      		 
      |    0.6171344999383525 |
      		 
      +-----------------------+

      The above result is wrong. The expected result is:

      DROP TABLE IF EXISTS t1;
      		 
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=InnoDB;
      		 
      INSERT INTO t1 VALUES (9999999999999999);
      		 
      INSERT INTO t1 VALUES (99999999999999998);
      		 
      INSERT INTO t1 VALUES (999999999999999997);
      		 
      SELECT RAND(a) FROM t1;


      +--------------------+
      		 
      | RAND(a)            |
      		 
      +--------------------+
      		 
      | 0.8896564653978277 |
      		 
      | 0.5010456205355428 |
      		 
      |  0.866585171657228 |
      		 
      +--------------------+

      Attachments

        Issue Links

          blocks

          Task - A task that needs to be done. MCOL-4361 Replace pow(10.0, (double)scale) expressions with a static dictionary lookup.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MCOL-4609 TreeNode::getIntVal() does not round: implicit DECIMAL->INT cast is not MariaDB compatible

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MCOL-4610 TreeNode::getUintVal() looses precision for narrow decimal

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MCOL-4619 TreeNode::getUintVal() does not round: Implicit DECIMAL->UINT conversion is not like in InnoDB

          • Major - Major loss of function.
          • Closed

          Activity

            People

              bar Alexander Barkov
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.