Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4650

TreeNode::getIntVal() looses precision for narrow decimal

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.6.1, 6.1.1
    • Fix Version/s: 6.1.1
    • Component/s: PrimProc
    • Labels:
      None

      Description

      The code in TreeNode::getIntVal() performs DECIMAL-to-INT conversion using pow(), so the calculation is done in double format. This leads to precision loss and therefore to MariaDB incompatible results.

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
      INSERT INTO t1 VALUES (9999999999999999);
      INSERT INTO t1 VALUES (99999999999999998);
      INSERT INTO t1 VALUES (999999999999999997);
      SELECT RAND(a) FROM t1;
      

      +-----------------------+
      | RAND(a)               |
      +-----------------------+
      |   0.13983957482486925 |
      | 0.0014118393896257872 |
      |    0.6171344999383525 |
      +-----------------------+
      

      The above result is wrong. The expected result is:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (9999999999999999);
      INSERT INTO t1 VALUES (99999999999999998);
      INSERT INTO t1 VALUES (999999999999999997);
      SELECT RAND(a) FROM t1;
      

      +--------------------+
      | RAND(a)            |
      +--------------------+
      | 0.8896564653978277 |
      | 0.5010456205355428 |
      |  0.866585171657228 |
      +--------------------+
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              bar Alexander Barkov
              Reporter:
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration