Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4610

TreeNode::getUintVal() looses precision for narrow decimal

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 5.5.1, 5.6.1, 6.1.1
    • 6.1.1
    • PrimProc
    • None
    • 2021-5

    Description

      The code for the DECIMAL data type in TreeNode::getUintVal() performs calculations in double format. This approach looses precision for huge numbers.

              case CalpontSystemCatalog::DECIMAL:
              case CalpontSystemCatalog::UDECIMAL:
              {
                  return (uint64_t)(fResult.decimalVal.value / pow((double)10, fResult.decimalVal.scale));
              }
      

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
      INSERT INTO t1 VALUES (9999999999999999);
      INSERT INTO t1 VALUES (99999999999999998);
      INSERT INTO t1 VALUES (999999999999999997);
      SELECT a, CAST(a DIV 1 AS UNSIGNED) FROM t1;
      

      +--------------------+---------------------------+
      | a                  | CAST(a DIV 1 AS UNSIGNED) |
      +--------------------+---------------------------+
      |   9999999999999999 |         10000000000000000 |
      |  99999999999999998 |        100000000000000000 |
      | 999999999999999997 |       1000000000000000000 |
      +--------------------+---------------------------+
      

      Looks wrong. Values in the first and the second columns are expected to be equal.

      Note:

      • the problem is repeatable for narrow decimal in 5.5 and 5.6
      • the problem is not repeatable for wide decimal in 5.6:

        DROP TABLE IF EXISTS t1;
        CREATE TABLE t1 (a DECIMAL(30,0)) ENGINE=ColumnStore;
        INSERT INTO t1 VALUES (9999999999999999);
        INSERT INTO t1 VALUES (99999999999999998);
        INSERT INTO t1 VALUES (999999999999999997);
        SELECT a, CAST(a DIV 1 AS UNSIGNED) FROM t1;
        

        +--------------------+---------------------------+
        | a                  | CAST(a DIV 1 AS UNSIGNED) |
        +--------------------+---------------------------+
        |   9999999999999999 |          9999999999999999 |
        |  99999999999999998 |         99999999999999998 |
        | 999999999999999997 |        999999999999999997 |
        +--------------------+---------------------------+
        

      Attachments

        Issue Links

          Activity

            People

              bar Alexander Barkov
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.