Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4640

Narrow DECIMAL precision loss in CAST(AS SIGNED) and CHAR()

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 5.6.1, 6.1.1
    • 6.1.1
    • PrimProc
    • None

    Description

      The patch for MCOL-4633 removed duplicate code by introducing TDecimal64::toSInt64Round() and reusing it in a few places:

      • Func_cast_signed::getIntVal
      • Func_char::getStrVal
      • Func_elt::getStrVal
      • makedate
      • Func_maketime::getStrVal

      The patch for MCOL-4633 did not change the behavior: narrow DECIMAL to INT conversion in TDecimal64::toSInt64Round() is still performed in double format by multiplying to pow(10,scale),
      like in all duplicate chunks before the patch. This causes precision loss on huge narrow decimal values.

      Note, the patch for MCOL-4633 intentionally did not change the behavior because the underlying code is very essential and fragile. Let's do one small thing at a time, to commit small and easy to review changes.

      Under terms of this task we'll do the next step - fix TDecimal64::toSInt64Round() not to use DOUBLE arithmetic, to fix precision loss in the following examples:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(17,1)) ENGINE=ColumnStore;
      INSERT INTO t1 VALUES (8999999999999999.0);
      SELECT
        CAST(a AS SIGNED),
        HEX(CHAR(a USING latin1))
      FROM t1;
      

      +-------------------+---------------------------+
      | CAST(a AS SIGNED) | HEX(CHAR(a USING latin1)) |
      +-------------------+---------------------------+
      |  8999999999999998 | CAFA7FFE                  |
      +-------------------+---------------------------+
      

      Looks wrong.

      The expected result is:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(17,1)) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (8999999999999999.0);
      SELECT
        CAST(a AS SIGNED),
        HEX(CHAR(a USING latin1))
      FROM t1;
      

      +-------------------+---------------------------+
      | CAST(a AS SIGNED) | HEX(CHAR(a USING latin1)) |
      +-------------------+---------------------------+
      |  8999999999999999 | CAFA7FFF                  |
      +-------------------+---------------------------+
      

      Attachments

        Issue Links

          Activity

            People

              bar Alexander Barkov
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.