Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4640

Narrow DECIMAL precision loss in CAST(AS SIGNED) and CHAR()

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.6.1, 6.1.1
    • Fix Version/s: 6.1.1
    • Component/s: PrimProc
    • Labels:
      None

      Description

      The patch for MCOL-4633 removed duplicate code by introducing TDecimal64::toSInt64Round() and reusing it in a few places:

      • Func_cast_signed::getIntVal
      • Func_char::getStrVal
      • Func_elt::getStrVal
      • makedate
      • Func_maketime::getStrVal

      The patch for MCOL-4633 did not change the behavior: narrow DECIMAL to INT conversion in TDecimal64::toSInt64Round() is still performed in double format by multiplying to pow(10,scale),
      like in all duplicate chunks before the patch. This causes precision loss on huge narrow decimal values.

      Note, the patch for MCOL-4633 intentionally did not change the behavior because the underlying code is very essential and fragile. Let's do one small thing at a time, to commit small and easy to review changes.

      Under terms of this task we'll do the next step - fix TDecimal64::toSInt64Round() not to use DOUBLE arithmetic, to fix precision loss in the following examples:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(17,1)) ENGINE=ColumnStore;
      INSERT INTO t1 VALUES (8999999999999999.0);
      SELECT
        CAST(a AS SIGNED),
        HEX(CHAR(a USING latin1))
      FROM t1;
      

      +-------------------+---------------------------+
      | CAST(a AS SIGNED) | HEX(CHAR(a USING latin1)) |
      +-------------------+---------------------------+
      |  8999999999999998 | CAFA7FFE                  |
      +-------------------+---------------------------+
      

      Looks wrong.

      The expected result is:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(17,1)) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (8999999999999999.0);
      SELECT
        CAST(a AS SIGNED),
        HEX(CHAR(a USING latin1))
      FROM t1;
      

      +-------------------+---------------------------+
      | CAST(a AS SIGNED) | HEX(CHAR(a USING latin1)) |
      +-------------------+---------------------------+
      |  8999999999999999 | CAFA7FFF                  |
      +-------------------+---------------------------+
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              bar Alexander Barkov
              Reporter:
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration