Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4611

MOD loses precision on huge narrow decimal

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.6.1, 6.1.1, 5.5.1
    • Fix Version/s: 6.3.1
    • Component/s: None
    • Labels:
      None

      Description

      Because of using double as a temporary storage for decimal in Func_mod::getDecimalVal, the MOD operator looses precision on huge narrow decimal values:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
      INSERT INTO t1 VALUES (999999999999999999);
      SELECT a MOD 10000 FROM t1;
      

      +-------------+
      | a MOD 10000 |
      +-------------+
      |           0 |
      +-------------+
      

      Looks wrong. The expected result is:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (999999999999999999);
      SELECT a MOD 10000 FROM t1;
      

      +-------------+
      | a MOD 10000 |
      +-------------+
      |        9999 |
      +-------------+
      

      The same problem presents in the template Func_mod::doDecimal:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=ColumnStore;
      INSERT INTO t1 VALUES (999999999999999999);
      SELECT RAND(9999), RAND(a MOD 10000) FROM t1;
      

      +--------------------+---------------------+
      | RAND(9999)         | RAND(a MOD 10000)   |
      +--------------------+---------------------+
      | 0.7361292641015065 | 0.15522042769493574 |
      +--------------------+---------------------+
      

      The result for the second column looks wrong.

      The expected result is:

      DROP TABLE IF EXISTS t1;
      CREATE TABLE t1 (a DECIMAL(18,0)) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (999999999999999999);
      SELECT RAND(9999), RAND(a MOD 10000) FROM t1;
      

      +--------------------+--------------------+
      | RAND(9999)         | RAND(a MOD 10000)  |
      +--------------------+--------------------+
      | 0.7361292641015065 | 0.7361292641015065 |
      +--------------------+--------------------+
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              bar Alexander Barkov
              Reporter:
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration