Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-3874

Convert gives excessive rounding error on long string fields

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.2.4
    • Fix Version/s: None
    • Component/s: PrimProc
    • Labels:
      None

      Description

      The built in function FORMAT is used to display a number with group separators for the locale.
      When the number is a very large string containing digits, the string is first converted to a double, then formatted. Both Server and Columnstore do it this way. However, the final output differs by quite a bit. This is because Server has their own double to string function my_fcvt() while columnstore uses snprintf(), which appears to have more rounding error.

      For server, see item_strfunc.cc – String *Item_func_format::val_str_ascii(String *str)
      For Columnstore see func_math.cpp – string Func_format::getStrVal()

      We should either copy my_fcvt or find another way to eliminate the rounding error.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            David.Hall David Hall
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: