Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-3874

Convert gives excessive rounding error on long string fields

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 1.2.4
    • Icebox
    • PrimProc
    • None

    Description

      The built in function FORMAT is used to display a number with group separators for the locale.
      When the number is a very large string containing digits, the string is first converted to a double, then formatted. Both Server and Columnstore do it this way. However, the final output differs by quite a bit. This is because Server has their own double to string function my_fcvt() while columnstore uses snprintf(), which appears to have more rounding error.

      For server, see item_strfunc.cc – String *Item_func_format::val_str_ascii(String *str)
      For Columnstore see func_math.cpp – string Func_format::getStrVal()

      We should either copy my_fcvt or find another way to eliminate the rounding error.

      Attachments

        Activity

          People

            Unassigned Unassigned
            David.Hall David Hall (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.