[MCOL-3874] Convert gives excessive rounding error on long string fields Created: 2020-03-06  Updated: 2023-11-21  Resolved: 2023-10-25

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 1.2.4
Fix Version/s: Icebox

Type: Bug Priority: Minor
Reporter: David Hall (Inactive) Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None

Epic Link: ColumnStore Compatibility Improvements

 Description   

The built in function FORMAT is used to display a number with group separators for the locale.
When the number is a very large string containing digits, the string is first converted to a double, then formatted. Both Server and Columnstore do it this way. However, the final output differs by quite a bit. This is because Server has their own double to string function my_fcvt() while columnstore uses snprintf(), which appears to have more rounding error.

For server, see item_strfunc.cc – String *Item_func_format::val_str_ascii(String *str)
For Columnstore see func_math.cpp – string Func_format::getStrVal()

We should either copy my_fcvt or find another way to eliminate the rounding error.



 Comments   
Comment by Gagan Goel (Inactive) [ 2020-04-03 ]

Once this is fixed, mysql/queries/working_tpch1_compareLogOnly/misc/bug3488.sql.ref.log needs to be changed to match output from innodb.

Comment by David Hall (Inactive) [ 2020-04-15 ]

working_tpch1/qa_fe_cnxFunctions/conv.sql.ref.log needs to be fixed as well

Generated at Thu Feb 08 02:46:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.