[MCOL-4633] Remove duplicate code for DECIMAL to int64_t rounding conversion Created: 2021-03-23  Updated: 2021-03-26  Resolved: 2021-03-25

Status: Closed
Project: MariaDB ColumnStore
Component/s: PrimProc
Affects Version/s: 6.1.1
Fix Version/s: 6.1.1

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Gagan Goel (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Blocks
blocks MCOL-4361 Replace pow(10.0, (double)scale) expr... Closed
blocks MCOL-4640 Narrow DECIMAL precision loss in CAST... Closed
Relates
relates to MCOL-4604 CHAR(negativeWideDecimal) not behavin... Closed

 Description   

This code in Func_cast_signed::getIntVal():

        case execplan::CalpontSystemCatalog::DECIMAL:
        case execplan::CalpontSystemCatalog::UDECIMAL:
        {
            IDB_Decimal d = parm[0]->data()->getDecimalVal(row, isNull);
 
            if (parm[0]->data()->resultType().colWidth == datatypes::MAXDECIMALWIDTH)
            {
                return static_cast<int64_t>(d.getPosNegRoundedIntegralPart(4));
            }
            else
            {
                double dscale = d.scale;
                int64_t value = d.value / pow(10.0, dscale);
                int lefto = (d.value - value * pow(10.0, dscale)) / pow(10.0, dscale - 1);
 
                if ( value >= 0 && lefto > 4 )
                    value++;
 
                if ( value < 0 && lefto < -4 )
                    value--;
 
                return value;
            }
        }

is also repeated in:

  • func_makedate.cpp - 2 times
  • func_maketime.cpp - 3 times
  • func_char.cpp - in Func_char::getStrVal() - not precisely the same, but a very similar code

In order to make the patch for MCOL-4631 smaller, let's remove this duplicate code as a separate change.
Note, under terms of MCOL-4633 the new shared code will still use pow(), which will be replaced to a dictionary lookup later - in the final patch for MCOL-4361.


Generated at Thu Feb 08 02:51:50 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.