Dynamic-column varints accept out-of-domain values through truncation

COLUMN_CHECK accepts malformed dynamic-column records whose ten-group varints exceed either 64 bits or the narrower charset-ID and decimal precision/scale domains. COLUMN_GET then truncates the metadata and decodes the records as valid low-bit values, including the string x and decimal value 1.2. This was reproduced with MariaDB 12.3.3 on Linux/amd64.

Run the following read-only SQL; it creates no database objects:

SELECT @@version AS server_version;

SELECT

  COLUMN_CHECK(UNHEX('000100010003801278'))

    AS valid_string,

  COLUMN_CHECK(UNHEX('000100010003809280808080808080800078'))

    AS eleven_groups,

  COLUMN_CHECK(UNHEX('0001000100038092808080808080800278'))

    AS u64_string_check,

  COLUMN_CHECK(UNHEX('0001000100038092808080808080800178'))

    AS charset_id_check,

  COLUMN_CHECK(UNHEX('00010001000489808080808080808002018000000102'))

    AS u64_decimal_check,

  COLUMN_CHECK(UNHEX('00010001000489808080808080808001018000000102'))

    AS decimal_domain_check;

SELECT

  COLUMN_GET(UNHEX('0001000100038092808080808080800278'), 1 AS CHAR)

    AS u64_string_value,

  COLUMN_GET(UNHEX('0001000100038092808080808080800178'), 1 AS CHAR)

    AS charset_id_value,

  COLUMN_GET(UNHEX('00010001000489808080808080808002018000000102'),

             1 AS DECIMAL(10,1))

    AS u64_decimal_value,

  COLUMN_GET(UNHEX('00010001000489808080808080808001018000000102'),

             1 AS DECIMAL(10,1))

    AS decimal_domain_value;

MariaDB 12.3.3 returns:

12.3.3-MariaDB-ubu2404

1	0	1	1	1	1

x	x	1.2	1.2

The canonical record is accepted and the eleven-group control is rejected, showing that the ten-group cap is active. All four malformed ten-group records are nevertheless reported as valid and decoded as x, x, 1.2, and 1.2.

Each malformed ten-group record should make COLUMN_CHECK return 0, or COLUMN_GET should raise ER_DYNCOL_FORMAT. A tenth payload group of 0x02 represents a value wider than ulonglong and must not be reduced modulo 2^64. A tenth payload group of 0x01 is representable as ulonglong, but the values exceed the uint charset-ID domain or the int decimal precision/scale domain and must not be narrowed to their low bits.

This occurs when COLUMN_CHECK or COLUMN_GET processes a malformed dynamic-column blob whose ten-group varint exceeds the serialized or consumer integer domain.