Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7650

COLUMN_CHECK() does not always detect data truncation

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Not a Bug
    • 10.0.16
    • N/A
    • Dynamic Columns
    • Server version: 10.0.16-MariaDB Homebrew
      Darwin fsb 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64

    Description

      Bug is fully described in the following test script:

      select @@max_allowed_packet;
      # 1073741824
       
      select column_check(column_create('a', 1, 'b', repeat('1234567890', 6554)));
      # 1
       
      set @txt = (select repeat('1234567890', 6554));
      select column_check(column_create('a', 1, 'txt', @txt));
      # 1
       
      select column_check(column_create('a', 1, 'css', @txt));
      # 1
       
      create database if not exists dyncol_test;
      use dyncol_test;
      drop table if exists t;
      create table t (a int, b varchar(255), dcols blob);
       
      insert into t (a, b, dcols) values (1, 'two', column_create('txt', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'txt' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # 1   1234567890123456789012345678901234567890    {"txt":"12345678901234567890123456789012
       
      delete from t;
      insert into t (a, b, dcols) values (1, 'two', column_create('one', 1, 'txt', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'txt' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # 1   1234567890123456789012345678901234567890    {"one":1,"txt":"123456789012345678901234
       
      delete from t;
      insert into t (a, b, dcols) values (1, 'two', column_create('css', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'css' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # 1   1234567890123456789012345678901234567890    {"css":"12345678901234567890123456789012
       
      delete from t;
      insert into t (a, b, dcols) values (1, 'two', column_create('one', 1, 'css', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'css' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # ERROR 1919 (HY000) at line 31: Encountered illegal format of dynamic column string
       
      # Looks like the conditions for the error are:
      # 1. dynamic column blob size greater than 64kB
      # 2. dynamic column blob is saved to a table
      # 3. dynamic column blob contains more than one dynamic columns
      # 4. the dynamic column with the long value is named 'css' and not 'txt' !!!

      Attachments

        Activity

          thefsb Tom created issue -

          Thanks for the report and the test case.

          elenst Elena Stepanova added a comment - Thanks for the report and the test case.
          elenst Elena Stepanova made changes -
          Field Original Value New Value
          Fix Version/s 10.1 [ 16100 ]
          Fix Version/s 10.0 [ 16000 ]
          Assignee Oleksandr Byelkin [ sanja ]
          Labels verified
          thefsb Tom added a comment - - edited

          Dynamic columns are packed into the blob in alphabetical order.

          create database if not exists dyncol_test;
          use dyncol_test;
          drop table if exists t;
          create table t (id int, dcols blob);
           
          set @txt = repeat('1234567890', 6551);
          insert into t (id, dcols) values 
              (1, column_create('aaa', 1, 'aab', @txt)),
              (2, column_create('aac', 1, 'aab', @txt)),
              (3, column_create('aaa', 1, 'aab', concat(@txt, '1234567890'))),
              (4, column_create('aac', 1, 'aab', concat(@txt, '1234567890')));
           
          select id, column_check(dcols) from t;
          >>>
          id	column_check(dcols)
          1	1
          2	1
          3	1
          4	0

          thefsb Tom added a comment - - edited Dynamic columns are packed into the blob in alphabetical order. create database if not exists dyncol_test; use dyncol_test; drop table if exists t; create table t (id int, dcols blob);   set @txt = repeat('1234567890', 6551); insert into t (id, dcols) values (1, column_create('aaa', 1, 'aab', @txt)), (2, column_create('aac', 1, 'aab', @txt)), (3, column_create('aaa', 1, 'aab', concat(@txt, '1234567890'))), (4, column_create('aac', 1, 'aab', concat(@txt, '1234567890')));   select id, column_check(dcols) from t; >>> id column_check(dcols) 1 1 2 1 3 1 4 0
          thefsb Tom made changes -
          Summary COLUMN_CREATE() produces illegal format of dynamic column string when dynamic column name is 'css' COLUMN_CREATE() produces illegal format of dynamic column string when dynamic column values exceed 64kB

          Right. Long ago there was a discussion about consequences of data truncation in dynamic columns (which happens here, all INSERTs produce the warning); but I can't find or remember what the outcome of that discussion was, so I leave it to Sanja to comment, he will remember for sure.

          elenst Elena Stepanova added a comment - Right. Long ago there was a discussion about consequences of data truncation in dynamic columns (which happens here, all INSERTs produce the warning); but I can't find or remember what the outcome of that discussion was, so I leave it to Sanja to comment, he will remember for sure.

          A clearer test case:

          MariaDB [test]> select column_check(left(column_create('one', 1, 'txt', @txt), 65535));
          +-----------------------------------------------------------------+
          | column_check(left(column_create('one', 1, 'txt', @txt), 65535)) |
          +-----------------------------------------------------------------+
          |                                                               1 |
          +-----------------------------------------------------------------+
          1 row in set (0.00 sec)
           
          MariaDB [test]> select column_check(left(column_create('one', 1, 'css', @txt), 65535));
          +-----------------------------------------------------------------+
          | column_check(left(column_create('one', 1, 'css', @txt), 65535)) |
          +-----------------------------------------------------------------+
          |                                                               0 |
          +-----------------------------------------------------------------+
          1 row in set (0.00 sec)

          serg Sergei Golubchik added a comment - A clearer test case: MariaDB [test]> select column_check(left(column_create('one', 1, 'txt', @txt), 65535)); +-----------------------------------------------------------------+ | column_check(left(column_create('one', 1, 'txt', @txt), 65535)) | +-----------------------------------------------------------------+ | 1 | +-----------------------------------------------------------------+ 1 row in set (0.00 sec)   MariaDB [test]> select column_check(left(column_create('one', 1, 'css', @txt), 65535)); +-----------------------------------------------------------------+ | column_check(left(column_create('one', 1, 'css', @txt), 65535)) | +-----------------------------------------------------------------+ | 0 | +-----------------------------------------------------------------+ 1 row in set (0.00 sec)
          thefsb Tom added a comment -

          I did not know that blobs were restricted to such a small size.

          This was therefore my foolish error. Caught out again by the crazy selection of MySQL types.

          I do not know how to delete or close this issue.

          thefsb Tom added a comment - I did not know that blobs were restricted to such a small size. This was therefore my foolish error. Caught out again by the crazy selection of MySQL types. I do not know how to delete or close this issue.

          While we can close it, I'd suggest to keep it as a COLUMN_CHECK bug. I'll change the subject to better reflect what this bug is about.

          serg Sergei Golubchik added a comment - While we can close it, I'd suggest to keep it as a COLUMN_CHECK bug. I'll change the subject to better reflect what this bug is about.
          serg Sergei Golubchik made changes -
          Summary COLUMN_CREATE() produces illegal format of dynamic column string when dynamic column values exceed 64kB COLUMN_CHECK() does not always detect data truncation
          ratzpo Rasmus Johansson (Inactive) made changes -
          Workflow MariaDB v2 [ 59850 ] MariaDB v3 [ 64055 ]

          It is not a bug. Dynamic columns goal was to keep everything compact but not check integrity.

          Length of the string is not stored, it detected by beginning of other dynamic field or by end of whole string where dynamic fields stored. So if last field happened to be string you can chop from string as much as string length without destroying internal data integrity.

          If one need to check integrity other means should be used (like SHA1 or control sum or even signature).

          sanja Oleksandr Byelkin added a comment - It is not a bug. Dynamic columns goal was to keep everything compact but not check integrity. Length of the string is not stored, it detected by beginning of other dynamic field or by end of whole string where dynamic fields stored. So if last field happened to be string you can chop from string as much as string length without destroying internal data integrity. If one need to check integrity other means should be used (like SHA1 or control sum or even signature).
          sanja Oleksandr Byelkin made changes -
          Fix Version/s N/A [ 14700 ]
          Fix Version/s 10.0 [ 16000 ]
          Fix Version/s 10.1 [ 16100 ]
          Resolution Not a Bug [ 6 ]
          Status Open [ 1 ] Closed [ 6 ]
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 64055 ] MariaDB v4 [ 148845 ]

          People

            sanja Oleksandr Byelkin
            thefsb Tom
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.