Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7650

COLUMN_CHECK() does not always detect data truncation

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Not a Bug
    • 10.0.16
    • N/A
    • Dynamic Columns
    • Server version: 10.0.16-MariaDB Homebrew
      Darwin fsb 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64

    Description

      Bug is fully described in the following test script:

      select @@max_allowed_packet;
      # 1073741824
       
      select column_check(column_create('a', 1, 'b', repeat('1234567890', 6554)));
      # 1
       
      set @txt = (select repeat('1234567890', 6554));
      select column_check(column_create('a', 1, 'txt', @txt));
      # 1
       
      select column_check(column_create('a', 1, 'css', @txt));
      # 1
       
      create database if not exists dyncol_test;
      use dyncol_test;
      drop table if exists t;
      create table t (a int, b varchar(255), dcols blob);
       
      insert into t (a, b, dcols) values (1, 'two', column_create('txt', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'txt' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # 1   1234567890123456789012345678901234567890    {"txt":"12345678901234567890123456789012
       
      delete from t;
      insert into t (a, b, dcols) values (1, 'two', column_create('one', 1, 'txt', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'txt' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # 1   1234567890123456789012345678901234567890    {"one":1,"txt":"123456789012345678901234
       
      delete from t;
      insert into t (a, b, dcols) values (1, 'two', column_create('css', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'css' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # 1   1234567890123456789012345678901234567890    {"css":"12345678901234567890123456789012
       
      delete from t;
      insert into t (a, b, dcols) values (1, 'two', column_create('one', 1, 'css', @txt));
      select column_check(dcols),
          substring(column_get(dcols, 'css' as char), 1, 40),
          substring(column_json(dcols), 1, 40) from t;
      # ERROR 1919 (HY000) at line 31: Encountered illegal format of dynamic column string
       
      # Looks like the conditions for the error are:
      # 1. dynamic column blob size greater than 64kB
      # 2. dynamic column blob is saved to a table
      # 3. dynamic column blob contains more than one dynamic columns
      # 4. the dynamic column with the long value is named 'css' and not 'txt' !!!

      Attachments

        Activity

          Thanks for the report and the test case.

          elenst Elena Stepanova added a comment - Thanks for the report and the test case.
          thefsb Tom added a comment - - edited

          Dynamic columns are packed into the blob in alphabetical order.

          create database if not exists dyncol_test;
          use dyncol_test;
          drop table if exists t;
          create table t (id int, dcols blob);
           
          set @txt = repeat('1234567890', 6551);
          insert into t (id, dcols) values 
              (1, column_create('aaa', 1, 'aab', @txt)),
              (2, column_create('aac', 1, 'aab', @txt)),
              (3, column_create('aaa', 1, 'aab', concat(@txt, '1234567890'))),
              (4, column_create('aac', 1, 'aab', concat(@txt, '1234567890')));
           
          select id, column_check(dcols) from t;
          >>>
          id	column_check(dcols)
          1	1
          2	1
          3	1
          4	0

          thefsb Tom added a comment - - edited Dynamic columns are packed into the blob in alphabetical order. create database if not exists dyncol_test; use dyncol_test; drop table if exists t; create table t (id int, dcols blob);   set @txt = repeat('1234567890', 6551); insert into t (id, dcols) values (1, column_create('aaa', 1, 'aab', @txt)), (2, column_create('aac', 1, 'aab', @txt)), (3, column_create('aaa', 1, 'aab', concat(@txt, '1234567890'))), (4, column_create('aac', 1, 'aab', concat(@txt, '1234567890')));   select id, column_check(dcols) from t; >>> id column_check(dcols) 1 1 2 1 3 1 4 0

          Right. Long ago there was a discussion about consequences of data truncation in dynamic columns (which happens here, all INSERTs produce the warning); but I can't find or remember what the outcome of that discussion was, so I leave it to Sanja to comment, he will remember for sure.

          elenst Elena Stepanova added a comment - Right. Long ago there was a discussion about consequences of data truncation in dynamic columns (which happens here, all INSERTs produce the warning); but I can't find or remember what the outcome of that discussion was, so I leave it to Sanja to comment, he will remember for sure.

          A clearer test case:

          MariaDB [test]> select column_check(left(column_create('one', 1, 'txt', @txt), 65535));
          +-----------------------------------------------------------------+
          | column_check(left(column_create('one', 1, 'txt', @txt), 65535)) |
          +-----------------------------------------------------------------+
          |                                                               1 |
          +-----------------------------------------------------------------+
          1 row in set (0.00 sec)
           
          MariaDB [test]> select column_check(left(column_create('one', 1, 'css', @txt), 65535));
          +-----------------------------------------------------------------+
          | column_check(left(column_create('one', 1, 'css', @txt), 65535)) |
          +-----------------------------------------------------------------+
          |                                                               0 |
          +-----------------------------------------------------------------+
          1 row in set (0.00 sec)

          serg Sergei Golubchik added a comment - A clearer test case: MariaDB [test]> select column_check(left(column_create('one', 1, 'txt', @txt), 65535)); +-----------------------------------------------------------------+ | column_check(left(column_create('one', 1, 'txt', @txt), 65535)) | +-----------------------------------------------------------------+ | 1 | +-----------------------------------------------------------------+ 1 row in set (0.00 sec)   MariaDB [test]> select column_check(left(column_create('one', 1, 'css', @txt), 65535)); +-----------------------------------------------------------------+ | column_check(left(column_create('one', 1, 'css', @txt), 65535)) | +-----------------------------------------------------------------+ | 0 | +-----------------------------------------------------------------+ 1 row in set (0.00 sec)
          thefsb Tom added a comment -

          I did not know that blobs were restricted to such a small size.

          This was therefore my foolish error. Caught out again by the crazy selection of MySQL types.

          I do not know how to delete or close this issue.

          thefsb Tom added a comment - I did not know that blobs were restricted to such a small size. This was therefore my foolish error. Caught out again by the crazy selection of MySQL types. I do not know how to delete or close this issue.

          While we can close it, I'd suggest to keep it as a COLUMN_CHECK bug. I'll change the subject to better reflect what this bug is about.

          serg Sergei Golubchik added a comment - While we can close it, I'd suggest to keep it as a COLUMN_CHECK bug. I'll change the subject to better reflect what this bug is about.

          It is not a bug. Dynamic columns goal was to keep everything compact but not check integrity.

          Length of the string is not stored, it detected by beginning of other dynamic field or by end of whole string where dynamic fields stored. So if last field happened to be string you can chop from string as much as string length without destroying internal data integrity.

          If one need to check integrity other means should be used (like SHA1 or control sum or even signature).

          sanja Oleksandr Byelkin added a comment - It is not a bug. Dynamic columns goal was to keep everything compact but not check integrity. Length of the string is not stored, it detected by beginning of other dynamic field or by end of whole string where dynamic fields stored. So if last field happened to be string you can chop from string as much as string length without destroying internal data integrity. If one need to check integrity other means should be used (like SHA1 or control sum or even signature).

          People

            sanja Oleksandr Byelkin
            thefsb Tom
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.