Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40696

JSON_ARRAYAGG returns up to two bytes more than group_concat_max_len, and returns them without a warning when the elements end exactly on the limit

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Minor
    • Resolution: Unresolved
    • 10.6, 10.11, 11.4, 11.8, 12.3, 13.0
    • None
    • JSON
    • None

    Description

      JSON_ARRAYAGG is documented in the same words as GROUP_CONCAT, which it
      derives from: "The maximum returned length in bytes is determined by the
      group_concat_max_len server system variable." GROUP_CONCAT ends on that
      limit exactly. JSON_ARRAYAGG ends two bytes past it.

      CREATE TABLE t1 (s VARCHAR(20));
      INSERT INTO t1 VALUES ('aaaaaaaaaa'), ('bbbbbbbbbb');
       
      SET SESSION group_concat_max_len = 25;
      SELECT JSON_ARRAYAGG(s) AS v, LENGTH(JSON_ARRAYAGG(s)) AS len FROM t1;
      v                             len
      ["aaaaaaaaaa","bbbbbbbbbb"]   27
      

      The limit is 25 and 27 bytes come back. No warning is raised: the
      elements ended exactly on the limit, so nothing was cut, and the two
      brackets went on afterwards.

      Where a cut does happen the answer is still over the limit, and there
      the warning is raised - for the overshoot that is left, not for the one
      the warning is about:

      SET SESSION group_concat_max_len = 12;
      SELECT JSON_ARRAYAGG(s) AS v, LENGTH(JSON_ARRAYAGG(s)) AS len FROM t1;
      v                len
      ["aaaaaaaaaa"]   14
      Warnings:
      Warning  1260  Row 2 was cut by JSON_ARRAYAGG()
      

      GROUP_CONCAT over the same rows and the same limit ends on 12:

      SET SESSION group_concat_max_len = 12;
      SELECT GROUP_CONCAT(s) AS v, LENGTH(GROUP_CONCAT(s)) AS len FROM t1;
      v              len
      aaaaaaaaaa,b   12
      Warnings:
      Warning  1260  Row 2 was cut by GROUP_CONCAT()
      

      How to repeat

      CREATE TABLE t1 (s VARCHAR(20));
      INSERT INTO t1 VALUES ('aaaaaaaaaa'), ('bbbbbbbbbb');
       
      SET SESSION group_concat_max_len = 25;
      SELECT LENGTH(JSON_ARRAYAGG(s)) AS len, @@group_concat_max_len AS cap
      FROM t1;
      len  cap
      27   25
       
      SET SESSION group_concat_max_len = 12;
      SELECT LENGTH(JSON_ARRAYAGG(s)) AS len, @@group_concat_max_len AS cap
      FROM t1;
      len  cap
      14   12
       
      DROP TABLE t1;
      

      In a character set that writes no character in one byte the overshoot
      is wider, the brackets being written in the set of the result:

      CREATE TABLE t2 (a VARCHAR(20)) CHARACTER SET ucs2;
      INSERT INTO t2 VALUES ('aaaa'), ('bbbb');
       
      SET SESSION group_concat_max_len = 26;
      SELECT LENGTH(JSON_ARRAYAGG(a ORDER BY a)) AS len,
             @@group_concat_max_len AS cap FROM t2;
      len  cap
      30   26
       
      DROP TABLE t2;
      

      Four bytes over, and again with no warning.

      Root cause

      Item_func_group_concat::dump_leaf_key() tests the accumulated result
      once per row, in sql/item_sum.cc:

        uint max_length= table->in_use->gconcat_max_len();
        ...
        /* stop if length of result more than max_length */
        if (result->length() > max_length)
        {
          ...
          item->cut_max_length(result, old_length, max_length);
      

      For GROUP_CONCAT what has been accumulated is the whole answer, so that
      test is exact. JSON_ARRAYAGG's brackets are not in that buffer: they
      are written afterwards, in Item_func_json_arrayagg::val_str(), when the
      group is asked for. So the row test is made against a body that is not
      yet the answer, and the answer is longer than the body it was tested by

      • by one bracket at each end, each as wide as the narrowest character of
        the result's character set.

      Where the body ends exactly on the limit the test never fires, so there
      is no cut and no warning, and the answer is over the limit in silence.

      Relationship to MDEV-39817

      MDEV-39817 (commit 59141cbd2c4, 2026-06-01) fixed a different fault in
      the same cut and added a test whose recorded answer is over the limit:

      set session group_concat_max_len = 9;
      select json_arrayagg(c order by c) from t;
      json_arrayagg(c order by c)
      ["ab","cd"]
      

      Eleven bytes against a limit of nine. That commit also widened the
      width the item declares, in Item_func_json_arrayagg::fix_fields():

        /* account for opening and closing brackets */
        max_length= MY_MIN(max_length + 2*collation.collation->mbminlen,
                           UINT_MAX32);
      

      which is visible to a caller:

      SET SESSION group_concat_max_len = 9;
      CREATE TABLE x AS SELECT JSON_ARRAYAGG(c) AS v FROM t;
      SHOW CREATE TABLE x;
        `v` varchar(11)
       
      CREATE TABLE y AS SELECT GROUP_CONCAT(c) AS v FROM t;
      SHOW CREATE TABLE y;
        `v` varchar(9)
      

      So the declared width already allows for the answer running past the
      limit. Whether that was meant to settle what the limit covers, or only
      to keep a materialised column from truncating what the function was
      already returning, the commit does not say; its message is about the
      off-by-one it fixes and says nothing about the limit.

      One case where the limit cannot be kept

      group_concat_max_len may be set as low as 4, and an empty array is as
      short as this function goes. In a character set of four bytes to the
      character that is eight bytes:

      CREATE TABLE t3 (a VARCHAR(20)) CHARACTER SET utf32;
      INSERT INTO t3 VALUES ('aa'), ('bb');
      SET SESSION group_concat_max_len = 4;
      SELECT LENGTH(JSON_ARRAYAGG(a)) AS len FROM t3;
      len
      8
      DROP TABLE t3;
      

      There is nothing left to cut, so any fix has this floor: what comes back
      is the limit or the two brackets, whichever is longer. The declared
      width quoted above already covers it.

      Affected versions

      JSON_ARRAYAGG was added in 10.5.0. The limit began to be applied to it
      in 10.5.4 (MDEV-22844), and the row test has been made against the
      bracket-free body ever since.

      Verified on 10.11 only (10.11.19-MariaDB-debug, commit 1dab253482d,
      which contains MDEV-39817). Other branches were not tested.

      Note

      Found while auditing the JSON functions against their documented
      contracts. No test in the suite compares LENGTH() of either JSON
      aggregate against @@group_concat_max_len, which is why it has not shown
      up: func_json_agg_limits records the bytes at many caps but never
      asserts them against the cap.

      JSON_OBJECTAGG has the same shape one byte wide, its closing brace being
      written in val_str() as well.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              arcivanov Arcadiy Ivanov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.