A character outside the first Unicode plane is written as one escape carrying both UTF-16 units, so the document a JSON function produces is not JSON and the server rejects its own output

A value going into a JSON document is escaped where the document's
character set cannot carry the character as it stands. JSON spells such
a character as the hex of its UTF-16 form, and a character outside the
first plane is two UTF-16 units, so it is written as two escapes, one
unit apiece.

MariaDB writes one escape and puts the figures of both units after it.
The second unit's four figures are then not an escape - they are four
more characters of the string - and a reader takes the one escape, finds
half a character, and stops.

What comes out is a document the server itself will not read back.

How to repeat

SELECT VERSION();

-- 10.11.19-MariaDB

SELECT JSON_SET(_latin1'{}', '$.b', _utf8mb4 0xF09F9880) AS v;

-- NULL

SHOW WARNINGS;

-- Warning  4038  Syntax error in JSON text in argument 1 to function

--                'json_set' at position 12

Argument 1 is an empty object, two characters long, and it is a valid
document. It has no position 12. The offset is an offset into what the
function itself wrote.

All five functions that write a value into a document they were handed
answer the same way, and it makes no difference which character set the
document is written in as long as it cannot carry the character:

SELECT JSON_INSERT(_latin1'{}', '$.b', _utf8mb4 0xF09F9880) AS ins,

       JSON_REPLACE(_latin1'{"b":1}', '$.b', _utf8mb4 0xF09F9880) AS rep;

-- NULL  NULL

SELECT JSON_ARRAY_APPEND(_latin1'[1]', '$', _utf8mb4 0xF09F9880) AS app,

       JSON_ARRAY_INSERT(_latin1'[1]', '$[0]', _utf8mb4 0xF09F9880) AS ins;

-- NULL  NULL

SELECT JSON_SET(_utf8mb3'{}', '$.b', _utf8mb4 0xF09F9880) AS v;

-- NULL

What is actually written

Composing an array reaches the same writing without reading the result
back afterwards, so the bytes can be seen:

SELECT HEX(JSON_ARRAY(_utf8mb4 0xF09F9880, _binary'x')) AS bytes;

-- 5B225C754438334444453030222C202278225D

5B  22  5C 75 44 38 33 44  44 45 30 30  22  2C 20  22 78 22  5D

[   "   \  u  D  8  3  D   D  E  0  0   "   ,      "  x  "   ]

The first four figures are the first UTF-16 unit and are correct. The
four after them are the second unit, standing in the string as text.
Each of the two units of this character should be carrying an escape of
its own.

The server does not accept the result:

SELECT JSON_VALID(JSON_ARRAY(_utf8mb4 0xF09F9880, _binary'x')) AS valid;

-- 0

A character inside the first plane is one UTF-16 unit and takes one
escape, which comes out right - which is why this has stayed quiet:

SELECT HEX(JSON_SET(_latin1'{}', '$.b', _utf8mb4 0xE4BDA0)) AS bmp;

-- 7B2262223A20225C7534463630227D

7B  22 62 22  3A 20  22  5C 75 34 46 36 30  22  7D

{   "  b  "   :      "   \  u  4  F  6  0   "   }

Root cause

json_escape() in strings/json_lib.c writes the backslash and the
u once, before it knows how many units the character takes. It then
converts the character with my_uni_utf16(), which answers two bytes
for a character of the first plane and four for any other, fills a
buffer with the hex of however many bytes came back, and appends the
whole buffer:

        uchar utf16buf[4];

        uchar code_str[8];

        int u_len= my_uni_utf16(0, c_chr, utf16buf, utf16buf + 4);

        code_str[0]= hexconv[utf16buf[0] >> 4];

        code_str[1]= hexconv[utf16buf[0] & 15];

        code_str[2]= hexconv[utf16buf[1] >> 4];

        code_str[3]= hexconv[utf16buf[1] & 15];

        if (u_len > 2)

        {

          code_str[4]= hexconv[utf16buf[2] >> 4];

          code_str[5]= hexconv[utf16buf[2] & 15];

          code_str[6]= hexconv[utf16buf[3] >> 4];

          code_str[7]= hexconv[utf16buf[3] & 15];

        }

        if ((c_len= json_append_ascii(json_cs, json, json_end,

                                      code_str, code_str+u_len*2)) > 0)

The count of figures follows the number of units. Nothing that
introduces the second unit follows it.

The code has read this way since it was written, in 27025221fe2
"MDEV-9143 JSON_xxx functions.", first released in 10.2.3.

The buffer the writing goes into is asked for at twelve characters
apiece by st_append_escaped() in sql/item_jsonfunc.cc, whose
comment names the two-escape form, so the room for the correct output
has been reserved all along:

  /*

    In the worst case one character from the 'a' string

    turns into '\uXXXX\uXXXX' which is 12.

  */

  int str_len= a->length() * 12 * s->charset()->mbmaxlen /

               a->charset()->mbminlen;

Which arguments reach it

Only a document whose character set cannot carry the character is
escaped this way, so the functions that meet it are the ones that take
their character set from the document they were handed and aggregate
nothing with the values going in: JSON_SET, JSON_INSERT,
JSON_REPLACE, JSON_ARRAY_APPEND and JSON_ARRAY_INSERT.

JSON_ARRAY and JSON_OBJECT aggregate the character sets of their
arguments, which normally leaves the result able to carry everything
that goes into it. A binary argument is the exception: it wins the
aggregation by precedence rather than by covering the others, and the
arguments it beats are not converted to it, so a wide value keeps its
own character set while the document around it is written as bytes.

JSON_QUOTE is not affected. It writes into utf8mb4, which carries
every character there is, so it never escapes a character for want of
somewhere to put it.

Note

This is the writing side. Reading the two-escape form back works: on
10.11.19 both JSON_UNQUOTE and JSON_VALUE return the character
correctly out of a document that carries the pair, and the two agree
with each other.