A JSON function allows two characters of room for each character of a value it writes as a string, but escaping one costs six, so a value escaped throughout is cut into invalid JSON or rejected outright

A JSON function has to name a length for its result before it has seen a
value, so it asks for one out of what its arguments say about themselves.

A value that is not already a document is written into the result as a JSON
string, and the room asked for it is twice its characters:

    length= static_cast<ulonglong>(arg->max_char_length()) * 2 + 2;

Twice does not cover the writing. A character the document's character set
cannot carry is not written as itself; it is written as the hex of its UTF-16
form, a backslash and a u and four figures, which is six characters. A
character outside the first plane is two UTF-16 units and each unit takes an
escape of its own, so that one costs twelve.

So a value whose characters all have to be escaped comes to six times what
was asked for, or twelve, and the result does not fit the length its own
function named.

Two character sets settle the cost, not one

A character is escaped only where the set the result is being written in
cannot carry it, and the pair of escapes is only ever reached where the
value's own set reaches past the first plane. Neither set decides this
alone, and both are known before any value is: the set being written in is
the collation the function has already aggregated or taken from its first
argument, and the value's is what that aggregation left the argument in.

A character set carries characters outside the first plane exactly when it is
four bytes wide at its widest. MY_CS_UNICODE_SUPPLEMENT does not answer
this question - it is set on utf8mb4 and not on utf16 or utf32, which carry
those characters just as well.

How to repeat

A VARCHAR(20) column holding twenty copies of CHAR(1), each of which is
written out as a six-character escape.

Written into a table of its own, the store rejects the statement:

CREATE TABLE t1 (v VARCHAR(10), s VARCHAR(20));

INSERT INTO t1 VALUES ('{"x":1}', REPEAT(CHAR(1),20)),

                      ('{"x":2}', REPEAT(CHAR(2),20));

CREATE TABLE d1 AS SELECT JSON_SET(v, '$.p', s) AS r FROM t1;

ERROR 22001: Data too long for column 'r' at row 1

Written into an internal temporary table, nothing is said at all. That store
leaves count_cuted_fields at CHECK_FIELD_IGNORE, so the store that
cuts the value answers 0 and warns nobody, and the width of that column was
chosen by nothing but the function's own declaration:

SET @@optimizer_switch='derived_merge=off';

SELECT LENGTH(JSON_SET(v, '$.p', s)) AS produced FROM t1;

SELECT LENGTH(r) AS kept, JSON_VALID(r) AS valid

  FROM (SELECT JSON_SET(v, '$.p', s) AS r FROM t1) AS d;

produced

137

kept    valid

75      0

Warnings:

Note  4037  Unexpected end of JSON text in argument 1 to function 'json_valid'

137 characters were produced and 75 were kept. What is left of the document
is not a document, and the only diagnostic is JSON_VALID complaining about the
already damaged value, and only because it was asked.

The functions that build a document rather than edit one understate it the
same way:

SELECT LENGTH(JSON_ARRAY(s)) AS produced FROM t1;

SELECT LENGTH(r) AS kept, JSON_VALID(r) AS valid

  FROM (SELECT JSON_ARRAY(s) AS r FROM t1) AS d;

produced

124

kept    valid

46      0

Which functions

Seven SQL functions ask for a value's room this way: JSON_ARRAY and
JSON_OBJECT, JSON_ARRAY_APPEND and JSON_ARRAY_INSERT, and JSON_SET,
JSON_INSERT and JSON_REPLACE.

JSON_QUOTE is not one of them. It writes into utf8mb4, which carries every
character there is, so it never escapes a character for want of somewhere to
put it and the handful JSON refuses literally are all inside the first plane.

Note on fixing it

Pricing this properly widens what these functions declare, and a declared
width decides whether a result is given a blob to live in
(CONVERT_IF_BIGGER_TO_BLOB, 512 characters). A blob cannot live in a
HEAP table, so an internal temporary table holding one is created on disk
instead of in memory. An ordinary column width is enough to cross that line
once an escaping is priced honestly, so the two questions are best settled
together.