In MySQL 5.7, this commit of WL#5757 introduced a 16383-byte in-page size limit for all InnoDB records. In our code, I would rename the constant REC_MAX_DATA_SIZE to something else, because it is confusingly defined as one byte too much (16384). Also, there is no need to limit the maximum record size with other than ROW_FORMAT=REDUNDANT. With other row formats, we only have to enforce a maximum length of 16383 bytes for variable-length columns when innodb_page_size=64k. That could be done by ensuring that dtuple_convert_big_rec() gets called and does move such columns off-page.

Contrary to what the MDEV-6075 commit message says, we are not limiting the ROW_FORMAT=REDUNDANT record size to 16383 bytes as we should.
I think that we can simply introduce 2 limits: a maximum field size limit of 16383 bytes, and a maximum ROW_FORMAT=REDUNDANT record size limit of 16383 bytes. Both these limits can only kick in with innodb_page_size=64k, because with smaller sizes, the maximum record size limit of half the innodb_page_size (minus header and trailer) would be less than 16383 bytes. The field-size limit does not need depend on ROW_FORMAT, even though that limit would never kick in on ROW_FORMAT=REDUNDANT.

Anything that would exceed these limist must be stored externally and converted by dtuple_convert_big_rec(). If the too long record or field is not for a clustered index leaf page, we must reject the record as too long.

bb-10.1-MDEV-11828

Sorry for the delay.
I would like some improvements to the tests. Also some minor code changes are needed.

Please check MDEV-12042 for a suggestion how to let buildbot run the tests with different innodb_page_size. I did not have time to study that yet.

commit 9284e8b2c64aba1ac108512ecc415a2ea563c387
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Wed Jul 12 19:34:55 2017 +0300

MDEV-11828: innodb_page_size=64k must reject ROW_FORMAT=REDUNDANT records longer than 16383 bytes

In all InnoDB row formats, the pointers or lengths stored in the record
header can be at most 14 bits, that is, count up to 16383.
In ROW_FORMAT=REDUNDANT, this limits the maximum possible record length
to 16383 bytes. In other ROW_FORMAT, it could merely limit the maximum
length of variable-length fields.

When MySQL 5.7 introduced innodb_page_size=32k and 64k, the maximum
record length was limited to 16383 bytes (I hope 16383, not 16384,
to be able to distinguish from a record whose length is 0 bytes).
This change is present in MariaDB Server 10.2.

btr_cur_optimistic_update(): Restrict maximum record size to 16K-1
for REDUNDANT and 64K page size.

dict_index_too_big_for_tree(): The maximum allowed record size
is half a B-tree page or 16K(-1 for REDUNDANT) for 64K page size.

convert_error_code_to_mysql(): Fix error message to print
correct limits.

my_error_innodb(): Fix error message to print correct limits.

page_zip_rec_needs_ext() : record size was already restricted to 16K.
Restrict REDUNDANT to 16K-1.

rem0rec.h: Introduce REDUNDANT_REC_MAX_DATA_SIZE (16K-1)
and COMPRESSED_REC_MAX_DATA_SIZE (16K).