The problematic SYS_TABLES clustered index record contents is as follows:

This looks valid to me. The fields MIX_ID, MIX_LEN and CLUSTER_NAME were originally unused and in very old InnoDB versions written as garbage, so for ROW_FORMAT=REDUNDANT we must ignore these fields. The fields were originally intended to be used for mixed clustered indexes, where a clustered index could contain records from multiple tables sharing a similar primary key.

The MIX_LEN field was repurposed for additional table flags in the InnoDB Plugin for MySQL 5.1: Applying InnoDB snapshot, fixes BUG#41609. Until MySQL 5.6, the only flag was DICT_TF2_TEMPORARY, intended to suppress warnings about non-found files for temporary tables. (Until MySQL 5.7 and MariaDB 10.2, InnoDB treated temporary tables as persistent ones.) 5.6 introduced further persistently stored flags in SYS_TABLES.MIX_LEN mainly for passing transient information during table creation.

The failing debug assertion seems bogus:

		/* If the table is not a predefined tablespace then it must

		be in a file-per-table tablespace.

		Note that flags2 is not available for REDUNDANT tables,

		so don't check those. */

		ut_ad(!DICT_TF_GET_COMPACT(flags)

		      || flags2 & DICT_TF2_USE_FILE_PER_TABLE);

The flag DICT_TF2_USE_FILE_PER_TABLE was introduced in SYS_TABLES.MIX_LEN, originally called DICT_TF2_USE_TABLESPACE in MySQL 5.6.5. It does not make sense to store this transient information ("was innodb_file_per_table set at the start of CREATE TABLE?") in a persistent data structure, but that is what was done. Finally, MySQL 5.7.6 introduced this bogus assertion in a change that was approved by me. (My bad, I should have remembered that the flag did not exist before 5.6.)

I believe that Oracle does not test upgrades with debug binaries. The fix is simple: remove the bogus debug assertion. We must only ensure that if a table that resides in an .ibd file is rebuilt, it will remain in an .ibd file even if innodb_file_per_table=0:

SET GLOBAL innodb_file_per_table=0;

ALTER TABLE t001 FORCE;

That actually was the case, after I removed one more failing debug assertion in dict_table_is_file_per_table():

	/* If the table is file-per-table and it is not redundant, then

	it should have the flags2 bit for DICT_TF2_USE_FILE_PER_TABLE. */

	ut_ad(!is_file_per_table

	      || !DICT_TF_GET_COMPACT(table->flags)

	      || DICT_TF2_FLAG_IS_SET(table, DICT_TF2_USE_FILE_PER_TABLE));

After these removals, the remaining references to DICT_TF2_USE_TABLESPACE look reasonable. It would seem reasonable to stop setting the flag in MIX_LEN altogether, but that would cause trouble when downgrading to earlier versions. So, we should basically treat the flag as garbage.

This bug only affected debug builds. Release builds may have been affected before MDEV-13084 (MariaDB 10.2.7).