Pull off 64k size limit of Extra2 segment in FRM file

Summary

Remove the approximately 64 KiB size limit of the MariaDB-specific Extra2 segment in FRM files.

Introduce an extensible Extra3 segment that:

• Can exceed 64 KiB within the existing total FRM-size limit.
• Supports individual items larger than 64 KiB.
• Preserves the current FRM layout when Extra3 is not needed.
• Allows new servers to read existing FRM files.
• Makes older servers reject FRM files containing required Extra3 data instead of silently ignoring it.

Rationale

The Extra2 segment stores MariaDB-specific table metadata that is not represented by the original FRM format.

Its total size is limited to approximately 64 KiB because both the segment length and the position of the following key-information block are stored as 2-byte values in the FRM header:

1. fileinfo + 4 stores the length of the Extra2 segment.
2. fileinfo + 6 stores the position of the key-information block, also known as disk_buff.

The key-information position is calculated as:

disk_buff_offset =

  FRM_HEADER_SIZE + extra2_size + sizeof(uint32);

Because disk_buff_offset is represented by 2 bytes, the effective maximum Extra2 size is:

65535 - FRM_HEADER_SIZE - 4

= 65535 - 64 - 4

= 65467 bytes

This restricts the amount of MariaDB-specific metadata that can be stored in an FRM file. The limit can be reached when metadata grows with the number of columns, indexes, engine options, data-type attributes, periods, foreign keys, or future table features.

The problem cannot be solved merely by changing the existing header fields from 2 bytes to 4 bytes. Such a change would alter the established FRM layout and cause compatibility problems for:

• Older MariaDB versions.
• Storage engines that inspect FRM files directly.
• External tools that parse FRM metadata.
• Existing FRM files and downgrade procedures.

A separate segment is therefore required.

Current Extra2 format

Extra2 starts immediately after the 64-byte FRM header.

There is no independent header at the beginning of the segment. The first byte is the type of the first Extra2 item.

For current MariaDB FRM files, the first item is normally:

EXTRA2_TABLEDEF_VERSION

Older FRM files may contain '/' at this position instead.

Each Extra2 item has the following format:

+-------------+------------------+-----------------+

| Type: 1 byte| Encoded length   | Value           |

+-------------+------------------+-----------------+

The current length encoding is:

• Lengths from 1 through 255: one byte.
• Longer lengths: a zero byte followed by a 2-byte little-endian length.

Conceptually:

if (length <= 255)

{

  write_uint8(length);

}

else

{

  write_uint8(0);

  write_uint16(length);

}

Consequently, the current format has two independent limits:

• The complete Extra2 segment must fit before a key-information offset represented by 2 bytes.
• Each individual Extra2 item is limited to 65535 bytes.

Increasing only the total segment limit would not allow an individual metadata item to exceed 64 KiB.

Current FRM layout

The relevant parts of the current FRM layout are:

The extra chunk contains data such as:

• Connection string.
• Storage-engine name.
• Partition information.
• Parser names.
• Table comment.

The default-record position is calculated from the key-information position and key-information length:

record_offset =

  uint2korr(frm_image + 6) +

  (uint2korr(frm_image + 14) == 0xffff

    ? uint4korr(frm_image + 47)

    : uint2korr(frm_image + 14));

The forminfo block is already addressed through a 4-byte position stored immediately after Extra2.

This permits an additional segment to be inserted between the existing extra chunk and forminfo without changing the 2-byte disk_buff position.

Proposed format

Introduce a new Extra3 segment in the unused extensible area after the existing extra chunk and before forminfo.

The resulting layout is:

The stored forminfo position must be updated to point past Extra3.

No existing offset or field in the FRM header changes size or meaning.

Extra3 descriptor

Add a new Extra2 item:

EXTRA2_EXTRA3

Its payload contains:

The payload format is:

+------------------------+------------------------+

| Extra3 offset: 4 bytes | Extra3 length: 4 bytes |

+------------------------+------------------------+

Both values use the byte order used by other integer values in the FRM format.

EXTRA2_EXTRA3 must use a type code greater than or equal to:

EXTRA2_ENGINE_IMPORTANT

An older server that does not recognize this item must therefore reject the FRM file as containing unknown required metadata.

It must not open the table while silently ignoring Extra3.

Extra3 item encoding

Extra3 uses a type-length-value representation similar to Extra2:

+-------------+-----------------+-----------------+

| Type: 1 byte| Length: 4 bytes | Value           |

+-------------+-----------------+-----------------+

The item length is stored as an unsigned 4-byte value.

Conceptually:

write_uint8(type);

write_uint32(length);

write_bytes(value, length);

This removes the 64 KiB per-item limit.

The implementation must provide dedicated helpers equivalent to the existing Extra2 helpers, for example:

extra3_write_len()

extra3_write()

extra3_write_str()

extra3_read_len()

The helpers must:

• Detect arithmetic overflow.
• Reject zero or otherwise invalid lengths according to the selected format rules.
• Verify that the complete item is within the declared Extra3 boundary.
• Never read or write beyond the FRM image.
• Reject truncated item headers and values.

Item types

Extra3 should use the existing metadata type namespace where practical so that an item can be moved between Extra2 and Extra3 without defining two unrelated meanings.

Unknown-item handling must preserve the existing importance rule:

• An unknown optional item can be skipped using its declared length.
• An unknown important item causes FRM parsing to fail.
• A known item that may occur only once causes an error if duplicated.
• A metadata type must not occur simultaneously in Extra2 and Extra3 unless that type explicitly permits multiple occurrences.

EXTRA2_EXTRA3 is a descriptor only. It must not occur inside Extra3.

Writer behavior

The FRM writer must determine which metadata items are stored in Extra2 and which are stored in Extra3.

The following rules apply:

• Small, frequently accessed, and compatibility-critical items remain in Extra2.
• Items that can make Extra2 exceed its limit are moved to Extra3.
• Items larger than 65535 bytes must be stored in Extra3.
• Extra3 is created only when at least one item is stored there.
• EXTRA2_EXTRA3 is written only when Extra3 exists.
• fileinfo + 4 continues to contain only the Extra2 length.
• fileinfo + 6 continues to identify the existing key-information block.
• The 4-byte forminfo position after Extra2 is adjusted to account for Extra3.
• The complete FRM image must remain within FRM_MAX_SIZE.

When every item fits in Extra2, the writer must continue to produce the current layout without an empty Extra3 segment.

Item placement

The implementation must define a deterministic placement policy.

An item should be placed in Extra3 when:

• Its value is larger than the maximum representable Extra2 item length.
• Keeping it in Extra2 would make the Extra2 segment or disk_buff position exceed its representable limit.
• Its metadata category is explicitly designated for Extra3.

When several movable items exist, the writer should move sufficiently large items to make Extra2 fit while keeping required compatibility metadata in Extra2.

The same table definition and server version should produce the same item placement and FRM image.

Reader behavior

The primary FRM reader must:

1. Read and validate Extra2 using the existing segment boundaries.
2. Detect EXTRA2_EXTRA3.
3. Decode its 4-byte offset and 4-byte length.
4. Validate the declared Extra3 range.
5. Parse all Extra3 items.
6. Merge recognized metadata into the same internal table-definition structures used for Extra2.
7. Reject conflicting, duplicated, malformed, or unsupported metadata.
8. Continue parsing forminfo and the remainder of the FRM image using the stored forminfo position.

The table definition must have the same in-memory representation regardless of whether an item originated from Extra2 or Extra3.

Boundary validation

The reader must validate all offsets and lengths before dereferencing them.

At minimum, it must verify that:

• The EXTRA2_EXTRA3 payload has exactly the expected size.
• Offset and length addition does not overflow.
• The Extra3 offset is within the FRM image.
• The complete Extra3 range is within the FRM image.
• Extra3 does not overlap the header.
• Extra3 does not overlap Extra2.
• Extra3 does not overlap the key-information block.
• Extra3 does not overlap the default-value record.
• Extra3 does not overlap the existing extra chunk.
• Extra3 ends no later than forminfo.
• The location is consistent with the layout produced by the writer.
• Each item header and value fits within the declared Extra3 range.
• Important unknown items cause failure.
• Duplicate singleton items cause failure.

Malformed metadata must produce the normal corrupted-FRM or invalid-table-definition error. It must not cause an assertion failure, out-of-bounds access, excessive allocation, or server crash.

Secondary FRM readers

All code that reads Extra2 or derives metadata directly from an FRM image must be reviewed.

This includes at least:

• TABLE_SHARE::init_from_binary_frm_image().
• ha_partition::get_def_part_engine().
• S3 get_tabledef_version_from_frm().
• dd_frm_type() and related data-dictionary inspection.
• Upgrade, repair, discovery, and metadata-copy paths.
• Tools or server components that inspect table-definition versions or engines without fully opening the table.

If metadata required by a secondary reader can be stored in Extra3, that reader must support EXTRA2_EXTRA3 and safely parse the required Extra3 item.

Alternatively, metadata required by early or partial readers may be permanently retained in Extra2. This placement requirement must be documented in the code.

No reader may return incomplete or incorrect metadata merely because the item was moved to Extra3.

Error handling

FRM creation must fail with ER_TABLE_DEFINITION_TOO_BIG or the appropriate existing error when:

• Extra2 still exceeds its representable limit after eligible items are moved.
• An individual item is too large for both the Extra3 representation and the complete FRM image.
• The total FRM image exceeds FRM_MAX_SIZE.
• An offset or length cannot be represented safely.
• Memory allocation fails.

FRM loading must fail with a corrupted-definition error when:

• The EXTRA2_EXTRA3 descriptor has an invalid length.
• The offset or length is outside the FRM image.
• Offset arithmetic overflows.
• The segment overlaps another FRM area.
• An item header or value is truncated.
• An item length exceeds the segment boundary.
• Required metadata occurs more than once.
• The same singleton metadata occurs in both Extra2 and Extra3.
• An unknown important item is encountered.

Errors must identify the affected table and must not leave a partially initialized TABLE_SHARE.

Compatibility

The implementation must preserve forward and backward compatibility as follows:

• A new server can open existing FRM files containing only Extra2.
• A new server writes the current FRM layout when Extra3 is unnecessary.
• FRM files without Extra3 should remain byte-identical for the same table definition and server version.
• An older server can open a new FRM file when it does not contain Extra3 or other unsupported metadata.
• An older server must reject an FRM file containing EXTRA2_EXTRA3 because the descriptor is marked as important.
• An older server must not silently open a table while ignoring metadata stored in Extra3.
• Existing meanings of fileinfo + 4 and fileinfo + 6 remain unchanged.
• The current Extra2 item format remains unchanged.
• Existing FRM tools must continue to work with files that do not require Extra3.

Directly widening fileinfo + 4 or fileinfo + 6 is not part of this task.

Atomic DDL behavior

DDL that creates or rewrites an FRM file containing Extra3 must use the same atomic replacement rules as existing FRM-writing operations.

A failed write must not leave:

• A truncated FRM file.
• An FRM file whose EXTRA2_EXTRA3 descriptor points outside the file.
• A mixed image containing old offsets and new segment contents.
• An installed table definition that cannot be reopened.

Temporary-file creation, synchronization, rename, cleanup, and recovery must follow the existing DDL path used for the corresponding operation.

ALTER TABLE behavior

ALTER TABLE must be able to transition in both directions:

• From an Extra2-only FRM to an FRM containing Extra3.
• From an FRM containing Extra3 back to an Extra2-only FRM when the metadata becomes small enough.

The resulting placement must be based on the final table definition rather than on the previous FRM layout.

The following operations must preserve all metadata:

• Copy-based ALTER.
• In-place ALTER that rewrites the FRM.
• Storage-engine conversion.
• Adding or dropping columns.
• Adding or dropping indexes.
• Adding or removing periods.
• Changing table and engine options.
• Partitioning changes.

Table-copy operations

Operations that copy or reconstruct table definitions must preserve metadata stored in Extra3, including:

• CREATE TABLE ... LIKE.
• Table rename.
• Partition-table FRM creation.
• Storage-engine table discovery where applicable.
• Table-definition cache reload.
• Backup and restore.
• Upgrade and downgrade checks.

A copied table may choose a different Extra2/Extra3 placement if its final metadata differs, but no recognized item may be lost.

Partitioning

Partitioning code has a partial FRM reader for determining the default partition storage engine.

The implementation must ensure that:

• The default partition engine remains accessible to ha_partition::get_def_part_engine().
• Partition metadata stored in Extra3 is read correctly.
• Partitioned tables can be created, reopened, altered, renamed, and dropped.
• Corrupted Extra3 metadata is rejected before partition handlers use it.
• Partition operations do not lose Extra3-only metadata.

If the default partition-engine item remains permanently in Extra2, this requirement must be enforced by the writer.

S3

The S3 storage engine reads table-definition metadata directly from FRM files.

The implementation must ensure that:

• get_tabledef_version_from_frm() continues to read the table-definition version.
• Required S3 metadata remains available when other metadata moves to Extra3.
• S3 tables containing Extra3 can be opened and validated.
• Corrupted Extra3 descriptors and items are rejected.
• Upload, discovery, reopen, ALTER, and restore paths preserve Extra3 metadata.

If the table-definition version remains permanently in Extra2, this requirement must be enforced and documented.

Resource limits

Parsing Extra3 must remain bounded by:

• The declared segment length.
• The actual FRM image length.
• FRM_MAX_SIZE.
• Existing table-definition memory limits.

The reader must not allocate memory based solely on an unvalidated item length.

The writer must use overflow-safe calculations when adding:

• Segment headers.
• Item headers.
• Item payloads.
• Offsets.
• Alignment or padding.
• The final forminfo area.

Remaining limits

The following limits are not removed by this task:

• The FRM_MAX_SIZE limit for the complete FRM image, currently 1 MiB.
• The 2-byte key-information length at fileinfo + 28.
• The corresponding “index information is too long” restriction.
• Other 2-byte counters and lengths in forminfo.
• Field-count, key-count, and record-length limits unrelated to Extra2.
• Storage-engine-specific metadata limits.

The explicit 0xFFFF restriction for the field data-type information image may be removed if that item is moved to Extra3, but unrelated FRM limits remain out of scope.

Acceptance criteria

The task is complete when all of the following are satisfied:

• A table whose combined MariaDB-specific metadata exceeds the Extra2 limit can be created.
• An individual metadata item larger than 65535 bytes can be stored in Extra3.
• The table can be reopened after closing and after server restart.
• The table can be queried, altered, renamed, copied, and dropped.
• Extra2 remains within its existing representable size.
• fileinfo + 4 and fileinfo + 6 retain their current formats and meanings.
• Extra3 is stored after the existing extra chunk and before forminfo.
• EXTRA2_EXTRA3 contains a valid 4-byte offset and length.
• The stored forminfo position accounts for Extra3.
• Extra3 item lengths use a 4-byte representation.
• Tables that do not require Extra3 use the existing layout.
• Existing FRM files can be opened by the new server.
• Older servers reject FRM files containing Extra3.
• Older servers do not silently ignore required Extra3 metadata.
• Corrupted offsets, lengths, and item boundaries are detected safely.
• Unknown important Extra3 items are rejected.
• Secondary FRM readers return correct metadata.
• Partitioned and S3 tables work with Extra3.
• No supported DDL operation loses Extra3 metadata.
• The total FRM_MAX_SIZE limit remains enforced.

Test coverage

Add tests covering:

• An Extra2-only table below the existing limit.
• A table whose combined metadata is just below the Extra2 limit.
• A table whose metadata crosses the Extra2 limit by one item.
• A table with substantially more than 64 KiB of applicable metadata.
• One Extra3 item larger than 65535 bytes.
• Multiple Extra3 items.
• A mixture of Extra2 and Extra3 items.
• Deterministic placement of items.
• Creation, close, reopen, and drop.
• Server restart and table-cache reload.
• SHOW CREATE TABLE after reopening.
• Copy-based and in-place ALTER TABLE.
• Transition from Extra2-only to Extra3.
• Transition from Extra3 back to Extra2-only.
• CREATE TABLE ... LIKE.
• RENAME TABLE.
• Storage-engine conversion.
• Backup and restore.
• A table created by an older server opened by the new server.
• A new Extra2-only FRM opened by an older compatible server.
• An older server refusing an FRM containing EXTRA2_EXTRA3.
• Byte comparison of Extra2-only FRM files with the current format.
• Invalid EXTRA2_EXTRA3 payload length.
• Extra3 offset before the permitted region.
• Extra3 offset beyond the end of the FRM image.
• Extra3 length extending beyond the FRM image.
• Offset-plus-length integer overflow.
• Extra3 overlapping the key-information block.
• Extra3 overlapping the default-value record.
• Extra3 overlapping the existing extra chunk.
• Extra3 overlapping forminfo.
• Truncated Extra3 item type or length.
• Truncated Extra3 item value.
• Zero or invalid item length.
• Duplicate singleton item in Extra3.
• Duplicate item across Extra2 and Extra3.
• Unknown optional item.
• Unknown important item.
• FRM image reaching FRM_MAX_SIZE.
• FRM image exceeding FRM_MAX_SIZE.
• Partitioned table creation and reopen.
• Partition ALTER operations.
• Default partition-engine lookup.
• S3 table creation and reopen.
• S3 table-definition-version lookup.
• Failure injection during FRM writing and replacement.
• Sanitizer builds to verify absence of out-of-bounds reads and writes.

Documentation

Add source-code documentation describing:

• The complete FRM layout with Extra3.
• The EXTRA2_EXTRA3 descriptor format.
• The Extra3 type-length-value encoding.
• Integer byte order.
• Item importance handling.
• Placement rules for Extra2 and Extra3.
• Compatibility behavior for older servers.
• Validation requirements for offsets and lengths.
• Metadata that must remain in Extra2 for partial readers.
• Remaining FRM limits.

Update developer documentation for every component that reads FRM metadata directly.