The maximum number of index fields varies as follows:

• Clustered index non-leaf pages: At most 17 (up to 16 fields of PRIMARY KEY and a child page number)
• Clustered index leaf pages: 1021 or 1024
• Secondary index non-leaf pages: 16+16+1 (up to 16 fields of KEY and 16 fields of PRIMARY KEY) and a child page number
• Secondary index leaf pages: 16+16

Also, in some contexts, such as accessing DB_TRX_ID or DB_ROLL_PTR, we will only need a prefix of the clustered index leaf page record fields.

I hope that we can avoid allocating 1024*2 bytes for the offsets in many cases.

This is one possible solution. https://godbolt.org/z/mQ9kTj

Accessing some offset is 2 pointer dereferences unless compiler will optimize it e.g. inside a loop.

Design notes:

• Only caller creates offsets either on stack or on heap.
• Caller should have an ability to create offsets on a heap, because of possible long recursion on FK checks.

I mostly liked your changes so far. For innodb_page_size=64k, you must check the actual maximum record size of ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC records. I am rather sure that it is more than 16383 bytes, which you are assuming. Extrapolating your changes to 10.3, which introduces the REC_OFFS_DEFAULT bit, it looks like the maximum record size would shrink to 8191 bytes, which would become a problem also with innodb_page_size=32k.

Perhaps we can do with a single flag bit, REC_OFFS_EXTERNAL = 0x8000? The REC_OFFS_SQL_NULL flag could be replaced with a magic value, such as 0x8000. We know that REC_OFFS_EXTERNAL can only be set if index->is_primary(), and hence we also know that the start offset of externally stored fields must be at least 6+7 (the combined size of the system columns DB_TRX_ID and DB_ROLL_PTR). For the rare case where a NULL value is updated in place to a NOT NULL value, or vice versa, for a ROW_FORMAT=REDUNDANT record, we can use rec_…_old() accessor functions directly. Similarly, in 10.3+, REC_OFFS_DEFAULT would no longer be a flag, but we could use a magic constant value 0x8001. Anything in the range of 0x8000 and 0x800c would be distinguishable from REC_OFFS_EXTERNAL.

As far as I understand, the following changes are yet to be made:

• In non-debug builds, make rec_offs_set_n_alloc() no-op and omit the corresponding element of the offsets header. The new first element of offsets would be rec_offs_n_fields().
• Remove the heap parameter of rec_get_offsets(), and possibly adjust some mem_heap_create() or mem_heap_alloc() calls accordingly.

I have refactored my patch so it now uses uppers bits of the offset differently. Every offset has 2 bits for flags. It can contain 4 different values. My patch uses it to store offset type (normal, off-page, NULL). One more type INSTANTLY ADDED can easily be added while merging patch to 10.3

Thanks, this looks better. Still, I’d like to see how we would set the flags in the offsets header in the 10.3 version. We should have more bits available in that header field; the maximum extra_size should be around 2005 bytes, which fits in 11 bits. So, we should have 5 flag bits at our disposal.

Also, I would like you to remove the heap parameter of rec_get_offsets().

Please file a separate 10.5 task for removing the heap parameter of rec_get_offsets(). It could be a too big change for GA versions.

The way this task was resolved conflicts with a system typedef in AIX. offset_t is already defined.

Can I suggest to use a more specific name? E.g. inno_offset_t.
I have open an issue MDEV-21595.