Refactor TABLE::vers_update_fields() via stored virtual columns

Summary

Refactor calculation of system-versioning fields to use the generic virtual-column and default-expression machinery.

Attach internal stored Virtual_column_info expressions to ROW START and ROW END fields when a system-versioned table is opened. Evaluate these expressions through the existing field-update paths instead of assigning system-field values manually in:

• TABLE::vers_update_fields().
• TABLE::vers_update_end().
• Field::save_in_field_default_value().
• Individual DML, replication, trigger, and table-copy call sites.

The internal expressions must not make system fields user-visible generated columns or change table metadata, SQL syntax, on-disk FRM format, dump output, or existing DML behavior.

Rationale

System-versioning field values are currently assigned by dedicated code distributed across several execution paths.

TABLE::vers_update_fields() initializes the period of a current row:

• ROW START is set to the query timestamp unless an explicit value must be preserved.
• ROW END is set to the maximum value representing a current row.
• Virtual columns are then reevaluated with VCOL_UPDATE_FOR_READ because they may depend on the system fields.

TABLE::vers_update_end() separately sets ROW END to the query timestamp when creating a historical row.

Additional special cases exist for:

• DEFAULT assignment.
• Single-table and multi-table DML.
• History-row creation.
• Row-based replication.
• ALTER TABLE row copying.
• Triggers.
• SQL metadata and table-definition processing.
• Storage-engine interaction.

The implementation contains many checks of:

• Field::vers_sys_field().
• VERS_ROW_START.
• VERS_ROW_END.

Maintaining system-field initialization independently in each execution path makes evaluation order, explicit-value handling, and error behavior difficult to keep consistent. This duplicated logic has contributed to problems such as MDEV-16937 and MDEV-19304.

The generic stored virtual-column machinery already provides infrastructure for:

• Attaching an expression to a field.
• Evaluating fields during row construction.
• Respecting dependencies between computed fields.
• Processing default expressions.
• Reusing the same calculation path across DML operations.

System-versioning fields should use this infrastructure internally while retaining their existing external semantics.

Goals

The refactoring must:

• Calculate ROW START and ROW END through internal stored virtual-column expressions.
• Use generic field-update and default-expression paths where possible.
• Remove TABLE::vers_update_fields().
• Remove TABLE::vers_update_end().
• Remove redundant system-field special cases.
• Preserve explicit values in authorized operations.
• Preserve correct timestamp-precise and transaction-precise behavior.
• Evaluate system fields before generated columns that depend on them.
• Preserve all SQL-visible metadata and behavior.
• Avoid changing the FRM format.

This is an internal refactoring. It must not introduce new SQL syntax or user-visible generated columns.

Current implementation

TABLE::vers_update_fields() is called from several paths, including:

• TABLE::update_generated_fields().
• vers_update_or_validate_fields() in the fill_record() path.
• multi_update::do_updates().
• Write_rows_log_event::write_row().
• Update_rows_log_event::do_exec_row().

It initializes current-row system fields and then reevaluates dependent virtual columns.

TABLE::vers_update_end() is used when creating a historical version, including from:

• vers_insert_history_row().
• TABLE::delete_row().
• Sql_cmd_update::update_single_table() when a delete is executed as a versioned-table update.

Field::save_in_field_default_value() contains a separate vers_sys_field() branch for applying DEFAULT to system fields.

Further system-field checks are distributed across files including:

• sql_base.cc.
• sql_table.cc.
• sql_trigger.cc.
• sql_show.cc.
• item.cc.
• handler.cc.
• unireg.cc.

The refactoring must audit these call sites and remove only those checks whose behavior is provided by the new generic path.

Proposed design

When a system-versioned table is opened, attach an internal Virtual_column_info object to its ROW START and ROW END fields.

The initialization should occur while opening the table from its binary definition, in or around:

TABLE_SHARE::init_from_binary_frm_image()

The internal virtual-column descriptor must have:

stored_in_db= 1;

Its expression must be represented by a dedicated item, such as:

Item_vers_sys_field_setter

The expression evaluates the value required for the system field according to:

• Whether the field is ROW START or ROW END.
• Whether the row is current or historical.
• Whether timestamp-precise or transaction-precise versioning is used.
• Whether an authorized explicit value has already been supplied.
• The current row operation.

The expression exists only in memory. It must not be serialized to the FRM definition.

Expression initialization

The internal expression must be created early enough for all generic virtual-column initialization and validation code.

In particular, code such as:

check_vcol_forward_refs()

parse_vcol_defs()

expects vcol->expr to be initialized.

The expression must therefore be attached before parse_vcol_defs() or any other operation that traverses virtual-column expressions.

Initialization must not depend on reparsing an expression stored in the FRM, because no user-visible generation expression is persisted.

Failure to allocate or initialize the internal expression must fail table opening cleanly without leaving partially initialized table metadata.

Current-row values

For timestamp-precise system versioning, the internal expressions normally produce:

The query timestamp must be the same value currently used by TABLE::vers_update_fields().

All rows affected by one operation must preserve the existing query-time semantics. The refactoring must not replace a statement-stable value with a separately evaluated wall-clock timestamp for every row.

The maximum ROW END value must remain exactly the value used to identify a current row before this refactoring.

History-row values

When an update or delete creates a historical version:

• The historical row retains its original ROW START.
• Its ROW END is set to the version boundary represented by the query timestamp.
• The replacement current row receives the appropriate new ROW START.
• The replacement current row receives the maximum ROW END value.

The operation-sensitive internal expression must produce the historical ROW END value currently assigned by:

TABLE::vers_update_end()

After conversion, history-row creation must not call vers_update_end() separately.

The same expression infrastructure may return different values for current-row construction and history-row construction, but this context must be explicit and local to the operation.

Explicit values

An authorized explicit value must take precedence over the internal expression.

The implementation must preserve the existing has_explicit_value() behavior for at least:

• History insertion enabled by @@system_versioning_insert_history.
• Row-based replication.
• Replication from an older primary.
• secure_timestamp processing.
• Write_rows_log_event::write_row().
• vers_fix_old_timestamp().
• Copy-based ALTER TABLE.
• Other operations that legitimately preserve existing period values.

The internal expression must not overwrite an explicit value merely because the field has an attached Virtual_column_info.

Normal user DML must remain unable to assign arbitrary system-period values unless an existing authorized mode permits it.

DEFAULT handling

Applying DEFAULT to a system-versioning field must use the generic default-expression path.

The dedicated vers_sys_field() handling in:

Field::save_in_field_default_value()

must be removed when the internal expression provides equivalent behavior.

For a system field, DEFAULT must produce the same result as before the refactoring.

The change must not:

• Treat the system field as an ordinary nullable column.
• Apply a user-defined literal default.
• Expose the internal expression as a SQL generation expression.
• Produce a generated-column assignment error where the current behavior produces a system-versioning warning or error.

Generated-column distinction

Although implementation uses Virtual_column_info, a system-versioning field must not become a generated column from the SQL user's perspective.

Generated-column restrictions must not be applied to internal system-field expressions.

In particular:

• A system field may participate in a primary key where currently supported.
• ROW END may continue to participate in internally extended unique keys.
• System versioning must continue to work with engines such as MEMORY that do not support ordinary generated columns.
• Generated-column capability checks must not reject system-versioned tables.
• Generated-column assignment diagnostics must not replace existing system-field diagnostics.
• Generated-column metadata must not be exposed for system fields.

Checks in areas such as:

mysql_prepare_create_table()

open_table_from_share()

must distinguish an internal system-field expression from a user-declared generated column.

Evaluation order

System-versioning fields must be evaluated before any virtual or stored generated column that depends on them.

For example:

CREATE TABLE t1 (

  id INT,

  row_start TIMESTAMP(6) GENERATED ALWAYS AS ROW START,

  row_end TIMESTAMP(6) GENERATED ALWAYS AS ROW END,

  active_flag INT AS (row_end = TIMESTAMP'2038-01-19 03:14:07.999999') STORED,

  PERIOD FOR SYSTEM_TIME (row_start, row_end)

) WITH SYSTEM VERSIONING;

active_flag must be calculated from the final row_end value for the row.

The dependency and evaluation order must support:

• A virtual column depending on ROW START.
• A virtual column depending on ROW END.
• A stored generated column depending on either system field.
• A chain of generated-column dependencies beginning with a system field.
• Current-row construction.
• History-row construction.
• Replication apply.
• Table-copy operations.

The existing reevaluation performed by vers_update_fields() with VCOL_UPDATE_FOR_READ may be removed only after the generic dependency order provides equivalent results.

Update paths

The refactoring must cover all row-construction paths, including:

• INSERT.
• Multi-row INSERT.
• INSERT ... SELECT.
• REPLACE.
• INSERT ... ON DUPLICATE KEY UPDATE.
• LOAD DATA.
• Single-table UPDATE.
• Multi-table UPDATE.
• Single-table DELETE.
• Multi-table DELETE.
• DML executed by triggers.
• Foreign-key cascades where applicable.
• History-row creation.
• Copy-based ALTER TABLE.
• Row-based replication events.

Every path must initialize the required system fields exactly once.

No path may leave a stale value from a previously used record buffer.

UPDATE assignment behavior

An explicit assignment to a system field in:

UPDATE t1 SET row_start=...;

UPDATE t1 SET row_end=...;

must retain its current behavior.

The refactoring must not replace the existing system-versioning diagnostic with an error intended for ordinary generated columns.

This applies to:

• Direct assignment.
• Assignment through DEFAULT.
• Multi-table update.
• Trigger assignment.
• Prepared statements.
• Replicated events.

Existing SQL modes and authorized history-insertion behavior must remain unchanged.

Transaction-precise versioning

For transaction-precise versioning, InnoDB supplies transaction identifiers used by the versioning fields.

The SQL-layer internal expression must not overwrite a value assigned by the storage engine.

The SQL layer must continue to initialize only the values it owns, including the maximum ROW END value where required for a current row.

The design must clearly distinguish:

• Values generated in the SQL layer.
• Values generated by the storage engine.
• Explicit values supplied through an authorized path.
• Values copied from an existing row.

Timestamp-based assumptions must not be applied to transaction identifiers.

Replication

Row-based replication must preserve explicit source values.

The internal expression must not replace period values contained in row events.

Existing behavior must be preserved for:

• Replication from servers predating this refactoring.
• Write_rows_log_event::write_row().
• Update_rows_log_event::do_exec_row().
• secure_timestamp.
• vers_fix_old_timestamp().
• Explicit history rows.
• Current rows.
• Mixed-version replication topologies supported by MariaDB.

Statement-based replication must continue to evaluate system fields using the existing statement timestamp semantics.

The change must not alter binary-log row images or event formats.

ALTER TABLE

Copy-based ALTER TABLE must preserve existing system-period values instead of regenerating them.

During a table rebuild:

• Existing current rows retain their period values.
• Existing historical rows retain their period values.
• Explicit copied values take precedence over internal expressions.
• Generated columns depending on period fields are evaluated consistently.
• Conversion between supported storage engines remains possible.
• The resulting table remains system-versioned.

An ALTER TABLE operation that adds system versioning must initialize period values according to existing behavior.

An operation that removes system versioning must not retain internal system-field expressions on ordinary columns.

Triggers

Triggers operating on system-versioned tables must preserve existing behavior.

The refactoring must correctly handle:

• BEFORE INSERT.
• AFTER INSERT.
• BEFORE UPDATE.
• AFTER UPDATE.
• BEFORE DELETE.
• AFTER DELETE.
• Trigger access to NEW period fields.
• Trigger access to OLD period fields.
• Trigger attempts to assign system fields.
• Trigger-generated DML against another versioned table.

System fields must be initialized at the same logical stage as before so that trigger-visible values do not change unexpectedly.

Metadata compatibility

The internal expressions must not appear in SQL metadata.

The following output must remain unchanged:

• SHOW CREATE TABLE.
• SHOW COLUMNS.
• DESCRIBE.
• INFORMATION_SCHEMA.COLUMNS.IS_GENERATED.
• INFORMATION_SCHEMA.COLUMNS.GENERATION_EXPRESSION.
• Connector metadata.
• mariadb-dump table definitions.

System fields must continue to be identified as ROW START and ROW END, not as user-generated columns.

GENERATION_EXPRESSION must not expose Item_vers_sys_field_setter or an equivalent internal representation.

FRM compatibility

The internal Virtual_column_info and its expression must not be written to the FRM file.

The change must not:

• Modify the FRM format.
• Require a table upgrade.
• Change the stored table definition.
• Prevent an older supported server from opening an otherwise compatible table.
• Require dump and restore to activate the refactoring.

The internal expression must be reconstructed whenever the table definition is opened.

A table-cache eviction and reopen must produce an equivalent internal expression.

Storage-engine compatibility

System versioning must remain usable with every currently supported storage engine.

The internal expression must not cause the server to require engine support for ordinary generated columns.

In particular, test engines where:

• Ordinary generated columns are unsupported or restricted.
• System fields participate in indexes.
• System versioning is implemented partly by the engine.
• Transaction-precise versioning is engine-managed.

The storage-engine table definition and field flags exposed to the engine must remain compatible unless an intentional internal API adjustment is required.

Special-case cleanup

After the generic expression path is complete, audit every use of:

vers_update_fields()

vers_update_end()

vers_sys_field()

VERS_ROW_START

VERS_ROW_END

Remove a special case only when its behavior is fully covered by the new implementation.

Retain system-field checks that are still required for:

• SQL syntax validation.
• Metadata presentation.
• Period definition validation.
• Storage-engine coordination.
• Authorized explicit-value handling.
• System-versioning-specific diagnostics.

The objective is not to remove all system-field awareness. It is to eliminate duplicated value-assignment logic.

Error handling

Expression initialization and evaluation must follow the existing error-handling rules for the corresponding operation.

An error must not leave:

• A partially initialized table object.
• An incomplete current row.
• A history row without a valid ROW END.
• A current row with a finite ROW END.
• A copied row with regenerated period values.
• Stale field values in a reused record buffer.

The original DML or table-open error must be preserved.

An internal expression failure must not expose a user-visible generated-column expression or diagnostic unless that diagnostic already applies to the operation.

Performance

The generic expression path must not introduce avoidable duplicate evaluation.

In particular:

• Each required system field should normally be evaluated once per row.
• Dependent generated columns should be evaluated after system fields without reevaluating unrelated columns.
• History-row construction should not evaluate current-row values and then overwrite them.
• Row-based replication should not calculate values that are already explicit in the event.
• Table opening should allocate the internal expression once per table definition or table instance according to the existing object model.

Performance should be compared with the existing dedicated implementation for insert, update, delete, and replication workloads.

Compatibility

The refactoring must not change:

• SQL syntax.
• Current-row period values.
• Historical-row period values.
• Query timestamp semantics.
• Authorized explicit history insertion.
• User attempts to assign system fields.
• Timestamp-precise versioning.
• Transaction-precise versioning.
• Trigger-visible values.
• Replication formats.
• Table-definition formats.
• Generated-column metadata.
• Storage-engine availability.
• SHOW CREATE TABLE output.
• Information Schema output.
• mariadb-dump output.

Existing test results in the versioning and period suites must remain unchanged unless a result currently exposes a confirmed bug fixed as part of this work.

Acceptance criteria

The task is complete when all of the following are satisfied:

• ROW START and ROW END have internal stored Virtual_column_info expressions after opening a system-versioned table.
• The expressions are initialized before generic virtual-column parsing and dependency analysis.
• Current-row ROW START is generated correctly.
• Current-row ROW END receives the maximum value.
• Historical-row ROW END receives the correct version boundary.
• Authorized explicit values take precedence over internal expressions.
• Transaction-precise values assigned by InnoDB are not overwritten.
• System fields are evaluated before generated columns that depend on them.
• DEFAULT uses the generic default-expression path.
• TABLE::vers_update_fields() is removed.
• TABLE::vers_update_end() is removed.
• Redundant system-field assignment checks are removed.
• Required semantic and metadata checks remain.
• System fields are not treated as SQL generated columns.
• System fields remain permitted in existing supported indexes and engines.
• All relevant DML paths produce unchanged results.
• Copy-based ALTER TABLE preserves period values.
• Row-based replication preserves explicit source values.
• Trigger behavior remains unchanged.
• SHOW CREATE TABLE output remains unchanged.
• Information Schema generated-column metadata remains unchanged.
• mariadb-dump output remains unchanged.
• No FRM format change is introduced.
• Existing tables open without upgrade or rebuild.
• All existing versioning and period tests pass without unintended result changes.

Test coverage

Add or extend tests covering at least:

• Table open and reopen.
• Table-cache eviction.
• Explicit ROW START and ROW END columns.
• Implicit system-versioning fields.
• Visible system fields.
• Invisible system fields.
• INSERT.
• Multi-row INSERT.
• INSERT ... SELECT.
• REPLACE.
• INSERT ... ON DUPLICATE KEY UPDATE.
• LOAD DATA.
• Single-table UPDATE.
• Multi-table UPDATE.
• Single-table DELETE.
• Multi-table DELETE.
• Repeated updates in one statement.
• Timestamp-precise versioning.
• Transaction-precise versioning.
• Fractional timestamp precision.
• Explicit system-field values.
• Rejected unauthorized system-field values.
• DEFAULT for ROW START.
• DEFAULT for ROW END.
• @@system_versioning_insert_history.
• secure_timestamp.
• vers_fix_old_timestamp().
• Row-based replication.
• Statement-based replication.
• Replication from an older primary.
• Minimal and full row images.
• Copy-based ALTER TABLE.
• In-place ALTER TABLE.
• Adding system versioning.
• Removing system versioning.
• Storage-engine conversion.
• Virtual columns depending on ROW START.
• Virtual columns depending on ROW END.
• Stored generated columns depending on system fields.
• Chained generated-column dependencies.
• A system field in a primary key where currently supported.
• Versioned tables using MEMORY.
• Other engines without ordinary generated-column support.
• InnoDB transaction-precise field assignment.
• BEFORE INSERT and AFTER INSERT triggers.
• BEFORE UPDATE and AFTER UPDATE triggers.
• BEFORE DELETE and AFTER DELETE triggers.
• Trigger attempts to assign system fields.
• Trigger-generated DML on another versioned table.
• Prepared statements.
• Error handling during table opening.
• Error handling during expression evaluation.
• Reuse of record buffers across rows.
• SHOW CREATE TABLE comparison before and after the refactoring.
• INFORMATION_SCHEMA.COLUMNS comparison.
• SHOW COLUMNS comparison.
• mariadb-dump comparison.
• Opening existing FRM definitions without table upgrade.
• Existing MDEV-16937 regression coverage.
• Existing MDEV-19304 regression coverage.
• Complete versioning test suite.
• Complete period test suite.

Documentation

No user-facing SQL documentation change should be required because the refactoring must not change visible behavior.

Add or update developer documentation and code comments to explain:

• Why system fields use internal Virtual_column_info objects.
• Why these fields must not be treated as user-generated columns.
• When the internal expressions are created.
• How operation context determines current-row and history-row values.
• How authorized explicit values override the expression.
• How transaction-precise values are coordinated with InnoDB.
• The required evaluation order for dependent generated columns.
• Why the expressions are not persisted in the FRM.
• Which remaining vers_sys_field() checks are intentional.