Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40510

System-versioned ROW_END values are never converted from 2038→2106 sentinel on ES replicas populated via row-based replication from a CS source

    XMLWordPrintable

Details

    Description

      Problem
      When a MariaDB Enterprise Server instance is provisioned fresh and set up as a replica of an existing Community Server source containing system-versioned tables, current-time queries against those tables (SELECT ... FOR SYSTEM_TIME AS OF CURRENT_TIMESTAMP, and by extension a plain SELECT * FROM table) return an empty set, even though the source clearly has current, non-deleted rows. Historical rows are being misidentified as "closed" because their ROW_END is still set to the old 2038-01-19 sentinel rather than the 2106-02-07 sentinel introduced by the TIMESTAMP range extension (MDEV-32188, MariaDB 11.5).

      This is not simply a CS-vs-ES version mismatch or a missed upgrade step — it's a fundamental conflict between the 2038→2106 conversion and row-based replication (RBR).

      Root mechanism (per Rich Meese's latest testing, RBR in use throughout):

      Under row-based replication, replicated rows carry the literal ROW_END value from the source binlog — including the old 2038 sentinel. Since the replica applies the row image verbatim rather than recomputing anything, a freshly-provisioned ES replica never converts ROW_END on its own, regardless of any mysql_version check on the replica side. This is true on both 11.4.9 ES and 11.8.6 ES.
      On 11.8, if you subsequently run mariadb_upgrade on that replica, it does appear to correctly rewrite ROW_END from 2038 to 2106 in the local table data. However, this then breaks replication going forward — the binlog stream from the source still carries the old 2038 ROW_END value in its row images, which no longer matches what's now stored in the upgraded replica's table, causing replication to fail (row image / table state mismatch).

      In short: leave it alone and current-time queries return nothing; fix it locally and replication breaks. There is currently no clean path through this for a live CS→ES migration using RBR.

      Untested potential workaround: switching to statement-based replication (SBR) instead of RBR might avoid this, since statements would be re-executed against the replica's own (already-converted) schema/data rather than copying row images verbatim — but this has not been tested and needs verification. Worth a specific test pass: replicate under binlog_format=STATEMENT or MIXED and see whether ROW_END converts correctly and stays consistent.

      Reproduction matrix (Rich Meese, isolated lab environment, Rocky Linux 8, single primary + 5 replicas, all via RBR)

      Setup: versioning_test.prices, a WITH SYSTEM VERSIONING table, created and modified (insert → update → delete → update) on an 11.4.2 Community source (Rocky8-1), replicated live to each of the following:

      Replica Role FOR SYSTEM_TIME AS OF CURRENT_TIMESTAMP
      Rocky8-2 11.4.9 Community (fresh, as replica) ✅ returns current rows
      Rocky8-3 11.4.9 Enterprise (fresh, as replica) ❌ Empty set
      Rocky8-4 11.8.6 Community (fresh, as replica) ✅ returns current rows
      Rocky8-5 11.8.6 Enterprise (fresh, as replica) ❌ Empty set
      Rocky8-6 11.8.6 Community as replica, then upgraded in-place to 11.8.6 Enterprise ✅ returns current rows (both before and after the in-place upgrade — this node was never populated via RBR onto an already-ES target, so it doesn't hit the conflict above)

      FOR SYSTEM_TIME ALL correctly shows all 5 historical + current rows on every node in every case — the data itself replicates correctly; only the current-time view is broken, and only on ES replicas that received system-versioned data via RBR while already running as ES.

      Sample failure output (Rocky8-3, 11.4.9-6-MariaDB-enterprise-log):

      SELECT * FROM prices;
      Empty set (0.001 sec)
       
      SELECT * FROM prices FOR SYSTEM_TIME ALL ORDER BY item_name;
      +---------+---------------------+--------+
      | item_id | item_name           | price  |
      +---------+---------------------+--------+
      |       1 | Mechanical Keyboard | 120.00 |
      |       1 | Mechanical Keyboard |  99.99 |
      |       3 | Monitor             | 300.00 |
      |       2 | Wireless Mouse      |  45.00 |
      |       2 | Wireless Mouse      |  50.00 |
      +---------+---------------------+--------+
      5 rows in set (0.000 sec)
      

      Identical behavior on Rocky8-5 (11.8.6-3-MariaDB-enterprise-log) — confirms this is not fixed in 11.8.6 either.

      Setup script (run on 11.4.2 CS source, Rocky8-1), for engineering to reproduce:

      CREATE DATABASE IF NOT EXISTS versioning_test;
      USE versioning_test;
       
      CREATE TABLE prices (
          item_id INT AUTO_INCREMENT PRIMARY KEY,
          item_name VARCHAR(50),
          price DECIMAL(10,2)
      ) WITH SYSTEM VERSIONING;
       
      INSERT INTO prices (item_name, price) VALUES
      ('Mechanical Keyboard', 120.00),
      ('Wireless Mouse', 45.00),
      ('Monitor', 300.00);
       
      DO SLEEP(2);
      UPDATE prices SET price = 99.99 WHERE item_name = 'Mechanical Keyboard';
      DO SLEEP(2);
      DELETE FROM prices WHERE item_name = 'Monitor';
      UPDATE prices SET price = 50.00 WHERE item_name = 'Wireless Mouse';
      

      Then observe SELECT * FROM prices; (current view) vs SELECT * FROM prices FOR SYSTEM_TIME ALL ORDER BY item_name; (full history) on each replica type above. To reproduce the second failure mode, run mariadb_upgrade on an 11.8.6 ES replica already exhibiting the empty-set symptom and attempt to continue replicating from the CS source.

      Background: earlier hypothesis (may still be a contributing factor, now secondary to the RBR finding above)
      Prior discussion with Elena Stepanova / Sergei Golubchik identified that the routine which flags a system-versioned table as needing ALTER TABLE ... FORCE conversion uses a mysql_version check altered during the 11.5→11.4 ES backport (from < 110500 to < 110400), correct for ES→ES upgrades but not CS→ES. Rich's RBR finding suggests the more fundamental issue is that this conversion logic — whatever its version-check condition — is only ever invoked via the upgrade code path, and is bypassed entirely when data arrives via row-based replication rather than through an actual version upgrade. Both the version-check condition and the RBR-conflict likely need to be addressed.

      Impact
      Blocks MariaDB Cloud go-live migrations from CS to ES for any customer using system-versioned tables with RBR — the standard cloud provisioning pattern (build fresh ES replica, replicate from customer's CS source, cut over) hits this every time. The only tested local fix (mariadb_upgrade on the replica) trades the query-visibility bug for a replication-breaking bug. No known clean workaround yet; SBR is an untested candidate worth a dedicated test pass.

      Test environment retained by Rich Meese (local lab, Rocky8-1 through Rocky8-6) for further testing if engineering needs it.

      Reported by: Richard Meese
      Escalated/retested by: Rafal Hernet (June 3, 11.4.9 CS→ES, confirmed ALTER TABLE...FORCE and mariadb-upgrade don't fix already-broken ROW_END values post-migration in that scenario)
      Discussed by: Elena Stepanova, Sergei Golubchik (Slack, March 31–June 23, 2026)

      Related tickets

      MDEV-32188 — parent epic, TIMESTAMP range extension to 2106 (community)
      MDEV-33310 — "mariadb-upgrade does not fix InnoDB versioned tables upon upgrade to 32-bit timestamp" (community-side sibling, worth checking for shared fix path)
      CENG-415 (internal corp Jira) — early investigation subtask, closed Done April 20 before root cause found; superseded by this ticket
      Zendesk 239071 — customer ticket

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ivenn Isaac Venn
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.