Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41197

DuckDB: support applying replicated row events without row lookup (read-free replica apply)

    XMLWordPrintable

Details

    Description

      Problem

      A replica applying row-based events (Delete_rows / Update_rows) to a DuckDB table stops with:

      [ERROR] Slave SQL: Could not execute Delete\_rows\_v1 event on table test.invoice; Storage engine DUCKDB of the table {{test}}.{{invoice}} doesn't have this option, Error\_code: 1031; handler error HA\_ERR\_WRONG\_COMMAND
      

      Root cause: Rows_log_event::find_key() unconditionally selects a unique PK for the row lookup (the HA_NOSAME branch does not consult index_flags()), and use_pk_position() is false for DuckDB, so find_row() calls ha_index_read_map() which is a HA_ERR_WRONG_COMMAND stub in the DuckDB handler. rnd_pos() is a stub as well, and there is no fallback to a table scan in this code path.

      Reproduced with a two-server MTR setup (master-slave.inc, binlog_format=MIXED): the master survives DELETE FROM (with the MDEV-40957 fix), but the replica SQL thread aborts with error 1031 on the first Delete_rows event.

      Proposed fix: positioning stubs on the replica (read-free apply)

      find_row() unpacks the event's before-image into record[0] before performing the lookup; the lookup merely overwrites record[0] with the row read from the table. If the positioning call is a no-op returning success, record[0] keeps the before-image from the event, which is sufficient for the DuckDB handler:

      • DELETE: ha_delete_row(record[0]) identifies the row by PK taken from the before-image;
      • UPDATE: the server saves record[0] to record[1], unpacks the after-image into record[0], and ha_update_row(record[1], record[0]) gets the old PK and the complete new row.

      Implementation sketch (engine-side only, no server changes):

      Add HA_PRIMARY_KEY_REQUIRED_FOR_POSITION to ha_duckdb::table_flags() so that use_pk_position() becomes true and find_row() takes the ha_rnd_pos_by_record(record[0]) path instead of ha_index_read_map().

      Override ha_duckdb::rnd_pos_by_record() to return 0 without touching record[0], guarded so that the fake success is only returned in the replication applier context (rgi_slave); any other caller (filesort, multi-table UPDATE/DELETE, window functions) must keep getting HA_ERR_WRONG_COMMAND to avoid silent data corruption.

      This mirrors what AliSQL does server-side for its DuckDB engine: a dedicated ROW_LOOKUP_DUCKDB algorithm whose do_duckdb_update() unpacks the before-image and calls do_apply_row() directly, with index_read_map()/rnd_pos() remaining stubs. The stub-based approach achieves the same without patching sql/ (similar in spirit to RocksDB read-free replication).

      Constraints and trade-offs

      • binlog_row_image=FULL is required: with MINIMAL the after-image is incomplete and the delete+insert update path would destroy non-updated columns. The engine must detect an incomplete row image and fail the apply with a clear error.
      • A primary key is required (duckdb_require_primary_key is already ON by default).
      • Semantics become idempotent rather than strict: a DELETE for a non-existent row silently affects 0 rows, an UPDATE re-inserts the row; master/replica drift is no longer detected by the applier. This matches RocksDB read-free replication behaviour and must be documented.

      Test

      Extend the replication MTR test from MDEV-40957 (two servers, DuckDB table with UUID PK, INSERT + DELETE FROM under MIXED and ROW binlog formats) so that both master and replica stay healthy and the replica row counts match the master.

      Attachments

        Issue Links

          Activity

            People

              drrtuy Roman
              drrtuy Roman
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.