DuckDB engine: read and INSERT support for Iceberg and DuckLake tables via DuckDB extensions

Expose Apache Iceberg (REST catalogs) and DuckLake tables as MariaDB tables through the DuckDB storage engine, using the upstream DuckDB iceberg and ducklake extensions (pinned to the commits used by DuckDB v1.5.5).

Design doc: storage/duckdb/docs/datalake-iceberg-ducklake-design.md

Scope (first version)

• SELECT (full select_handler pushdown, joins with local DuckDB and cross-engine tables) and INSERT / INSERT ... SELECT / LOAD DATA.
• No UPDATE, DELETE, TRUNCATE, ALTER, or lake-side DDL.

Key design points

• Per-table mapping through table options: ENGINE=DuckDB LAKE_CATALOG='..' LAKE_SCHEMA='..' LAKE_TABLE='..'. Columns are validated against the lake schema at CREATE TABLE (exact type match, no PK/UNIQUE, no AUTO_INCREMENT, no generated columns).
• Read path: a DuckDB view "db"."t" in the main catalog over "_mdb_lake<cat>".schema.table, so verbatim-SQL pushdown works unchanged; the view applies the same COLLATE mapping as local tables.
• Write path: insert-only DeltaAppender path with duckdb::Appender(con, catalog, schema, table).
• Catalogs and secrets are registered with admin functions (duckdb_lake_attach, duckdb_lake_detach, duckdb_lake_secret_create, duckdb_lake_secret_drop, SUPER only). The registry lives in duckdb.db; secrets are DuckDB persistent secrets in a private secret_directory. No credentials in .frm / SHOW CREATE TABLE. Catalogs are re-attached at plugin init.
• Transactions: a DuckDB transaction can write to only one attached database, so mixing writes to local DuckDB tables and lake tables in one transaction is rejected early. CREATE ... SELECT into a lake mapping is rejected.
• Replication: new sysvar duckdb_lake_replica_apply = SKIP|APPLY (default SKIP) prevents double writes into a shared lake.
• New sysvars: duckdb_lake_enabled (read-only, OFF), duckdb_lake_secret_directory, duckdb_lake_replica_apply.

Milestones

• M0: build spike: statically link httpfs, avro, iceberg, ducklake, aws; resolve third-party dependencies and OpenSSL/wolfSSL symbol clashes; vendor sources for offline builds; answer the open questions in the design doc.
• M1: registry and admin functions, DuckLake mapping tables (DuckDB-file metadata), read/insert paths, write-target guard, replication sysvar. MTR tests run offline.
• M2: Iceberg REST (OAuth2/bearer/SigV4). Opt-in MTR tests against a REST fixture with MinIO.

Known limitations

• No atomic commit across a lake and InnoDB/local DuckDB; the lake catalog commit can fail after the binlog commit.
• Secret values passed to duckdb_lake_secret_create() appear in the general log and slow log if they are enabled.