S3: transient definition read errors evict local .frm & encoded names prevent rediscovery

Problem

The S3 engine can turn a temporary failure to fetch a table definition into a false "table does not exist" error and delete its local .frm. A second, related discovery bug prevents recovery for identifiers requiring filesystem encoding, even while all archived objects remain present.

Reproduction A: temporary definition read failure

Configure an isolated S3 endpoint/bucket and run on unpatched 12.3.3:

CREATE DATABASE s3_repro;

USE s3_repro;

CREATE TABLE t1 (a INT) ENGINE=Aria;

INSERT INTO t1 VALUES (9);

ALTER TABLE t1 ENGINE=S3;

SELECT * FROM t1;

1. After the successful SELECT, make an HTTP proxy return 503 for GET requests to the existing bucket/s3_repro/t1/frm object. Keep this failure active through the following statement, including any rediscovery request. Do not remove or alter the actual S3 object.
2. Run SELECT * FROM t1 again, using the cached table handler.
3. Observe error 1146 and that the local s3_repro/t1.frm has been removed. A direct request to the backing S3 fixture, bypassing the fault proxy, still retrieves the original object.
4. Clear the fault. With this ordinary ASCII name, a later SELECT can rediscover the table and recover its data.

This was reproduced with an actual HTTP 503 against the upstream 12.3.3 functional baseline. It is not a simulated 404 or a claim that the S3 service deleted the archive.

Expected: report the read/connection failure, retain the local definition and remote archive, and allow a later statement to retry. A verified UUID change or genuine S3 404 should retain existing rediscovery/deleted-table semantics.

Reproduction B: filesystem-encoded identifiers

CREATE DATABASE `s3_repro-encoded`;

USE `s3_repro-encoded`;

CREATE TABLE `t-name` (a INT) ENGINE=Aria;

INSERT INTO `t-name` VALUES (1);

ALTER TABLE `t-name` ENGINE=S3;

FLUSH TABLES;

Remove only the local s3_repro@002dencoded/t@002dname.frm from the disposable datadir, leaving S3 unchanged, then SELECT * FROM `t-name`.

The archive writer uses the existing encoded prefix s3_repro@002dencoded/t@002dname/, but discovery looks up logical names s3_repro-encoded/t-name/. Discovery therefore returns 1146 despite the archived definition and data still existing. Similar inconsistencies affect existence probes, listings, base-table version checks, and metadata-only ALTER updates. Unicode names and literal @ sequences also need consistent conversion without double encoding. Partition parent metadata must use the same established namespace.

Root cause in 12.3.3

• storage/maria/s3_func.c: s3_check_frm_version() does not distinguish a confirmed 404 from other definition fetch failures. Separately, s3_discover_table() in ha_s3.cc maps every s3_get_def() failure to HA_ERR_NO_SUCH_TABLE.
• storage/maria/ha_s3.cc: discover_check_version() maps every nonzero version-check result to HA_ERR_TABLE_DEF_CHANGED. Several discovery/metadata callbacks use logical identifiers instead of the filesystem-encoded names used by archival.
• sql/sql_base.cc: the cached-open caller requests OT_DISCOVER for every nonzero discover_check_version() result. Merely returning a different error from the engine is insufficient: this caller must distinguish an actual definition change from a read failure.
• The cold Aria/S3 open path also needs to handle version-read errors instead of ignoring them.
• Existence checks collapse HEAD errors into absence, and listing errors can be presented as a successful empty list.

Source: https://github.com/MariaDB/server/blob/mariadb-12.3.3/storage/maria/ha_s3.cc
Source: https://github.com/MariaDB/server/blob/mariadb-12.3.3/storage/maria/s3_func.c
Source: https://github.com/MariaDB/server/blob/mariadb-12.3.3/sql/sql_base.cc

Impact: intact archives become inaccessible and missing-table diagnostics can mislead external recovery/maintenance software. The minimal reproduction demonstrates definition eviction and loss of access, not automatic deletion of archived data.

Proposed patch for review

A source patch has been prepared against tag mariadb-12.3.3, commit 83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e:

• Encode logical identifiers at discovery, existence, version-check and metadata-update boundaries, and decode S3 listing names. Keep filesystem-derived paths as they are.
• Distinguish matching/changed UUIDs, confirmed 404, malformed definitions and read failures. Preserve local metadata on read failures and malformed remote definitions, and fail the statement rather than silently returning stale data.
• Request OT_DISCOVER only for HA_ERR_TABLE_DEF_CHANGED; report other handler errors without discarding the local definition. Apply consistent handling to cold Aria/S3 opens and partition handlers.
• Fail closed on uncertain existence probes and report listing/optional .par read failures.
• Check definition length before inspecting its header, including empty/truncated responses.

Compatibility requirement: preserve the current filesystem-encoded namespace, object layout, .frm/.par formats, UUID representation, Aria headers, data/index blocks and compression. No migration, renaming, copying or re-archiving of compatible existing objects is required. Rebuild the server and S3 plugin together because the internal version-check status handling changes on both sides.

Patch filename: mariadb-12.3.3-s3-discovery.patch
SHA-256: fd8a0907b407abf0d0bf412e464ca01549727b2ac3db0705290c155a404a0851

Validation

• Full Debug server, S3 plugin, archive-copy utility and backup tool built successfully.
• Full local S3 suite: 23 passed; 1 AWS-only test skipped.
• After adding partition fault cases, both new regression tests and existing discovery were rerun without recording results: 3 passed.
• 79 HTTP/compatibility assertions passed, covering real 403/503 responses and dropped connections; cached/cold/discovery/existence/listing paths; malformed metadata; optional .par failures; genuine 404 semantics; recovery; encoded identifiers; compressed and partitioned archives.
• All 30 objects written by original 12.3.3 retained identical keys and SHA-256 hashes after patched warm reads and cold rediscovery.
• Original 12.3.3 read ordinary, compressed and partitioned archives written by the patched build. Its existing encoded-base-name discovery bug remains in the original binary.
• Both changed S3 translation units passed DBUG_OFF syntax/warning checks with debug tracing disabled. Full runtime testing used Debug builds.
• The patch applies cleanly to pristine source files from the exact tag and passes whitespace checks.

Runtime tests used Moto 5.2.0 over real HTTP, not AWS/MinIO. The existing s3_no_content_type=1 option avoided Moto consuming curl's default form content type as an empty body; the patch makes no HTTP-header changes.