mariabackup fails with "Attempted to open a previously opened tablespace" likely due to race condition during atomic RENAME TABLE operations

Summary

mariabackup --backup fails during tablespace enumeration with "Attempted to open a previously opened tablespace" when it encounters tables mid-RENAME during atomic DDL operations. This is a race condition where mariabackup scans the filesystem/data dictionary during the brief window when atomic RENAME TABLE is transitioning table names and space IDs.

Full Command Line used

/usr/bin/mariabackup --backup --stream=xbstream --parallel=4 --incremental-history-name=lsnstore --history=lsnstore

Full Error Output

[00] 2025-12-30 23:03:08 mariabackup: Generating a list of tablespaces

2025-12-30 23:04:40 0 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace: ./lists/pds_normalization_term_66706_temp.ibd uses space ID: 86104750. Cannot open filepath: ./lists/pds_normalization_term_66706.ibd which uses the same space ID.

[00] FATAL ERROR: 2025-12-30 23:04:40 Failed to validate first page of the file lists/pds_normalization_term_66706, error 39

Reproduction

Status: Occurs daily during backup window when RENAME operations are active

Confirmed Pattern:

Database contains 133,441 tables
Tables are regularly rebuilt and swapped using atomic RENAME TABLE:

   -- Application pattern:

   CREATE TABLE pds_normalization_term_XXXXX_temp ...

   -- Populate temp table

   RENAME TABLE 

     pds_normalization_term_XXXXX TO pds_normalization_term_XXXXX_old,

     pds_normalization_term_XXXXX_temp TO pds_normalization_term_XXXXX;

   DROP TABLE pds_normalization_term_XXXXX_old;

3. mariabackup runs concurrently with these operations
4. Backup fails when scanning tablespaces during mid-RENAME state

Investigation Results

*NO orphaned files found:*

• All .ibd files on filesystem have corresponding entries in data dictionary
• No cleanup issues detected

*NO space ID duplicates found:*

• Manual queries of INNODB_SYS_TABLES show no duplicate space IDs
• Space ID allocation is working correctly
• Not approaching 32-bit limit

*Timing confirmed:*

• Error only occurs during backup
• Manual inspection after backup failure shows no conflicts
• Tables appear normal in data dictionary post-failure

Root Cause Analysis

This is likely a race condition between mariabackup and atomic DDL operations:

Timeline of Events:

T0: Application begins RENAME TABLE operation (atomic DDL)

    RENAME TABLE 

      pds_normalization_term_66706 TO pds_normalization_term_66706_old,

      pds_normalization_term_66706_temp TO pds_normalization_term_66706;

T1: Atomic DDL enters intermediate state:

    - File pds_normalization_term_66706_temp.ibd exists on disk

    - File pds_normalization_term_66706.ibd exists on disk  

    - Data dictionary metadata is being updated

    - Space ID reassignment in progress

T2: mariabackup "Generating a list of tablespaces" scan occurs

    - Scans filesystem for .ibd files

    - Reads page 0 headers from each file

    - Encounters both files during DDL transition

    - Detects space ID conflict in transient state

T3: mariabackup aborts with error 39

T4: RENAME TABLE completes successfully

    - Space IDs properly reassigned

    - Old table cleaned up

    - Database state is consistent

T5: Manual investigation shows no issues (DDL completed)

Expected Behavior

mariabackup should handle transient states during atomic DDL operations by:

Acquiring appropriate locks to prevent seeing inconsistent states, OR
Detecting and safely handling mid-DDL tables, OR
Retrying tablespace enumeration if conflicts detected, OR
Coordinating with backup locks to avoid scanning during DDL

Actual Behavior

mariabackup scans tablespaces without coordination with atomic DDL
Encounters transient space ID conflicts during RENAME operations
Immediately fails with error 39 (Data structure corruption)
No retry or conflict resolution mechanism

Impact

Severity: High
Frequency: Daily backup failures
Production database with cannot be backed up reliably
Backup failures coincide with application table rebuild operations
Manual intervention required to retry backups during quiet periods

Database Characteristics

Table count: 133,441 tables in schema
Filesystem: XFS
DDL Pattern: Continuous atomic RENAME TABLE operations
Temp table naming: pds_normalization_term_XXXXX_temp
Active temp tables: Multiple temp tables exist at any given time during normal operations
Space ID range: Normal, no exhaustion concerns