Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-37663

Clustered MariaDB fails to start from backup recovery

    XMLWordPrintable

Details

    Description

      We have clustered MariaDB with 3 hosts where each host runs containerised server.
      We took a replica backup with mariabackup as follows on master host.

      mariabackup \
              --defaults-file="${REPLICA_MY_CNF}" \
              --backup \
              --stream=mbstream \
              --history="${LAST_FULL_DATE}" \
              | gzip > "${BACKUP_PATH}/${BACKUP_FILE}"
      

      where REPLICA_MY_CONF is config file for mariabackup that contains backup user's credential. Backup file was saved as .mbs.gz at ${BACKUP_PATH}/${BACKUP_FILE}

      Later, the backup file was uncompressed and mbstream was used to extract contents. Then, the backup was prepared to be restored

      mbstream -x -C /backup/restore/full < backupfile.mbs
      mariabackup --prepare --target-dir /backup/restore/full
      

      All 3 MariaDB containers were stopped, then backup was copied back

      rm -rf /var/lib/mysql/*
      rm -rf /var/lib/mysql/\.[^\.]*
      mariabackup --copy-back --target-dir /backup/restore/full
      

      After that, only the server at master host was bootstrapped. However, it failed to start with following logs

      2025-09-16 15:27:10 0 [Note] Starting MariaDB 11.4.8-MariaDB-ubu2404-log source revision c4ed889b74ed95fa8b4eaf96610d4c1480b346a4 server_uid oP7jeIwWKccpCJJpW17/FGbNMhE= as process 199
      2025-09-16 15:27:10 0 [Note] InnoDB: Compressed tables use zlib 1.3
      2025-09-16 15:27:10 0 [Note] InnoDB: Number of transaction pools: 1
      2025-09-16 15:27:10 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 
      2025-09-16 15:27:10 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
      2025-09-16 15:27:10 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).
      create_uring failed: falling back to libaio
      2025-09-16 15:27:10 0 [Note] InnoDB: Using Linux native AIO
      2025-09-16 15:27:10 0 [Note] InnoDB: innodb_buffer_pool_size_max=6400m, innodb_buffer_pool_size=6397m
      2025-09-16 15:27:10 0 [Note] InnoDB: Completed initialization of buffer pool
      2025-09-16 15:27:10 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
      2025-09-16 15:27:10 0 [Note] InnoDB: End of log at LSN=19015308
      2025-09-16 15:27:10 0 [Note] InnoDB: Transaction 26204 was in the XA prepared state.
      2025-09-16 15:27:10 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
      2025-09-16 15:27:10 0 [Note] InnoDB: Trx id counter is 26206
      2025-09-16 15:27:10 0 [Note] InnoDB: Resizing redo log from 12.016KiB to 96.000MiB; LSN=19015308
      2025-09-16 15:27:10 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
      2025-09-16 15:27:10 0 [Note] InnoDB: Opened 3 undo tablespaces
      2025-09-16 15:27:10 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
      2025-09-16 15:27:10 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
      2025-09-16 15:27:10 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      2025-09-16 15:27:10 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
      2025-09-16 15:27:10 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
      2025-09-16 15:27:10 0 [Note] InnoDB: log sequence number 19015308; transaction id 26207
      2025-09-16 15:27:10 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
      2025-09-16 15:27:10 0 [Note] Plugin 'FEEDBACK' is disabled.
      2025-09-16 15:27:10 0 [Note] InnoDB: Starting recovery for XA transactions...
      2025-09-16 15:27:10 0 [Note] InnoDB: Transaction 26204 in prepared state after recovery
      2025-09-16 15:27:10 0 [Note] InnoDB: Transaction contains changes to 1 rows
      2025-09-16 15:27:10 0 [Note] InnoDB: 1 transactions in prepared state after recovery
      2025-09-16 15:27:10 0 [Note] Found 1 prepared transaction(s) in InnoDB
      2025-09-16 15:27:10 0 [Note] WSREP: Last wsrep seqno to be recovered 1115
      2025-09-16 15:27:10 0 [ERROR] Found 1 prepared transactions! It means that server was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start server with --tc-heuristic-recover switch to commit or rollback pending transactions.
      2025-09-16 15:27:10 0 [ERROR] Aborting'
      

      which I'm not sure if this is expected. We're just testing backup recovery of MariaDB 11.4, so this MariaDB cluster was fresh and this is the only backup we made.

      This maybe related to MDEV-36233 but they used --no-lock while we did not.

      Attachments

        Activity

          People

            Unassigned Unassigned
            seunghun1ee Seunghun Lee
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.