Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41693

mariadb-backup 11.4.12 full backup fails with "Was only able to copy log ... try increasing innodb_log_file_size" although the redo log is far from full (MDEV-36159 fix in place)

    XMLWordPrintable

Details

    • Bug
    • Status: Needs Feedback (View Workflow)
    • Major
    • Resolution: Unresolved
    • 11.4.12
    • None
    • mariabackup
    • None
    • Rocky Linux 10.2 (kernel 6.12.0-211.26.1.el10_2), MariaDB-server/MariaDB-backup 11.4.12-1.el10, XFS, innodb_log_file_size=16G

    Description

      A scheduled full backup with mariadb-backup 11.4.12 failed at the end of the copy phase with
      the "try increasing innodb_log_file_size" error. The redo log was nowhere near full during the
      backup, and the defaults introduced by the MDEV-36159 fix (fixed in 11.4.9) are active. So it
      looks like that fix does not cover every case. MDEV-36237 (11.4.4, 11.4.5) and MDEV-37964 were
      closed as duplicates of MDEV-36159 with resolution "Incomplete". MDEV-38396 reports the same
      failure on 10.11.15, which also contains the fix.

      Environment

      • MariaDB-server and MariaDB-backup 11.4.12-1.el10 (official MariaDB repository), Rocky Linux 10.2, kernel 6.12.0-211.26.1.el10_2.x86_64
      • 24 vCPU, 251 GB RAM, XFS datadir (~1.3 TB of InnoDB data; compressed full backup ~180 GB)
      • innodb_log_file_size = 16G, innodb_log_buffer_size = 64M, innodb_buffer_pool_size = 180G
      • innodb_flush_method = O_DIRECT, innodb_flush_log_at_trx_commit = 1
      • innodb_log_file_mmap = OFF, innodb_log_file_buffering = OFF, innodb_log_write_ahead_size = 512
      • innodb_io_capacity = 200, innodb_max_dirty_pages_pct_lwm = 0
      • kernel.io_uring_disabled = 2, so mariadb-backup falls back to libaio
      • mariadb-backup's own defaults, as printed by --help --verbose: innodb-log-file-mmap FALSE, innodb-log-checkpoint-now FALSE
      • Command (streamed):
        mariadb-backup --defaults-extra-file=... --backup --stream=mbstream --extra-lsndir=<dir> | zstd -3 -T<n> > backup.mbstream.zst

      What happened
      The full backup started at 00:10:03 and streamed all tablespaces for 1 h 47 min. At BACKUP STAGE BLOCK_DDL it failed:

      [00] 2026-10-18 01:57:47 Started BACKUP STAGE BLOCK_DDL
      [00] 2026-10-18 01:57:47 Acquired locks for BACKUP STAGE BLOCK_DDL
      [00] 2026-10-18 01:57:47 Getting InnoDB LSN
      [00] 2026-10-18 01:57:47 InnoDB LSN: 128396462925059, Flushing Logs
      [00] 2026-10-18 01:57:47 Flushed Logs
      ...
      [00] 2026-10-18 01:57:48 Finished backing up non-InnoDB tables and files
      [00] 2026-10-18 01:57:48 Waiting for log copy thread to read lsn 128396462925059
      [00] 2026-10-18 01:57:50 Retrying read of log at LSN=128394766232720
      [00] 2026-10-18 01:57:53 Was only able to copy log from 128391090802399 to 128394766232720, not 128396462925059; try increasing innodb_log_file_size
      mariabackup: Stopping log copying thread.[00] 2026-10-18 01:57:53 Retrying read of log at LSN=128394766232720

      Why this is not a redo log capacity problem

      • The whole span the backup had to copy, from 128391090802399 to 128396462925059, is about 5.4 GB. The circular redo log is 16 GB.
      • The log copy thread stopped at 128394766232720, about 1.7 GB short of the target LSN. It kept retrying the read at that same LSN.
      • Measured redo generation on this server is about 28-37 MB/min, which matches about 5 GB over the 107-minute backup. The log could not have wrapped.
      • Nothing relevant appears in the server error log around that time.

      Frequency
      Intermittent. Earlier weekly full backups on the same server, version and configuration
      completed successfully. A rerun on 2026-10-21 is in progress.

      Impact
      Because the full backup has no mariadb_backup_checkpoints, every following incremental backup
      of that week has no base and fails. One failed run therefore leaves the server without a new
      backup until a full backup is taken by hand.

      Related
      MDEV-36159 (closed, Fixed in 11.4.9), MDEV-36237 (11.4.4 / 11.4.5, closed as Incomplete, duplicate of MDEV-36159), MDEV-38396 (10.11.15, Needs Feedback), MDEV-37964, MDEV-34062.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              gokhan Gokhan Demir
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.