Details
-
Bug
-
Status: Needs Feedback (View Workflow)
-
Major
-
Resolution: Unresolved
-
11.4.12
-
None
-
None
-
Rocky Linux 10.2 (kernel 6.12.0-211.26.1.el10_2), MariaDB-server/MariaDB-backup 11.4.12-1.el10, XFS, innodb_log_file_size=16G
Description
A scheduled full backup with mariadb-backup 11.4.12 failed at the end of the copy phase with
the "try increasing innodb_log_file_size" error. The redo log was nowhere near full during the
backup, and the defaults introduced by the MDEV-36159 fix (fixed in 11.4.9) are active. So it
looks like that fix does not cover every case. MDEV-36237 (11.4.4, 11.4.5) and MDEV-37964 were
closed as duplicates of MDEV-36159 with resolution "Incomplete". MDEV-38396 reports the same
failure on 10.11.15, which also contains the fix.
Environment
- MariaDB-server and MariaDB-backup 11.4.12-1.el10 (official MariaDB repository), Rocky Linux 10.2, kernel 6.12.0-211.26.1.el10_2.x86_64
- 24 vCPU, 251 GB RAM, XFS datadir (~1.3 TB of InnoDB data; compressed full backup ~180 GB)
- innodb_log_file_size = 16G, innodb_log_buffer_size = 64M, innodb_buffer_pool_size = 180G
- innodb_flush_method = O_DIRECT, innodb_flush_log_at_trx_commit = 1
- innodb_log_file_mmap = OFF, innodb_log_file_buffering = OFF, innodb_log_write_ahead_size = 512
- innodb_io_capacity = 200, innodb_max_dirty_pages_pct_lwm = 0
- kernel.io_uring_disabled = 2, so mariadb-backup falls back to libaio
- mariadb-backup's own defaults, as printed by --help --verbose: innodb-log-file-mmap FALSE, innodb-log-checkpoint-now FALSE
- Command (streamed):
mariadb-backup --defaults-extra-file=... --backup --stream=mbstream --extra-lsndir=<dir> | zstd -3 -T<n> > backup.mbstream.zst
What happened
The full backup started at 00:10:03 and streamed all tablespaces for 1 h 47 min. At BACKUP STAGE BLOCK_DDL it failed:
[00] 2026-10-18 01:57:47 Started BACKUP STAGE BLOCK_DDL
|
[00] 2026-10-18 01:57:47 Acquired locks for BACKUP STAGE BLOCK_DDL
|
[00] 2026-10-18 01:57:47 Getting InnoDB LSN
|
[00] 2026-10-18 01:57:47 InnoDB LSN: 128396462925059, Flushing Logs
|
[00] 2026-10-18 01:57:47 Flushed Logs
|
...
|
[00] 2026-10-18 01:57:48 Finished backing up non-InnoDB tables and files
|
[00] 2026-10-18 01:57:48 Waiting for log copy thread to read lsn 128396462925059
|
[00] 2026-10-18 01:57:50 Retrying read of log at LSN=128394766232720
|
[00] 2026-10-18 01:57:53 Was only able to copy log from 128391090802399 to 128394766232720, not 128396462925059; try increasing innodb_log_file_size
|
mariabackup: Stopping log copying thread.[00] 2026-10-18 01:57:53 Retrying read of log at LSN=128394766232720
|
Why this is not a redo log capacity problem
- The whole span the backup had to copy, from 128391090802399 to 128396462925059, is about 5.4 GB. The circular redo log is 16 GB.
- The log copy thread stopped at 128394766232720, about 1.7 GB short of the target LSN. It kept retrying the read at that same LSN.
- Measured redo generation on this server is about 28-37 MB/min, which matches about 5 GB over the 107-minute backup. The log could not have wrapped.
- Nothing relevant appears in the server error log around that time.
Frequency
Intermittent. Earlier weekly full backups on the same server, version and configuration
completed successfully. A rerun on 2026-10-21 is in progress.
Impact
Because the full backup has no mariadb_backup_checkpoints, every following incremental backup
of that week has no base and fails. One failed run therefore leaves the server without a new
backup until a full backup is taken by hand.
Related
MDEV-36159 (closed, Fixed in 11.4.9), MDEV-36237 (11.4.4 / 11.4.5, closed as Incomplete, duplicate of MDEV-36159), MDEV-38396 (10.11.15, Needs Feedback), MDEV-37964, MDEV-34062.
Attachments
Issue Links
- duplicates
-
MDEV-39468 Is there a solution for mariadb-backup? I haven't been able to sync nodes or create a backup for the last 18 months on the 10.11 version.
-
- Closed
-
- relates to
-
MDEV-36237 mariabackup fails with 'try increasing innodb_log_file_size'
-
- Closed
-