XML

Word

Printable

Details

Type: New Feature
Status: Stalled (View Workflow)
Priority: Critical
Resolution: Unresolved
Fix Version/s: 13.2
Component/s: Backup
Labels:
- INNODB26

Epic Link:
In-Server Backup, phase 1
Sprint:
Q1/2026 Server Development, Q1/2026 Server Maintenance, Q2/2026 Server Maintenance, Q2/2026 Server Development, Q3/2026 Server Maintenance, Q3/2026 Server Development

Description

The purpose of this work is to improve current situation around backups by implementing a SQL command that makes a backup of the running server to a mounted directory path or to a tar stream (MDEV-38362):

BACKUP SERVER TO '/path/to/directory';

BACKUP SERVER TO '/path/to/directory' 1 CONCURRENT;

BACKUP SERVER WITH 'command';

BACKUP SERVER WITH 1 CONCURRENT 'command';

In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to.

For the second variant, 'command' must be the name of a script or command that will be executed in a child process. The standard input of that command will be in a format that is compatible with GNU tar --format=oldgnu (and also BSD `tar` variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, each concurrent stream can write a separate file.

This statement will create the requested target directory and fill it with a consistent snapshot of the database. Additionally, the directory will contain a file backup.cnf, which includes some variables that are specific to restoring the backup.

To prepare a backup (analogous to mariadb-backup --prepare), MDEV-39061 introduces a script that executes roughly the following:

echo 'SELECT … INTO OUTFILE 'ib_logfile0';shutdown;'|mariadbd \

--defaults-file=/path/to/directory/backup.cnf \

--datadir=/path/to/directory --bootstrap

rm /path/to/directory/ib_*.log

That is, it will apply the backed up log and create an empty ib_logfile0 that corresponds to the final log sequence number (LSN). The … is based on this comment of ~~MDEV-27208~~.

This step is optional. Alternatively, one could just move the backup to its final storage location and start up MariaDB Server on it, with the correct innodb_log_recovery_start to avoid corruption. By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database!

We may assume that when log_bin is enabled, binlog_storage_engine=innodb (~~MDEV-34705~~) will be enabled as well. In this way, there should be no need to include a binlog position or GTID in the backup.cnf.

Initially only full backup

For the duration of the backup execution, the server will be configured with innodb_log_archive=ON (~~MDEV-37949~~). We will avoid copying any data files created after innodb_lsn_archived, because those files can be reconstructed based on log records.

Unlike mariadb-backup, the initial implementation of BACKUP SERVER only supports a full backup. Partial backup will be covered in MDEV-40163 and incremental backup in MDEV-39054 (log streaming) and MDEV-39089 (copying changed pages).

Storage engine interface

Most of the actual work is done by the storage engines. There are 3 new handlerton function pointers backup_start, backup_step, backup_end as well as new data types: backup_target to identify a backup target directory, backup_sink comprising the storage engine context and the target stream, and backup_phase (BACKUP_PHASE_FINISH, BACKUP_PHASE_ABORT, BACKUP_PHASE_PREPARE_START or one of the phases that are tightly coupled with backup lock modes, between BACKUP_PHASE_START for MDL_BACKUP_START and BACKUP_PHASE_NO_COMMIT for MDL_BACKUP_WAIT_COMMIT.

For most phases, handlerton::backup_start() and handlerton::backup_end() will be invoked in the thread in which the execution of the BACKUP SERVER statement started. Between these calls, handlerton::backup_step() may be invoked in multiple threads, corresponding to the CONCURRENT clause.

The currently implemented phases are as follows:

BACKUP_PHASE_START:
- innodb_backup_start(): enables innodb_log_archive=ON for the duration of the backup and initializes a work queue for innodb_backup_step()
- innodb_backup_step(): copies one InnoDB log or data file at a time
BACKUP_PHASE_NO_COMMIT:
- this is where ENGINE=RocksDB backup (MDEV-39091) would be implemented based on rocksdb_create_checkpoint
- innodb_backup_start(): determine the end LSN of the backup, and add the remaining log files to the work queue
- innodb_backup_step(): hard-link, copy or stream the remaining log files
- innodb_backup_end(): release the InnoDB resources (for example, restore innodb_log_archive=OFF)
- aria_backup_start(): traverses directories to collect file names
- aria_backup_end(): copies or streams all files in a loop (yes, in a single thread for now, while the server is maximally blocked!); to be improved in MDEV-39092
BACKUP_PHASE_FINISH: executed after releasing locks; another BACKUP SERVER may execute concurrently
- For ENGINE=RocksDB (MDEV-39091) this is where the entire backup would be streamed
- no special handling in innodb_backup_start()
- innodb_backup_step(): copy the remaining log files
- innodb_backup_end(): if the last log file was hard-linked, duplicate it; write or stream the backup.cnf containing essential information

The streaming backup (MDEV-38362) does not currently make use of any working directory for the backup. A log-based ENGINE=Aria backup should be possible with the use of some temporary delete-on-close files.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

xtrabackup.png
100 kB
2018-02-24 18:14

Issue Links

blocks

MDEV-39054 InnoDB-only, DML-only incremental backup

Open

MDEV-39055 Multi-threaded innodb_log_archive parsing

Open

MDEV-39061 mariadb-backup compatible wrappers for BACKUP SERVER

In Testing

MDEV-39089 Incremental BACKUP SERVER based on copying changed pages

Open

causes

MXS-6249 Research how to remote backup mariadb besides mysqldump

Open

includes

MDEV-37949 Implement innodb_log_archive

Closed

MDEV-38362 Develop an efficient alternative to mbstream

In Review

MDEV-39091 BACKUP SERVER of ENGINE=RocksDB to the local file system

In Review

MDEV-39092 BACKUP SERVER of ENGINE=Aria to the local file system

In Review

MDEV-39101 Make BACKUP SERVER mutually exclusive with itself and BACKUP STAGE

Closed

MDEV-39988 Copy non-ACID engine files in BACKUP SERVER

Stalled

is blocked by

MDEV-35248 in server backup: Research, scope & prototyping

Closed

MDEV-39122 Fix my_realpath to treat empty string as current directory

Closed

MDEV-39225 Assertion failures with innodb_log_file_size>4g

Closed

MDEV-39772 SET GLOBAL innodb_log_archive=OFF breaks recovery

Closed

MDEV-39861 Recovery with innodb_log_recovery_target wrongly opens log in read-write mode

Closed

MDEV-40063 Corruption due to race condition in SET GLOBAL innodb_log_archive

Closed

relates to

MDEV-22096 Mariabackup copied too old page or too new checkpoint

Closed

MDEV-23947 reflink support for mariabackup

Open

MDEV-27424 mariabackup ignores physically corrupt first pages

Closed

MDEV-27621 Backup fails with FATAL ERROR: Was only able to copy log from .. to .., not ..

Closed

MDEV-27812 Allow innodb_log_file_size to change without server restart

Closed

MDEV-28994 Backup produces garbage when using memory-mapped log (PMEM)

Closed

MDEV-33980 mariadb-backup --backup is missing retry logic for undo tablespaces

Closed

MDEV-34062 mariadb-backup --backup is extremely slow at copying ib_logfile0

Closed

MDEV-35791 mariadb-backup 10.11.10 failed to create backup

Closed

MDEV-36159 mariabackup failed after upgrade 10.11.10

Closed

MDEV-39541 mem_pressure::~mem_pressure() causes a crash on bootstrap

Closed

MDEV-5336 Implement BACKUP STAGE for safe external backups

Closed

MDEV-7502 Automatic provisioning of slave

Open

MDEV-13833 implement --innodb-track-changed-pages

Open

MDEV-14425 Change the InnoDB redo log format to reduce write amplification

Closed

MDEV-18336 Remove backup_fix_ddl() during backup

Open

MDEV-18985 Remove support for XtraDB's changed page bitmap from Mariabackup in 10.2+

Closed

MDEV-19492 Mariabackup hangs if table populated with INSERT... SELECT while it runs

Stalled

MDEV-19749 MDL scalability regression after backup locks

Closed

MDEV-21105 Port clone plugin API (MYSQL_CLONE_PLUGIN) from MySQL

Closed

MDEV-27551 mariabackup --backup aborts if a file is deleted during enumerate_ibd_files()

Open

MDEV-29115 mariabackup.mdev-14447 started failing in a new way in CIs

Closed

MDEV-30026 incremental backup creates broken files if there is a high load during backup

Closed

MDEV-31410 mariadb-backup prepare crash with InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT

Closed

MDEV-31446 mariabackup loop on Read redo log up to LSN

Open

MDEV-33367 FATAL ERROR: <time> Was only able to copy log from <seq_no> to <seq_no>, not <seq_no>; try increasing innodb_log_file_size

Closed

MDEV-36159 mariabackup failed after upgrade 10.11.10

Closed

PERF-444 Loading...

split to

MDEV-35248 in server backup: Research, scope & prototyping

Closed

mentioned in: Page Loading...

(6 includes, 6 is blocked by, 28 relates to, 1 split to, 1 mentioned in)

Activity

People

Assignee:: Marko Mäkelä

Reporter:: Vladislav Vaintroub

Votes:: 17 Vote for this issue

Watchers:: 43 Start watching this issue

Dates

Created:: 2018-01-19 07:53

Updated:: 4 days ago 14:35

Time Tracking

Estimated:

50d

Remaining:

6d 6h 50m

Logged:

43d 3h 25m

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.