Implement innodb_log_archive_file_size, innodb_log_archive_path

The InnoDB write-ahead log file (ib_logfile0) is pre-allocated to innodb_log_file_size and written as a ring buffer. This is good for write performance and space management, but unsuitable for arbitrary point-in-time recovery or for facilitating incremental backup.

As noted in MDEV-14992, it would be better if the server took care of saving a sufficient amount of InnoDB write-ahead log to support backup. The SQL interface with a backup program would be something like the following:

-- At the start of the backup, we check if log archiving is already running.

SELECT @@GLOBAL.innodb_log_archive_file_size, variable_value

FROM information_schema.global_status

WHERE variable_name = 'INNODB_LSN_ARCHIVED';

-- If BETWEEN 1 AND the LSN of the previous backup, skip copying InnoDB files (incremental backup)

-- Else, request the log to be archived (in append-only mode, from the current LSN onwards).

SET GLOBAL innodb_log_archive_file_size=12288;

-- In this case, at the end of the backup, we would disable the log archiving again:

SET GLOBAL innodb_log_archive_file_size=0;

Log archiving could also be enabled indefinitely, to facilitate arbitrary point-in-time recovery of anything that is covered by the InnoDB write-ahead log. When MDEV-34705 is implemented and enabled, this would include the binlog. However, point-in-time recovery of DDL operations would not work without limitations even for an InnoDB-only deployment, because part of the data dictionary is stored in .frm files, whose creation is covered by a separate log (see MDEV-17567).

The default value of innodb_log_archive_file_size is 0, meaning that the log archiving is disabled.

If innodb_log_archive_file_size=12288 (the special value that corresponds to the size of a log header), the archived log will be appended to a single file, similar to the ib_logfile0 that mariadb-backup --backup is generating today. The file name will correspond to the new status variable innodb_lsn_archived, such as ib_000000001234abcd.log if the current LSN at the time the append-only log archive was enabled was 0x1234abcd or 305,441,471.

If innodb_log_archive_file_size is set to a larger nonzero value, that value will be the preallocated size of a log file snippet. Each preallocated log file will be updated in place until the size is reached and we switch to the next file. When innodb_log_archive_file_size is changed between these larger nonzero values, the change will not have impact until the current log archive file runs out of space.

Log archiving will maintain an append-only "manifest" file ib_manifest.log, with contents similar to the following:

In the above example, the server would apparently have been initialized with the setting innodb_log_archive_file_size=1G, because the first and the second archived log file are that many LSN apart. The server bootstrap completed at LSN 50000, and the first checkpoints of the subsequently started server were 305441471 and 1074389568. The latter entry is starting 0x9e240 bytes into the second archived log file. At the start of each archived log file, we’d also record a pointer to this first checkpoint, so that it will be possible to reconstruct the ib_manifest.log from individual log files.

The append-only file ib_manifest.log will be in a format that facilitates editing by a DBA, such as CSV. We are not too worried about corruption due to a file system crash or due to human error, because this file may is optional and may be reconstructed from individual ib_*.log files.

For efficient recovery from archived log files instead of ib_logfile0, two further start-up parameters will be introduced:

A backup client may write these into a .cnf file or pass these in a command line when invoking mariadbd. The idea of these parameters is to limit the scope of recovery and to avoid replaying an archived log from the very beginning to the very end.

We must keep in mind that the creation and modification of some files, such as .frm files that form part of the data dictionary, are not covered by the InnoDB write-ahead log. Therefore, it is important to be able to stop the recovery at a specific LSN.