Implement innodb_log_archive

The InnoDB write-ahead log file (ib_logfile0) is pre-allocated to innodb_log_file_size and written as a ring buffer. This is good for write performance and space management, but unsuitable for arbitrary point-in-time recovery or for facilitating incremental backup.

As noted in MDEV-14992, it would be better if the server took care of saving a sufficient amount of InnoDB write-ahead log to support backup. The SQL interface with a backup program would be something like the following:

-- At the start of the backup, we check if log archiving is already running.

SELECT @@GLOBAL.innodb_log_archive, variable_value

FROM information_schema.global_status

WHERE variable_name = 'INNODB_LSN_ARCHIVED';

-- If BETWEEN 1 AND the LSN of the previous backup, skip copying InnoDB files (incremental backup)

-- Else if necessary, request the log to be archived from the current LSN onwards.

SET GLOBAL innodb_log_archive=ON;

-- In this case, at the end of the backup, we would disable the log archiving again:

SET GLOBAL innodb_log_archive=OFF;

Log archiving could also be enabled indefinitely, to facilitate arbitrary point-in-time recovery of anything that is covered by the InnoDB write-ahead log. When MDEV-34705 is implemented and enabled, this would include the binlog. However, point-in-time recovery of DDL operations would not work without limitations even for an InnoDB-only deployment, because part of the data dictionary is stored in .frm files, whose creation is covered by a separate log (see MDEV-17567).

The default value of innodb_log_archive is OFF, meaning that the log archiving is disabled.

When innodb_log_archive=ON, changes of the parameter innodb_log_file_size will take place when the current log file is about to be filled up and a new file is being created and allocated. The log resizing logic (MDEV-27812) as well as the creation of a redundant log file ib_logfile101 will remain in use when innodb_log_archive=OFF.

Log archiving will maintain an append-only "manifest" file ib_manifest.log, with contents similar to the following:

In the above example, the server would apparently have been initialized with the setting innodb_log_file_size=1G, because the first and the second archived log file are that many LSN apart. The server bootstrap completed at LSN 50000, and the first checkpoints of the subsequently started server were 305441471 and 1074389568. The latter entry is starting 0x9e240 bytes into the second archived log file. At the start of each archived log file, we’d also record a pointer to this first checkpoint, so that it will be possible to reconstruct the ib_manifest.log from individual log files.

The append-only file ib_manifest.log will be in a user-readable format. We are not too worried about corruption due to a file system crash or human error, because this file may is optional and may be reconstructed from individual ib_*.log files.

For efficient recovery from archived log files instead of ib_logfile0, two further start-up parameters will be introduced:

A backup client may write these into a .cnf file or pass these in a command line when invoking mariadbd. The idea of these parameters is to limit the scope of recovery and to avoid replaying an archived log from the very beginning to the very end.

We must keep in mind that the creation and modification of some files, such as .frm files that form part of the data dictionary, are not covered by the InnoDB write-ahead log. Therefore, it is important to be able to stop the recovery at a specific LSN.