I've had issues with mariabackup/xtrabackup/innobackupex from day one. Can't remember the last time I successfully backed up and restored using these tools.

Last couple of years mariabackup has been the tool I use, but it's never working. We always end up restoring from Veeam instead.

I'd love to resolve this once and for all. I can't believe the mariabackup version I have is completely broken. If it is, of course we'll have to plan an upgrade.

I've attached a transcript of my latest failed attempt at restoring incremental backups.

I've followed this document: https://mariadb.com/kb/en/incremental-backup-and-restore-with-mariabackup/

To create the base backup I run:
mariabackup --defaults-file="$mysql_defaults_file" --socket="$mysql_socket" --extra-lsndir="$backup_config_dir" --stream=xbstream --backup | ssh

On one of the master DB nodes.

To create subsequent incrementals I run the same command but with;
--backup --incremental-lsn="$to_lsn" arguments appended

$to_lsn is extracted from;
checkpoints_file="${backup_config_dir}/xtrabackup_checkpoints"
to_lsn="$(grep ^to_lsn "$checkpoints_file"|cut -d= -f2|tr -d ' ')"

This appears to work successfully for many days.

It's only the restore that gives me errors. See the attached transcript for those details.

Note that in the transcript I attempted a solution I found here: https://tutel.me/c/dba/questions/184185/percona+xtrabackup+transaction+log+corrupted
but it did not help.