When we run mariabackup on a galera cluster nodes with a lot of data/databases (over 1500 databases with 700GB of data at the moment), if the servers are under moderate use mariabackup tool fails with the following log messages:

Aug 21 17:44:51 maria3 -innobackupex-backup[25752]: 2018-08-21 17:44:51 140002715002816 [Note] InnoDB: Read redo log up to LSN=1691316337664

Aug 21 17:44:51 maria3 -innobackupex-backup[25752]: mariabackup: Error: xtrabackup_copy_logfile() failed.

Aug 21 17:44:52 maria3 -wsrep-sst-donor[25825]: mariabackup finished with error: 1.  Check /data/maria//innobackup.backup.log

Aug 21 17:44:52 maria3 -wsrep-sst-donor[25826]: Cleanup after exit with status:22

The cluster is under load, but I think these are happening when schema is being modified (databases created, dropped and so on). It could be related to tables flooded with records, because when we create databases we populate them with a lot of records.

This happens both when running mariabackup manually or when it's triggered by SST.

The log file mentioned in the message "innobackup.backup.log" cannot be found anywhere on the system. So we have no more info about the error.

Unfortunately, this is a production cluster and we can't do much in a way of debugging on these machines.