[MDEV-21048] Mariabackup: huge speed degradation Created: 2019-11-14  Updated: 2023-08-02

Status: Open
Project: MariaDB Server
Component/s: mariabackup
Affects Version/s: 10.3.18
Fix Version/s: 10.3

Type: Bug Priority: Major
Reporter: Vadim Kalsin Assignee: Vladislav Lesin
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Debian 10.1

Attachments: PNG File image-2019-11-14-10-38-28-508.png    

 Description   

Hello

We are trying to make a backup to setup one more slave, and we are using the `mariabackup` for this. We do not have enough space on the first slave server, so we are using a streaming (xbstream) backup to second slave. The database is about 1.5TB.
The problem is: the backup process starts without problems. But after some hours the backup speed reduced hundreds of times. It may takes days!
To exclude problems from the `mbstream` side, we connected the NFS share and started up without xbstream. And we saw the same picture:

The first backup was started with the command:

mariabackup --backup --slave-info --safe-slave-backup --user=root --target-dir=/var/lib/mysql-nfs --rsync --parallel=8 2> dump-nfs.log

When we found that the rate has fallen, we restart the process with some small changes:

mariabackup --backup --slave-info --safe-slave-backup --innodb_checksum_algorithm=innodb --user=root --target-dir=/var/lib/mysql-nfs --rsync --parallel=8 2> dump-nfs.log

As you can see, the backup process begins at a good speed, but after a while the speed drops a hundred times.
May it be related to the InnoDB table compression, which we are using? How can we avoid this behavior?

 Comments   
Comment by Jonas Krauss [ 2023-08-02 ]

We are seeing similar issues, if they are related I cannot tell. The pattern is that the backup starts fast and quickly falls down to a rate of a couple of 100 kilobytes per second which makes taking a backup of 3T impossible.

My best guess is that it gets stuck at some InnoDB table at some point. We are using a mix of InnoDB and RocksDB storage engines but it never even gets to the point where RocksDB data is being processed.

I can provide additional information if someone let's me know what would be necessary to drill down on this issue. The log does not show anything suspicious, it just keeps printing lines how much of the innodb tx log it has scanned.

Below are basic information of our setup:

Environment:
Ubuntu 20.04.1 LTS

$ mariabackup --version
mariabackup based on MariaDB server 10.6.12-MariaDB debian-linux-gnu (x86_64)

$ mariadb --version
mariadb Ver 15.1 Distrib 10.6.12-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

Command:
mariabackup --backup --target-dir="$TARGET_DIR" --no-backup-locks --no-lock --parallel=2 --tables-exclude='...' > "$LOGFILE" 2>&1

Generated at Thu Feb 08 09:04:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.