Fedora Server 27, Galera cluster with 3 nodes, DB size ~ 1.4TB

There is not a big difference in Mariabackup from 10.2 to 10.3, apart from necessary code changes to make it compilable due to innodb refactoring.

(you can compare extra/mariabackup subtrees in the source code)

To exclude the external factors, such as networking, can you peterw17 backup the same dataset in 10.2 and 10.3, locally, i,e without --stream, and measure the difference
As well as --prepare ,and measure the difference,

Hi Vladislav,
Thank you for the quick answer. We don't have a 10.2 installation after upgrading it to 10.3. So I will do a local backup of the 10.3 tonight and send you the log file tomorrow.

Hi Vladislav,
the local backup with 10.3 is finished now and took about 6 hours so it takes about twice the time as the 10.2. Is it possible that the --use-memory parameter doesn't work correctly?

what takes more time as compared to 10.2 --backup. or --prepare?
--use-memory has no effect on --backup, because it does not use bufferpool

The --backup took much longer, the --prepare not so much. So it's not --use-memory.

can you attach the exact command line, server's my.cnf file.

There are 3 additional files attached now. The my.cnf of the DB server, backup-DB.sh which runs on the backup server and ssh-starts the backup-mysql.sh script on one of the DB nodes.

I switched back to our original backup setup with --stream. And the --prepare part takes much longer with --stream than it did during the local backup I did for testing. So maybe the MariaDB client software is slowing the backup? There is no my.cnf configuration for the client on the backup server and I think there was was none before the update to 10.3 too. Are there chances to improve the client configuration?

--stream has nothing to do with mariadb client software. stream is storing output to the stdout, not to a directory.
Mariadb client is used only minimalistically, there is no more than maybe a couple of queries during backup.

If there is something that slows down things, it is can be anything on your network, because people like to combine stream with netcat etc.
slower streaming leads to slower backup . slower backup leads to larger redo log. larger redo log leads to increased recovery times. increased recovery times lead to slower prepare.

So the real problem is the --backup part which takes in 10.3 twice the time than in 10.2 for us. And this is the same with --stream or local backup. We only made the update from 10.2 to 10.3 and didn't change anything else in the configs of the DB, OS and network. Could you find anything suspect in the attached config and log files?

To be honest, since there is not enough information I could gather from this bug report. There is no "how to reproduce" info, some speculations about --stream, and "it is slow for me" section.As I mentioned, there is no difference between 10.2 and 10.3.
mariabackup's parameter innodb_checksum_algorithm changed from "innodb" to "crc32" (in 10.2.16 / 10.3.8 due to MDEV-16457), and this could affect performance, but you would have to check it for yourself, what happens if you set the parameter back.

If you would like further help, I think you'd need to add "how to reproduce" section, or collect profiler info. Also, we would need the information on the size of the backup (the data, and the redo log). The current bug description is not very actionable.

For the moment I have no additional information for you but one last question. If you compare the beginnings of the attached logs 10_2_14.log and 10_3_8.log there are in 10.3 many more "log scanned up to ..." lines than in 10.2. Is it possible that the size(s) of some buffer(s) should be increased? I tried --use-memory and --innodb-buffer-pool-size, both didn't change this. Do you have any other suggestions for increasings?

I gave the suggestion last time , changing the innodb_checksum_algorithm in mariabackup, to innodb and see if it fares better. Because the default has changed, and if you happen to have lot of pages checksummed with old checksum algorithm (which is possible if you upgraded e.g from the 10.1), then
the new default will slow down checksumming process, which is relatively time consuming. Maybe your case is backwards to MDEV-16457, which was solved correctly to favor new 10.2 style CRC32 page checksums.

"log scanned up to.." is given by the redo log copying thread periodically, as a progress message.
.I do not think size of buffers need to be increased. I'm not even aware of any large buffers during --backup, except the ones that are used for copying files, and those have not changed. innodb buffer pool has nothing to do during --backup.

you can use --parallel to speed up the backup processing. You can backup when there is not a lot of write activity. I do not have suggestion other than that for the moment. a bug database is not right place for suggestions, you can try mailing list for example, this is better.

winstone, could you try to reproduce "very slow" mariabackup in 10.3 vs 10.2? If you do not find anything, feel free to close this one

peterw17
hello Peter,
you mentioned that you deploy 3 Galera Nodes, but you have 5 Nodes in configuration.Can you check wsrep_cluster_size variable ?

Hello winstone,
running SHOW GLOBAL STATUS LIKE 'wsrep_%'; I get:

.
.

.
.

The wsrep_incoming_addresses fit to "wsrep_cluster_address=gcomm://10.0.50.240,10.0.50.241,10.0.50.242" in my.cnf. Is there any other configuration for the cluster members that I missed?

peterw17 ,
can you please send the output of the du --summarize /var/lib/mysql/
and
du --summarize $BACKUPDIR/$1/$WEEKDAY
from your backups from versions 10_2_14 and 10_3_8 ?

Hello winstone,
because our servers were upgraded from 10_2_14 to 10_3_8, I don't have a 10_2_14 anymore. For 10_3_8 I get for the backup last night:

root@dbcluster02:/var/backup/DB/db02#du --summarize Tue/
1542557240 Tue/

and the current disk usage of the DB is:

root@dbcluster02:/var/backup/DB/db02#du --summarize /var/lib/mysql/
1551101564 /var/lib/mysql/

We had a similar issue with mysqldump being 2-3 times slower after the upgrade (issue https://jira.mariadb.org/browse/MDEV-14772 ). The fix was to use "stdbuf -o 10MB" in front of the backup command.

Hello Stijn,
thank you very much for your hint. Unfortunately the "stdbuf -o 10MB" did not speed up our backup, it took nearly the same time as before.

hello stijn, excuse me for delayed feedback. You are operating evidently with increasing data volumes. I intended to try to reproduce with loaded data near to you size,
but unfortunately on environment with network bottlenecks, and in addition such huge data load is performing very slowly.
So, I started measuring mariabackup time on smaller size DBs.
From test results however it's seen that there is not diffrence in times when get the backup with or without --stream=xbstream option ,
but if run backup with options --rsync --parallel= the performance time become several times better .
I checked for regression on the best time, and it's was not seen .

 10.2.14-MariaDB  

 mariabackup  on 80G

*+Backup  without stream option+*          Duration: 1 hours 29 minutes 15 seconds

Prepare                                Duration: 0 hours 0 minutes 1 seconds

*+Backup  with --stream=xbstream option+*

                                                       Duration: 1 hours 29 minutes 25 seconds

*+mdbstrem  operation  +*            Duration: 0 hours 26 minutes 47 seconds

+*Backup done  with options --rsync  --parallel=6*+   Duration: 0 hours 39 minutes 13 seconds

 10.3.10-MariaDB 

+*Backup done  with options --rsync  --parallel=6*+  

Backup done                  Duration: 0 hours 35 minutes 3 seconds

Backup prepare done     Duration: 0 hours 0 minutes 2 seconds

I'm going to close this one, because nobody could localize the problem. Backup time in your case can depend on many factors, such as network load and database load.

Check if something will change if you use --innodb_checksum_algorithm=innodb (for mariabackup), it could have an impact since MDEV-16457 changed the default. use --parallel to speedup the backup time. Backup when database load is low. Those are generic tips, since there is nothing specific known about your case. And to know anything specific, we'd need some kind of performance data, which you maybe could collect with perf (while doing on-disk backup, to isolate it from external networking problems). IF you have some performance data, you can attach it to the bug report, and we can look further.

As I mentioned, there is practically no difference in backup from 10.2 to 10.3, so the difference you see will mostly likely be something external.