mariadb-backup --prepare with larger --use-memory may be slower

This came up while I was testing MDEV-33817. In one environment, preparing a backup (applying the ib_logfile0 to it) would be significantly faster when using a smaller --use-memory so that a multi-batch recovery will be needed. I could not reproduce exactly that on my local system, but I did reproduce a case where a smaller memory is faster.

I made a backup during a Sysbench workload, which resulted in a 6.2 GiB ib_logfile0 and 9.6 GiB total backup directory size (4.1 GiB after preparing the backup). When I used a large enough buffer pool so that the recovery can be completed in a single batch, it took 78 seconds:

2024-05-03 14:34:49 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=46878

2024-05-03 14:35:04 0 [Note] InnoDB: Read redo log up to LSN=1853421805

2024-05-03 14:35:19 0 [Note] InnoDB: Read redo log up to LSN=3570573075

2024-05-03 14:35:34 0 [Note] InnoDB: Read redo log up to LSN=5203858038

2024-05-03 14:35:48 0 [Note] InnoDB: End of log at LSN=6621000189

2024-05-03 14:35:53 0 [Note] InnoDB: To recover: 255860 pages

[00] 2024-05-03 14:36:07 Last binlog file , position 0

[00] 2024-05-03 14:36:07 completed OK!

I went on to test the mariadb-backup --prepare time with various buffer pool sizes:

All these were run under perf record on a 6.7.12-amd64 kernel compiled with GCC 13.2.0.

This test shows that recv_sys_t::garbage_collect() is a significant bottleneck for the multi-batch recovery case. We should also make better use of resources. On a system with 144 CPU cores, we only seemed to be employing less than 3 of them on the average. While MDEV-29911 was a step into the right direction, we will need to do more.