[MDEV-30026] incremental backup creates broken files if there is a high load during backup Created: 2022-11-17  Updated: 2023-10-02  Resolved: 2023-10-02

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.6.11
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Timofey Turenko Assignee: Timofey Turenko
Resolution: Not a Bug Votes: 0
Labels: None

Attachments: File backup.tar.gz    
Issue Links:
Relates
relates to MDEV-14992 BACKUP: in-server backup Open
relates to MDEV-29685 Incremental backup does not work Closed

 Description   

test scenario:

1. create 1M file (e.g file1M.bin), file can contain zeros or random data
2. start thread which executes
`INSERT INTO Test

{i}

VALUES (LOAD_FILE('/tmp/file1M.bin'))`
in the loop

3. in another thread:

create table (e.g `CREATE TABLE table1 (i INT, s VARCHAR(1024))`) and put some number of rows into it (in my test 10000)

make a first part of backup:
`sudo mariadb-backup -utest_user -pct570c3521fCCR# --backup --target-dir=backup/base_backup_dir `

4. create another table, put data into it, make a next part of the backup:
`sudo mariadb-backup -utest_user -pct570c3521fCCR# --backup --incremental-basedir=backup/base_backup_dir --target-dir=backup/inc_backup_dir_2 `

5. repeat step 4 several times

6. try to prepare backup for restore:

sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir
 
sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_2 
sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_3 
....

Result:
two first parts are ok, 3rd failed:

sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_3 
mariadb-backup based on MariaDB server 10.6.11-MariaDB debian-linux-gnu (x86_64)
 
[00] 2022-11-17 15:28:47 incremental backup from 1311892709 is enabled.
 
[00] 2022-11-17 15:28:47 cd to /home/ubuntu/backup_prepare/base_backup_dir/
 
[00] 2022-11-17 15:28:47 open files limit requested 0, set to 1024
 
[00] 2022-11-17 15:28:47 This target seems to be already prepared.
 
[00] 2022-11-17 15:28:47 error: This incremental backup seems not to be proper for the target. Check 'to_lsn' of the target and 'from_lsn' of the incremental.

attached backup.tar.gz - files created by mariadb-backup

The error looks similar to MDEV-29685, which is reproducible only for 10.3 and 10.4



 Comments   
Comment by Marko Mäkelä [ 2022-11-19 ]

Is this simply because the backup process cannot keep up with the server? That is, would it work if you configured the server with a much larger innodb_log_file_size?

Comment by Timofey Turenko [ 2022-11-24 ]

`innodb_log_file_size=1073741824` does not help

Comment by Timofey Turenko [ 2023-10-02 ]

after detailed re-testing: under load backup can be extremely slow and it can cause ssh connection disconnection (ssh between test machine and mariadb-backup/server machine)

now test is changed, it can survive very big delays. Closing this issue.

Generated at Thu Feb 08 10:13:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.