Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30026

incremental backup creates broken files if there is a high load during backup

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Not a Bug
    • 10.6.11
    • N/A
    • Backup
    • None

    Description

      test scenario:

      1. create 1M file (e.g file1M.bin), file can contain zeros or random data
      2. start thread which executes
      `INSERT INTO Test

      {i}

      VALUES (LOAD_FILE('/tmp/file1M.bin'))`
      in the loop

      3. in another thread:

      create table (e.g `CREATE TABLE table1 (i INT, s VARCHAR(1024))`) and put some number of rows into it (in my test 10000)

      make a first part of backup:
      `sudo mariadb-backup -utest_user -pct570c3521fCCR# --backup --target-dir=backup/base_backup_dir `

      4. create another table, put data into it, make a next part of the backup:
      `sudo mariadb-backup -utest_user -pct570c3521fCCR# --backup --incremental-basedir=backup/base_backup_dir --target-dir=backup/inc_backup_dir_2 `

      5. repeat step 4 several times

      6. try to prepare backup for restore:

      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir
      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_2 
      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_3 
      ....
      

      Result:
      two first parts are ok, 3rd failed:

      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_3 
      mariadb-backup based on MariaDB server 10.6.11-MariaDB debian-linux-gnu (x86_64)
      [00] 2022-11-17 15:28:47 incremental backup from 1311892709 is enabled.
      [00] 2022-11-17 15:28:47 cd to /home/ubuntu/backup_prepare/base_backup_dir/
      [00] 2022-11-17 15:28:47 open files limit requested 0, set to 1024
      [00] 2022-11-17 15:28:47 This target seems to be already prepared.
      [00] 2022-11-17 15:28:47 error: This incremental backup seems not to be proper for the target. Check 'to_lsn' of the target and 'from_lsn' of the incremental.
      

      attached backup.tar.gz - files created by mariadb-backup

      The error looks similar to MDEV-29685, which is reproducible only for 10.3 and 10.4

      Attachments

        Issue Links

          Activity

            Is this simply because the backup process cannot keep up with the server? That is, would it work if you configured the server with a much larger innodb_log_file_size?

            marko Marko Mäkelä added a comment - Is this simply because the backup process cannot keep up with the server? That is, would it work if you configured the server with a much larger innodb_log_file_size ?

            `innodb_log_file_size=1073741824` does not help

            tturenko Timofey Turenko added a comment - `innodb_log_file_size=1073741824` does not help

            after detailed re-testing: under load backup can be extremely slow and it can cause ssh connection disconnection (ssh between test machine and mariadb-backup/server machine)

            now test is changed, it can survive very big delays. Closing this issue.

            tturenko Timofey Turenko added a comment - after detailed re-testing: under load backup can be extremely slow and it can cause ssh connection disconnection (ssh between test machine and mariadb-backup/server machine) now test is changed, it can survive very big delays. Closing this issue.

            People

              tturenko Timofey Turenko
              tturenko Timofey Turenko
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.