Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30026

incremental backup creates broken files if there is a high load during backup

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Not a Bug
    • 10.6.11
    • N/A
    • Backup
    • None

    Description

      test scenario:

      1. create 1M file (e.g file1M.bin), file can contain zeros or random data
      2. start thread which executes
      `INSERT INTO Test

      {i}

      VALUES (LOAD_FILE('/tmp/file1M.bin'))`
      in the loop

      3. in another thread:

      create table (e.g `CREATE TABLE table1 (i INT, s VARCHAR(1024))`) and put some number of rows into it (in my test 10000)

      make a first part of backup:
      `sudo mariadb-backup -utest_user -pct570c3521fCCR# --backup --target-dir=backup/base_backup_dir `

      4. create another table, put data into it, make a next part of the backup:
      `sudo mariadb-backup -utest_user -pct570c3521fCCR# --backup --incremental-basedir=backup/base_backup_dir --target-dir=backup/inc_backup_dir_2 `

      5. repeat step 4 several times

      6. try to prepare backup for restore:

      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir
      		 
      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_2 
      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_3 
      ....

      Result:
      two first parts are ok, 3rd failed:

      sudo mariadb-backup --prepare --target-dir=backup_prepare/base_backup_dir --incremental-dir=backup_prepare/inc_backup_dir_3 
      mariadb-backup based on MariaDB server 10.6.11-MariaDB debian-linux-gnu (x86_64)
      		 
      [00] 2022-11-17 15:28:47 incremental backup from 1311892709 is enabled.
      		 
      [00] 2022-11-17 15:28:47 cd to /home/ubuntu/backup_prepare/base_backup_dir/
      		 
      [00] 2022-11-17 15:28:47 open files limit requested 0, set to 1024
      		 
      [00] 2022-11-17 15:28:47 This target seems to be already prepared.
      		 
      [00] 2022-11-17 15:28:47 error: This incremental backup seems not to be proper for the target. Check 'to_lsn' of the target and 'from_lsn' of the incremental.

      attached backup.tar.gz - files created by mariadb-backup

      The error looks similar to MDEV-29685, which is reproducible only for 10.3 and 10.4

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. MDEV-14992 BACKUP: in-server backup

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29685 Incremental backup does not work

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            tturenko Timofey Turenko created issue -
            tturenko Timofey Turenko made changes -
            Field Original Value New Value
            tturenko Timofey Turenko made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä added a comment -

            Is this simply because the backup process cannot keep up with the server? That is, would it work if you configured the server with a much larger innodb_log_file_size?

            marko Marko Mäkelä added a comment - Is this simply because the backup process cannot keep up with the server? That is, would it work if you configured the server with a much larger innodb_log_file_size ?
            tturenko Timofey Turenko added a comment -

            `innodb_log_file_size=1073741824` does not help

            tturenko Timofey Turenko added a comment - `innodb_log_file_size=1073741824` does not help
            elenst Elena Stepanova made changes -
            Fix Version/s 10.6 [ 24028 ]
            tturenko Timofey Turenko made changes -
            Assignee Alexander Barkov [ bar ] Timofey Turenko [ tturenko ]
            tturenko Timofey Turenko added a comment -

            after detailed re-testing: under load backup can be extremely slow and it can cause ssh connection disconnection (ssh between test machine and mariadb-backup/server machine)

            now test is changed, it can survive very big delays. Closing this issue.

            tturenko Timofey Turenko added a comment - after detailed re-testing: under load backup can be extremely slow and it can cause ssh connection disconnection (ssh between test machine and mariadb-backup/server machine) now test is changed, it can survive very big delays. Closing this issue.
            tturenko Timofey Turenko made changes -
            issue.field.resolutiondate 2023-10-02 14:24:50.0 2023-10-02 14:24:49.604
            tturenko Timofey Turenko made changes -
            Fix Version/s N/A [ 14700 ]
            Fix Version/s 10.6 [ 24028 ]
            Resolution Not a Bug [ 6 ]
            Status Open [ 1 ] Closed [ 6 ]

            People

              tturenko Timofey Turenko
              tturenko Timofey Turenko
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.