Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24197

Prepare step of backup failing on large backup

Details

    Description

      Customer is running backup on very large dataset (18TB). Backup completes successfully, but fails during prepare step.
      Here is the planned backup process as the customer has detailed:

        1. On Source:
          mariabackup --user=root --backup --target-dir=/opt/database/backup --stream=xbstream --parallel=12 | pigz -c | socat - TCP-LISTEN:4444,reuseaddr
        2. On Target
          rm -rf /var/lib/mysql/*
          rm -rf /var/log/mysql/*
          cd /var/lib/mysql
          socat -u TCP:192.168.111.184:4444 STDOUT | pigz -dc | mbstream -x
          mariabackup --user=root --use-memory=100G --prepare --target-dir=/var/lib/mysql/
          rm -rf ib_logfile*
          chown -R mysql:mysql /var/lib/mysql/
          systemctl start mariadb

      The prepare fails as such:
      2020-11-03 15:08:46 0 [Note] InnoDB: To recover: 2407553 pages from log
      2020-11-03 15:08:54 0 [ERROR] InnoDB: Not applying INSERT_REUSE_REDUNDANT due to corruption on [page id: space=0, page number=120831]
      2020-11-03 15:08:54 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25328 --innodb command line option causes mariabackup to fail

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25031 Not applying INSERT_REUSE_REDUNDANT due to corruption on page

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-17289 Multi-pass recovery fails to apply some redo log records

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21724 Optimize page_cur_insert_rec_low() redo logging

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-24719 backport MDEV-24705 from 10.5 to 10.2, 10.3, 10.4

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25031 Not applying INSERT_REUSE_REDUNDANT due to corruption on page

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Task - A task that needs to be done. MDEV-24705 add check that LSN of the last skipped log record equals to FIL_PAGE_LSN field

          • Major - Major loss of function.
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            Ascending order - Click to sort in descending order
            View 7 older comments
            marko Marko Mäkelä added a comment -

            For 10.5, this could have been fixed by MDEV-25031, which fixed a regression that was introduced in 10.5.
            For 10.2.27, I have no idea of the cause.

            marko Marko Mäkelä added a comment - For 10.5, this could have been fixed by MDEV-25031 , which fixed a regression that was introduced in 10.5. For 10.2.27, I have no idea of the cause.
            marko Marko Mäkelä added a comment -

            Pull request #1772 would implement innodb_force_recovery=1 for mariabackup --prepare, and I think solve this issue.

            marko Marko Mäkelä added a comment - Pull request #1772 would implement innodb_force_recovery=1 for mariabackup --prepare , and I think solve this issue.
            marko Marko Mäkelä added a comment -

            The contributed fix is basically only a work-around: it allows the option --innodb-force-recovery=1 to be specified when applying logs (preparing a backup).

            The root cause of the failure is always something else (such as MDEV-25031 for the case that is in the Description). This work-around allows an attempt to restore a backup even if some corruption is present.

            marko Marko Mäkelä added a comment - The contributed fix is basically only a work-around: it allows the option --innodb-force-recovery=1 to be specified when applying logs (preparing a backup). The root cause of the failure is always something else (such as MDEV-25031 for the case that is in the Description). This work-around allows an attempt to restore a backup even if some corruption is present.
            rpizzi Rick Pizzi (Inactive) added a comment -

            marko will the backup still be consistent when restoring with force-recovery=1? especially regarding the position resp. the master (in case we want to make this server a replica)?

            rpizzi Rick Pizzi (Inactive) added a comment - marko will the backup still be consistent when restoring with force-recovery=1? especially regarding the position resp. the master (in case we want to make this server a replica)?
            marko Marko Mäkelä added a comment -

            The purpose of innodb_force_recovery=1 is to ignore some errors (corrupted pages or missing data files). Depending on where the errors occur, it may or may not be safe. In the general case, I would not recommend to continue using a server where recovery errors had been ignored in the past.

            When it comes to MariaDB Server (not Mariabackup), it would be nice if there was a way to set innodb_force_recovery to the values 2 or 3 in such a way that the recovery will run in strict mode. If no crash recovery is involved, those values of the parameter are not expected to cause further corruption. The main purpose of innodb_force_recovery is to allow data to be extracted from a corrupted database.

            marko Marko Mäkelä added a comment - The purpose of innodb_force_recovery=1 is to ignore some errors (corrupted pages or missing data files). Depending on where the errors occur, it may or may not be safe. In the general case, I would not recommend to continue using a server where recovery errors had been ignored in the past. When it comes to MariaDB Server (not Mariabackup), it would be nice if there was a way to set innodb_force_recovery to the values 2 or 3 in such a way that the recovery will run in strict mode. If no crash recovery is involved, those values of the parameter are not expected to cause further corruption. The main purpose of innodb_force_recovery is to allow data to be extracted from a corrupted database.

            People

              marko Marko Mäkelä
              ivenn Isaac Venn (Inactive)
              Votes:
              7 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.