Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24197

Prepare step of backup failing on large backup

Details

    Description

      Customer is running backup on very large dataset (18TB). Backup completes successfully, but fails during prepare step.
      Here is the planned backup process as the customer has detailed:

        1. On Source:
          mariabackup --user=root --backup --target-dir=/opt/database/backup --stream=xbstream --parallel=12 | pigz -c | socat - TCP-LISTEN:4444,reuseaddr
        2. On Target
          rm -rf /var/lib/mysql/*
          rm -rf /var/log/mysql/*
          cd /var/lib/mysql
          socat -u TCP:192.168.111.184:4444 STDOUT | pigz -dc | mbstream -x
          mariabackup --user=root --use-memory=100G --prepare --target-dir=/var/lib/mysql/
          rm -rf ib_logfile*
          chown -R mysql:mysql /var/lib/mysql/
          systemctl start mariadb

      The prepare fails as such:
      2020-11-03 15:08:46 0 [Note] InnoDB: To recover: 2407553 pages from log
      2020-11-03 15:08:54 0 [ERROR] InnoDB: Not applying INSERT_REUSE_REDUNDANT due to corruption on [page id: space=0, page number=120831]
      2020-11-03 15:08:54 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25328 --innodb command line option causes mariabackup to fail

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25031 Not applying INSERT_REUSE_REDUNDANT due to corruption on page

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-17289 Multi-pass recovery fails to apply some redo log records

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21724 Optimize page_cur_insert_rec_low() redo logging

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-24719 backport MDEV-24705 from 10.5 to 10.2, 10.3, 10.4

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25031 Not applying INSERT_REUSE_REDUNDANT due to corruption on page

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Task - A task that needs to be done. MDEV-24705 add check that LSN of the last skipped log record equals to FIL_PAGE_LSN field

          • Major - Major loss of function.
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            Transition Time In Source Status Execution Times
            Julien Fritsch made transition -
            Open In Progress
            		91d 13h 8m 1
            Marko Mäkelä made transition -
            In Progress Stalled
            		45d 23h 24m 1
            Marko Mäkelä made transition -
            Stalled In Progress
            		6m 15s 1
            Marko Mäkelä made transition -
            In Progress Closed
            		3d 2h 21m 1

            People

              marko Marko Mäkelä
              ivenn Isaac Venn (Inactive)
              Votes:
              7 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.