Details
Description
Customer is running backup on very large dataset (18TB). Backup completes successfully, but fails during prepare step.
Here is the planned backup process as the customer has detailed:
-
- On Source:
mariabackup --user=root --backup --target-dir=/opt/database/backup --stream=xbstream --parallel=12 | pigz -c | socat - TCP-LISTEN:4444,reuseaddr - On Target
rm -rf /var/lib/mysql/*
rm -rf /var/log/mysql/*
cd /var/lib/mysql
socat -u TCP:192.168.111.184:4444 STDOUT | pigz -dc | mbstream -x
mariabackup --user=root --use-memory=100G --prepare --target-dir=/var/lib/mysql/
rm -rf ib_logfile*
chown -R mysql:mysql /var/lib/mysql/
systemctl start mariadb
- On Source:
The prepare fails as such:
2020-11-03 15:08:46 0 [Note] InnoDB: To recover: 2407553 pages from log
2020-11-03 15:08:54 0 [ERROR] InnoDB: Not applying INSERT_REUSE_REDUNDANT due to corruption on [page id: space=0, page number=120831]
2020-11-03 15:08:54 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
Attachments
Issue Links
- causes
-
MDEV-25328 --innodb command line option causes mariabackup to fail
-
- Closed
-
- is blocked by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
MDEV-24719 backport MDEV-24705 from 10.5 to 10.2, 10.3, 10.4
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- mentioned in
-
Page Loading...
-
-
-
-
-
Activity
Pull request #1772 would implement innodb_force_recovery=1 for mariabackup --prepare, and I think solve this issue.
The contributed fix is basically only a work-around: it allows the option --innodb-force-recovery=1 to be specified when applying logs (preparing a backup).
The root cause of the failure is always something else (such as MDEV-25031 for the case that is in the Description). This work-around allows an attempt to restore a backup even if some corruption is present.
marko will the backup still be consistent when restoring with force-recovery=1? especially regarding the position resp. the master (in case we want to make this server a replica)?
The purpose of innodb_force_recovery=1 is to ignore some errors (corrupted pages or missing data files). Depending on where the errors occur, it may or may not be safe. In the general case, I would not recommend to continue using a server where recovery errors had been ignored in the past.
When it comes to MariaDB Server (not Mariabackup), it would be nice if there was a way to set innodb_force_recovery to the values 2 or 3 in such a way that the recovery will run in strict mode. If no crash recovery is involved, those values of the parameter are not expected to cause further corruption. The main purpose of innodb_force_recovery is to allow data to be extracted from a corrupted database.
For 10.5, this could have been fixed by
MDEV-25031, which fixed a regression that was introduced in 10.5.For 10.2.27, I have no idea of the cause.