Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
None
-
None
Description
Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other words if data directory has leftover data/log files from old Server instance - various problems may occur.
Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous).
So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare.
Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be most likely just overwritten.
Change in behavior:
Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which will delete old innodb log files, if any).
For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs).
Attachments
Issue Links
- causes
-
-
- Closed
-
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other worlds if data directory has leftover data/log files from old Server instance - various problems may occur.
Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous). So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare. Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be overwritten. *Change in behavior:* Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which will remove old innodb log files if any). For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs). |
Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other words if data directory has leftover data/log files from old Server instance - various problems may occur.
Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous). So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare. Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be overwritten. *Change in behavior:* Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which will remove old innodb log files if any). For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs). |
| Description |
Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other words if data directory has leftover data/log files from old Server instance - various problems may occur.
Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous). So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare. Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be overwritten. *Change in behavior:* Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which will remove old innodb log files if any). For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs). |
Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other words if data directory has leftover data/log files from old Server instance - various problems may occur.
Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous). So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare. Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be overwritten. *Change in behavior:* Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which most likely will overwrite old innodb log files, if any). For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs). |
| Description |
Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other words if data directory has leftover data/log files from old Server instance - various problems may occur.
Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous). So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare. Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be overwritten. *Change in behavior:* Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which most likely will overwrite old innodb log files, if any). For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs). |
Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other words if data directory has leftover data/log files from old Server instance - various problems may occur.
Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous). So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare. Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be most likely just overwritten. *Change in behavior:* Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which will delete old innodb log files, if any). For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs). |
| Assignee | Ralf Gebhardt [ ralf.gebhardt@mariadb.com ] | Ian Gilfillan [ greenman ] |
| Assignee | Ian Gilfillan [ greenman ] |
| Assignee | Andrii Nikitin [ anikitin ] |
| Fix Version/s | 10.2.8 [ 22544 ] |
| Attachment | 0001-Mariabackup-Write-a-dummy-empty-redo-log-after-prepa.patch [ 43941 ] |
I would prefer not to change this.
I did spend some hours on 0001-Mariabackup-Write-a-dummy-empty-redo-log-after-prepa.patch
that creates a dummy logically empty redo log file in --prepare and adds code to InnoDB startup to detect this special redo log file. I thought that this solution is too risky for a GA release. Such file format changes are also prohibited in beta releases.
What is the problem of using the --prepare option instead of copying files using low-level tools?
| Fix Version/s | 10.2.8 [ 22544 ] |
So, it should be a trade between easiness-to-use and protection from incorrect (and potentially dangerous) action in cases 2 vs 3 below :
Case 1 : dba uses copy-back option - the only recommended procedure
No issue here
Case 2: dba just copies data from prepared backup folder into empty data directory
Case 3: dba overwrites existing data directory with prepared backup
I think the best solution will be creating just empty file ib_logfile0 during backup prepare phase (and skip that file in copy-back / move-back ) .
The observed behavior will be:
Case 1 : dba uses copy-back option - the only recommended procedure.
Old logs are removed, empty ib_logfile0 is not copied - everything in good shape
Case 2: dba just copies data from prepared backup folder into empty data directory
Innodb fails to start complaining that ib_logfile0 has zero size.
Dba just needs to remove that ib_logfile0 to let server start.
This is inconvenient annoyance , but required step to prevent potential data corruption in case 3 below:
Case 3: dba overwrites existing data directory with prepared backup.
The biggest danger here is that InnoDB will attempt to use old log files, which potentially may lead to data corruption.
With current proposal old ib_logfile0 gets overwritten with empty one from backup.
InnoDB fails to start and needs removal of all log files
marko do you think such trade is correct behavior to protect users from incorrect action?
If recommended procedure is used - everything is good. Otherwise data will be safe with easy way to start server.
| Priority | Major [ 3 ] | Critical [ 2 ] |
| Assignee | Andrii Nikitin [ anikitin ] | Marko Mäkelä [ marko ] |
I think that copying files directly is inherently unsafe, and the --copy-back option should be preferred.
If someone implements direct copying, they should use the appropriate options, such as rsync --delete. Yes, it would be nice to have some safety mechanism against this.
An option that I did not consider so far would be that --prepare truncates the ib_logfile0 to zero bytes on completion, and --copy-back will ignore this zero-length file in the prepared backup directory. Then, if the user copies files by low-level means, the zero-length ib_logfile0 should prevent server startup.
anikitin, did you imply something like that in your case 2? I do not see how else you would get a zero-length ib_logfile0 file. I think that InnoDB should refuse to start up with a zero-length redo log file. If not, we can implement that.
I provided the patch 0001-Mariabackup-Write-a-dummy-empty-redo-log-after-prepa.patch to create a 'dummy' redo log file that does not change the system LSN. This is important, so that incremental backups will not be broken. This patch has not been tested much, and it changes the redo log format in an incompatible way. We cannot do that within a beta or GA release (MariaDB 10.2). Soon we cannot do that in MariaDB 10.3 either.
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Fix Version/s | 10.2.10 [ 22615 ] | |
| Fix Version/s | 10.3.3 [ 22644 ] | |
| Fix Version/s | 10.2 [ 14601 ] | |
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Closed [ 6 ] |
| Link |
This issue causes |
| Workflow | MariaDB v3 [ 81656 ] | MariaDB v4 [ 152476 ] |
Documented in https://mariadb.com/kb/en/mariadb/mariadb-1027-release-notes/ and https://mariadb.com/kb/en/mariadb/mariadb-backup/