Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13311

Presence of old logs in 10.2.7 will corrupt restored instance (change in behavior)

Details

    Description

      Restore of backups (created with XtraBackup/Maria Backup) into empty data directory is the only valid scenario where success may be guaranteed. In other words if data directory has leftover data/log files from old Server instance - various problems may occur.

      Nevertheless, on practice restore into non-empty directory may easily succeed without any issues (despite being dangerous).

      So far both xtrabackup and mariabackup 10.1 did generate empty log files during --prepare.
      Thus if user does simple 'copy' of prepared backup into non-empty data directory - old logs would be most likely just overwritten.

      Change in behavior:
      Mariabackup 10.2.7 doesn't create empty log files and relies on --copy-back action executed by user. (which will delete old innodb log files, if any).

      For situations where users don't follow mentioned recommendations (i.e. neither use --copy-back or make sure that data directory is empty before restore) - backups created mariabackup 10.2.7 are definitely expected to become inconsistent/corrupted. (because of presence of old leftover InnoDB logs).

      Attachments

        Issue Links

          Activity

            greenman Ian Gilfillan added a comment - Documented in https://mariadb.com/kb/en/mariadb/mariadb-1027-release-notes/ and https://mariadb.com/kb/en/mariadb/mariadb-backup/

            I would prefer not to change this.
            I did spend some hours on 0001-Mariabackup-Write-a-dummy-empty-redo-log-after-prepa.patch that creates a dummy logically empty redo log file in --prepare and adds code to InnoDB startup to detect this special redo log file. I thought that this solution is too risky for a GA release. Such file format changes are also prohibited in beta releases.

            What is the problem of using the --prepare option instead of copying files using low-level tools?

            marko Marko Mäkelä added a comment - I would prefer not to change this. I did spend some hours on 0001-Mariabackup-Write-a-dummy-empty-redo-log-after-prepa.patch that creates a dummy logically empty redo log file in --prepare and adds code to InnoDB startup to detect this special redo log file. I thought that this solution is too risky for a GA release. Such file format changes are also prohibited in beta releases. What is the problem of using the --prepare option instead of copying files using low-level tools?
            anikitin Andrii Nikitin (Inactive) added a comment - - edited

            So, it should be a trade between easiness-to-use and protection from incorrect (and potentially dangerous) action in cases 2 vs 3 below :

            Case 1 : dba uses copy-back option - the only recommended procedure
            No issue here
            Case 2: dba just copies data from prepared backup folder into empty data directory
            Case 3: dba overwrites existing data directory with prepared backup

            I think the best solution will be creating just empty file ib_logfile0 during backup prepare phase (and skip that file in copy-back / move-back ) .
            The observed behavior will be:
            Case 1 : dba uses copy-back option - the only recommended procedure.
            Old logs are removed, empty ib_logfile0 is not copied - everything in good shape

            Case 2: dba just copies data from prepared backup folder into empty data directory
            Innodb fails to start complaining that ib_logfile0 has zero size.
            Dba just needs to remove that ib_logfile0 to let server start.
            This is inconvenient annoyance , but required step to prevent potential data corruption in case 3 below:

            Case 3: dba overwrites existing data directory with prepared backup.
            The biggest danger here is that InnoDB will attempt to use old log files, which potentially may lead to data corruption.
            With current proposal old ib_logfile0 gets overwritten with empty one from backup.
            InnoDB fails to start and needs removal of all log files

            marko do you think such trade is correct behavior to protect users from incorrect action?
            If recommended procedure is used - everything is good. Otherwise data will be safe with easy way to start server.

            anikitin Andrii Nikitin (Inactive) added a comment - - edited So, it should be a trade between easiness-to-use and protection from incorrect (and potentially dangerous) action in cases 2 vs 3 below : Case 1 : dba uses copy-back option - the only recommended procedure No issue here Case 2: dba just copies data from prepared backup folder into empty data directory Case 3: dba overwrites existing data directory with prepared backup I think the best solution will be creating just empty file ib_logfile0 during backup prepare phase (and skip that file in copy-back / move-back ) . The observed behavior will be: Case 1 : dba uses copy-back option - the only recommended procedure. Old logs are removed, empty ib_logfile0 is not copied - everything in good shape Case 2: dba just copies data from prepared backup folder into empty data directory Innodb fails to start complaining that ib_logfile0 has zero size. Dba just needs to remove that ib_logfile0 to let server start. This is inconvenient annoyance , but required step to prevent potential data corruption in case 3 below: Case 3: dba overwrites existing data directory with prepared backup. The biggest danger here is that InnoDB will attempt to use old log files, which potentially may lead to data corruption. With current proposal old ib_logfile0 gets overwritten with empty one from backup. InnoDB fails to start and needs removal of all log files marko do you think such trade is correct behavior to protect users from incorrect action? If recommended procedure is used - everything is good. Otherwise data will be safe with easy way to start server.

            I think that copying files directly is inherently unsafe, and the --copy-back option should be preferred.
            If someone implements direct copying, they should use the appropriate options, such as rsync --delete. Yes, it would be nice to have some safety mechanism against this.

            An option that I did not consider so far would be that --prepare truncates the ib_logfile0 to zero bytes on completion, and --copy-back will ignore this zero-length file in the prepared backup directory. Then, if the user copies files by low-level means, the zero-length ib_logfile0 should prevent server startup.

            anikitin, did you imply something like that in your case 2? I do not see how else you would get a zero-length ib_logfile0 file. I think that InnoDB should refuse to start up with a zero-length redo log file. If not, we can implement that.

            I provided the patch 0001-Mariabackup-Write-a-dummy-empty-redo-log-after-prepa.patch to create a 'dummy' redo log file that does not change the system LSN. This is important, so that incremental backups will not be broken. This patch has not been tested much, and it changes the redo log format in an incompatible way. We cannot do that within a beta or GA release (MariaDB 10.2). Soon we cannot do that in MariaDB 10.3 either.

            marko Marko Mäkelä added a comment - I think that copying files directly is inherently unsafe, and the --copy-back option should be preferred. If someone implements direct copying, they should use the appropriate options, such as rsync --delete. Yes, it would be nice to have some safety mechanism against this. An option that I did not consider so far would be that --prepare truncates the ib_logfile0 to zero bytes on completion, and --copy-back will ignore this zero-length file in the prepared backup directory. Then, if the user copies files by low-level means, the zero-length ib_logfile0 should prevent server startup. anikitin , did you imply something like that in your case 2? I do not see how else you would get a zero-length ib_logfile0 file. I think that InnoDB should refuse to start up with a zero-length redo log file. If not, we can implement that. I provided the patch 0001-Mariabackup-Write-a-dummy-empty-redo-log-after-prepa.patch to create a 'dummy' redo log file that does not change the system LSN. This is important, so that incremental backups will not be broken. This patch has not been tested much, and it changes the redo log format in an incompatible way. We cannot do that within a beta or GA release (MariaDB 10.2). Soon we cannot do that in MariaDB 10.3 either.

            People

              marko Marko Mäkelä
              anikitin Andrii Nikitin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.