Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23394

mariadb --prepare --export fails with "Upgrade after crash is not supported" and error 11

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.4.13, 10.5.4
    • 10.5, 10.11
    • Documentation, mariabackup
    • None
    • Ubuntu 20.04
      Google Cloud

    Description

      Customer reported following error when they tried to restore databases to a mariadb installation with version 10.5.4 installed with a backup taken from an instance running 10.4.13.

      I'm attempting to restore databases to a mariadb installation with version 10.5.4 installed with a backup taken from an instance running 10.4.13. I will paste a copy of the error at the end of this description. Is there some method to force mariabackup on the new installation to do a conversion or upgrade of the backup so it is able to properly use it for restoration?
       
      Here is the output of the attempted restoration job run:
       
      root@db-staging-primary-2:/opt/mariadb# mariabackup --prepare --export --target_dir=/opt/mariadb/backup-scratch
      mariabackup based on MariaDB server 10.5.4-MariaDB debian-linux-gnu (x86_64)
      [00] 2020-08-03 23:04:49 cd to /opt/mariadb/backup-scratch/
      [00] 2020-08-03 23:04:49 This target seems to be not prepared yet.
      [00] 2020-08-03 23:04:49 mariabackup: using the following InnoDB configuration for recovery:
      [00] 2020-08-03 23:04:49 innodb_data_home_dir = .
      [00] 2020-08-03 23:04:49 innodb_data_file_path = ibdata1:10M:autoextend
      [00] 2020-08-03 23:04:49 innodb_log_group_home_dir = .
      [00] 2020-08-03 23:04:49 InnoDB: Using Linux native AIO
      [00] 2020-08-03 23:04:49 Starting InnoDB instance for recovery.
      [00] 2020-08-03 23:04:49 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
      2020-08-03 23:04:49 0 [Note] InnoDB: Uses event mutexes
      2020-08-03 23:04:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2020-08-03 23:04:49 0 [Note] InnoDB: Number of pools: 1
      2020-08-03 23:04:49 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
      mariabackup: O_TMPFILE is not supported on /tmp (disabling future attempts)
      2020-08-03 23:04:49 0 [Note] InnoDB: Initializing buffer pool, total size = 104857600, chunk size = 104857600
      2020-08-03 23:04:49 0 [Note] InnoDB: Completed initialization of buffer pool
      2020-08-03 23:04:49 0 [Note] InnoDB: page_cleaner coordinator priority: -20
      2020-08-03 23:04:49 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.4.13-MariaDB.
      2020-08-03 23:04:49 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
      [00] FATAL ERROR: 2020-08-03 23:04:49 mariabackup: innodb_init() returned 11 (Generic error).
      

      Attachments

        Issue Links

          Activity

            What's news about bug ...

            Jeremy Jeremy Lachevre added a comment - What's news about bug ...

            Backups are supposed to be prepared using the same major version of the backup as the originating server. The InnoDB redo log record format and all code related to it was changed in 10.5 (MDEV-12353). Later, in 10.8 (MDEV-14425) the redo log file format was changed. To deal with the old format, only a small consistency check was implemented, to see if all log had been applied.

            The error message and the documentation could perhaps be clarified.

            The bug title incorrectly mentioned "signal 11" a.k.a. SIGSEGV. I only see an "error 11" in the posted output, which is something completely different.

            marko Marko Mäkelä added a comment - Backups are supposed to be prepared using the same major version of the backup as the originating server. The InnoDB redo log record format and all code related to it was changed in 10.5 ( MDEV-12353 ). Later, in 10.8 ( MDEV-14425 ) the redo log file format was changed. To deal with the old format, only a small consistency check was implemented, to see if all log had been applied. The error message and the documentation could perhaps be clarified. The bug title incorrectly mentioned "signal 11" a.k.a. SIGSEGV. I only see an "error 11" in the posted output, which is something completely different.
            stefan.hinz Stefan Hinz added a comment -

            Hi Julien,

            I don’t have a glimpse of an idea what to do with that task. Is this something we should mention in the release notes? If so, what should we be saying?

            Cheers,

            Stefan

            stefan.hinz Stefan Hinz added a comment - Hi Julien, I don’t have a glimpse of an idea what to do with that task. Is this something we should mention in the release notes? If so, what should we be saying? Cheers, Stefan

            As far as I can tell, there is a design problem. Backup is a separate program, instead of being part of the server (MDEV-14992). What mariadb-backup --backup generates is almost the same as what you would get if you killed the mariadbd process and made a copy of its data directory. For a DML-only workload, there probably is not much difference.

            Much of the time it may feel redundant to execute mariadb-backup --prepare, and it might happen to work if you just started mariadbd right on the backup directory. This will definitely not work

            • with --inclremental backup
            • when any DDL operations were executed during the backup
            • when the format of ib_logfile0 has been changed, like it was in MariaDB Server 10.2, 10.5, 10.8

            Our documentation needs to make it clear that after a backup has been made, it would be best to run mariadb-backup --prepare as soon as possible. Note that this can be run on a different system than the database server, to reduce memory and I/O bandwidth. There are at least the following benefits of this:

            • any corruption in the backup will be noticed immediately, and not some time later when you really would want to restore it
            • restoring the backup will be faster, and will work with any future version of the server that might not support the current log format
            marko Marko Mäkelä added a comment - As far as I can tell, there is a design problem. Backup is a separate program, instead of being part of the server ( MDEV-14992 ). What mariadb-backup --backup generates is almost the same as what you would get if you killed the mariadbd process and made a copy of its data directory. For a DML-only workload, there probably is not much difference. Much of the time it may feel redundant to execute mariadb-backup --prepare , and it might happen to work if you just started mariadbd right on the backup directory. This will definitely not work with --inclremental backup when any DDL operations were executed during the backup when the format of ib_logfile0 has been changed, like it was in MariaDB Server 10.2, 10.5, 10.8 Our documentation needs to make it clear that after a backup has been made, it would be best to run mariadb-backup --prepare as soon as possible. Note that this can be run on a different system than the database server, to reduce memory and I/O bandwidth. There are at least the following benefits of this: any corruption in the backup will be noticed immediately, and not some time later when you really would want to restore it restoring the backup will be faster, and will work with any future version of the server that might not support the current log format
            stefan.hinz Stefan Hinz added a comment -

            marko julien.fritsch What about adding this, based on Marko's comment?
            It is best to run mariadb-backup --prepare as soon as possible after a backup. The command can be run on a different machine than the database server, to reduce memory usage and I/O bandwidth one the machine running the server. Advantages of doing so are:

            • Any corruption in the backup will be noticed immediately, rather than some time later when you want to restore from the backup.
            • Restoring the backup will be faster, and will work with any future version of the server that may not support the current log format.

            Now where to put that? Somewhere under https://mariadb.com/kb/en/mariabackup/ I guess. I could put it here: https://mariadb.com/kb/en/full-backup-and-restore-with-mariabackup/ ... Unless you have a better idea.

            stefan.hinz Stefan Hinz added a comment - marko julien.fritsch What about adding this, based on Marko's comment? It is best to run mariadb-backup --prepare as soon as possible after a backup. The command can be run on a different machine than the database server, to reduce memory usage and I/O bandwidth one the machine running the server. Advantages of doing so are: Any corruption in the backup will be noticed immediately, rather than some time later when you want to restore from the backup. Restoring the backup will be faster, and will work with any future version of the server that may not support the current log format. Now where to put that? Somewhere under https://mariadb.com/kb/en/mariabackup/ I guess. I could put it here: https://mariadb.com/kb/en/full-backup-and-restore-with-mariabackup/ ... Unless you have a better idea.

            People

              stefan.hinz Stefan Hinz
              allen.lee@mariadb.com Allen Lee (Inactive)
              Votes:
              6 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.