[MDEV-23394] mariadb --prepare --export fails with "Upgrade after crash is not supported" and error 11 Created: 2020-08-04  Updated: 2023-12-14

Status: Open
Project: MariaDB Server
Component/s: mariabackup
Affects Version/s: 10.4.13, 10.5.4
Fix Version/s: 10.5, 10.11

Type: Bug Priority: Major
Reporter: Allen Lee (Inactive) Assignee: Alexander Barkov
Resolution: Unresolved Votes: 5
Labels: None
Environment:

Ubuntu 20.04
Google Cloud


Issue Links:
Duplicate
is duplicated by MDEV-30670 Partial backup/restore operation fail... Open
Problem/Incident
is caused by MDEV-23718 Mariabackup should check version of t... Open

 Description   

Customer reported following error when they tried to restore databases to a mariadb installation with version 10.5.4 installed with a backup taken from an instance running 10.4.13.

I'm attempting to restore databases to a mariadb installation with version 10.5.4 installed with a backup taken from an instance running 10.4.13. I will paste a copy of the error at the end of this description. Is there some method to force mariabackup on the new installation to do a conversion or upgrade of the backup so it is able to properly use it for restoration?
 
Here is the output of the attempted restoration job run:
 
root@db-staging-primary-2:/opt/mariadb# mariabackup --prepare --export --target_dir=/opt/mariadb/backup-scratch
mariabackup based on MariaDB server 10.5.4-MariaDB debian-linux-gnu (x86_64)
[00] 2020-08-03 23:04:49 cd to /opt/mariadb/backup-scratch/
[00] 2020-08-03 23:04:49 This target seems to be not prepared yet.
[00] 2020-08-03 23:04:49 mariabackup: using the following InnoDB configuration for recovery:
[00] 2020-08-03 23:04:49 innodb_data_home_dir = .
[00] 2020-08-03 23:04:49 innodb_data_file_path = ibdata1:10M:autoextend
[00] 2020-08-03 23:04:49 innodb_log_group_home_dir = .
[00] 2020-08-03 23:04:49 InnoDB: Using Linux native AIO
[00] 2020-08-03 23:04:49 Starting InnoDB instance for recovery.
[00] 2020-08-03 23:04:49 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
2020-08-03 23:04:49 0 [Note] InnoDB: Uses event mutexes
2020-08-03 23:04:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-03 23:04:49 0 [Note] InnoDB: Number of pools: 1
2020-08-03 23:04:49 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
mariabackup: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-08-03 23:04:49 0 [Note] InnoDB: Initializing buffer pool, total size = 104857600, chunk size = 104857600
2020-08-03 23:04:49 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-03 23:04:49 0 [Note] InnoDB: page_cleaner coordinator priority: -20
2020-08-03 23:04:49 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.4.13-MariaDB.
2020-08-03 23:04:49 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
[00] FATAL ERROR: 2020-08-03 23:04:49 mariabackup: innodb_init() returned 11 (Generic error).



 Comments   
Comment by Elena Stepanova [ 2020-08-04 ]

I don't know whether it could ever work – as the message says, crash upgrade 10.4=>10.5 is not supported, and if mariabackup restoration is basically a crash upgrade scenario, it will be forbidden as well. Assigning to vlad.lesin for an expert opinion.

Comment by Jeremy Lachevre [ 2020-12-14 ]

Hello,
We are facing to the same issue since Mariadb 10.5.6-MariaDB Linux (x86_64) installation (installed from scratch). The restoration of transportable tablespaces done from Mariadb 10.4 should be compatible with that version.
We always use partial backup / restore on many important projects since several years.

Could you give us a feedback on that worrying situation ?

Thanks,
Regards,
Lachevre Jeremy

see attached traces:
------------------
mariabackup based on MariaDB server 10.5.6-MariaDB Linux (x86_64)
[00] 2020-12-14 11:56:53 cd to /local/backup/dbr/customer_db/
[00] 2020-12-14 11:56:53 This target seems to be not prepared yet.
[00] 2020-12-14 11:56:53 mariabackup: using the following InnoDB configuration for recovery:
[00] 2020-12-14 11:56:53 innodb_data_home_dir = .
[00] 2020-12-14 11:56:53 innodb_data_file_path = ibdata1:10M:autoextend
[00] 2020-12-14 11:56:53 innodb_log_group_home_dir = .
[00] 2020-12-14 11:56:53 InnoDB: Using Linux native AIO
[00] 2020-12-14 11:56:53 Starting InnoDB instance for recovery.
[00] 2020-12-14 11:56:53 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
2020-12-14 11:56:53 0 [Note] InnoDB: Uses event mutexes
2020-12-14 11:56:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-12-14 11:56:53 0 [Note] InnoDB: Number of pools: 1
2020-12-14 11:56:53 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
2020-12-14 11:56:53 0 [Note] InnoDB: Initializing buffer pool, total size = 104857600, chunk size = 104857600
2020-12-14 11:56:53 0 [Note] InnoDB: Completed initialization of buffer pool
2020-12-14 11:56:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-12-14 11:56:53 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.4.14-MariaDB.
2020-12-14 11:56:53 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
[00] FATAL ERROR: 2020-12-14 11:56:53 mariabackup: innodb_init() returned 11 (Generic error).

Comment by Christophe Lafaille (ASN - Nokia) [ 2021-05-31 ]

Hi, if I well understand : if an application use MariaDB 10.4.x (running RHEL7.x), it'll be never possible to upgrade to a 10.5 or 10.6 version (running RHEL8.x)...

It's totally restrictive and very strange for MariaDB... this kind of problem doesn't exist with MySQL ==> so, the solution seems to use MySQL !

Regards

Comment by Christophe Lafaille (ASN - Nokia) [ 2021-12-21 ]

Hi,

is it planned to be corrected or not (the problem is logged since 20 months ) ?

The problem is also traced on reddit ( https://www.reddit.com/r/unRAID/comments/pbtumq/problems_with_mariadb_after_update_any_ideas/ ), I've added a link to this JIRA on the reddit page.

I think it's important to indicate on your site that MariaDB is only a "one version server"... if you use 10.4, you're blocked on this version (migration to upper version is not possible).
The comment from Elena Stepanova is clear: upgrade from 10.4 to 10.5 is forbidden (I can't understand how customers can accept this base feature is missing! )

In 2024, 10.4 support will be stopped... so, we have 2 years for a correction or to replace mariadb by another database.

Comment by Jeremy Lachevre [ 2021-12-21 ]

I have exactly the same bug than Mr Lafaille.

Nokia for which we are part uses many Mariadb products so I don't understand why that bug is not resolved yet.

Comment by Naresh Chandra [ 2022-01-17 ]

Even we are also also facing the issue as we are trying to upgrade from 10.4.22 to 10.5.13.

Please find the error details.

[root@test301]# /usr/bin/mariabackup --prepare --target-dir=/db/backup/full_backup_01172021
/usr/bin/mariabackup based on MariaDB server 10.5.13-9-MariaDB Linux (x86_64)
[00] 2022-01-16 21:45:42 cd to /db/backup/full_backup_01172021/
[00] 2022-01-16 21:45:42 open files limit requested 0, set to 1024
[00] 2022-01-16 21:45:44 This target seems to be not prepared yet.
[00] 2022-01-16 21:45:44 mariabackup: using the following InnoDB configuration for recovery:
[00] 2022-01-16 21:45:44 innodb_data_home_dir = .
[00] 2022-01-16 21:45:44 innodb_data_file_path = ibdata1:12M:autoextend
[00] 2022-01-16 21:45:44 innodb_log_group_home_dir = .
[00] 2022-01-16 21:45:44 InnoDB: Using Linux native AIO
[00] 2022-01-16 21:45:44 Starting InnoDB instance for recovery.
[00] 2022-01-16 21:45:44 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
2022-01-16 21:45:44 0 [Note] InnoDB: Uses event mutexes
2022-01-16 21:45:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2022-01-16 21:45:44 0 [Note] InnoDB: Number of pools: 1
2022-01-16 21:45:44 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-01-16 21:45:44 0 [Note] InnoDB: Using Linux native AIO
2022-01-16 21:45:44 0 [Note] InnoDB: Initializing buffer pool, total size = 104857600, chunk size = 104857600
2022-01-16 21:45:44 0 [Note] InnoDB: Completed initialization of buffer pool
2022-01-16 21:45:44 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with Backup 10.4.22-14-MariaDB.
2022-01-16 21:45:44 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
[00] FATAL ERROR: 2022-01-16 21:45:44 mariabackup: innodb_init() returned 11 (Generic error).

Comment by Jeremy Lachevre [ 2022-08-03 ]

What's news about bug ...

Comment by Marko Mäkelä [ 2023-12-14 ]

Backups are supposed to be prepared using the same major version of the backup as the originating server. The InnoDB redo log record format and all code related to it was changed in 10.5 (MDEV-12353). Later, in 10.8 (MDEV-14425) the redo log file format was changed. To deal with the old format, only a small consistency check was implemented, to see if all log had been applied.

The error message and the documentation could perhaps be clarified.

The bug title incorrectly mentioned "signal 11" a.k.a. SIGSEGV. I only see an "error 11" in the posted output, which is something completely different.

Generated at Thu Feb 08 09:22:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.