[MDEV-29110] mariabackup has wrong or missing plugin-dir default? Created: 2022-07-15  Updated: 2023-11-13  Resolved: 2023-11-09

Status: Closed
Project: MariaDB Server
Component/s: Backup, mariabackup
Affects Version/s: 10.5
Fix Version/s: 10.4.33, 10.5.24, 10.6.17, 10.11.7, 11.0.5, 11.1.4, 11.2.3, 11.3.2, 11.4.1

Type: Bug Priority: Critical
Reporter: Hartmut Holzgraefe Assignee: Alexander Barkov
Resolution: Fixed Votes: 1
Labels: None


 Description   

Preparing a backup taken on a server with data-at-rest encryption enabled fails with

mariabackup: Can't open shared library '/file_key_management.so' (errno: 2, cannot open shared object file: No such file or directory)

The prepare only succeeds after explicitly adding --plugin-dir=/usr/lib/mysql/plugins, which should not be necessary as it is the default path anyway.

Full log output:

# mariabackup --prepare
 
 
 
mariabackup based on MariaDB server 10.5.16-MariaDB debian-linux-gnu (x86_64)
 
[00] 2022-07-15 13:30:22 cd to /root/xtrabackup_backupfiles/
 
[00] 2022-07-15 13:30:22 open files limit requested 0, set to 1024
 
[00] 2022-07-15 13:30:22 Loading encryption plugin from file_key_management=file_key_management
 
[00] 2022-07-15 13:30:22 Loading encryption plugin
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--socket=/run/mysqld/mysqld.sock'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--pid-file=/run/mysqld/mysqld.pid'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--basedir=/usr'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--bind-address=127.0.0.1'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--expire_logs_days=10'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--character-set-server=utf8mb4'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--collation-server=utf8mb4_general_ci'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--plugin_load_add=file_key_management'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--file_key_management_filename=/etc/mysql//keyfile.enc'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--file_key_management_filekey=FILE:/etc/mysql/keyfile.key'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--file_key_management_encryption_algorithm=AES_CTR'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--innodb_encrypt_tables=FORCE'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--innodb_encrypt_log=ON'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--innodb_encrypt_temporary_tables=ON'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--innodb_encryption_threads=4'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--innodb_encryption_rotate_key_age=1'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--innodb_default_encryption_key_id=1'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--aria_encrypt_tables=ON'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--encrypt_tmp_disk_tables=ON'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--encrypt_tmp_files=ON'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--encrypt_binlog=ON'
 
[00] 2022-07-15 13:30:22 	 Encryption plugin parameter :  '--prepare'
 
mariabackup: Can't open shared library '/file_key_management.so' (errno: 2, cannot open shared object file: No such file or directory)
 
2022-07-15 13:30:22 0 [ERROR] Couldn't load plugin 'file_key_management' from 'file_key_management.so'.
 
[00] 2022-07-15 13:30:22 This target seems to be not prepared yet.
 
[00] 2022-07-15 13:30:22 mariabackup: using the following InnoDB configuration for recovery:
 
[00] 2022-07-15 13:30:22 innodb_data_home_dir = .
 
[00] 2022-07-15 13:30:22 innodb_data_file_path = ibdata1:10M:autoextend
 
[00] 2022-07-15 13:30:22 innodb_log_group_home_dir = .
 
[00] 2022-07-15 13:30:22 InnoDB: Using Linux native AIO
 
[00] 2022-07-15 13:30:22 Starting InnoDB instance for recovery.
 
[00] 2022-07-15 13:30:22 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
 
2022-07-15 13:30:22 0 [Note] InnoDB: Uses event mutexes
 
2022-07-15 13:30:22 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2022-07-15 13:30:22 0 [Note] InnoDB: Number of pools: 1
 
2022-07-15 13:30:22 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
2022-07-15 13:30:22 0 [Note] InnoDB: Using Linux native AIO
 
2022-07-15 13:30:22 0 [Note] InnoDB: Initializing buffer pool, total size = 104857600, chunk size = 104857600
 
2022-07-15 13:30:22 0 [Note] InnoDB: Completed initialization of buffer pool
 
2022-07-15 13:30:22 0 [ERROR] InnoDB: Obtaining redo log encryption key version 1 failed (4294967295). Maybe the key or the required encryption key management plugin was not found.
 
2022-07-15 13:30:22 0 [ERROR] InnoDB: Reading checkpoint encryption info failed.
 
2022-07-15 13:30:22 0 [ERROR] InnoDB: No valid checkpoint found (corrupted redo log). You can try --innodb-force-recovery=6 as a last resort.
 
2022-07-15 13:30:22 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
 
[00] FATAL ERROR: 2022-07-15 13:30:22 mariabackup: innodb_init() returned 11 (Generic error).



 Comments   
Comment by Hartmut Holzgraefe [ 2022-07-15 ]

Plugin dir indeed defaults to NULL here,

proposed diff (against 10.6 branch):

index dbaa67e1324..d21b409b543 100644
 
--- a/extra/mariabackup/encryption_plugin.cc
 
+++ b/extra/mariabackup/encryption_plugin.cc
 
@@ -31,7 +31,7 @@ extern struct st_maria_plugin *mysql_mandatory_plugins[];
 
 static void encryption_plugin_init(int argc, char **argv);
 
 
 extern char *xb_plugin_load;
 
-extern char *xb_plugin_dir;
 
+extern const char *xb_plugin_dir;
 
 
 const int PLUGIN_MAX_ARGS = 1024;
 
 std::vector<std::string> backup_plugins_args;
 
diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc
 
index ad902bdee59..db55d3d952f 100644
 
--- a/extra/mariabackup/xtrabackup.cc
 
+++ b/extra/mariabackup/xtrabackup.cc
 
@@ -268,7 +268,7 @@ ulong xtrabackup_innodb_force_recovery = 0;
 
 lsn_t  flushed_lsn= 0;
 
 
 ulong xb_open_files_limit= 0;
 
-char *xb_plugin_dir;
 
+const char *xb_plugin_dir = PLUGINDIR;
 
 char *xb_plugin_load;
 
 my_bool xb_close_files;
 
 
diff --git a/extra/mariabackup/xtrabackup.h b/extra/mariabackup/xtrabackup.h
 
index a8c4486b74c..4821aca14ea 100644
 
--- a/extra/mariabackup/xtrabackup.h
 
+++ b/extra/mariabackup/xtrabackup.h
 
@@ -72,7 +72,7 @@ extern char           *xtrabackup_incremental_basedir;
 
 extern char            *innobase_data_home_dir;
 
 extern char            *innobase_buffer_pool_filename;
 
 extern char            *buffer_pool_filename;
 
-extern char            *xb_plugin_dir;
 
+extern const char      *xb_plugin_dir;
 
 extern char            *xb_rocksdb_datadir;
 
 extern my_bool xb_backup_rocksdb;

Comment by Daniel Black [ 2022-07-18 ]

Note we've got the same plugin_dir option being used for both server and client plugins which I don't think are guaranteed to be the same.

Client side is in extra/mariabackup/backup_mysql.cc mysql_options(connection, MYSQL_PLUGIN_DIR, xb_plugin_dir);.

I don't have a good solution to the predicament either. Suggestions welcome.

Comment by Elena Stepanova [ 2022-07-20 ]

Mariabackup prepare is supposed to take plugin_dir from backup-my.cnf stored along with the backup in the target directory, and it usually does so.
The problem occurs when the target-dir option is not provided explicitly, as it was in the case above, and the default location is used. Then, at the moment when mariabackup would normally load options from backup-my.cnf stored in the target_dir, target_dir remains unitialized, and the options are not loaded, including the plugin-dir.
This needs to be fixed.

Comment by Hartmut Holzgraefe [ 2022-07-20 ]

Uh, is it actually a good idea to take this specific setting from backup-my.cnf? Prepare could happen on a different machine than the actual backup, and even though it is likely that both have the same install prefix it is not guaranteed ...

Comment by Elena Stepanova [ 2022-07-20 ]

The same can be said about using a hard default, it's not guaranteed at all that mariabackup is of the same version as mariadb/mysql plugins installed system-wide, or that anything is even installed system-wide at all. I don't think there is a universal solution, somebody will always have to override the value, either via the command line (if it's possible, I vaguely remember some problems with it) or by editing the config file.

Comment by Hartmut Holzgraefe [ 2022-07-20 ]

It's more likely that a compiled in default is matching the actual install path when running backup and prepare on different machines then to assume that the paths are the same on backup and prepare systems.

Most likely both are the same as even when a user is preparing on a different machine than the one the backup was taken from is using same OS release, generalCPU type, etc.

But if prepare is run on a system where the default plugin dir path differs (e.g. /usr/lib/mysql/plugin vs. /usr/lib64/mysql/plugin) it is more likely that the compiled in default is correct than that the path is the same as on the machine the backup step had been run on

So compiled in default looks more safe to me than value from backup-my.cnf. It will break for setups that deliberately chose to install everything, including libraries, under non-standard paths, but then again such users should know what they've done and that they might need to configure paths themselves accordingly

Comment by Alexander Barkov [ 2022-11-14 ]

Hello hholzgra.

I can see two ways:
1. Close this issue as Won't fix. Any changes can fix the problem for some installations, but introduce incompatibilities for other inatallations.

2. Try to identify the machine somehow and detect implicit plugin-dir differently:

  • If --backup and --prepare is done on the same machine, then plugin-dir can be taken from backup-my.cnf.
  • Otherwise, taking plugin-dir from backup-my.cnf is rather meaningless. We could use the compiled default PLUGINDIR in such case (like you proposed in your patch).

Can you please comment?

Comment by Hartmut Holzgraefe [ 2022-12-02 ]

When I created the patch I was not aware of the backup-my.cnf logic, and how this makes things behave differently depending on whether --target-dir was given explicitly or not.

I'm actually not sure whether backup-my.cnf or compile time plugin-dir, or target machine my.cnf setting if available, should take precedence ...

Comment by Hartmut Holzgraefe [ 2022-12-02 ]

Maybe a different / additional approach:

When loading a plugin, make sure plugin_dir is not empty or

{/}

, and produce a more meaningful error message than: 

mariabackup: Can't open shared library '/file_key_management.so' (errno: 2, cannot open shared object file: No such file or directory)

Something more like:

Can't open plugin library as plugin_dir is not configured, try to set it explicitly with --plugin-dir option

Comment by Sergei Golubchik [ 2023-05-17 ]

As far as I understand, the first bug that the customer complained about was explained in this comment. It should be fixed first. Then, it seems, the next bug to fix will be MDEV-23116, but let's fix the first one and wait for the confirmation.

Comment by Sergei Golubchik [ 2023-11-07 ]

0c27922db7f is ok to push, thanks!

Generated at Thu Feb 08 10:05:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.