[MDEV-17787] Unable to backup using mariabackup , if the MariaDB server is using AWS key management service plugin Created: 2018-11-21  Updated: 2019-02-17  Resolved: 2018-11-21

Status: Closed
Project: MariaDB Server
Component/s: mariabackup, Plugin - AWS key management
Affects Version/s: 10.3.11
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Akshaya Srinivasan Assignee: Unassigned
Resolution: Not a Bug Votes: 0
Labels: mariabackup
Environment:

centos7

Issue Links:
Relates
relates to MDEV-18607 MariaDB Galera Cluster SST fails whil... Open

 Description   

Hi ,

My MariaDB server is configured to use AWS key management service encryption plugin. Since xtrabackup is not supported with MariaDB 10.3, I am using mariabackup to take the backup, but it fails with below error. However without encryption mariabackup goes fine. Please can you let me know if any additional options passed to mariabackup will resolve the issue?

[root@mongotest2lin bin]# /usr/bin/mariabackup --defaults-file=/etc/my.cnf -p'mysql' --user=root --backup --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --parallel=4 --extra-lsndir="/clone" --databases "testenc" --stream=xbstream >/dev/null
181121 12:34:35 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: /var/lib/mysql/mysql.sock
Using server version 10.3.11-MariaDB
/usr/bin/mariabackup based on MariaDB server 10.3.11-MariaDB Linux (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /var/lib/mysql/
Loading encryption plugin
Encryption plugin parameter : '--aws_key_management_key_spec=AES_128'
Encryption plugin parameter : '--aws_key_management_log_level=Off'
Encryption plugin parameter : '--aws_key_management_master_key_id=alias/mariadbEncryption'
Encryption plugin parameter : '--aws_key_management_region=us-east-2'
Encryption plugin parameter : '--aws_key_management_request_timeout=0'
Encryption plugin parameter : '--aws_key_management_rotate_key=0'
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup: innodb_data_home_dir =
mariabackup: innodb_data_file_path = ibdata1:12M:autoextend
mariabackup: innodb_log_group_home_dir = ./
2018-11-21 12:34:35 0x7f4638c78880 InnoDB: Using Linux native AIO
2018-11-21 12:34:35 0 [Note] InnoDB: Number of pools: 1
/usr/bin/mariabackup: AWS KMS plugin : KMS Client API 'Decrypt' failed : MissingAuthenticationTokenException - Missing Authentication Token
/usr/bin/mariabackup: AWS KMS plugin: key 1, version 1 could not be decrypted
2018-11-21 12:34:40 0 [ERROR] InnoDB: Obtaining redo log encryption key version 1 failed (4294967295). Maybe the key or the required encryption key management plugin was not found.
2018-11-21 12:34:40 0 [ERROR] InnoDB: Reading checkpoint encryption info failed.
2018-11-21 12:34:40 0 [ERROR] InnoDB: Obtaining redo log encryption key version 1 failed (4294967295). Maybe the key or the required encryption key management plugin was not found.
2018-11-21 12:34:40 0 [ERROR] InnoDB: Reading checkpoint encryption info failed.
2018-11-21 12:34:40 0 [ERROR] InnoDB: No valid checkpoint found (corrupted redo log). You can try --innodb-force-recovery=6 as a last resort.

 Comments   
Comment by Vladislav Vaintroub [ 2018-11-21 ]

mariabackup process obviously needs to authenticate in AWS. If it uses the same credentials as the server process it would work

Comment by Akshaya Srinivasan [ 2018-11-21 ]

Thanks Vladislav.

After I exported below values backup succeeded. Is there a way I can pass the aws encryption key id too? here the id was 3 when I created the table.

aws_access_key_id = XXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXX

MariaDB [test]> create table t1 (id serial, v varchar(32)) ENCRYPTION_KEY_ID=3;
Query OK, 0 rows affected (0.91 sec)

Comment by Vladislav Vaintroub [ 2018-11-21 ]

aws_key_management plugin does not have dedicated variables for aws_access_key_id or aws_secret_access_key.

Ideally you provide the credentials using one of the methods here
https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/credentials.html

But you can set env.variables as per document above.

Note that your server already authenticates against AWS, because aws_key_management plugin works. How exactly it works in your case, I do not known, it probably uses profile file.

Comment by Vladislav Vaintroub [ 2018-11-21 ]

You do not need to pass key ids, mariabackup can and will backup files using the same keys as server uses them.

Comment by Akshaya Srinivasan [ 2018-11-21 ]

Thanks Vladislav.

When I created the table, below log line was seen, where key id 3 was loaded.

Nov 21 11:42:12 mongotest2lin mysqld[52326]: 2018-11-21 11:42:12 8 [Note] mysqld: AWS KMS plugin: loaded key 3, version 1, key length 128 bit

But when I took a backup of the table using mariabackup, below line was seen when key 1 was loaded instead of key 3

2018-11-21 14:20:44 0 [Note] InnoDB: Number of pools: 1
/usr/bin/mariabackup: AWS KMS plugin: loaded key 1, version 1, key length 128 bit

Comment by Vladislav Vaintroub [ 2018-11-21 ]

Do not look at the "loaded" messages, look at what is actually in the table after prepare. USe "SHOW CREATE TABLE" is for that.

Different keys can be loaded at different times into the server (or backup), as needed. mariabackup does not have the intelligence to renumber the keys of a table.

Generated at Thu Feb 08 08:39:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.