[MDEV-28819] Statically compiled encryption plugins do not work in mariadb-backup Created: 2022-06-13  Updated: 2023-07-18  Resolved: 2022-06-20

Status: Closed
Project: MariaDB Server
Component/s: Compiling
Affects Version/s: 10.9, 10.10
Fix Version/s: 10.4.26, 10.5.17, 10.6.9, 10.7.5, 10.8.4, 10.9.2

Type: Bug Priority: Minor
Reporter: Michael Widenius Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None


 Description   

In 10.9, when compiling hashicorp statically many InnoDB and mariabackup tests fails.

To repeat:
compile MariaDB with hashicorp statically and run:
mtr mariabackup.encrypted_page_corruption

Test fails with:
mysqltest: At line 76: command "rmdir" failed with error: 1 my_errno: 2 errno: 2
and var/log/mysqld.1.err contains a lot of failures like:

2022-06-13 12:14:35 0 [ERROR] InnoDB: The page [page id: space=5, page number=3] in file './test/t1.ibd' cannot be decrypted.
2022-06-13 12:14:35 0 [Note] InnoDB: However key management plugin or used key_version 1 is not found or used encryption algorithm or method does not match.



 Comments   
Comment by Marko Mäkelä [ 2022-06-14 ]

I analyzed this with a build with cmake -DPLUGIN_FILE_KEY_MANAGEMENT=STATIC -DPLUGIN_PERFSCHEMA=NO and the following change to a test:

diff --git a/mysql-test/suite/mariabackup/xb_compressed_encrypted.test b/mysql-test/suite/mariabackup/xb_compressed_encrypted.test
index 2cca740d7dd..24a28034acd 100644
--- a/mysql-test/suite/mariabackup/xb_compressed_encrypted.test
+++ b/mysql-test/suite/mariabackup/xb_compressed_encrypted.test
@@ -22,7 +22,7 @@ echo # xtrabackup backup;
 let $targetdir=$MYSQLTEST_VARDIR/tmp/backup;
 exec $INNOBACKUPEX --defaults-file=$MYSQLTEST_VARDIR/my.cnf --no-timestamp $targetdir;
 drop table t1;
-exec $INNOBACKUPEX --apply-log $targetdir;
+exec rr record $INNOBACKUPEX --apply-log $targetdir;
 
 -- source include/restart_and_restore.inc
 --enable_result_log

mkdir /dev/shm/rr
_RR_TRACE_DIR=/dev/shm/rr ./mtr xb_compressed_encrypted,4k

This would fail as follows:

10.9 6dea701e0f666b8643d6103972bda2844bae8b5f

mariabackup.xb_compressed_encrypted '4k,innodb' [ fail ]
        Test ended at 2022-06-14 09:51:10
 
CURRENT_TEST: mariabackup.xb_compressed_encrypted
mysqltest: At line 25: exec of 'rr record /dev/shm/10.9/extra/mariabackup/mariabackup --innobackupex --apply-log /dev/shm/10.9/mysql-test/var/tmp/backup 2>&1' failed, error: 256, status: 1, errno: 11
Output from before failure:
220614 09:51:09 innobackupex: Starting the apply-log operation
--innobackupex: Can't open shared library '/dev/shm/10.9/mysql-test/var/plugins/file_key_management.so' (errno: 2, cannot open shared object file: No such file or directory)
2022-06-14  9:51:09 0 [ERROR] Couldn't load plugins from 'file_key_management.so'.
2022-06-14  9:51:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=78881
2022-06-14  9:51:09 0 [ERROR] InnoDB: Cannot apply log to [page id: space=5, page number=0] of corrupted file './test/t1.ibd'
[00] 2022-06-14 09:51:09 mariadb-backup: srv_start() returned 37 (Data structure corruption).

The file test/t1.ibd is filled all with zero bytes, as expected after MDEV-24626. The recovery will fail, because the encryption key cannot be retrieved:

#0  no_key () at /mariadb/10.9/sql/encryption.cc:35
#1  0x000055a2db657bb4 in fil_space_crypt_t::key_get_latest_version (
    this=this@entry=0x55a2de44b300)
    at /mariadb/10.9/storage/innobase/fil/fil0crypt.cc:139
#2  0x000055a2db65b6ea in fil_space_crypt_t::fil_space_crypt_t (
    this=0x55a2de44b300, new_type=0, new_min_key_version=<optimized out>, 
    new_key_id=<optimized out>, new_encryption=FIL_ENCRYPTION_ON)
    at /mariadb/10.9/storage/innobase/include/fil0crypt.h:129
#3  0x000055a2db657c36 in fil_space_create_crypt_data (type=type@entry=0, 
    encrypt_mode=FIL_ENCRYPTION_ON, min_key_version=min_key_version@entry=0, 
    key_id=1) at /mariadb/10.9/storage/innobase/fil/fil0crypt.cc:221
#4  0x000055a2db657c59 in fil_space_create_crypt_data (
    encrypt_mode=<optimized out>, key_id=<optimized out>)
    at /mariadb/10.9/storage/innobase/fil/fil0crypt.cc:245
#5  0x000055a2db658b56 in fil_space_read_crypt_data (
    zip_size=zip_size@entry=2048, page=page@entry=0x7f3262710000 "")
    at /mariadb/10.9/storage/innobase/fil/fil0crypt.cc:313
#6  0x000055a2db70fa4f in recv_sys_t::recover_deferred (
    this=this@entry=0x55a2dc251960 <recv_sys>, p=
      {first = {m_id = 21474836481}, second = {state = page_recv_t::RECV_WILL_NOT_READ, last_offset = 1, log = {head = 0x7f32620d7d50, tail = 0x7f32626ab570}}}, name="./test/t1.ibd", free_block=@0x7fffce6c5fd0: 0x0)
    at /mariadb/10.9/storage/innobase/log/log0recv.cc:944

I hope that someone else knows how to make statically linked encryption plugins work with mariadb-backup. The ./mtr --suite=encryption works just fine, including some crash recovery tests.

Comment by Vladislav Lesin [ 2022-06-14 ]

BTW, there is MDEV-28607 with proposed fix https://github.com/MariaDB/server/tree/10.6-MDEV-28607-rr-backup to run mariabackup under rr without test file fixing.

Comment by Marko Mäkelä [ 2022-06-15 ]

A work-around was pushed to 10.9.

As far as I can tell, this only affects mariadb-backup, which for some reason fails to initialize statically compiled encryption plugins. The encryption tests of the server work just fine, including crash recovery tests.

Comment by Vladislav Vaintroub [ 2022-06-20 ]

I downgrade this to minor. I do not see how it is a blocker,critical or major.

Generated at Thu Feb 08 10:03:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.