[MDEV-29241] Hashicorp Plugin: Provide Key rotation Created: 2022-08-03  Updated: 2023-11-28

Status: Stalled
Project: MariaDB Server
Component/s: Encryption, Plugin - Hashicorp Key Management
Affects Version/s: 10.9.1
Fix Version/s: 10.11

Type: Bug Priority: Major
Reporter: Simon Stier Assignee: Julius Goryavsky
Resolution: Unresolved Votes: 1
Labels: None

Issue Links:
Issue split
split to MDEV-30847 Hashicorp Plugin: Provide cache flush... Stalled
split to MDEV-30849 Hashicorp Plugin: enable key version ... Stalled
Problem/Incident

 Description   

The Hashicorp Plugin seems to recognise new encryption keys in vault - but does not rotate them.

Reproduce by:

  • Install mariadb-server & mariadb-plugin-hashicorp-key-management 10.9.1
  • Install hashicorp vault, init and unseal vault
  • create secret engine & set a secret key:

    vault secrets enable -path /mariadb -version=2 kv
    vault kv put /mariadb/1 data=$(openssl rand -hex 32)
    

  • enable encryption by adding the following block to mariadb section in /etc/mysql/mariadb.conf.d/50-server.cnf:

    [mariadb]
     
    plugin_load_add = hashicorp_key_management
    hashicorp-key-management-vault-url=http://127.0.0.1:8200/v1/mariadb
    hashicorp-key-management-token=xxxxxxxxxxxxxxxxx
     
    innodb_encrypt_tables = FORCE
    innodb_encrypt_log = ON
    innodb_encrypt_temporary_tables = ON
     
    encrypt_tmp_disk_tables = ON
    encrypt_tmp_files = ON
    encrypt_binlog = ON
    aria_encrypt_tables = ON
     
    innodb_encryption_threads = 4
    innodb_encryption_rotation_iops = 2000
    log_error=server.log
    

  • set new key version

    vault kv put /mariadb/1 data=$(openssl rand -hex 32)
    

  • service mariadb restart
  • set a new version of the encryption key to vault:

    vault kv put /mariadb/1 data=$(openssl rand -hex 32)
    

    information_schema.INNODB_TABLESPACES_ENCRYPTION shows the following content now:

    MariaDB [(none)]> select * from information_schema.INNODB_TABLESPACES_ENCRYPTION;
    +-------+----------------------------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+----------------------+
    | SPACE | NAME                       | ENCRYPTION_SCHEME | KEYSERVER_REQUESTS | MIN_KEY_VERSION | CURRENT_KEY_VERSION | KEY_ROTATION_PAGE_NUMBER | KEY_ROTATION_MAX_PAGE_NUMBER | CURRENT_KEY_ID | ROTATING_OR_FLUSHING |
    +-------+----------------------------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+----------------------+
    |     0 | innodb_system              |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
    |     1 | mysql/innodb_table_stats   |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
    |     2 | mysql/innodb_index_stats   |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
    |     4 | mysql/gtid_slave_pos       |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
    |     3 | mysql/transaction_registry |                 1 |                  0 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
    +-------+----------------------------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+----------------------+
    5 rows in set (0.008 sec)
    

CURRENT_KEY_VERSION -> 2
CURRENT_KEY_ID -> 1

there seems to be no possibility to re-encrypt the tables.
In the documentation the Hashicorp Plugin ist not listed as "with-" nor "without Key Rotation Support":
https://mariadb.com/kb/en/encryption-key-management/#support-for-key-rotation-in-encryption-plugins



 Comments   
Comment by Sergei Golubchik [ 2023-01-06 ]

sysprg, why is it not possible to re-encrypt the table? As far as I can see from the plugin code, it caches the latest key version, so it might immediately notice that the key was rotated, but the cache expires in cache_version_timeout milliseconds (which is 0 by default, btw). Why the server doesn't notice that the key was rotated in the vault?

Comment by Sergei Golubchik [ 2023-02-22 ]

Ok, so the problem is not that it's "not possible to re-encrypt the table". The issue is that hashicorp_key_management plugin caches replies from Hashicorp Vault and these values never expire.

By default the caching is enabled, but cache_version_timeout is zero.

The workaround for the user is to set cache_version_timeout. The fix for MariaDB is to set the default to some meaningful value.

Comment by Sebastian [ 2023-03-13 ]

I'm also interested in the key rotation mechanism of this plugin. The official documentation is not clear if this plugin supports it or not and I'm trying to find it out manually, but an official statement and an example would be highly appreciated.

Comment by Sergei Golubchik [ 2023-03-13 ]

As far as I understand, the plugin supports key rotation just fine. But if you have enabled key caching (so that the plugin wouldn't need to query Hashicorp Vault all the time) and you have disabled key expiration timeout, then the key will stay forever in the cache and the plugin will not notice that the key was rotated in the Hashicorp Vault until you restart the server.

Key caching is generally a good idea, an infinite timeout is not.

Comment by Sebastian [ 2023-03-14 ]

Indeed I see that when I write a new version of the enc key in Vault, MariaDB sees it if I select data from information_schema.INNODB_TABLESPACES_ENCRYPTION (i.e. Current Key Version corresponds to the latest version in Vault, while Min Key Version is the older one used in encryption). But how do I trigger MariaDB to encrypt the tables using the newer version?

Comment by Sergei Golubchik [ 2023-03-14 ]

It's beyond the hashicorp plugin. Generally it should be explained here: https://mariadb.com/kb/en/innodb-background-encryption-threads/ You'll need innodb_encryption_threads (like the original reporter did) and innodb_encryption_rotate_key_age (which is 1 by default, so, likely, ok).

We're now trying to repeat this issue

Generated at Thu Feb 08 10:06:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.