Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29241

Hashicorp Plugin: Provide Key rotation

Details

    Description

      The Hashicorp Plugin seems to recognise new encryption keys in vault - but does not rotate them.

      Reproduce by:

      • Install mariadb-server & mariadb-plugin-hashicorp-key-management 10.9.1
      • Install hashicorp vault, init and unseal vault
      • create secret engine & set a secret key:

        vault secrets enable -path /mariadb -version=2 kv
        vault kv put /mariadb/1 data=$(openssl rand -hex 32)
        

      • enable encryption by adding the following block to mariadb section in /etc/mysql/mariadb.conf.d/50-server.cnf:

        [mariadb]
         
        plugin_load_add = hashicorp_key_management
        hashicorp-key-management-vault-url=http://127.0.0.1:8200/v1/mariadb
        hashicorp-key-management-token=xxxxxxxxxxxxxxxxx
         
        innodb_encrypt_tables = FORCE
        innodb_encrypt_log = ON
        innodb_encrypt_temporary_tables = ON
         
        encrypt_tmp_disk_tables = ON
        encrypt_tmp_files = ON
        encrypt_binlog = ON
        aria_encrypt_tables = ON
         
        innodb_encryption_threads = 4
        innodb_encryption_rotation_iops = 2000
        log_error=server.log
        

      • set new key version

        vault kv put /mariadb/1 data=$(openssl rand -hex 32)
        

      • service mariadb restart
      • set a new version of the encryption key to vault:

        vault kv put /mariadb/1 data=$(openssl rand -hex 32)
        

        information_schema.INNODB_TABLESPACES_ENCRYPTION shows the following content now:

        MariaDB [(none)]> select * from information_schema.INNODB_TABLESPACES_ENCRYPTION;
        +-------+----------------------------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+----------------------+
        | SPACE | NAME                       | ENCRYPTION_SCHEME | KEYSERVER_REQUESTS | MIN_KEY_VERSION | CURRENT_KEY_VERSION | KEY_ROTATION_PAGE_NUMBER | KEY_ROTATION_MAX_PAGE_NUMBER | CURRENT_KEY_ID | ROTATING_OR_FLUSHING |
        +-------+----------------------------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+----------------------+
        |     0 | innodb_system              |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
        |     1 | mysql/innodb_table_stats   |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
        |     2 | mysql/innodb_index_stats   |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
        |     4 | mysql/gtid_slave_pos       |                 1 |                  1 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
        |     3 | mysql/transaction_registry |                 1 |                  0 |               1 |                   2 |                     NULL |                         NULL |              1 |                    0 |
        +-------+----------------------------+-------------------+--------------------+-----------------+---------------------+--------------------------+------------------------------+----------------+----------------------+
        5 rows in set (0.008 sec)
        

      CURRENT_KEY_VERSION -> 2
      CURRENT_KEY_ID -> 1

      there seems to be no possibility to re-encrypt the tables.
      In the documentation the Hashicorp Plugin ist not listed as "with-" nor "without Key Rotation Support":
      https://mariadb.com/kb/en/encryption-key-management/#support-for-key-rotation-in-encryption-plugins

      Attachments

        Issue Links

          Activity

            Slm0n Simon Stier created issue -
            serg Sergei Golubchik made changes -
            Field Original Value New Value
            Issue Type Task [ 3 ] Bug [ 1 ]
            serg Sergei Golubchik made changes -
            Affects Version/s 10.9.1 [ 27114 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.9 [ 26905 ]
            serg Sergei Golubchik made changes -
            Assignee Julius Goryavsky [ sysprg ]
            serg Sergei Golubchik made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            sysprg Julius Goryavsky made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Component/s Plugin - Hashicorp Key Management [ 19200 ]
            sysprg Julius Goryavsky made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            sysprg Julius Goryavsky made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            sysprg Julius Goryavsky made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            sysprg Julius Goryavsky made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            sysprg Julius Goryavsky made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            sysprg Julius Goryavsky made changes -
            sysprg Julius Goryavsky made changes -
            Priority Critical [ 2 ] Minor [ 4 ]
            sysprg Julius Goryavsky made changes -
            Priority Minor [ 4 ] Major [ 3 ]

            sysprg, why is it not possible to re-encrypt the table? As far as I can see from the plugin code, it caches the latest key version, so it might immediately notice that the key was rotated, but the cache expires in cache_version_timeout milliseconds (which is 0 by default, btw). Why the server doesn't notice that the key was rotated in the vault?

            serg Sergei Golubchik added a comment - sysprg , why is it not possible to re-encrypt the table? As far as I can see from the plugin code, it caches the latest key version, so it might immediately notice that the key was rotated, but the cache expires in cache_version_timeout milliseconds (which is 0 by default, btw). Why the server doesn't notice that the key was rotated in the vault?

            Ok, so the problem is not that it's "not possible to re-encrypt the table". The issue is that hashicorp_key_management plugin caches replies from Hashicorp Vault and these values never expire.

            By default the caching is enabled, but cache_version_timeout is zero.

            The workaround for the user is to set cache_version_timeout. The fix for MariaDB is to set the default to some meaningful value.

            serg Sergei Golubchik added a comment - Ok, so the problem is not that it's "not possible to re-encrypt the table". The issue is that hashicorp_key_management plugin caches replies from Hashicorp Vault and these values never expire. By default the caching is enabled, but cache_version_timeout is zero. The workaround for the user is to set cache_version_timeout. The fix for MariaDB is to set the default to some meaningful value.
            Sebieee Sebastian added a comment -

            I'm also interested in the key rotation mechanism of this plugin. The official documentation is not clear if this plugin supports it or not and I'm trying to find it out manually, but an official statement and an example would be highly appreciated.

            Sebieee Sebastian added a comment - I'm also interested in the key rotation mechanism of this plugin. The official documentation is not clear if this plugin supports it or not and I'm trying to find it out manually, but an official statement and an example would be highly appreciated.

            As far as I understand, the plugin supports key rotation just fine. But if you have enabled key caching (so that the plugin wouldn't need to query Hashicorp Vault all the time) and you have disabled key expiration timeout, then the key will stay forever in the cache and the plugin will not notice that the key was rotated in the Hashicorp Vault until you restart the server.

            Key caching is generally a good idea, an infinite timeout is not.

            serg Sergei Golubchik added a comment - As far as I understand, the plugin supports key rotation just fine. But if you have enabled key caching (so that the plugin wouldn't need to query Hashicorp Vault all the time) and you have disabled key expiration timeout, then the key will stay forever in the cache and the plugin will not notice that the key was rotated in the Hashicorp Vault until you restart the server. Key caching is generally a good idea, an infinite timeout is not.
            Sebieee Sebastian added a comment -

            Indeed I see that when I write a new version of the enc key in Vault, MariaDB sees it if I select data from information_schema.INNODB_TABLESPACES_ENCRYPTION (i.e. Current Key Version corresponds to the latest version in Vault, while Min Key Version is the older one used in encryption). But how do I trigger MariaDB to encrypt the tables using the newer version?

            Sebieee Sebastian added a comment - Indeed I see that when I write a new version of the enc key in Vault, MariaDB sees it if I select data from information_schema.INNODB_TABLESPACES_ENCRYPTION (i.e. Current Key Version corresponds to the latest version in Vault, while Min Key Version is the older one used in encryption). But how do I trigger MariaDB to encrypt the tables using the newer version?

            It's beyond the hashicorp plugin. Generally it should be explained here: https://mariadb.com/kb/en/innodb-background-encryption-threads/ You'll need innodb_encryption_threads (like the original reporter did) and innodb_encryption_rotate_key_age (which is 1 by default, so, likely, ok).

            We're now trying to repeat this issue

            serg Sergei Golubchik added a comment - It's beyond the hashicorp plugin. Generally it should be explained here: https://mariadb.com/kb/en/innodb-background-encryption-threads/ You'll need innodb_encryption_threads (like the original reporter did) and innodb_encryption_rotate_key_age (which is 1 by default, so, likely, ok). We're now trying to repeat this issue
            sysprg Julius Goryavsky made changes -
            sysprg Julius Goryavsky made changes -
            sysprg Julius Goryavsky made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            julien.fritsch Julien Fritsch made changes -
            julien.fritsch Julien Fritsch made changes -
            Labels Cloned
            ralf.gebhardt Ralf Gebhardt made changes -
            Labels Cloned
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.11 [ 27614 ]
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.9 [ 26905 ]

            People

              sysprg Julius Goryavsky
              Slm0n Simon Stier
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.