Details
-
Bug
-
Status: Confirmed (View Workflow)
-
Critical
-
Resolution: Unresolved
-
11.4.1, 10.11
-
None
Description
how to reproduce:
- create data-at-rest encryption setup with hashicorp key mgmt plugin
- create a key on the hashicorp vault side
- create a table and encrypty it using the generated key
- create two new key versions on the vault server (two are needed at minimum for background key encoding to kick in)
while re-encryption is ongoing the mariadbd process takes about 10-15% CPU in my test setup
after re-encryption is complete the mariadbd process still shows CPU usage fluctuating between 10-30% even though the server should be completely idle, and in per-thread view in "top" it is clear that it is the background encryption threads that consume the CPU time
then changing hashicorp_key_management_cache_version_timeout from its default 0 to 100 milliseconds the CPU usage goes up to some 100 to 130%.
my config settings:
[mysqld]
|
performance_schema=ON
|
|
|
plugin_load_add = hashicorp_key_management.so
|
hashicorp-key-management
|
hashicorp-key-management-vault-url="..."
|
hashicorp-key-management-token="...."
|
|
|
innodb_encrypt_tables = FORCE
|
innodb_encrypt_log = ON
|
innodb_encrypt_temporary_tables = ON
|
innodb_encryption_threads = 4
|
innodb_encryption_rotate_key_age = 1
|
innodb_default_encryption_key_id = 1
|
innodb_encryption_rotation_iops=100
|
|
|
aria_encrypt_tables = ON
|
|
|
encrypt_tmp_disk_tables = ON
|
encrypt_tmp_files = ON
|
|
|
log-bin
|
encrypt_binlog = ON
|
Attachments
Issue Links
- is caused by
-
MDEV-19281 Vault Key Management Plugin
-
- Closed
-
Activity
marko, new versions are created manually, using the vault command. I'm sure, the bug reporter means that threads consume CPU after the reencryption (caused by such vault command) has finished, but before a new version was generated by another vault command.
It's affected by hashicorp_key_management_cache_version_timeout, so it seems that they're constantly querying the vault for new key versions?
Not sure whether it's exactly the same, but at least seems to have the same root cause as MDEV-29328
The CPU usage only really grows after re-encryption of tables using the new key version has finished, aka.
SELECT *
|
FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION
|
WHERE ROTATING_OR_FLUSHING = 1;
|
does no longer show any result rows.
I see that the Hashicorp encryption plugin was added in
MDEV-19281. It should exist also in the 10.11 long-term-support release.If I understood it correctly, the Hashicorp encryption plugin supports key versioning. The simpler file_key_management does not, and thanks to
MDEV-14180it behaves differently. Could it perhaps be that new encryption key versions are being introduced constantly, and the database is simply busy making dummy writes to pages in order to re-encrypt them with newer key versions?