[MDEV-29238] Hashicorp Plugin: Key rotation creates infinity request loop to vault (dos) Created: 2022-08-03  Updated: 2023-11-28

Status: Stalled
Project: MariaDB Server
Component/s: Encryption
Affects Version/s: 10.9.1
Fix Version/s: 10.11

Type: Bug Priority: Critical
Reporter: Simon Stier Assignee: Julius Goryavsky
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Ubuntu 22.04
Using apt packages:
mariadb-server 1:10.9.1+maria~jammy
mariadb-plugin-hashicorp-key-management 1:10.9.1+maria~jammy
Vault v1.11.2



 Description   

Hi,

during evaluation of the Hashicorp Plugin I noticed the problem, that after a rotation of the encryption key in vault (set version 2 of the secret) and a restart of mariadbd I got an request loop (dos) from the hashicorp plugin against vault.

Reproduce by:

  • Install mariadb-server & mariadb-plugin-hashicorp-key-management 10.9.1
  • Install hashicorp vault, init and unseal vault
  • create secret engine & set a secret key and enable audit_log for debugging purpose:

    vault secrets enable -path /mariadb -version=2 kv
    vault kv put /mariadb/1 data=$(openssl rand -hex 32)
    vault audit enable file file_path=/hashicorp/vault_audit.log
    

  • enable encryption by adding the following block to mariadb section in /etc/mysql/mariadb.conf.d/50-server.cnf:

    [mariadb]
     
    plugin_load_add = hashicorp_key_management
    hashicorp-key-management-vault-url=http://127.0.0.1:8200/v1/mariadb
    hashicorp-key-management-token=xxxxxxxxxxxxxxxxx
     
    innodb_encrypt_tables = FORCE
    innodb_encrypt_log = ON
    innodb_encrypt_temporary_tables = ON
     
    encrypt_tmp_disk_tables = ON
    encrypt_tmp_files = ON
    encrypt_binlog = ON
    aria_encrypt_tables = ON
     
    innodb_encryption_threads = 4
    innodb_encryption_rotation_iops = 2000
    log_error=server.log
    

  • set new key version

    vault kv put /mariadb/1 data=$(openssl rand -hex 32)
    

  • service mariadb restart
  • check /hashicorp/vault_audit.log mariadbd sends an endless loop of requests to vault
    Load of my 1vCPU Instance increases to 7 due to high cpu load of vault & mariadbd


 Comments   
Comment by Simon Stier [ 2022-08-03 ]

Update: The issue is related to the configuration of "innodb_encryption_threads = 4", which I copy/pasted from a tutorial. If I set it to 1 for my 1vCPU instance its all fine.
But this error should be catched somehow - after it starts a DOS to the Vault server.

Generated at Thu Feb 08 10:06:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.