Improve documentation of data at rest encryption

MariaDB's documentation for encryption-related features needs to be improved. Some examples of things that need to be improved in the documentation are:

• Counterintuitive key rotations:

The background encryption threads can automatically encrypt unencrypted tables in the background when the value of innodb_encrypt_tables changes to ON or FORCE. However, this is technically considered a "key rotation", so for this to work, it requires both innodb_encryption_threads>0 and innodb_encryption_rotate_key_age>0.

Note that the inverse is also true: if innodb_encrypt_tables is changed to OFF, then it is also considered a "key rotation" for the background thread to automatically decrypt encrypted tables, so for this to work, it also requires both innodb_encryption_threads>0 and innodb_encryption_rotate_key_age>0.

See MDEV-14398.

• System tablespace:

The system tablespace (i.e. the tablespace defined by innodb_data_file_path) can only currently be encrypted by the background encryption threads, so for this to be encrypted, it requires both innodb_encryption_threads>0 and innodb_encryption_rotate_key_age>0.

See MDEV-14571 and MDEV-14610.

• innodb_encryption_threads:

The documentation for innodb_encryption_threads doesn't contain much information:

Description: Number of threads performing background key rotation and scrubbing. See Table and Tablespace Encryption.

https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_encryption_threads

This might lead users to believe that they can safely set innodb_encryption_threads=0 if they never rotate their encryption keys and if they don't need data scrubbing. However, this is not exactly true. See the points about counterintuitive key rotations and system tablespace for more information about what the background encryption threads do.

• innodb_encryption_rotate_key_age:

The documentation for innodb_encryption_rotate_key_age doesn't contain much information:

Description: Re-encrypt in background any page having a key older than this. See Table and Tablespace Encryption.

https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_encryption_rotate_key_age

This might lead users to believe that they can safely set innodb_encryption_rotate_key_age=0 if they never rotate their encryption keys. However, this is not exactly true. See the points about counterintuitive key rotations and system tablespace for more information about what kinds of operations are considered "key rotations".

• information_schema.INNODB_TABLESPACES_ENCRYPTION:

The documentation for information_schema.INNODB_TABLESPACES_ENCRYPTION does not explicitly mention that the system tablespace has the "special" name "innodb_system". I'm not sure if this is the only "special" name used in this table.

The documentation also does not mention how unencrypted tablespaces are represented in this table. I've actually noticed two different behaviors--sometimes an unencrypted tablespace has no row at all in information_schema.INNODB_TABLESPACES_ENCRYPTION, and other times an unencrypted tablespace does have a row in the table, and the row shows that the tablespace is not encrypted by showing ENCRYPTION_SCHEME=0 and CURRENT_KEY_VERSION=0. It is not entirely clear to me when each behavior is used.

The documentation also does not mention the ROTATING_OR_FLUSHING column at all, so this field should be added and explained:

MariaDB [information_schema]> SHOW CREATE TABLE INNODB_TABLESPACES_ENCRYPTION\G

*************************** 1. row ***************************

       Table: INNODB_TABLESPACES_ENCRYPTION

Create Table: CREATE TEMPORARY TABLE `INNODB_TABLESPACES_ENCRYPTION` (

  `SPACE` int(11) unsigned NOT NULL DEFAULT '0',

  `NAME` varchar(655) DEFAULT NULL,

  `ENCRYPTION_SCHEME` int(11) unsigned NOT NULL DEFAULT '0',

  `KEYSERVER_REQUESTS` int(11) unsigned NOT NULL DEFAULT '0',

  `MIN_KEY_VERSION` int(11) unsigned NOT NULL DEFAULT '0',

  `CURRENT_KEY_VERSION` int(11) unsigned NOT NULL DEFAULT '0',

  `KEY_ROTATION_PAGE_NUMBER` bigint(21) unsigned DEFAULT NULL,

  `KEY_ROTATION_MAX_PAGE_NUMBER` bigint(21) unsigned DEFAULT NULL,

  `CURRENT_KEY_ID` int(11) unsigned NOT NULL DEFAULT '0',

  `ROTATING_OR_FLUSHING` int(1) unsigned NOT NULL DEFAULT '0'

) ENGINE=MEMORY DEFAULT CHARSET=utf8

1 row in set (0.00 sec)

https://mariadb.com/kb/en/library/information-schema-innodb_tablespaces_encryption-table/

See MDEV-14399.

The documentation should probably also answer the following questions:

• If innodb_encrypt_threads=0 or innodb_encryption_rotate_key_age=0 is set, what tables will not be encrypted and what other encryption-related features will be disabled?

I believe the answer is that if innodb_encrypt_threads=0 or innodb_encryption_rotate_key_age=0 is set, then no key rotations are done, which also means that tables cannot be automatically encrypted or decrypted in the background. See counterintuitive key rotations above. Tables can still be manually encrypted by executing the following:

ALTER TABLE tab ENCRYPTION=YES;

However, the problem also applies to the system tablespace, and there is no way to manually encrypt that at the moment. See system tablespace above.

• How do you know when it is safe to disable key rotation by setting innodb_encryption_rotate_key_age=0?

If you never plan to rotate your encryption keys, then it would make sense to disable key rotation checks, because those extra checks can hurt performance. However, you have to make sure that all background encryption operations (which are considered key rotations) are complete before disabling them. How do you know when this is complete? I believe the answer is that if the tablespace's row in information_schema.INNODB_TABLESPACES_ENCRYPTION still has ROTATING_OR_FLUSHING=1, then that means that the background encryption threads are still working on the tablespace. When ROTATING_OR_FLUSHING=0, that means that the background threads are finished with the tablespace. As an additional detail, if MIN_KEY_VERSION=0 for a given tablespace, it looks like that might mean that the tablespace still has unencrypted pages.

• If you only want to use the background thread for data scrubbing, and you don't care about encryption, what is the best way to optimize the configuration of the background threads?

Is it enough to just set innodb_encryption_rotate_key_age=0?

• For a normal page encryption operation (i.e. not a key rotation operation by a background encryption thread), when is the actual page encryption done?

Is the page encrypted right before the page is flushed to disk, similar to how page compression works? i.e.: https://mariadb.com/kb/en/library/compression/#monitoring-compression

The following pages/entries may need to be updated:

https://mariadb.com/kb/en/library/data-at-rest-encryption/

https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_encryption_threads

https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_encryption_rotate_key_age

https://mariadb.com/kb/en/library/information-schema-innodb_tablespaces_encryption-table/

And the following page may also need some updates, and it may be good to create some linkage between it and the encryption page:

https://mariadb.com/kb/en/library/xtradb-innodb-data-scrubbing/