[MDEV-24426] fil_crypt_thread keep spinning even if innodb_encryption_rotate_key_age=0 Created: 2020-12-17  Updated: 2021-01-04  Resolved: 2020-12-17

Status: Closed
Project: MariaDB Server
Component/s: Encryption, Storage Engine - InnoDB
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6
Fix Version/s: 10.5.9, 10.6.0

Type: Bug Priority: Major
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: performance

Issue Links:
Relates
relates to MDEV-14180 Automatically disable key rotation ch... Closed
relates to MDEV-11738 Mariadb uses 100% of several of my 8 ... Closed
relates to MDEV-21452 Use condition variables and normal mu... Closed
relates to MDEV-22258 Limit innodb_encryption_threads to 255 Closed

 Description   

With the following patch:

diff --git a/mysql-test/suite/mariabackup/xb_compressed_encrypted.opt b/mysql-test/suite/mariabackup/xb_compressed_encrypted.opt
 
index 8baef973470..265ee9fe2e0 100644
 
--- a/mysql-test/suite/mariabackup/xb_compressed_encrypted.opt
 
+++ b/mysql-test/suite/mariabackup/xb_compressed_encrypted.opt
 
@@ -1,5 +1,5 @@
 
 --innodb-encryption-rotate-key-age=2
 
---innodb-encryption-threads=4
 
+--innodb-encryption-threads=200
 
 --innodb-tablespaces-encryption
 
 --plugin-load-add=$FILE_KEY_MANAGEMENT_SO
 
 --loose-file-key-management

and with the invocation

./mtr --start mariabackup.xb_compressed_encrypted

I see high CPU usage on GNU/Linux on the 10.5 server. On the 10.6 server with MDEV-21452 fixed, the CPU usage is more moderate on GNU/Linux, but there is a huge amount of system calls. On Microsoft Windows, the changed system call pattern make the test run much slower, causing timeouts. This was successfully worked around by setting innodb_encryption_threads=1.

The anomaly is caused by the fact that fil_crypt_return_iops() is waking up all fil_crypt_thread even though innodb_encryption_rotate_key_age=0 and there is no work to do.

Unfortunately, for other settings than --innodb-encryption-rotate-key-age=0 a proper fix of this would require implementing a check whether the encryption plugin supports key versioning.

 Comments   
Comment by Marko Mäkelä [ 2020-12-18 ]

For the record, on 10.6 ever since MDEV-21452 was fixed, the encryption threads employ pthread_cond_wait() without any timeout. Before that, the threads would wake up 10 times per second even if there is no work to do. On 10.6, I checked the following on Debian GNU/Linux:

./mtr --start mariabackup.xb_compressed_encrypted &
 
sudo powertop

The powertop main screen would report the wake-ups for each thread. I took 3 samples of the output while the server process that was started by mtr was idle. The biggest thread with wakeups was the timer_handler, which is used by InnoDB as well, ever since MDEV-16264. That was 4.5 to 4.6 wakeups per second, in pthread_cond_timedwait(). I think that InnoDB should request 1.2 wakeups per second ever since MDEV-24280 was fixed. No InnoDB threads were waking up during the measurement period, thanks to this fix (and the test now employing innodb_encryption_rotate_key_age=0) as well as the previous fixes MDEV-24278, MDEV-24270.

Apart from the timer_handler, only two other threads were reported by powertop: ma_checkpoint_background (0.05 wakeups/second), and a random sample (0.00 wakeups/second) for tpool::thread_pool_generic::worker_main using a timed wait in tpool::thread_pool_generic::wait_for_tasks.

Comment by Matthias Leich [ 2020-12-18 ] 

Develop the RQG test
 
- grammar     conf/mariadb/innodb_compression_encryption1.yy
 
- setup           conf/mariadb/innodb_compression_encryption2.cc
 
in order to cover areas of functionality which are especially
 
affected by MDEV-24426.
 
Check that test+setup etc.

Comment by Marko Mäkelä [ 2021-01-04 ]

The remaining problem should be addressed by MDEV-14180.

Generated at Thu Feb 08 09:29:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.