Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5, 10.6
-
None
Description
Scenario is like the following:
- When one of the encryption thread does decryption of tablespace. (flushing of rotated pages happening)
- Test case sets INNODB_ENCRYPT_TABLES = 1
- Other encryption thread does check whether the space is eligible for key rotation
and removes it from default encrypt list.
|
|
encryption.innodb_encryption_filekeys 'cbc,innodb' w2 [ fail ]
|
Test ended at 2021-06-21 18:39:22
|
|
|
CURRENT_TEST: encryption.innodb_encryption_filekeys
|
mysqltest: At line 116: Timeout waiting for encryption threads
|
|
|
CREATE TABLE t2 (pk INT PRIMARY KEY AUTO_INCREMENT, c VARCHAR(256)) ENGINE=INNODB ENCRYPTED=YES;
|
CREATE TABLE t3 (pk INT PRIMARY KEY AUTO_INCREMENT, c VARCHAR(256)) ENGINE=INNODB ENCRYPTED=NO;
|
CREATE TABLE t4 (pk INT PRIMARY KEY AUTO_INCREMENT, c VARCHAR(256)) ENGINE=INNODB ENCRYPTED=YES ENCRYPTION_KEY_ID=4;
|
SET GLOBAL innodb_encrypt_tables = on;
|
# Wait max 10 min for key encryption threads to encrypt required all spaces
|
SELECT NAME,ENCRYPTION_SCHEME,MIN_KEY_VERSION, ROTATING_OR_FLUSHING FROM INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION;
|
NAME ENCRYPTION_SCHEME MIN_KEY_VERSION ROTATING_OR_FLUSHING
|
innodb_system 0 0 0
|
mysql/innodb_table_stats 1 1 0
|
mysql/innodb_index_stats 1 1 0
|
mysql/transaction_registry 1 1 0
|
test/t1 1 1 0
|
test/t2 1 1 0
|
test/t3 0 0 0
|
test/t4 1 1 0
|
SHOW STATUS LIKE 'innodb_encryption%';
|
Variable_name Value
|
Innodb_encryption_rotation_pages_read_from_cache 3065
|
Innodb_encryption_rotation_pages_read_from_disk 32
|
Innodb_encryption_rotation_pages_modified 2958
|
Innodb_encryption_rotation_pages_flushed 3274
|
Innodb_encryption_rotation_estimated_iops 64159
|
Innodb_encryption_key_rotation_list_length 0
|
Innodb_encryption_n_merge_blocks_encrypted 0
|
Innodb_encryption_n_merge_blocks_decrypted 0
|
Innodb_encryption_n_rowlog_blocks_encrypted 0
|
Innodb_encryption_n_rowlog_blocks_decrypted 0
|
|
Solution could be the only the last active encryption thread working on the thread
should be allowed to remove the tablespace from default encrypt list.
Right now, the Workaround can be set global innodb_encrypt_tables= ON/OFF; as per user wish.
Attachments
Issue Links
- is caused by
-
-
- Closed
-
Activity
Raising the priority because the test is failing very frequently in buildbot (10-20 times a day on average), it needs to be fixed.
Added the patch to avoid the failures in buildbot as of now. It is a workaround patch to stop the noise in buildbot. Patch
is in bb-10.2-MDEV-25998, bb-10.6-MDEV-25998
I see that the 10.2 version is a test-only change, while the 10.6 version is a code-only change.
In the 10.6 code-only change, the new function fil_crypt_space_remove_list() is reading srv_encrypt_tables multiple times, which looks suboptimal to me, but should not be a correctness issue, because both that function and SET GLOBAL innodb_encrypt_tables are protected by fil_system.mutex. I would appreciate an assertion to document the mutex ownership. That change passed 3,400 runs of encryption.innodb_encryption_keys without any problems for me. So, it should be OK to push after some cleanup.
#0 __GI___pthread_getspecific (key=1) at pthread_getspecific.c:30#1 0x000055d3c9a8ab78 in my_get_thread_local (key=1) at /home/thiru/mariarepo/10.6/10.6-sample/storage/perfschema/my_thread.h:41#2 0x000055d3c9a8ac60 in my_thread_get_THR_PFS () at /home/thiru/mariarepo/10.6/10.6-sample/storage/perfschema/pfs.cc:1371#3 0x000055d3c9a8ea8b in pfs_start_mutex_wait_v1 (state=0x7f1bba7fb720, mutex=0x7f1bf27655c0, op=PSI_MUTEX_LOCK,src_file=0x55d3ca6208e0 "/home/thiru/mariarepo/10.6/10.6-sample/storage/innobase/buf/buf0flu.cc", src_line=1522)at /home/thiru/mariarepo/10.6/10.6-sample/storage/perfschema/pfs.cc:2627#4 0x000055d3ca005215 in psi_mutex_lock (that=0x55d3cae4fc00 <buf_pool>,file=0x55d3ca6208e0 "/home/thiru/mariarepo/10.6/10.6-sample/storage/innobase/buf/buf0flu.cc", line=1522)at /home/thiru/mariarepo/10.6/10.6-sample/mysys/my_thr_init.c:484#5 0x000055d3c9e641b9 in inline_mysql_mutex_lock (that=0x55d3cae4fc00 <buf_pool>,src_file=0x55d3ca6208e0 "/home/thiru/mariarepo/10.6/10.6-sample/storage/innobase/buf/buf0flu.cc", src_line=1522)at /home/thiru/mariarepo/10.6/10.6-sample/include/mysql/psi/mysql_thread.h:746#6 0x000055d3c9e6ab82 in buf_flush_list_space (space=0x55d3cbe15768, n_flushed=0x7f1bba7fb7f8)at /home/thiru/mariarepo/10.6/10.6-sample/storage/innobase/buf/buf0flu.cc:1522#7 0x000055d3c9ee5ee2 in fil_crypt_flush_space (state=0x7f1bba7fbd60)at /home/thiru/mariarepo/10.6/10.6-sample/storage/innobase/fil/fil0crypt.cc:1972#8 0x000055d3c9ee6386 in fil_crypt_complete_rotate_space (state=0x7f1bba7fbd60)at /home/thiru/mariarepo/10.6/10.6-sample/storage/innobase/fil/fil0crypt.cc:2059#9 0x000055d3c9ee6658 in fil_crypt_thread () at /home/thiru/mariarepo/10.6/10.6-sample/storage/innobase/fil/fil0crypt.cc:2131#10 0x000055d3c9e702d0 in std::__invoke_impl<void, void (*)()> (__f=@0x7f1bb41f1be8: 0x55d3c9ee6419 <fil_crypt_thread()>)at /usr/include/c++/8/bits/invoke.h:60#11 0x000055d3c9e700b2 in std::__invoke<void (*)()> (__fn=@0x7f1bb41f1be8: 0x55d3c9ee6419 <fil_crypt_thread()>)at /usr/include/c++/8/bits/invoke.h:95#12 0x000055d3c9e70844 in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x7f1bb41f1be8)at /usr/include/c++/8/thread:244#13 0x000055d3c9e7081a in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x7f1bb41f1be8)at /usr/include/c++/8/thread:253#14 0x000055d3c9e707fe in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x7f1bb41f1be0)at /usr/include/c++/8/thread:196#15 0x00007f1bf0d3bd80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6#16 0x00007f1bf124a6db in start_thread (arg=0x7f1bba7fc700) at pthread_create.c:463#17 0x00007f1bf03e571f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95Above stack trace when innodb_encrypt_tables is 1.
Other encryption thread is skipping the tablespace to encrypt because of the following code:
In fil_crypt_space_needs_rotation():/* prevent threads from starting to rotate space */if (crypt_data->rotate_state.starting) {/* recheck this space later */*recheck = true;break;}/* prevent threads from starting to rotate space */if (space->is_stopping()) {break;}if (crypt_data->rotate_state.flushing) {break;}