[MDEV-24442] Assertion `space->referenced()' failed in fil_crypt_space_needs_rotation Created: 2020-12-18  Updated: 2020-12-19  Resolved: 2020-12-19

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.7, 10.5.8
Fix Version/s: 10.5.9, 10.6.0

Type: Bug Priority: Major
Reporter: Matthias Leich Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: rr-profile-analyzed

Issue Links:
Problem/Incident
is caused by MDEV-23855 InnoDB log checkpointing causes regre... Closed

 Description   

Assert hit during RQG testing
mysqld: storage/innobase/fil/fil0crypt.cc:1170: bool fil_crypt_space_needs_rotation(rotate_thread_t*, key_state_t*, bool*): Assertion `space->referenced()' failed.
# 2020-12-18T02:48:16 [1976796] | [rr 1977817 487007]201218  2:46:36 [rr 1977817 487010][ERROR] mysqld got signal 6 ;
...
(rr) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f1e3fe5a859 in __GI_abort () at abort.c:79
#2  0x00007f1e3fe5a729 in __assert_fail_base (fmt=0x7f1e3fff0588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x561cf33d19a0 "space->referenced()", file=0x561cf33d19e0 "/Server/10.5A/storage/innobase/fil/fil0crypt.cc", line=1170, 
    function=<optimized out>) at assert.c:92
#3  0x00007f1e3fe6bf36 in __GI___assert_fail (assertion=assertion@entry=0x561cf33d19a0 "space->referenced()", file=file@entry=0x561cf33d19e0 "/Server/10.5A/storage/innobase/fil/fil0crypt.cc", line=line@entry=1170, 
    function=function@entry=0x561cf33d6b20 "bool fil_crypt_space_needs_rotation(rotate_thread_t*, key_state_t*, bool*)") at assert.c:101
#4  0x0000561cf25eba19 in fil_crypt_space_needs_rotation (state=state@entry=0x32f96fd10d60, key_state=key_state@entry=0x32f96fd10d40, recheck=recheck@entry=0x32f96fd10d30) at /usr/include/c++/9/bits/atomic_base.h:413
#5  0x0000561cf25ecdd0 in fil_crypt_find_space_to_rotate (key_state=key_state@entry=0x32f96fd10d40, state=state@entry=0x32f96fd10d60, recheck=recheck@entry=0x32f96fd10d30) at /Server/10.5A/storage/innobase/fil/fil0crypt.cc:1595
#6  0x0000561cf25ed6d4 in fil_crypt_thread () at /Server/10.5A/storage/innobase/fil/fil0crypt.cc:1145
#7  0x000040ab6031d609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007f1e3ff57293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(rr)
 
origin/10.5 a1974d1991f808146c11ca0ada3724e8bafd5fb8 2020-12-18T10:00:57+02:00
 
RQG
-------
git clone https://github.com/mleich1/rqg --branch experimental RQG
 
perl rqg.pl \                  
--no-mask \
--seed=1608288279 \
--duration=300 \
--engine=InnoDB \
--queries=100000000 \
--mysqld=--loose_innodb_use_native_aio=1 \
--reporters=Backtrace,ErrorLog,Deadlock1,RestartConsistency \
--restart_timeout=120 \
--mysqld=--log_output=none \
--sqltrace=MarkErrors \
--grammar=conf/mariadb/innodb_compression_encryption1.yy \
--gendata=conf/mariadb/innodb_compression_encryption.zz \
--max_gd_duration=1500 \
--mysqld=--loose-innodb-use-atomic-writes \
--mysqld=--plugin-load-add=file_key_management.so \
--mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
--mysqld=--loose-innodb_buffer_pool_instances=1 \
--mysqld=--loose-innodb_doublewrite=1 \
--mysqld=--loose-innodb-encryption-threads=7 \
--mysqld=--loose-innodb_encryption_rotate_key_age=0 \
--threads=13 \
--mysqld=--innodb-encrypt-log \
--mysqld=--innodb-encrypt-tables \
--duration=300 \
--no_mask \
--workdir=<local settings> \
--vardir=<local settings> \
--mtr-build-thread=<local settings> \
--basedir1=<local settings> \
--script_debug=_nix_ \
--rr=Extended \
--rr_options=--chaos



 Comments   
Comment by Matthias Leich [ 2020-12-18 ]

sdp:/RQG2/storage/1608287267/TBR-798/dev/shm/vardir/1608287267/34/1/rr
_RR_TRACE_DIR="." rr replay --mark-stdio

Comment by Marko Mäkelä [ 2020-12-18 ]

I think that this bug (a missing check of the STOPPING flag) in fil_space_t::acquire_if_not_stopped() was introduced in MDEV-23855. It causes a race condition between DROP TABLE (or any table-rebuilding DDL that causes the deletion of .ibd files) and encryption threads for innodb_encrypt_tables.

Generated at Thu Feb 08 09:30:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.