[MDEV-16523] Assertion `!level_and_file.second->being_compacted' failed in LevelCompactionBuilder::SetupInitialFiles Created: 2018-06-19  Updated: 2023-01-20

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: None

Attachments: HTML File threads     HTML File threads_full    
Issue Links:
Blocks
is blocked by MDEV-15164 Assertion `ikey_.type == kTypeValue' ... Confirmed

 Description   

https://travis-ci.org/elenst/travis-tests/jobs/393093617

10.2 b8514c94f61f37757a24df1c6b9a7dd530b3de12

mysqld: /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143: void rocksdb::{anonymous}::LevelCompactionBuilder::SetupInitialFiles(): Assertion `!level_and_file.second->being_compacted' failed.
180616 19:22:08 [ERROR] mysqld got signal 6 ;
 
#7  0x00007f9379b21ca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x00007f93718f4214 in rocksdb::(anonymous namespace)::LevelCompactionBuilder::SetupInitialFiles (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143
#9  0x00007f93718f46dd in rocksdb::(anonymous namespace)::LevelCompactionBuilder::PickCompaction (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1232
#10 0x00007f93718f5497 in rocksdb::LevelCompactionPicker::PickCompaction (this=0x55ab0393e100, cf_name="default", mutable_cf_options=..., vstorage=0x7f9358034290, log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1431
#11 0x00007f93718c534a in rocksdb::ColumnFamilyData::PickCompaction (this=0x55ab0393bb10, mutable_options=..., log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/column_family.cc:838
#12 0x00007f9371605f96 in rocksdb::DBImpl::BackgroundCompaction (this=0x55ab0392e6f0, made_progress=0x7f936575706f, job_context=0x7f93657570c0, log_buffer=0x7f93657573a0, prepicked_compaction=0x0) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1541
#13 0x00007f937160519f in rocksdb::DBImpl::BackgroundCallCompaction (this=0x55ab0392e6f0, prepicked_compaction=0x0, bg_thread_pri=rocksdb::Env::LOW) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1358
#14 0x00007f93716042fd in rocksdb::DBImpl::BGWorkCompaction (arg=0x7f9358008b80) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1177
#15 0x00007f937195f96c in rocksdb::ThreadPoolImpl::__lambda0::operator() (__closure=0x7f93580096b0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:441
#16 0x00007f937195fdfa in std::_Function_handler<void(), rocksdb::ThreadPoolImpl::Schedule(void (*)(void*), void*, void*, void (*)(void*))::__lambda0>::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/4.8/functional:2071
#17 0x00007f9371859b96 in std::function<void ()>::operator()() const (this=0x7f9365757db0) at /usr/include/c++/4.8/functional:2471
#18 0x00007f937195ed54 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55ab036c4b90, thread_id=0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:240
#19 0x00007f937195eeb7 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55ab0392db00) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:278
#20 0x00007f9371967cde in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1732
#21 0x00007f9371967be9 in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::operator()() (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1720
#22 0x00007f9371967b82 in std::thread::_Impl<std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)> >::_M_run() (this=0x55ab0392db38) at /usr/include/c++/4.8/thread:115
#23 0x00007f937a488a60 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#24 0x00007f937a6e3184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#25 0x00007f9379befffd in clone () from /lib/x86_64-linux-gnu/libc.so.6

travis-workarounds c1710043b35269d4bae46f892bfccc02cd7cf2e2

perl /home/travis/rqg/runall-new.pl --basedir=/home/travis/server --vardir=/home/travis/logs/vardir --engine=RocksDB --duration=350 --threads=6 --seed=1529176799 --reporters=Backtrace,ErrorLog,Deadlock --validators=TransformerNoComparator --views --redefine=conf/mariadb/alter_table.yy --redefine=conf/mariadb/bulk_insert.yy --mysqld=--log_output=FILE --mysqld=--lock-wait-timeout=10 --mysqld=--loose-innodb-lock-wait-timeout=5 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --mysqld=--default-storage-engine=RocksDB --mysqld=--plugin-load-add=ha_rocksdb --grammar=conf/runtime/performance_schema.yy --mysqld=--performance-schema --ps-protocol --transformers=ExecuteAsCTE,ExecuteAsDeleteReturning,ExecuteAsInsertSelect,ExecuteAsUnion,ExecuteAsUpdateDelete,ExecuteAsView,ExecuteAsSPTwice

Not reproducible so far.
Stack traces are attached.
Coredump, datadir etc. are available.



 Comments   
Comment by Elena Stepanova [ 2018-07-29 ]

Still happens:
https://travis-ci.org/elenst/travis-tests/jobs/409140012

Comment by tom wang [ 2019-08-28 ]

I had this same problem, but I don't know why.

Comment by Elena Stepanova [ 2022-08-25 ]

The test from the description fails fairly easily still, but it fails with MDEV-15164 instead of the failure reported here; MDEV-15164 is another assertion failure in compaction, which is apparently more probably to happen.

If we want to get to analyzing the failure described here, MDEV-15164 needs to be fixed first. Either it will fix both at once, or at least will remove the obstacle on the way to reproduce and debug.

MDEV-15164 needs a new test case though, something must have changed on the locking side since it was filed, so the test case from there does not work the same way as it did before.

Comment by Richard Stracke [ 2022-11-04 ]

https://review.spdk.io/gerrit/plugins/gitiles/spdk/rocksdb/+/ab8f6c01a6c48fd7b8c752a3ef0ef8640065dd48

Generated at Thu Feb 08 08:29:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.