Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
10.11.9, 10.4(EOL)
-
None
Description
https://travis-ci.org/elenst/travis-tests/jobs/393093617
|
10.2 b8514c94f61f37757a24df1c6b9a7dd530b3de12 |
mysqld: /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143: void rocksdb::{anonymous}::LevelCompactionBuilder::SetupInitialFiles(): Assertion `!level_and_file.second->being_compacted' failed.
|
180616 19:22:08 [ERROR] mysqld got signal 6 ;
|
|
|
#7 0x00007f9379b21ca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
|
#8 0x00007f93718f4214 in rocksdb::(anonymous namespace)::LevelCompactionBuilder::SetupInitialFiles (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1143
|
#9 0x00007f93718f46dd in rocksdb::(anonymous namespace)::LevelCompactionBuilder::PickCompaction (this=0x7f9365756510) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1232
|
#10 0x00007f93718f5497 in rocksdb::LevelCompactionPicker::PickCompaction (this=0x55ab0393e100, cf_name="default", mutable_cf_options=..., vstorage=0x7f9358034290, log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/compaction_picker.cc:1431
|
#11 0x00007f93718c534a in rocksdb::ColumnFamilyData::PickCompaction (this=0x55ab0393bb10, mutable_options=..., log_buffer=0x7f93657573a0) at /home/travis/src/storage/rocksdb/rocksdb/db/column_family.cc:838
|
#12 0x00007f9371605f96 in rocksdb::DBImpl::BackgroundCompaction (this=0x55ab0392e6f0, made_progress=0x7f936575706f, job_context=0x7f93657570c0, log_buffer=0x7f93657573a0, prepicked_compaction=0x0) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1541
|
#13 0x00007f937160519f in rocksdb::DBImpl::BackgroundCallCompaction (this=0x55ab0392e6f0, prepicked_compaction=0x0, bg_thread_pri=rocksdb::Env::LOW) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1358
|
#14 0x00007f93716042fd in rocksdb::DBImpl::BGWorkCompaction (arg=0x7f9358008b80) at /home/travis/src/storage/rocksdb/rocksdb/db/db_impl_compaction_flush.cc:1177
|
#15 0x00007f937195f96c in rocksdb::ThreadPoolImpl::__lambda0::operator() (__closure=0x7f93580096b0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:441
|
#16 0x00007f937195fdfa in std::_Function_handler<void(), rocksdb::ThreadPoolImpl::Schedule(void (*)(void*), void*, void*, void (*)(void*))::__lambda0>::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/4.8/functional:2071
|
#17 0x00007f9371859b96 in std::function<void ()>::operator()() const (this=0x7f9365757db0) at /usr/include/c++/4.8/functional:2471
|
#18 0x00007f937195ed54 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55ab036c4b90, thread_id=0) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:240
|
#19 0x00007f937195eeb7 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55ab0392db00) at /home/travis/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:278
|
#20 0x00007f9371967cde in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1732
|
#21 0x00007f9371967be9 in std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::operator()() (this=0x55ab0392db50) at /usr/include/c++/4.8/functional:1720
|
#22 0x00007f9371967b82 in std::thread::_Impl<std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)> >::_M_run() (this=0x55ab0392db38) at /usr/include/c++/4.8/thread:115
|
#23 0x00007f937a488a60 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
|
#24 0x00007f937a6e3184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
|
#25 0x00007f9379befffd in clone () from /lib/x86_64-linux-gnu/libc.so.6
|
|
travis-workarounds c1710043b35269d4bae46f892bfccc02cd7cf2e2 |
perl /home/travis/rqg/runall-new.pl --basedir=/home/travis/server --vardir=/home/travis/logs/vardir --engine=RocksDB --duration=350 --threads=6 --seed=1529176799 --reporters=Backtrace,ErrorLog,Deadlock --validators=TransformerNoComparator --views --redefine=conf/mariadb/alter_table.yy --redefine=conf/mariadb/bulk_insert.yy --mysqld=--log_output=FILE --mysqld=--lock-wait-timeout=10 --mysqld=--loose-innodb-lock-wait-timeout=5 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --mysqld=--default-storage-engine=RocksDB --mysqld=--plugin-load-add=ha_rocksdb --grammar=conf/runtime/performance_schema.yy --mysqld=--performance-schema --ps-protocol --transformers=ExecuteAsCTE,ExecuteAsDeleteReturning,ExecuteAsInsertSelect,ExecuteAsUnion,ExecuteAsUpdateDelete,ExecuteAsView,ExecuteAsSPTwice
|
Not reproducible so far.
Stack traces are attached.
Coredump, datadir etc. are available.
Attachments
Issue Links
- is blocked by
-
-
- Closed
-
- relates to
-
MDEV-30610 Update RocksDB to the latest upstream version
-
- Open
-
Activity
The test from the description fails fairly easily still, but it fails with MDEV-15164 instead of the failure reported here; MDEV-15164 is another assertion failure in compaction, which is apparently more probably to happen.
If we want to get to analyzing the failure described here, MDEV-15164 needs to be fixed first. Either it will fix both at once, or at least will remove the obstacle on the way to reproduce and debug.
MDEV-15164 needs a new test case though, something must have changed on the locking side since it was filed, so the test case from there does not work the same way as it did before.
Right. Updating to RocksDB 6.29 isn't "just update the submodule" - I can already see APIs changed and files moved, but it is still easier than update to RocksDB 8.x.
https://github.com/MariaDB/server/tree/bb-10.11-mdev16253-update-rocksdb-6.29fb - this updates RocksDB to https://github.com/facebook/rocksdb/tree/6.29.fb .
There are some test failures still (I also had some before the RocksDB update)
Ok, the commit
|
|
commit 281dc05bb764c737d35b51c9da25b7ddb636d51e (HEAD -> bb-10.11.11-mdev16253-rocksdb-6.29fb, origin/bb-10.11.11-mdev16253-rocksdb-6.29fb)
|
Author: Sergei Petrunia <sergey@mariadb.com>
|
Date: Fri Mar 21 11:43:31 2025 +0200
|
|
|
MDEV-16523, In MariaDB 10.11.11, update RocksDB to 6.29fb. (try3)
|
seems to be good. Waiting for https://buildbot.mariadb.org/#/grid?branch=bb-10.11.11-mdev16253-rocksdb-6.29fb to finish for exotic platforms.
... it has finished and it looks Ok. We're ready to create a custom build from this.
Running a generic/broad RocksDB test on bb-10.11.11-mdev16253-rocksdb-6.29fb as discussed.
Non-actionable, FYI
Created MDEV-36611 (SIGSEGV & UBASAN null-pointer-use in myrocks::Rdb_transaction_impl::get_write_batch): also present in current versions
Created MDEV-36624 (UBSAN: invalid-shift-exponent: shift exponent 32 is too large in myrocks::ha_rocksdb::info): also present in current versions
Created MDEV-36632 (Assertion `check_field_pointers(table)' failed and SIGSEGV): also present in current versions
Created MDEV-36640 (RocksDB Corruption/errors/assert/SIGSEGV upon CHECK TABLE): also present in current versions
Created MENT-2277 (ES 10.6 Optimized/Release has significant differences when it comes to RocksDB stability): no connection with this 6.29fb branch
Various other pre-existing RocksDB bugs also seen in bb-10.11.11-mdev16253-rocksdb-6.29fb
Actionable
None
I am still working through the test results, and there are a few RocksDB issues remaining, however they are either low frequency, look benign or look related to other [now] known issues. Otherwise the summary is as per the last comment; all new bugs are also present in current versions.
I also tested MDEV-15164, and it is considered fixed in bb-10.11.11-mdev16253-rocksdb-6.29fb. As for this bug (MDEV-16523), no occurences of being_compacted were seen.
If the push of bb-10.11.11-mdev16253-rocksdb-6.29fb is urgent, I believe we can proceed now.
In any case, I will roundup the runs.
Update: runs complete. All other remaining RocksDB issues were present in base versions as well. Fine to push, which already happened.
Still happens:
https://travis-ci.org/elenst/travis-tests/jobs/409140012