[MDEV-20861] Assertion `ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion' failed in rocksdb::CompactionIterator::PrepareOutput Created: 2019-10-18  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 1
Labels: None


 Description   

The failure happens pretty often in my tests, but it's badly reproducible. I could make a further effort for creating at least a concurrent test case, but it doesn't make much sense at this point, since upstream has a probably bugfix already and it just needs to be merged. If the failure continues happening after the merge, then we'll chase it further.

10.2 fa929f7c ASAN

 
mysqld: /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:632: void rocksdb::CompactionIterator::PrepareOutput(): Assertion `ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion' failed.
 
191018  5:53:57 [ERROR] mysqld got signal 6 ;
 
 
 
#13 0x00007fa0f622ebd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7fa0ea895060 "ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion", file=file@entry=0x7fa0ea894a60 "/home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc", line=line@entry=632, function=function@entry=0x7fa0ea895fe0 <rocksdb::CompactionIterator::PrepareOutput()::__PRETTY_FUNCTION__> "void rocksdb::CompactionIterator::PrepareOutput()") at assert.c:92
 
#14 0x00007fa0f622ec82 in __GI___assert_fail (assertion=0x7fa0ea895060 "ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion", file=0x7fa0ea894a60 "/home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc", line=632, function=0x7fa0ea895fe0 <rocksdb::CompactionIterator::PrepareOutput()::__PRETTY_FUNCTION__> "void rocksdb::CompactionIterator::PrepareOutput()") at assert.c:101
 
#15 0x00007fa0ea56c96d in rocksdb::CompactionIterator::PrepareOutput (this=0x6170000dd300) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:632
 
#16 0x00007fa0ea5673a8 in rocksdb::CompactionIterator::Next (this=0x6170000dd300) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:155
 
#17 0x00007fa0ea57a6ad in rocksdb::CompactionJob::ProcessKeyValueCompaction (this=0x7fa0e86511a0, sub_compact=0x61400143f640) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:954
 
#18 0x00007fa0ea5754d1 in rocksdb::CompactionJob::Run (this=0x7fa0e86511a0) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:590
 
#19 0x00007fa0e9ea575c in rocksdb::DBImpl::BackgroundCompaction (this=0x622000004900, made_progress=0x7fa0e86518b0, job_context=0x7fa0e8651bf0, log_buffer=0x7fa0e8651df0, prepicked_compaction=0x603001397710, thread_pri=rocksdb::Env::LOW) at /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2689
 
#20 0x00007fa0e9e9fffc in rocksdb::DBImpl::BackgroundCallCompaction (this=0x622000004900, prepicked_compaction=0x603001397710, bg_thread_pri=rocksdb::Env::LOW) at /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2249
 
#21 0x00007fa0e9e9c8e2 in rocksdb::DBImpl::BGWorkCompaction (arg=0x60200002b030) at /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2024
 
#22 0x00007fa0ea6a4f33 in std::__invoke_impl<void, void (*&)(void*), void*&> (__f=@0x602000091f50: 0x7fa0e9e9c6fe <rocksdb::DBImpl::BGWorkCompaction(void*)>, __args#0=@0x602000091f58: 0x60200002b030) at /usr/include/c++/7/bits/invoke.h:60
 
#23 0x00007fa0ea6a49e4 in std::__invoke<void (*&)(void*), void*&> (__fn=@0x602000091f50: 0x7fa0e9e9c6fe <rocksdb::DBImpl::BGWorkCompaction(void*)>, __args#0=@0x602000091f58: 0x60200002b030) at /usr/include/c++/7/bits/invoke.h:95
 
#24 0x00007fa0ea6a3b39 in std::_Bind<void (*(void*))(void*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x602000091f50, __args=...) at /usr/include/c++/7/functional:467
 
#25 0x00007fa0ea6a2037 in std::_Bind<void (*(void*))(void*)>::operator()<, void>() (this=0x602000091f50) at /usr/include/c++/7/functional:551
 
#26 0x00007fa0ea69fcac in std::_Function_handler<void (), std::_Bind<void (*(void*))(void*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
 
#27 0x00007fa0e9c34336 in std::function<void ()>::operator()() const (this=0x7fa0e8652bf0) at /usr/include/c++/7/bits/std_function.h:706
 
#28 0x00007fa0ea6970e7 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x611000000a40, thread_id=0) at /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:266
 
#29 0x00007fa0ea697365 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x602000000bd0) at /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:307
 
#30 0x00007fa0ea69e8b7 in std::__invoke_impl<void*, void* (*)(void*), rocksdb::BGThreadMetadata*> (__f=@0x603000030800: 0x7fa0ea69720c <rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)>, __args#0=@0x6030000307f8: 0x602000000bd0) at /usr/include/c++/7/bits/invoke.h:60
 
#31 0x00007fa0ea69ca85 in std::__invoke<void* (*)(void*), rocksdb::BGThreadMetadata*> (__fn=@0x603000030800: 0x7fa0ea69720c <rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)>, __args#0=@0x6030000307f8: 0x602000000bd0) at /usr/include/c++/7/bits/invoke.h:96
 
#32 0x00007fa0ea6a5789 in std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::_M_invoke<0ul, 1ul> (this=0x6030000307f8) at /usr/include/c++/7/thread:234
 
#33 0x00007fa0ea6a56f5 in std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::operator() (this=0x6030000307f8) at /usr/include/c++/7/thread:243
 
#34 0x00007fa0ea6a5656 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> > >::_M_run (this=0x6030000307f0) at /usr/include/c++/7/thread:186
 
#35 0x00007fa0f6bbc1a0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 
#36 0x00007fa0f6ed66ba in start_thread (arg=0x7fa0e8653700) at pthread_create.c:333
 
#37 0x00007fa0f630841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
 
 
==8066==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000021b806e bp 0x7fa0e864cdb0 sp 0x7fa0e864ccc0 T3)
 
==8066==The signal is caused by a WRITE memory access.
 
    #0 0x21b806d in my_read /home/vsts/src/mysys/my_read.c:63
 
    #1 0xed056d in output_core_info /home/vsts/src/sql/signal_handler.cc:66
 
    #2 0xed1681 in handle_fatal_signal /home/vsts/src/sql/signal_handler.cc:339
 
    #3 0x7fa0f6ee038f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
 
    #4 0x7fa0f6236427 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35427)
 
    #5 0x7fa0f6238029 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37029)
 
    #6 0x7fa0f622ebd6  (/lib/x86_64-linux-gnu/libc.so.6+0x2dbd6)
 
    #7 0x7fa0f622ec81 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x2dc81)
 
    #8 0x7fa0ea56c96c in rocksdb::CompactionIterator::PrepareOutput() /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:632
 
    #9 0x7fa0ea5673a7 in rocksdb::CompactionIterator::Next() /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:155
 
    #10 0x7fa0ea57a6ac in rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:954
 
    #11 0x7fa0ea5754d0 in rocksdb::CompactionJob::Run() /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:590
 
    #12 0x7fa0e9ea575b in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2689
 
    #13 0x7fa0e9e9fffb in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2249
 
    #14 0x7fa0e9e9c8e1 in rocksdb::DBImpl::BGWorkCompaction(void*) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2024
 
    #15 0x7fa0ea6a4f32 in void std::__invoke_impl<void, void (*&)(void*), void*&>(std::__invoke_other, void (*&)(void*), void*&) /usr/include/c++/7/bits/invoke.h:60
 
    #16 0x7fa0ea6a49e3 in std::__invoke_result<void (*&)(void*), void*&>::type std::__invoke<void (*&)(void*), void*&>(void (*&)(void*), void*&) /usr/include/c++/7/bits/invoke.h:95
 
    #17 0x7fa0ea6a3b38 in void std::_Bind<void (*(void*))(void*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/7/functional:467
 
    #18 0x7fa0ea6a2036 in void std::_Bind<void (*(void*))(void*)>::operator()<, void>() /usr/include/c++/7/functional:551
 
    #19 0x7fa0ea69fcab in std::_Function_handler<void (), std::_Bind<void (*(void*))(void*)> >::_M_invoke(std::_Any_data const&) /usr/include/c++/7/bits/std_function.h:316
 
    #20 0x7fa0e9c34335 in std::function<void ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706
 
    #21 0x7fa0ea6970e6 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:266
 
    #22 0x7fa0ea697364 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:307
 
    #23 0x7fa0ea69e8b6 in void* std::__invoke_impl<void*, void* (*)(void*), rocksdb::BGThreadMetadata*>(std::__invoke_other, void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) /usr/include/c++/7/bits/invoke.h:60
 
    #24 0x7fa0ea69ca84 in std::__invoke_result<void* (*)(void*), rocksdb::BGThreadMetadata*>::type std::__invoke<void* (*)(void*), rocksdb::BGThreadMetadata*>(void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) /usr/include/c++/7/bits/invoke.h:96
 
    #25 0x7fa0ea6a5788 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)())) std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/7/thread:234
 
    #26 0x7fa0ea6a56f4 in std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::operator()() /usr/include/c++/7/thread:243
 
    #27 0x7fa0ea6a5655 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> > >::_M_run() /usr/include/c++/7/thread:186
 
    #28 0x7fa0f6bbc19f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd019f)
 
    #29 0x7fa0f6ed66b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
    #30 0x7fa0f630841c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
 
    #0 0x7fa0f871dd6f in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d6f)
 
    #1 0x7fa0f6bbc434 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0434)
 
    #2 0x7fa0ea697a93 in rocksdb::ThreadPoolImpl::Impl::StartBGThreads() /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:339
 
    #3 0x7fa0ea69760f in rocksdb::ThreadPoolImpl::Impl::SetBackgroundThreadsInternal(int, bool) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:325
 
    #4 0x7fa0ea6990eb in rocksdb::ThreadPoolImpl::IncBackgroundThreadsIfNeeded(int) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:461
 
    #5 0x7fa0ea514c5a in IncBackgroundThreadsIfNeeded /home/vsts/src/storage/rocksdb/rocksdb/env/env_posix.cc:1006
 
    #6 0x7fa0e9effab5 in rocksdb::SanitizeOptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::DBOptions const&) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_open.cc:61
 
    #7 0x7fa0e9dacee3 in rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl.cc:229
 
    #8 0x7fa0e9f1264e in rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_open.cc:1261
 
    #9 0x7fa0ea44f95e in rocksdb::TransactionDB::Open(rocksdb::DBOptions const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::TransactionDB**) /home/vsts/src/storage/rocksdb/rocksdb/utilities/transactions/pessimistic_transaction_db.cc:249
 
    #10 0x7fa0e9bb3545 in rocksdb_init_func /home/vsts/src/storage/rocksdb/ha_rocksdb.cc:5587
 
    #11 0xed7c65 in ha_initialize_handlerton(st_plugin_int*) /home/vsts/src/sql/handler.cc:555
 
    #12 0x89eccc in plugin_initialize /home/vsts/src/sql/sql_plugin.cc:1417
 
    #13 0x8a09f5 in plugin_init(int*, char**, int) /home/vsts/src/sql/sql_plugin.cc:1698
 
    #14 0x629fd5 in init_server_components /home/vsts/src/sql/mysqld.cc:5333
 
    #15 0x62c20a in mysqld_main(int, char**) /home/vsts/src/sql/mysqld.cc:5929
 
    #16 0x614206 in main /home/vsts/src/sql/main.cc:25
 
    #17 0x7fa0f622182f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Also happens on 10.5.

 Comments   
Comment by PD [ 2021-11-02 ]

Issue still exists. mariaDB 10.3.22 crashed without any reason after nearly one year runtime with the same error message.

Assertion failed: (ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion), function PrepareOutput, file /home/prod/mariadb-10.3.22/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc, line 632.
 
211023 21:08:58 [ERROR] mysqld got signal 6 ;

Comment by PD [ 2022-01-15 ]

Just crashed again.

Assertion failed: (ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion), function PrepareOutput, file /home/prod/mariadb-10.3.22/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc, line 632.
 
220115 16:17:02 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.3.22-MariaDB-log
 
key_buffer_size=4294967296
 
read_buffer_size=131072
 
max_used_connections=151
 
max_threads=152
 
thread_count=9
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6707401 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
0xaf311e <my_print_stacktrace+0x2e> at /opt/mysql/bin/mysqld
 
0x831aa5 <handle_fatal_signal+0x295> at /opt/mysql/bin/mysqld
 
0x801542946 <pthread_sigmask+0x536> at /lib/libthr.so.3
 
0x801541eb2 <pthread_getspecific+0xe12> at /lib/libthr.so.3
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

Generated at Thu Feb 08 09:02:43 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.