Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20861

Assertion `ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion' failed in rocksdb::CompactionIterator::PrepareOutput

    XMLWordPrintable

    Details

      Description

      The failure happens pretty often in my tests, but it's badly reproducible. I could make a further effort for creating at least a concurrent test case, but it doesn't make much sense at this point, since upstream has a probably bugfix already and it just needs to be merged. If the failure continues happening after the merge, then we'll chase it further.

      10.2 fa929f7c ASAN

      mysqld: /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:632: void rocksdb::CompactionIterator::PrepareOutput(): Assertion `ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion' failed.
      191018  5:53:57 [ERROR] mysqld got signal 6 ;
       
      #13 0x00007fa0f622ebd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7fa0ea895060 "ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion", file=file@entry=0x7fa0ea894a60 "/home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc", line=line@entry=632, function=function@entry=0x7fa0ea895fe0 <rocksdb::CompactionIterator::PrepareOutput()::__PRETTY_FUNCTION__> "void rocksdb::CompactionIterator::PrepareOutput()") at assert.c:92
      #14 0x00007fa0f622ec82 in __GI___assert_fail (assertion=0x7fa0ea895060 "ikey_.type != kTypeDeletion && ikey_.type != kTypeSingleDeletion", file=0x7fa0ea894a60 "/home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc", line=632, function=0x7fa0ea895fe0 <rocksdb::CompactionIterator::PrepareOutput()::__PRETTY_FUNCTION__> "void rocksdb::CompactionIterator::PrepareOutput()") at assert.c:101
      #15 0x00007fa0ea56c96d in rocksdb::CompactionIterator::PrepareOutput (this=0x6170000dd300) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:632
      #16 0x00007fa0ea5673a8 in rocksdb::CompactionIterator::Next (this=0x6170000dd300) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:155
      #17 0x00007fa0ea57a6ad in rocksdb::CompactionJob::ProcessKeyValueCompaction (this=0x7fa0e86511a0, sub_compact=0x61400143f640) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:954
      #18 0x00007fa0ea5754d1 in rocksdb::CompactionJob::Run (this=0x7fa0e86511a0) at /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:590
      #19 0x00007fa0e9ea575c in rocksdb::DBImpl::BackgroundCompaction (this=0x622000004900, made_progress=0x7fa0e86518b0, job_context=0x7fa0e8651bf0, log_buffer=0x7fa0e8651df0, prepicked_compaction=0x603001397710, thread_pri=rocksdb::Env::LOW) at /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2689
      #20 0x00007fa0e9e9fffc in rocksdb::DBImpl::BackgroundCallCompaction (this=0x622000004900, prepicked_compaction=0x603001397710, bg_thread_pri=rocksdb::Env::LOW) at /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2249
      #21 0x00007fa0e9e9c8e2 in rocksdb::DBImpl::BGWorkCompaction (arg=0x60200002b030) at /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2024
      #22 0x00007fa0ea6a4f33 in std::__invoke_impl<void, void (*&)(void*), void*&> (__f=@0x602000091f50: 0x7fa0e9e9c6fe <rocksdb::DBImpl::BGWorkCompaction(void*)>, __args#0=@0x602000091f58: 0x60200002b030) at /usr/include/c++/7/bits/invoke.h:60
      #23 0x00007fa0ea6a49e4 in std::__invoke<void (*&)(void*), void*&> (__fn=@0x602000091f50: 0x7fa0e9e9c6fe <rocksdb::DBImpl::BGWorkCompaction(void*)>, __args#0=@0x602000091f58: 0x60200002b030) at /usr/include/c++/7/bits/invoke.h:95
      #24 0x00007fa0ea6a3b39 in std::_Bind<void (*(void*))(void*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x602000091f50, __args=...) at /usr/include/c++/7/functional:467
      #25 0x00007fa0ea6a2037 in std::_Bind<void (*(void*))(void*)>::operator()<, void>() (this=0x602000091f50) at /usr/include/c++/7/functional:551
      #26 0x00007fa0ea69fcac in std::_Function_handler<void (), std::_Bind<void (*(void*))(void*)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/7/bits/std_function.h:316
      #27 0x00007fa0e9c34336 in std::function<void ()>::operator()() const (this=0x7fa0e8652bf0) at /usr/include/c++/7/bits/std_function.h:706
      #28 0x00007fa0ea6970e7 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x611000000a40, thread_id=0) at /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:266
      #29 0x00007fa0ea697365 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x602000000bd0) at /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:307
      #30 0x00007fa0ea69e8b7 in std::__invoke_impl<void*, void* (*)(void*), rocksdb::BGThreadMetadata*> (__f=@0x603000030800: 0x7fa0ea69720c <rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)>, __args#0=@0x6030000307f8: 0x602000000bd0) at /usr/include/c++/7/bits/invoke.h:60
      #31 0x00007fa0ea69ca85 in std::__invoke<void* (*)(void*), rocksdb::BGThreadMetadata*> (__fn=@0x603000030800: 0x7fa0ea69720c <rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)>, __args#0=@0x6030000307f8: 0x602000000bd0) at /usr/include/c++/7/bits/invoke.h:96
      #32 0x00007fa0ea6a5789 in std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::_M_invoke<0ul, 1ul> (this=0x6030000307f8) at /usr/include/c++/7/thread:234
      #33 0x00007fa0ea6a56f5 in std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::operator() (this=0x6030000307f8) at /usr/include/c++/7/thread:243
      #34 0x00007fa0ea6a5656 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> > >::_M_run (this=0x6030000307f0) at /usr/include/c++/7/thread:186
      #35 0x00007fa0f6bbc1a0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      #36 0x00007fa0f6ed66ba in start_thread (arg=0x7fa0e8653700) at pthread_create.c:333
      #37 0x00007fa0f630841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
       
      ==8066==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000021b806e bp 0x7fa0e864cdb0 sp 0x7fa0e864ccc0 T3)
      ==8066==The signal is caused by a WRITE memory access.
          #0 0x21b806d in my_read /home/vsts/src/mysys/my_read.c:63
          #1 0xed056d in output_core_info /home/vsts/src/sql/signal_handler.cc:66
          #2 0xed1681 in handle_fatal_signal /home/vsts/src/sql/signal_handler.cc:339
          #3 0x7fa0f6ee038f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
          #4 0x7fa0f6236427 in gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35427)
          #5 0x7fa0f6238029 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37029)
          #6 0x7fa0f622ebd6  (/lib/x86_64-linux-gnu/libc.so.6+0x2dbd6)
          #7 0x7fa0f622ec81 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x2dc81)
          #8 0x7fa0ea56c96c in rocksdb::CompactionIterator::PrepareOutput() /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:632
          #9 0x7fa0ea5673a7 in rocksdb::CompactionIterator::Next() /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_iterator.cc:155
          #10 0x7fa0ea57a6ac in rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:954
          #11 0x7fa0ea5754d0 in rocksdb::CompactionJob::Run() /home/vsts/src/storage/rocksdb/rocksdb/db/compaction/compaction_job.cc:590
          #12 0x7fa0e9ea575b in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2689
          #13 0x7fa0e9e9fffb in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2249
          #14 0x7fa0e9e9c8e1 in rocksdb::DBImpl::BGWorkCompaction(void*) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2024
          #15 0x7fa0ea6a4f32 in void std::__invoke_impl<void, void (*&)(void*), void*&>(std::__invoke_other, void (*&)(void*), void*&) /usr/include/c++/7/bits/invoke.h:60
          #16 0x7fa0ea6a49e3 in std::__invoke_result<void (*&)(void*), void*&>::type std::__invoke<void (*&)(void*), void*&>(void (*&)(void*), void*&) /usr/include/c++/7/bits/invoke.h:95
          #17 0x7fa0ea6a3b38 in void std::_Bind<void (*(void*))(void*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/7/functional:467
          #18 0x7fa0ea6a2036 in void std::_Bind<void (*(void*))(void*)>::operator()<, void>() /usr/include/c++/7/functional:551
          #19 0x7fa0ea69fcab in std::_Function_handler<void (), std::_Bind<void (*(void*))(void*)> >::_M_invoke(std::_Any_data const&) /usr/include/c++/7/bits/std_function.h:316
          #20 0x7fa0e9c34335 in std::function<void ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706
          #21 0x7fa0ea6970e6 in rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:266
          #22 0x7fa0ea697364 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:307
          #23 0x7fa0ea69e8b6 in void* std::__invoke_impl<void*, void* (*)(void*), rocksdb::BGThreadMetadata*>(std::__invoke_other, void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) /usr/include/c++/7/bits/invoke.h:60
          #24 0x7fa0ea69ca84 in std::__invoke_result<void* (*)(void*), rocksdb::BGThreadMetadata*>::type std::__invoke<void* (*)(void*), rocksdb::BGThreadMetadata*>(void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) /usr/include/c++/7/bits/invoke.h:96
          #25 0x7fa0ea6a5788 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)())) std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/include/c++/7/thread:234
          #26 0x7fa0ea6a56f4 in std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> >::operator()() /usr/include/c++/7/thread:243
          #27 0x7fa0ea6a5655 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void* (*)(void*), rocksdb::BGThreadMetadata*> > >::_M_run() /usr/include/c++/7/thread:186
          #28 0x7fa0f6bbc19f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd019f)
          #29 0x7fa0f6ed66b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
          #30 0x7fa0f630841c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
          #0 0x7fa0f871dd6f in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d6f)
          #1 0x7fa0f6bbc434 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0434)
          #2 0x7fa0ea697a93 in rocksdb::ThreadPoolImpl::Impl::StartBGThreads() /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:339
          #3 0x7fa0ea69760f in rocksdb::ThreadPoolImpl::Impl::SetBackgroundThreadsInternal(int, bool) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:325
          #4 0x7fa0ea6990eb in rocksdb::ThreadPoolImpl::IncBackgroundThreadsIfNeeded(int) /home/vsts/src/storage/rocksdb/rocksdb/util/threadpool_imp.cc:461
          #5 0x7fa0ea514c5a in IncBackgroundThreadsIfNeeded /home/vsts/src/storage/rocksdb/rocksdb/env/env_posix.cc:1006
          #6 0x7fa0e9effab5 in rocksdb::SanitizeOptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::DBOptions const&) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_open.cc:61
          #7 0x7fa0e9dacee3 in rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl.cc:229
          #8 0x7fa0e9f1264e in rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool) /home/vsts/src/storage/rocksdb/rocksdb/db/db_impl/db_impl_open.cc:1261
          #9 0x7fa0ea44f95e in rocksdb::TransactionDB::Open(rocksdb::DBOptions const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::TransactionDB**) /home/vsts/src/storage/rocksdb/rocksdb/utilities/transactions/pessimistic_transaction_db.cc:249
          #10 0x7fa0e9bb3545 in rocksdb_init_func /home/vsts/src/storage/rocksdb/ha_rocksdb.cc:5587
          #11 0xed7c65 in ha_initialize_handlerton(st_plugin_int*) /home/vsts/src/sql/handler.cc:555
          #12 0x89eccc in plugin_initialize /home/vsts/src/sql/sql_plugin.cc:1417
          #13 0x8a09f5 in plugin_init(int*, char**, int) /home/vsts/src/sql/sql_plugin.cc:1698
          #14 0x629fd5 in init_server_components /home/vsts/src/sql/mysqld.cc:5333
          #15 0x62c20a in mysqld_main(int, char**) /home/vsts/src/sql/mysqld.cc:5929
          #16 0x614206 in main /home/vsts/src/sql/main.cc:25
          #17 0x7fa0f622182f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
      

      Also happens on 10.5.

        Attachments

          Activity

            People

            Assignee:
            psergey Sergei Petrunia
            Reporter:
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: