[MDEV-24791] Assertion `!level_file.second->being_compacted' failed. Created: 2021-02-05  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.3.27
Fix Version/s: 10.4, 10.5

Type: Bug Priority: Major
Reporter: Valerii Kravchuk Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 0
Labels: crash


 Description   

The following assertion failure happens while importing data into RocksDB on 10.3.27:

mysqld: /build/mariadb-10.3-Zg5BW3/mariadb-10.3-10.3.27/
storage/rocksdb/rocksdb/db/compaction/compaction_picker_level.cc:136: 
rocksdb::{anonymous}::LevelCompactionBuilder::PickExpiredTtlFiles()::
<lambda(std::pair<int, rocksdb::FileMetaData*>)>: 
Assertion `!level_file.second->being_compacted' failed.

mysqld: /build/mariadb-10.3-Zg5BW3/mariadb-10.3-10.3.27/storage/rocksdb/rocksdb/db/compaction/compaction_picker_level.cc:136: rocksdb::{anonymous}::LevelCompactionBuilder::PickExpiredTtlFiles()::<lambda(std::pair<int, rocksdb::FileMetaData*>)>: Assertion `!level_file.second->being_compacted' failed.
210204 20:57:48 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.27-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=13
max_threads=153
thread_count=19
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467424 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561718be391e]
/usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x5617187122ed]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fdf06f2f730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fdf064ea7bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fdf064d5535]
/lib/x86_64-linux-gnu/libc.so.6(+0x2240f)[0x7fdf064d540f]
/lib/x86_64-linux-gnu/libc.so.6(+0x30102)[0x7fdf064e3102]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb21LevelCompactionPicker14PickCompactionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_16MutableCFOptionsEPNS_18VersionStorageInfoEPNS_9LogBufferEm+0x18da)[0x7fdef104f9fa]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb16ColumnFamilyData14PickCompactionERKNS_16MutableCFOptionsEPNS_9LogBufferE+0x58)[0x7fdef10275f8]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl20BackgroundCompactionEPbPNS_10JobContextEPNS_9LogBufferEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0x228a)[0x7fdef0e1974a]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl24BackgroundCallCompactionEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0x13b)[0x7fdef0e1c12b]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl16BGWorkCompactionEPv+0x97)[0x7fdef0e1c8f7]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm+0x25a)[0x7fdef0fbfb1a]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv+0x51)[0x7fdef0fbfca1]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbbb2f)[0x7fdf068b4b2f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fdf06f24fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdf065ac4cf]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /srv/data/mariadb
Resource Limits:
Fatal signal 11 while backtracingmysqld: /build/mariadb-10.3-Zg5BW3/mariadb-10.3-10.3.27/storage/rocksdb/rocksdb/db/compaction/compaction_picker_level.cc:136: rocksdb::{anonymous}::LevelCompactionBuilder::PickExpiredTtlFiles()::<lambda(std::pair<int, rocksdb::FileMetaData*>)>: Assertion `!level_file.second->being_compacted' failed.
210204 20:57:48 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.27-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=13
max_threads=153
thread_count=19
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467424 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x561718be391e]
/usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x5617187122ed]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fdf06f2f730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fdf064ea7bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fdf064d5535]
/lib/x86_64-linux-gnu/libc.so.6(+0x2240f)[0x7fdf064d540f]
/lib/x86_64-linux-gnu/libc.so.6(+0x30102)[0x7fdf064e3102]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb21LevelCompactionPicker14PickCompactionERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_16MutableCFOptionsEPNS_18VersionStorageInfoEPNS_9LogBufferEm+0x18da)[0x7fdef104f9fa]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb16ColumnFamilyData14PickCompactionERKNS_16MutableCFOptionsEPNS_9LogBufferE+0x58)[0x7fdef10275f8]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl20BackgroundCompactionEPbPNS_10JobContextEPNS_9LogBufferEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0x228a)[0x7fdef0e1974a]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl24BackgroundCallCompactionEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0x13b)[0x7fdef0e1c12b]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb6DBImpl16BGWorkCompactionEPv+0x97)[0x7fdef0e1c8f7]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm+0x25a)[0x7fdef0fbfb1a]
/usr/lib/x86_64-linux-gnu/mariadb19/plugin/ha_rocksdb.so(_ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv+0x51)[0x7fdef0fbfca1]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbbb2f)[0x7fdf068b4b2f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fdf06f24fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fdf065ac4cf]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /srv/data/mariadb
Resource Limits:
Fatal signal 11 while backtracing

Related settings are:

plugin-load-add=ha_rocksdb.so
rocksdb_flush_log_at_trx_commit=2
rocksdb_max_row_locks=100000000
rocksdb_block_cache_size=32G
rocksdb_wal_recovery_mode=2 

I see some issues with similar assertion or related crash:

https://github.com/facebook/mysql-5.6/issues/186
https://github.com/facebook/rocksdb/issues/3723

but they are closed. I can not find any MDEV for this.



 Comments   
Comment by Sergei Petrunia [ 2021-02-12 ]

About the mentioned RocksDB issues: issue #186 was closed long ago and we have a patch for it , issue #3723 was closed with a conclusion that the user's comparator was invalid (and so should not affect MyRocks).

Comment by Sergei Petrunia [ 2021-02-12 ]

Stack trace after c++filt:

(rocksdb::LevelCompactionPicker::PickCompaction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::MutableCFOptions const&, rocksdb::VersionStorageInfo*, rocksdb::LogBuffer*, unsigned long)+0x18da)[0x7fdef104f9fa]
(rocksdb::ColumnFamilyData::PickCompaction(rocksdb::MutableCFOptions const&, rocksdb::LogBuffer*)+0x58)[0x7fdef10275f8]
(rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x228a)[0x7fdef0e1974a]
(rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0x13b)[0x7fdef0e1c12b]
(rocksdb::DBImpl::BGWorkCompaction(void*)+0x97)[0x7fdef0e1c8f7]
(rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x25a)[0x7fdef0fbfb1a]
(rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x51)[0x7fdef0fbfca1]

Generated at Thu Feb 08 09:32:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.