[MDEV-20315] MyRocks tests produce valgrind failures Created: 2019-08-11  Updated: 2019-09-04  Resolved: 2019-09-04

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2, 10.3, 10.4
Fix Version/s: 10.2.27, 10.3.18, 10.4.8

Type: Bug Priority: Major
Reporter: Sergei Petrunia Assignee: Sergei Petrunia
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-12439 MariaRocks produces numerous (spuriou... Closed
relates to MDEV-20322 rocksdb doesn't free memory on shutdown Open

 Description   

MyRocks tests produce several valgrind failures. We need to fix them.

Remaining failures for the test:

./mtr --valgrind rocksdb_sys_vars.rocksdb_lock_wait_timeout_basic


==30698== 24 bytes in 1 blocks are still reachable in loss record 2 of 19
 
==30698==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 
==30698==    by 0xD0C3865: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:111)
 
==30698==    by 0xD0BD988: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:436)
 
==30698==    by 0xD54057B: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2107)
 
==30698==    by 0xD53FEE8: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:354)
 
==30698==    by 0xD53F1DC: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2092)
 
==30698==    by 0xD53E04E: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2071)
 
==30698==    by 0xD53C8DA: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false>*) (hashtable.h:1718)
 
==30698==    by 0xD53B85C: std::pair<std::__detail::_Node_iterator<rocksdb::ThreadStatusData*, true, false>, bool> std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert<rocksdb::ThreadStatusData* const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false> > > >(rocksdb::ThreadStatusData* const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false> > > const&, std::integral_constant<bool, true>) (hashtable.h:1822)
 
==30698==    by 0xD53AAAF: std::__detail::_Insert_base<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::insert(rocksdb::ThreadStatusData* const&) (hashtable_policy.h:843)
 
==30698==    by 0xD539FBA: std::unordered_set<rocksdb::ThreadStatusData*, std::hash<rocksdb::ThreadStatusData*>, std::equal_to<rocksdb::ThreadStatusData*>, std::allocator<rocksdb::ThreadStatusData*> >::insert(rocksdb::ThreadStatusData* const&) (unordered_set.h:420)
 
==30698==    by 0xD537E44: rocksdb::ThreadStatusUpdater::RegisterThread(rocksdb::ThreadStatus::ThreadType, unsigned long) (thread_status_updater.cc:25)
 
==30698==    by 0xD2FD533: rocksdb::ThreadStatusUtil::RegisterThread(rocksdb::Env const*, rocksdb::ThreadStatus::ThreadType) (thread_status_util.cc:24)
 
==30698==    by 0xD56227A: rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) (threadpool_imp.cc:300)
 
==30698==    by 0xD567D72: void* std::__invoke_impl<void*, void* (*)(void*), rocksdb::BGThreadMetadata*>(std::__invoke_other, void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) (invoke.h:60)
 
==30698==    by 0xD567303: std::__invoke_result<void* (*)(void*), rocksdb::BGThreadMetadata*>::type std::__invoke<void* (*)(void*), rocksdb::BGThreadMetadata*>(void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) (invoke.h:96)
 
==30698== 


==30698== 24 bytes in 1 blocks are still reachable in loss record 4 of 19
 
==30698==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 
==30698==    by 0xD0C3865: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:111)
 
==30698==    by 0xD0BD988: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:436)
 
==30698==    by 0xD5407B5: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, false> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2107)
 
==30698==    by 0xD540192: std::_Hashtable<void const*, std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, std::allocator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo> >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:354)
 
==30698==    by 0xD53F756: std::_Hashtable<void const*, std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, std::allocator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo> >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2092)
 
==30698==    by 0xD53E5FE: std::_Hashtable<void const*, std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, std::allocator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo> >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2071)
 
==30698==    by 0xD53D1CE: std::_Hashtable<void const*, std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, std::allocator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo> >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, false>*) (hashtable.h:1718)
 
==30698==    by 0xD53BF6F: std::pair<std::__detail::_Node_iterator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, false, false>, bool> std::_Hashtable<void const*, std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, std::allocator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo> >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_emplace<std::piecewise_construct_t const&, std::tuple<void const*>, std::tuple<void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >(std::integral_constant<bool, true>, std::piecewise_construct_t const&, std::tuple<void const*>&&, std::tuple<void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&&) (hashtable.h:1666)
 
==30698==    by 0xD53B3A3: std::pair<std::__detail::_Node_iterator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, false, false>, bool> std::_Hashtable<void const*, std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, std::allocator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo> >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::emplace<std::piecewise_construct_t const&, std::tuple<void const*>, std::tuple<void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >(std::piecewise_construct_t const&, std::tuple<void const*>&&, std::tuple<void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&&) (hashtable.h:736)
 
==30698==    by 0xD53A861: std::pair<std::__detail::_Node_iterator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo>, false, false>, bool> std::unordered_map<void const*, rocksdb::ConstantColumnFamilyInfo, std::hash<void const*>, std::equal_to<void const*>, std::allocator<std::pair<void const* const, rocksdb::ConstantColumnFamilyInfo> > >::emplace<std::piecewise_construct_t const&, std::tuple<void const*>, std::tuple<void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >(std::piecewise_construct_t const&, std::tuple<void const*>&&, std::tuple<void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&&) (unordered_map.h:387)
 
==30698==    by 0xD538B4F: rocksdb::ThreadStatusUpdater::NewColumnFamilyInfo(void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (thread_status_updater.cc:226)
 
==30698==    by 0xD2FD94B: rocksdb::ThreadStatusUtil::NewColumnFamilyInfo(rocksdb::DB const*, rocksdb::ColumnFamilyData const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env const*) (thread_status_util.cc:128)
 
==30698==    by 0xD1679BA: rocksdb::DBImpl::NewThreadStatusCfInfo(rocksdb::ColumnFamilyData*) const (db_impl.cc:2750)
 
==30698==    by 0xD1C9745: rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool) (db_impl_open.cc:1106)
 
==30698==    by 0xD44F516: rocksdb::TransactionDB::Open(rocksdb::DBOptions const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::TransactionDB**) (pessimistic_transaction_db.cc:228)
 
==30698== 


==30698== 24 bytes in 1 blocks are still reachable in loss record 5 of 19
 
==30698==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 
==30698==    by 0xD0C3865: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:111)
 
==30698==    by 0xD0BD988: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:436)
 
==30698==    by 0xD540397: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, false> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2107)
 
==30698==    by 0xD53FAC0: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:354)
 
==30698==    by 0xD53E932: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2092)
 
==30698==    by 0xD53D48E: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2071)
 
==30698==    by 0xD53C29E: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, false>*) (hashtable.h:1718)
 
==30698==    by 0xD53B4D0: std::__detail::_Map_base<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](void const* const&) (hashtable_policy.h:728)
 
==30698==    by 0xD53A890: std::unordered_map<void const*, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> >, std::hash<void const*>, std::equal_to<void const*>, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > > >::operator[](void const* const&) (unordered_map.h:973)
 
==30698==    by 0xD538B75: rocksdb::ThreadStatusUpdater::NewColumnFamilyInfo(void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (thread_status_updater.cc:228)
 
==30698==    by 0xD2FD94B: rocksdb::ThreadStatusUtil::NewColumnFamilyInfo(rocksdb::DB const*, rocksdb::ColumnFamilyData const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env const*) (thread_status_util.cc:128)
 
==30698==    by 0xD1679BA: rocksdb::DBImpl::NewThreadStatusCfInfo(rocksdb::ColumnFamilyData*) const (db_impl.cc:2750)
 
==30698==    by 0xD1C9745: rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool) (db_impl_open.cc:1106)
 
==30698==    by 0xD44F516: rocksdb::TransactionDB::Open(rocksdb::DBOptions const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::TransactionDB**) (pessimistic_transaction_db.cc:228)
 
==30698==    by 0xD064E94: myrocks::rocksdb_init_func(void*) (ha_rocksdb.cc:5141)



 Comments   
Comment by Sergei Petrunia [ 2019-08-11 ]

Imported valgrind.supp from the upstream.
Which valgrind.supp to use?

  • 10.2 has the new MyRocks, merged in June, 2019
  • 10.5 still has an older MyRocks, merged in Aug, 2018.

Differences in valgrind.supp in the upstream:

diff -u  ~/valgrind.supp-aug23-2018  ~/valgrind.supp-jun14-2019
 
--- /home/psergey/valgrind.supp-aug23-2018      2019-08-11 16:48:39.992730979 +0300
 
+++ /home/psergey/valgrind.supp-jun14-2019      2019-08-11 16:53:46.776180725 +0300
 
@@ -1276,6 +1276,20 @@
 
 }
 
 
 {
 
+   TLS data on main thread is not cleaned up properly
 
+   Memcheck:Leak
 
+   match-leak-kinds: reachable
 
+   ...
 
+   fun:vasprintf
 
+   fun:asprintf
 
+   fun:dlerror
 
+   fun:dlfcn_load
 
+   fun:DSO_load
 
+   fun:DSO_dsobyaddr
 
+   ...
 
+}
 
+
 
+{
 
    Valgrind complains about std::unordered_map ctor in ha_rocksdb
 
    Memcheck:Leak
 
    match-leak-kinds: definite

So, using older/newer valgrind.supp doesn't make a difference for the failures we are getting.

Comment by Sergei Petrunia [ 2019-08-11 ]

Looking at the last failure, 

==30698== 24 bytes in 1 blocks are still reachable in loss record 5 of 19
 
==30698==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 
==30698==    by 0xD0C3865: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:111)
 
==30698==    by 0xD0BD988: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:436)
 
==30698==    by 0xD540397: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, false> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2107)
 
==30698==    by 0xD53FAC0: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:354)
 
==30698==    by 0xD53E932: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2092)
 
==30698==    by 0xD53D48E: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2071)
 
==30698==    by 0xD53C29E: std::_Hashtable<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, false>*) (hashtable.h:1718)
 
==30698==    by 0xD53B4D0: std::__detail::_Map_base<void const*, std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > >, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > >, std::__detail::_Select1st, std::equal_to<void const*>, std::hash<void const*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](void const* const&) (hashtable_policy.h:728)
 
==30698==    by 0xD53A890: std::unordered_map<void const*, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> >, std::hash<void const*>, std::equal_to<void const*>, std::allocator<std::pair<void const* const, std::unordered_set<void const*, std::hash<void const*>, std::equal_to<void const*>, std::allocator<void const*> > > > >::operator[](void const* const&) (unordered_map.h:973)
 
==30698==    by 0xD538B75: rocksdb::ThreadStatusUpdater::NewColumnFamilyInfo(void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (thread_status_updater.cc:228)
 
==30698==    by 0xD2FD94B: rocksdb::ThreadStatusUtil::NewColumnFamilyInfo(rocksdb::DB const*, rocksdb::ColumnFamilyData const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env const*) (thread_status_util.cc:128)
 
==30698==    by 0xD1679BA: rocksdb::DBImpl::NewThreadStatusCfInfo(rocksdb::ColumnFamilyData*) const (db_impl.cc:2750)
 
==30698==    by 0xD1C9745: rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool) (db_impl_open.cc:1106)
 
==30698==    by 0xD44F516: rocksdb::TransactionDB::Open(rocksdb::DBOptions const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::TransactionDB**) (pessimistic_transaction_db.cc:228)
 
==30698==    by 0xD064E94: myrocks::rocksdb_init_func(void*) (ha_rocksdb.cc:5141)

and the first valgrind suppression:

{
 
   Still reachable for once-per-process initializations
 
   Memcheck:Leak
 
   match-leak-kinds: reachable
 
   ...
 
   fun:_ZN7rocksdb16ThreadStatusUtil19NewColumnFamilyInfoEPKNS_2DBEPKNS_16ColumnFamilyDataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKNS_3EnvE
 
   fun:_ZNK7rocksdb6DBImpl21NewThreadStatusCfInfoEPNS_16ColumnFamilyDataE
 
   fun:_ZN7rocksdb2DB4OpenERKNS_9DBOptionsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorINS_22ColumnFamilyDescriptorESaISD_EEPSC_IPNS_18ColumnFamilyHandleESaISJ_EEPPS0_
 
   fun:_ZN7rocksdb13TransactionDB4OpenERKNS_9DBOptionsERKNS_20TransactionDBOptionsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorINS_22ColumnFamilyDescriptorESaISG_EEPSF_IPNS_18ColumnFamilyHandleESaISM_EEPPS0_
 
   fun:_ZN7myrocksL17rocksdb_init_funcEPv
 
}

Passing suppression code through c++filt :

   fun:rocksdb::DBImpl::NewThreadStatusCfInfo(rocksdb::ColumnFamilyData*) const
 
   fun:rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)
 
   fun:rocksdb::TransactionDB::Open(rocksdb::DBOptions const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::TransactionDB**)
 
   fun:myrocks::rocksdb_init_func(void*)

The difference is this line:

rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool) (db_impl_open.cc:1106)
 
fun:rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)

DBImpl::Open vs DB::Open (also the list of arguments is slightly different).

Comment by Sergei Petrunia [ 2019-08-11 ]

.. and the second reason that suppression doesn't work is that we are hitting the --num-callers=16 limit that MTR sets for valgrind. If I change DB::Open to DBImpl::Open and comment out the 

   fun:_ZN7myrocksL17rocksdb_init_funcEPv

it works .

Comment by Sergei Petrunia [ 2019-08-11 ]

The other failure

==32648== 24 bytes in 1 blocks are still reachable in loss record 2 of 19
 
==32648==    at 0x4C3017F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 
==32648==    by 0xD0C3865: __gnu_cxx::new_allocator<std::__detail::_Hash_node_base*>::allocate(unsigned long, void const*) (new_allocator.h:111)
 
==32648==    by 0xD0BD988: std::allocator_traits<std::allocator<std::__detail::_Hash_node_base*> >::allocate(std::allocator<std::__detail::_Hash_node_base*>&, unsigned long) (alloc_traits.h:436)
 
==32648==    by 0xD54057B: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false> > >::_M_allocate_buckets(unsigned long) (hashtable_policy.h:2107)
 
==32648==    by 0xD53FEE8: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_allocate_buckets(unsigned long) (hashtable.h:354)
 
==32648==    by 0xD53F1DC: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash_aux(unsigned long, std::integral_constant<bool, true>) (hashtable.h:2092)
 
==32648==    by 0xD53E04E: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:2071)
 
==32648==    by 0xD53C8DA: std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false>*) (hashtable.h:1718)
 
==32648==    by 0xD53B85C: std::pair<std::__detail::_Node_iterator<rocksdb::ThreadStatusData*, true, false>, bool> std::_Hashtable<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert<rocksdb::ThreadStatusData* const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false> > > >(rocksdb::ThreadStatusData* const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<rocksdb::ThreadStatusData*, false> > > const&, std::integral_constant<bool, true>) (hashtable.h:1822)
 
==32648==    by 0xD53AAAF: std::__detail::_Insert_base<rocksdb::ThreadStatusData*, rocksdb::ThreadStatusData*, std::allocator<rocksdb::ThreadStatusData*>, std::__detail::_Identity, std::equal_to<rocksdb::ThreadStatusData*>, std::hash<rocksdb::ThreadStatusData*>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::insert(rocksdb::ThreadStatusData* const&) (hashtable_policy.h:843)
 
==32648==    by 0xD539FBA: std::unordered_set<rocksdb::ThreadStatusData*, std::hash<rocksdb::ThreadStatusData*>, std::equal_to<rocksdb::ThreadStatusData*>, std::allocator<rocksdb::ThreadStatusData*> >::insert(rocksdb::ThreadStatusData* const&) (unordered_set.h:420)
 
==32648==    by 0xD537E44: rocksdb::ThreadStatusUpdater::RegisterThread(rocksdb::ThreadStatus::ThreadType, unsigned long) (thread_status_updater.cc:25)
 
==32648==    by 0xD2FD533: rocksdb::ThreadStatusUtil::RegisterThread(rocksdb::Env const*, rocksdb::ThreadStatus::ThreadType) (thread_status_util.cc:24)
 
==32648==    by 0xD56227A: rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) (threadpool_imp.cc:300)
 
==32648==    by 0xD567D72: void* std::__invoke_impl<void*, void* (*)(void*), rocksdb::BGThreadMetadata*>(std::__invoke_other, void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) (invoke.h:60)
 
==32648==    by 0xD567303: std::__invoke_result<void* (*)(void*), rocksdb::BGThreadMetadata*>::type std::__invoke<void* (*)(void*), rocksdb::BGThreadMetadata*>(void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) (invoke.h:96)

there is a suppression to address it:

{
 
   Still reachable for once-per-process initializations
 
   Memcheck:Leak
 
   match-leak-kinds: reachable
 
   ...
 
   fun:_ZN7rocksdb19ThreadStatusUpdater14RegisterThreadENS_12ThreadStatus10ThreadTypeEm
 
   fun:_ZN7rocksdb16ThreadStatusUtil14RegisterThreadEPKNS_3EnvENS_12ThreadStatus10ThreadTypeE
 
   fun:_ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv
 
   fun:_ZNSt12_Bind_simpleIFPFPvS0_EPN7rocksdb16BGThreadMetadataEEE9_M_invokeIILm0EEEES0_St12_Index_tupleIIXspT_EEE
 
   ...
 
}

but it doesn't match, because the last stack frame is

   fun:_ZNSt12_Bind_simpleIFPFPvS0_EPN7rocksdb16BGThreadMetadataEEE9_M_invokeIILm0EEEES0_St12_Index_tupleIIXspT_EEE

un-mangled:

   fun:void* std::_Bind_simple<void* (*(rocksdb::BGThreadMetadata*))(void*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>)

while the execution has this:

==32648==    by 0xD567D72: void* std::__invoke_impl<void*, void* (*)(void*), rocksdb::BGThreadMetadata*>(std::__invoke_other, void* (*&&)(void*), rocksdb::BGThreadMetadata*&&) (invoke.h:60)

Comment by Sergei Petrunia [ 2019-09-04 ]

Closing as currently the failures are suppressed (like in the upstream)

Generated at Thu Feb 08 08:58:31 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.