Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12445

Rocksdb does not shutdown worker threads and aborts in memleak check on server shutdown

    Details

      Description

      rocksdb.validate_datadic fails on Windows in buildbot:
      http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/3295/steps/test/logs/stdio

      The issue is not easily reproducible.

      Based on the above, it fails an assertion here:

      mysqld!my_sigabrt_handler [d:\winx64-debug\build\src\mysys\my_thr_init.c @ 488]
      mysqld!raise [d:\th\minkernel\crts\ucrt\src\appcrt\misc\signal.cpp @ 516]
      mysqld!abort [d:\th\minkernel\crts\ucrt\src\appcrt\startup\abort.cpp @ 71]
      mysqld!common_assert_to_stderr<wchar_t> [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 149]
        wchar_t * expression = 0x00007ff7`1def0900 "global_status_var.global_memory_used == 0"
        wchar_t * file_name = 0x00007ff7`1def08b0 "D:\winx64-debug\build\src\sql\mysqld.cc"
        unsigned int line_number = 0x86a
      (Inline Function) --------`-------- mysqld!common_assert [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 383]
      mysqld!_wassert [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 404]
      mysqld!mysqld_exit [d:\winx64-debug\build\src\sql\mysqld.cc @ 2154]
      mysqld!win_main [d:\winx64-debug\build\src\sql\mysqld.cc @ 6064]
      mysqld!mysql_service [d:\winx64-debug\build\src\sql\mysqld.cc @ 6086]
      mysqld!mysqld_main [d:\winx64-debug\build\src\sql\mysqld.cc @ 6279]
      mysqld!main [d:\winx64-debug\build\src\sql\main.cc @ 26]
      (Inline Function) --------`-------- mysqld!invoke_main [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 64]
      mysqld!__scrt_common_main_seh [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253]
      

      This is in mysqld_exit() here:

        if (!opt_debugging)
        {
      =>   DBUG_ASSERT(global_status_var.global_memory_used == 0);
        }
        cleanup_tls();
        DBUG_LEAVE;
        if (opt_endinfo && global_status_var.global_memory_used)
          fprintf(stderr, "Warning: Memory not freed: %ld\n",
                  (long) global_status_var.global_memory_used);
        sd_notify(0, "STATUS=MariaDB server is down");
        exit(exit_code); /* purecov: inspected */
      }
      

      An observation by Vladislav Vaintroub: below, we also see that one other thread is still running, it's here:

      (Inline Function) --------`-------- ha_rocksdb!Concurrency::details::stl_condition_variable_win7::wait_for [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216]
      ha_rocksdb!Concurrency::details::stl_condition_variable_win7::wait(class Concurrency::details::stl_critical_section_interface * lock = <Value unavailable error>) [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210]
      ha_rocksdb!do_wait(struct _Cnd_internal_imp_t * cond = 0x00000036`aa929788, struct _Mtx_internal_imp_t * mtx = 0x00000036`aa929738, struct xtime * target = 0x00000000`00000000) [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77]
      ha_rocksdb!std::_Cnd_waitX(struct _Cnd_internal_imp_t * _Cnd = 0x00000036`aa929788, struct _Mtx_internal_imp_t * _Mtx = 0x00000036`aa929738) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
      ha_rocksdb!std::condition_variable::wait(class std::unique_lock<std::mutex> * _Lck = 0x00000036`ad97f6f8) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 566]
      ha_rocksdb!rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned int64 thread_id = 0) [d:\winx64-debug\build\src\storage\rocksdb\rocksdb\util\threadpool_imp.cc @ 180]
      ha_rocksdb!rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void * arg = 0x00000036`aab6bec0) [d:\winx64-debug\build\src\storage\rocksdb\rocksdb\util\threadpool_imp.cc @ 262]
      ha_rocksdb!std::_Invoker_functor::_Call<void * __ptr64 (<function> ** _Obj = 0x00000036`aab06388, struct rocksdb::BGThreadMetadata ** <_Args_0> = 0x00000036`aab06390) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1377]
      

      this does not look normal. One would expect that at this point the plugin is unloaded and its threads stopped.

      • Could it be that that other thread has allocated (and is keeping allocated) something and this is why we see the assert to fire?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                psergey Sergei Petrunia
                Reporter:
                psergey Sergei Petrunia
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: