Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12445

Rocksdb does not shutdown worker threads and aborts in memleak check on server shutdown

    XMLWordPrintable

Details

    Description

      rocksdb.validate_datadic fails on Windows in buildbot:
      http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/3295/steps/test/logs/stdio

      The issue is not easily reproducible.

      Based on the above, it fails an assertion here:

      mysqld!my_sigabrt_handler [d:\winx64-debug\build\src\mysys\my_thr_init.c @ 488]
      mysqld!raise [d:\th\minkernel\crts\ucrt\src\appcrt\misc\signal.cpp @ 516]
      mysqld!abort [d:\th\minkernel\crts\ucrt\src\appcrt\startup\abort.cpp @ 71]
      mysqld!common_assert_to_stderr<wchar_t> [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 149]
        wchar_t * expression = 0x00007ff7`1def0900 "global_status_var.global_memory_used == 0"
        wchar_t * file_name = 0x00007ff7`1def08b0 "D:\winx64-debug\build\src\sql\mysqld.cc"
        unsigned int line_number = 0x86a
      (Inline Function) --------`-------- mysqld!common_assert [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 383]
      mysqld!_wassert [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 404]
      mysqld!mysqld_exit [d:\winx64-debug\build\src\sql\mysqld.cc @ 2154]
      mysqld!win_main [d:\winx64-debug\build\src\sql\mysqld.cc @ 6064]
      mysqld!mysql_service [d:\winx64-debug\build\src\sql\mysqld.cc @ 6086]
      mysqld!mysqld_main [d:\winx64-debug\build\src\sql\mysqld.cc @ 6279]
      mysqld!main [d:\winx64-debug\build\src\sql\main.cc @ 26]
      (Inline Function) --------`-------- mysqld!invoke_main [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 64]
      mysqld!__scrt_common_main_seh [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253]
      

      This is in mysqld_exit() here:

        if (!opt_debugging)
        {
      =>   DBUG_ASSERT(global_status_var.global_memory_used == 0);
        }
        cleanup_tls();
        DBUG_LEAVE;
        if (opt_endinfo && global_status_var.global_memory_used)
          fprintf(stderr, "Warning: Memory not freed: %ld\n",
                  (long) global_status_var.global_memory_used);
        sd_notify(0, "STATUS=MariaDB server is down");
        exit(exit_code); /* purecov: inspected */
      }
      

      An observation by wlad: below, we also see that one other thread is still running, it's here:

      (Inline Function) --------`-------- ha_rocksdb!Concurrency::details::stl_condition_variable_win7::wait_for [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216]
      ha_rocksdb!Concurrency::details::stl_condition_variable_win7::wait(class Concurrency::details::stl_critical_section_interface * lock = <Value unavailable error>) [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210]
      ha_rocksdb!do_wait(struct _Cnd_internal_imp_t * cond = 0x00000036`aa929788, struct _Mtx_internal_imp_t * mtx = 0x00000036`aa929738, struct xtime * target = 0x00000000`00000000) [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77]
      ha_rocksdb!std::_Cnd_waitX(struct _Cnd_internal_imp_t * _Cnd = 0x00000036`aa929788, struct _Mtx_internal_imp_t * _Mtx = 0x00000036`aa929738) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
      ha_rocksdb!std::condition_variable::wait(class std::unique_lock<std::mutex> * _Lck = 0x00000036`ad97f6f8) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 566]
      ha_rocksdb!rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned int64 thread_id = 0) [d:\winx64-debug\build\src\storage\rocksdb\rocksdb\util\threadpool_imp.cc @ 180]
      ha_rocksdb!rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void * arg = 0x00000036`aab6bec0) [d:\winx64-debug\build\src\storage\rocksdb\rocksdb\util\threadpool_imp.cc @ 262]
      ha_rocksdb!std::_Invoker_functor::_Call<void * __ptr64 (<function> ** _Obj = 0x00000036`aab06388, struct rocksdb::BGThreadMetadata ** <_Args_0> = 0x00000036`aab06390) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1377]
      

      this does not look normal. One would expect that at this point the plugin is unloaded and its threads stopped.

      • Could it be that that other thread has allocated (and is keeping allocated) something and this is why we see the assert to fire?

      Attachments

        Issue Links

          Activity

            People

              psergei Sergei Petrunia
              psergei Sergei Petrunia
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.