Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12445

Rocksdb does not shutdown worker threads and aborts in memleak check on server shutdown

Details

    Description

      rocksdb.validate_datadic fails on Windows in buildbot:
      http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/3295/steps/test/logs/stdio

      The issue is not easily reproducible.

      Based on the above, it fails an assertion here:

      mysqld!my_sigabrt_handler [d:\winx64-debug\build\src\mysys\my_thr_init.c @ 488]
      mysqld!raise [d:\th\minkernel\crts\ucrt\src\appcrt\misc\signal.cpp @ 516]
      mysqld!abort [d:\th\minkernel\crts\ucrt\src\appcrt\startup\abort.cpp @ 71]
      mysqld!common_assert_to_stderr<wchar_t> [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 149]
        wchar_t * expression = 0x00007ff7`1def0900 "global_status_var.global_memory_used == 0"
        wchar_t * file_name = 0x00007ff7`1def08b0 "D:\winx64-debug\build\src\sql\mysqld.cc"
        unsigned int line_number = 0x86a
      (Inline Function) --------`-------- mysqld!common_assert [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 383]
      mysqld!_wassert [d:\th\minkernel\crts\ucrt\src\appcrt\startup\assert.cpp @ 404]
      mysqld!mysqld_exit [d:\winx64-debug\build\src\sql\mysqld.cc @ 2154]
      mysqld!win_main [d:\winx64-debug\build\src\sql\mysqld.cc @ 6064]
      mysqld!mysql_service [d:\winx64-debug\build\src\sql\mysqld.cc @ 6086]
      mysqld!mysqld_main [d:\winx64-debug\build\src\sql\mysqld.cc @ 6279]
      mysqld!main [d:\winx64-debug\build\src\sql\main.cc @ 26]
      (Inline Function) --------`-------- mysqld!invoke_main [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 64]
      mysqld!__scrt_common_main_seh [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253]
      

      This is in mysqld_exit() here:

        if (!opt_debugging)
        {
      =>   DBUG_ASSERT(global_status_var.global_memory_used == 0);
        }
        cleanup_tls();
        DBUG_LEAVE;
        if (opt_endinfo && global_status_var.global_memory_used)
          fprintf(stderr, "Warning: Memory not freed: %ld\n",
                  (long) global_status_var.global_memory_used);
        sd_notify(0, "STATUS=MariaDB server is down");
        exit(exit_code); /* purecov: inspected */
      }
      

      An observation by wlad: below, we also see that one other thread is still running, it's here:

      (Inline Function) --------`-------- ha_rocksdb!Concurrency::details::stl_condition_variable_win7::wait_for [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216]
      ha_rocksdb!Concurrency::details::stl_condition_variable_win7::wait(class Concurrency::details::stl_critical_section_interface * lock = <Value unavailable error>) [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210]
      ha_rocksdb!do_wait(struct _Cnd_internal_imp_t * cond = 0x00000036`aa929788, struct _Mtx_internal_imp_t * mtx = 0x00000036`aa929738, struct xtime * target = 0x00000000`00000000) [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77]
      ha_rocksdb!std::_Cnd_waitX(struct _Cnd_internal_imp_t * _Cnd = 0x00000036`aa929788, struct _Mtx_internal_imp_t * _Mtx = 0x00000036`aa929738) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
      ha_rocksdb!std::condition_variable::wait(class std::unique_lock<std::mutex> * _Lck = 0x00000036`ad97f6f8) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 566]
      ha_rocksdb!rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned int64 thread_id = 0) [d:\winx64-debug\build\src\storage\rocksdb\rocksdb\util\threadpool_imp.cc @ 180]
      ha_rocksdb!rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void * arg = 0x00000036`aab6bec0) [d:\winx64-debug\build\src\storage\rocksdb\rocksdb\util\threadpool_imp.cc @ 262]
      ha_rocksdb!std::_Invoker_functor::_Call<void * __ptr64 (<function> ** _Obj = 0x00000036`aab06388, struct rocksdb::BGThreadMetadata ** <_Args_0> = 0x00000036`aab06390) [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1377]
      

      this does not look normal. One would expect that at this point the plugin is unloaded and its threads stopped.

      • Could it be that that other thread has allocated (and is keeping allocated) something and this is why we see the assert to fire?

      Attachments

        Issue Links

          Activity

            psergei Sergei Petrunia created issue -
            psergei Sergei Petrunia made changes -
            Field Original Value New Value
            elenst Elena Stepanova made changes -
            Fix Version/s 10.2 [ 14601 ]
            Affects Version/s 10.2 [ 14601 ]
            Assignee Sergei Petrunia [ psergey ]
            psergei Sergei Petrunia made changes -
            Summary rocksdb.validate_datadic fails on Windows rocksdb.validate_datadic fails on some platforms
            wlad Vladislav Vaintroub made changes -
            Summary rocksdb.validate_datadic fails on some platforms Rocksdb does not shutdown worker threads and aborts memleak check on server shutdown
            wlad Vladislav Vaintroub made changes -
            Summary Rocksdb does not shutdown worker threads and aborts memleak check on server shutdown Rocksdb does not shutdown worker threads and aborts in memleak check on server shutdown
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            wlad Vladislav Vaintroub made changes -
            wlad Vladislav Vaintroub made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 80242 ] MariaDB v4 [ 140273 ]

            People

              psergei Sergei Petrunia
              psergei Sergei Petrunia
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.