Vladislav Vaintroub added a comment - 2016-10-12 14:29 - edited Crash callstack (from a mysqld.dmp on buildbot) ntdll.dll!RtlpWaitOnCriticalSection() Unknown ntdll.dll!RtlpEnterCriticalSectionContended() Unknown mysqld.exe!Statement_map::~Statement_map() Line 3963 C++ mysqld.exe!THD::~THD() Line 1790 C++ mysqld.exe!THD::`scalar deleting destructor'(unsigned int) C++ > mysqld.exe!do_handle_bootstrap(THD * thd=0x0000008c80cf6508) Line 1086 C++ mysqld.exe!handle_bootstrap(void * arg=0x0000008c80cf6508) Line 1066 C++ mysqld.exe!pthread_start(void * p=0x0000000000000000) Line 62 C mysqld.exe!_callthreadstartex() Line 376 C mysqld.exe!_threadstartex(void * ptd=0x0000008c80cfeff0) Line 354 C kernel32.dll!BaseThreadInitThunk() Unknown ntdll.dll!RtlUserThreadStart() Unknown Mostly likely, there is a memory overwrite here that corrupts critical section. callstack shows RtlpEnterCriticalSectionContended, but this critical section cannot be contended, as there is only a single active thread.

Andrii Nikitin (Inactive) added a comment - 2017-02-10 08:45 - edited I see at http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/2542/steps/test/logs/stdio that main thread is being aborted when crash happens: mysqld!my_thread_global_end(void) [d:\winx64-debug\build\src\mysys\my_thr_init.c @ 203] mysqld!my_end(int infoflag = 0n0) [d:\winx64-debug\build\src\mysys\my_init.c @ 210] mysqld!mysqld_exit(int exit_code = 0n0) [d:\winx64-debug\build\src\sql\mysqld.cc @ 2149] mysqld!unireg_abort(int exit_code = 0n0) [d:\winx64-debug\build\src\sql\mysqld.cc @ 2123] mysqld!win_main(int argc = 0n54, char ** argv = 0x00000075`f9b15cf8) [d:\winx64-debug\build\src\sql\mysqld.cc @ 6018] mysqld!mysql_service(void * p = 0x00000000`00000000) [d:\winx64-debug\build\src\sql\mysqld.cc @ 6156] mysqld!mysqld_main(int argc = 0n54, char ** argv = 0x00000075`f9b03710) [d:\winx64-debug\build\src\sql\mysqld.cc @ 6349] Edited: it looks that crash happens because mutex LOCK_prepared_stmt_count is referenced from ~Statement_map() of bootstrap thread is already destroyed in mysqld_exit()

Andrii Nikitin (Inactive) added a comment - 2017-02-10 10:23 - edited It looks I can repeat the crash manually by injecting few Sleep() calls as below: diff --git a/sql/mysqld.cc b/sql/mysqld.cc index 23ac568d95..bc908740a4 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -2345,6 +2345,8 @@ static void clean_up_mutexes() mysql_rwlock_destroy(&LOCK_system_variables_hash); mysql_mutex_destroy(&LOCK_short_uuid_generator); mysql_mutex_destroy(&LOCK_prepared_stmt_count); + bzero(&LOCK_prepared_stmt_count, sizeof (LOCK_prepared_stmt_count)); + Sleep(5000); mysql_mutex_destroy(&LOCK_error_messages); mysql_cond_destroy(&COND_thread_count); mysql_cond_destroy(&COND_thread_cache); @@ -6384,6 +6386,7 @@ static void bootstrap(MYSQL_FILE *file) bootstrap_error=-1; DBUG_VOID_RETURN; } + Sleep(3000); /* Wait for thread to die */ mysql_mutex_lock(&LOCK_thread_count); while (in_bootstrap) diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc index 5348245449..58f6cc7a4e 100644 --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -1085,6 +1085,7 @@ void do_handle_bootstrap(THD *thd) end: in_bootstrap= FALSE; + Sleep(4000); delete thd; mysql_mutex_lock(&LOCK_thread_count); mysql_cond_broadcast(&COND_thread_count); In my understanding this may happen only on single (or very overloaded) CPU when do_handle_bootstrap() completes before main thread executes "while (in_bootstrap)"