[MDEV-15936] Crashes with varying stack trace on Windows with Spider engine Created: 2018-04-20  Updated: 2020-08-25  Resolved: 2018-07-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.2.14
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Nilnandan Joshi Assignee: Jacob Mathew (Inactive)
Resolution: Duplicate Votes: 3
Labels: None
Environment:

Windows

Attachments: Text File out_analyze.txt     Text File out_unique.txt    
Issue Links:
Duplicate
duplicates MDEV-12900 spider tests failed in buildbot with ... Closed
is duplicated by MDEV-16002 Crash in table_cache code Closed
Relates
relates to MDEV-12900 spider tests failed in buildbot with ... Closed
relates to MDEV-16348 the suddenly crashing in MariaDB10.1.29 Closed

 Description   

There have been a total of ten crashes by now, with eight more or less distinct backtraces.
When having a closer look at the backtraces they don't really seem to make sense, showing call sequences that include functions that can't possibly call the next function in the trace.

So it looks as if we're actually facing a stack smashing bug here.

Below are the extracted stack traces, demangled with the help of https://demangler.com/

Full error log is available on demand, and core-dump has now been added to the configuration to hopefully get better debug info from that.

Trace #1

mysqld.exe!private: void __cdecl QUERY_PROFILE::new_status(char const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!public: void __cdecl THD::enter_stage(struct PSI_stage_info_v1 const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!set_thd_stage_info()
 
mysqld.exe!public: BOOL __cdecl Query_cache::try_lock(class THD * __ptr64,enum Query_cache::Cache_try_lock_mode) __ptr64
 
mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_send_query()
 
mysqld.exe!mysql_real_query()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #2

mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_close_slow_part()
 
mysqld.exe!mysql_close()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!int __cdecl ha_finalize_handlerton(struct st_plugin_int * __ptr64)
 
mysqld.exe!BOOL __cdecl mysql_uninstall_plugin(class THD * __ptr64,struct st_mysql_lex_string const * __ptr64,struct st_mysql_lex_string const * __ptr64)
 
mysqld.exe!public: unsigned char * __ptr64 __cdecl sys_var_pluginvar::real_value_ptr(class THD * __ptr64,enum enum_var_type) __ptr64
 
mysqld.exe!void __cdecl plugin_shutdown(void)
 
mysqld.exe!public: char * __ptr64 __cdecl String::c_ptr_safe(void) __ptr64
 
mysqld.exe!void __cdecl unireg_end(void)
 
mysqld.exe!void __cdecl kill_mysql(class THD * __ptr64)
 
mysqld.exe!handle_shutdown()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64

Trace #3

mysqld.exe!private: void __cdecl QUERY_PROFILE::new_status(char const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!public: void __cdecl THD::enter_stage(struct PSI_stage_info_v1 const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!set_thd_stage_info()
 
mysqld.exe!public: BOOL __cdecl Query_cache::try_lock(class THD * __ptr64,enum Query_cache::Cache_try_lock_mode) __ptr64
 
mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_send_query()
 
mysqld.exe!mysql_real_query()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #4

mysqld.exe!private: void __cdecl QUERY_PROFILE::new_status(char const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!public: void __cdecl THD::enter_stage(struct PSI_stage_info_v1 const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!set_thd_stage_info()
 
mysqld.exe!public: BOOL __cdecl Query_cache::try_lock(class THD * __ptr64,enum Query_cache::Cache_try_lock_mode) __ptr64
 
mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_select_db()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #5

mysqld.exe!public: unsigned int __cdecl Query_cache_block::headers_len(void) __ptr64
 
mysqld.exe!public: unsigned char * __ptr64 __cdecl Query_cache_block::data(void) __ptr64
 
mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_send_query()
 
mysqld.exe!mysql_real_query()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #6

mysqld.exe!protected: char __cdecl Query_cache::append_result_data(struct Query_cache_block * __ptr64 * __ptr64,unsigned long,unsigned char * __ptr64,struct Query_cache_block * __ptr64) __ptr64
 
mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_send_query()
 
mysqld.exe!mysql_real_query()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #7

mysqld.exe!public: unsigned int __cdecl Query_cache_block::headers_len(void) __ptr64
 
mysqld.exe!public: unsigned char * __ptr64 __cdecl Query_cache_block::data(void) __ptr64
 
mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_select_db()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #8

mysqld.exe!public: __cdecl Trigger::~Trigger(void) __ptr64
 
mysqld.exe!public: __cdecl Table_triggers_list::~Table_triggers_list(void) __ptr64
 
mysqld.exe!public: __cdecl Table_cache_instance::~Table_cache_instance(void) __ptr64
 
mysqld.exe!void __cdecl tc_add_table(class THD * __ptr64,struct TABLE * __ptr64)
 
mysqld.exe!BOOL __cdecl open_table(class THD * __ptr64,struct TABLE_LIST * __ptr64,class Open_table_context * __ptr64)
 
mysqld.exe!BOOL __cdecl open_and_lock_tables(class THD * __ptr64,struct DDL_options_st const & __ptr64,struct TABLE_LIST * __ptr64,BOOL,unsigned int,class Prelocking_strategy * __ptr64)
 
mysqld.exe!BOOL __cdecl open_tables(class THD * __ptr64,struct DDL_options_st const & __ptr64,struct TABLE_LIST * __ptr64 * __ptr64,unsigned int * __ptr64,unsigned int,class Prelocking_strategy * __ptr64)
 
mysqld.exe!BOOL __cdecl open_and_lock_tables(class THD * __ptr64,struct DDL_options_st const & __ptr64,struct TABLE_LIST * __ptr64,BOOL,unsigned int,class Prelocking_strategy * __ptr64)
 
mysqld.exe!void __cdecl execute_init_command(class THD * __ptr64,struct st_mysql_lex_string * __ptr64,struct st_mysql_rwlock * __ptr64)
 
mysqld.exe!int __cdecl mysql_execute_command(class THD * __ptr64)
 
mysqld.exe!void __cdecl mysql_parse(class THD * __ptr64,char * __ptr64,unsigned int,class Parser_state * __ptr64,BOOL,BOOL)
 
mysqld.exe!BOOL __cdecl dispatch_command(enum enum_server_command,class THD * __ptr64,char * __ptr64,unsigned int,BOOL,BOOL)
 
mysqld.exe!BOOL __cdecl do_command(class THD * __ptr64)
 
mysqld.exe!void __cdecl pool_of_threads_scheduler(struct scheduler_functions * __ptr64,unsigned long * __ptr64,unsigned int * __ptr64)
 
mysqld.exe!void __cdecl tp_callback(struct TP_connection * __ptr64)
 
ntdll.dll!RtlReleaseSRWLockExclusive()
 
ntdll.dll!RtlReleaseSRWLockExclusive()
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #9

mysqld.exe!thd_increment_bytes_sent()
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_send_query()
 
mysqld.exe!mysql_real_query()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()

Trace #10

mysqld.exe!private: void __cdecl QUERY_PROFILE::new_status(char const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!public: void __cdecl THD::enter_stage(struct PSI_stage_info_v1 const * __ptr64,char const * __ptr64,char const * __ptr64,unsigned int) __ptr64
 
mysqld.exe!set_thd_stage_info()
 
mysqld.exe!public: BOOL __cdecl Query_cache::try_lock(class THD * __ptr64,enum Query_cache::Cache_try_lock_mode) __ptr64
 
mysqld.exe!public: void __cdecl Query_cache::insert(class THD * __ptr64,struct Query_cache_tls * __ptr64,char const * __ptr64,unsigned long,unsigned int) __ptr64
 
mysqld.exe!void __cdecl query_cache_insert(void * __ptr64,char const * __ptr64,unsigned long,unsigned int)
 
mysqld.exe!net_real_write()
 
mysqld.exe!net_flush()
 
mysqld.exe!net_write_command()
 
mysqld.exe!cli_advanced_command()
 
mysqld.exe!mysql_send_query()
 
mysqld.exe!mysql_real_query()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_deinit()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
ha_spider.dll!spider_flush_table_mon_cache_init()
 
mysqld.exe!pthread_join()
 
mysqld.exe!public: virtual void __cdecl Item::traverse_cond(void (__cdecl*)(class Item const * __ptr64,void * __ptr64),void * __ptr64,enum Item::traverse_order) __ptr64
 
KERNEL32.DLL!BaseThreadInitThunk()
 
ntdll.dll!RtlUserThreadStart()



 Comments   
Comment by Jacob Mathew (Inactive) [ 2018-04-25 ]

According to the stack trace, spider_flush_table_mon_cache_deinit() calls mysql_real_query(). In fact, spider_flush_table_mon_cache_deinit() is an empty function that doesn't call anything. So the stack trace is absolutely wrong. Is it possible that this is a build from source code and that there is something wrong with the Windows build?

So it is not clear that this is a Spider problem. If there is a way to reproduce this problem, we or someone in support could determine the correct stack trace and we would have more information about how to proceed.

Comment by Hartmut Holzgraefe [ 2018-04-27 ]

Title and description has been rewritten completely after seeing that we're actually facing multiple different and implausible crash backtraces in the error log

Comment by Antoine Girardi [ 2018-04-30 ]

Hi,

we got same error on 10.1.25 version

180427 12:46:30 [ERROR] mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.25-MariaDB
key_buffer_size=65536
read_buffer_size=131072
max_used_connections=1983
max_threads=1001
thread_count=1499
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1207304 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x787fe19ed8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!thd_increment_bytes_sent()
mysqld.exe!net_real_write()
mysqld.exe!net_flush()
mysqld.exe!net_write_command()
mysqld.exe!cli_advanced_command()
mysqld.exe!mysql_send_query()
mysqld.exe!mysql_real_query()
ha_spider.dll!spider_flush_table_mon_cache_init()
ha_spider.dll!???
ha_spider.dll!spider_flush_table_mon_cache_init()
ha_spider.dll!???
ha_spider.dll!???
mysqld.exe!pthread_join()
mysqld.exe!strxnmov()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 76840
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

Comment by Elena Stepanova [ 2018-05-31 ]

jacob-mathew, please also take a look at MDEV-12725 and issues linked to it – we had a problem in Federated(X) engines which caused similar crashes; it was reproduced and fixed, maybe it will give some hints.

Comment by Jacob Mathew (Inactive) [ 2018-06-09 ]

It is possible that this problem is a duplicate of MDEV-12900, which I fixed in MariaDB Server 10.2.16. I recommend that the customer install MariaDB Server 10.2.16 when it is released, or a daily 10.2 build that includes fixes for MDEV-12900 and related bug MDEV-7914.

Comment by Jacob Mathew (Inactive) [ 2018-07-24 ]

Waiting for the customer to confirm after installing 10.2.16 or a later release of MariaDB Server.

Comment by Nilnandan Joshi [ 2018-07-24 ]

Hi Jacob,

Customer confirmed that "After the installation of the fix 10.2.16, we got no more unexpected exceptions." so I think you can set this to be fixed.

Regards

Comment by Jacob Mathew (Inactive) [ 2018-07-24 ]

Customer has confirmed that they have encountered no further unexpected exceptions after installing MariaDB Server 10.2.16. I am therefore closing this bug as a duplicate of bug MDEV-12900.

Generated at Thu Feb 08 08:25:08 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.