[MDEV-12156] 10.1.21 crash Created: 2017-03-01  Updated: 2021-04-28  Resolved: 2021-04-28

Status: Closed
Project: MariaDB Server
Component/s: Query Cache
Affects Version/s: 10.1.21
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Andreas Schnederle-Wagner Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: crash

Attachments: File my.cnf    
Issue Links:
Duplicate
is duplicated by MDEV-25365 Server hangs or crashes in Query_cach... Confirmed
Relates
relates to MDEV-5924 MariaDB could crash after changing th... Closed
relates to MDEV-10498 10.1.9-MariaDB-log MariaDB Server cr... Closed
relates to MDEV-10826 Assertion `thd->get_stmt_da()->is_eof... Confirmed
relates to MDEV-11622 Mariadb= 10.1.17-MariaDB get crashed... Closed

 Description   

Hello,
tonight one of our MariaDB Server crashed out of nowhere. Searched the Reason for a few Hours now - but was not able to narrow the cause down! :-/
Maybe it's a Bug? Maybe you can tell me a bit more from those Errors?

OS: CentOS Linux release 7.3.1611 (Core)

Thank you

170301  0:40:01 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.21-MariaDB
 
key_buffer_size=3221225472
 
read_buffer_size=6291456
 
max_used_connections=502
 
max_threads=602
 
thread_count=79
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 85762103 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f6e58b00a38
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f67b2a2ccf0 thread_stack 0x30000
 
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x7f6e565ce80e]
 
/usr/libexec/mysqld(handle_fatal_signal+0x305)[0x7f6e560f5da5]
 
/lib64/libpthread.so.0(+0xf370)[0x7f6e55712370]
 
/usr/libexec/mysqld(_ZN11Query_cache12move_by_typeEPPhPP17Query_cache_blockPmS3_+0x730)[0x7f6e55f3b920]
 
/usr/libexec/mysqld(_ZN11Query_cache10pack_cacheEv+0x71)[0x7f6e55f3bb21]
 
/usr/libexec/mysqld(_ZN11Query_cache4packEP3THDmj+0x4e)[0x7f6e55f3be0e]
 
/usr/libexec/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x690)[0x7f6e5605e970]
 
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x1400)[0x7f6e55f716a0]
 
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x332)[0x7f6e55f79682]
 
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x243c)[0x7f6e55f7c99c]
 
/usr/libexec/mysqld(_Z10do_commandP3THD+0x14a)[0x7f6e55f7d1fa]
 
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f6e5604445a]
 
/usr/libexec/mysqld(handle_one_connection+0x40)[0x7f6e56044600]
 
/lib64/libpthread.so.0(+0x7dc5)[0x7f6e5570adc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f6e53f8373d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f68c40008e0): FLUSH QUERY CACHE
 
Connection ID (thread ID): 568895
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off



 Comments   
Comment by Andreas Schnederle-Wagner [ 2017-03-01 ]

added config file

Comment by Daniel Black [ 2017-03-01 ]

Definitely a bug. A server should never crash due to a query. Even one like FLUSH QUERY CACHE. No doing this SQL query will probably not affect any server behaviour at all.
The output from this may be useful too.

 SHOW GLOBAL STATUS LIKE 'Qcache%';

edit: server config provided

Comment by Andreas Schnederle-Wagner [ 2017-03-01 ]

Uptime: 7 Hours 

SHOW GLOBAL STATUS LIKE 'Qcache%' 
 
 
Qcache_free_blocks 	2174
 
Qcache_free_memory 	104877056
 
Qcache_hits 	3047970
 
Qcache_inserts 	627341
 
Qcache_lowmem_prunes 	63490
 
Qcache_not_cached 	366437
 
Qcache_queries_in_cache 	15284
 
Qcache_total_blocks 	32979

Uptime: 2 Days, 8 Hours, 30 Minutes

SHOW GLOBAL STATUS LIKE 'Qcache%' 
 
 
Qcache_free_blocks 	212
 
Qcache_free_memory 	67424424
 
Qcache_hits 	28510680
 
Qcache_inserts 	4346831
 
Qcache_lowmem_prunes 	682635
 
Qcache_not_cached 	2673469
 
Qcache_queries_in_cache 	36561
 
Qcache_total_blocks 	73617

Comment by Andreas Schnederle-Wagner [ 2017-03-03 ]

are there any further information I can supply?

Comment by Andreas Schnederle-Wagner [ 2017-03-13 ]

Hello,
during the weekend the crash happened 2 more times:

12.03.2017 08:00:11:

170312  8:00:02 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.21-MariaDB
 
key_buffer_size=3221225472
 
read_buffer_size=6291456
 
max_used_connections=503
 
max_threads=602
 
thread_count=73
 
It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 85762103 K  bytes of memory Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f56d1ef2c48
 
Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong...
 
stack_bottom = 0x7f54ffc8ccf0 thread_stack 0x30000 /usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x7f56cfbaf80e]
 
/usr/libexec/mysqld(handle_fatal_signal+0x305)[0x7f56cf6d6da5]
 
/lib64/libpthread.so.0(+0xf370)[0x7f56cecf3370]
 
/usr/libexec/mysqld(_ZN11Query_cache12move_by_typeEPPhPP17Query_cache_blockPmS3_+0x730)[0x7f56cf51c920]
 
/usr/libexec/mysqld(_ZN11Query_cache10pack_cacheEv+0x71)[0x7f56cf51cb21]
 
/usr/libexec/mysqld(_ZN11Query_cache4packEP3THDmj+0x4e)[0x7f56cf51ce0e]
 
/usr/libexec/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x690)[0x7f56cf63f970]
 
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x1400)[0x7f56cf5526a0]
 
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x332)[0x7f56cf55a682]
 
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x243c)[0x7f56cf55d99c]
 
/usr/libexec/mysqld(_Z10do_commandP3THD+0x14a)[0x7f56cf55e1fa]
 
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f56cf62545a]
 
/usr/libexec/mysqld(handle_one_connection+0x40)[0x7f56cf625600]
 
/lib64/libpthread.so.0(+0x7dc5)[0x7f56cecebdc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f56cd56473d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f528c0008e0): FLUSH QUERY CACHE Connection ID (thread ID): 292308
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

Comment by Andreas Schnederle-Wagner [ 2017-03-13 ]

13.03.2017 03:10:01:

170313  3:10:01 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.21-MariaDB
 
key_buffer_size=3221225472
 
read_buffer_size=6291456
 
max_used_connections=417
 
max_threads=602
 
thread_count=48
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 85762103 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f135263aea8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f118c344cf0 thread_stack 0x30000
 
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x7f134e99880e]
 
/usr/libexec/mysqld(handle_fatal_signal+0x305)[0x7f134e4bfda5]
 
/lib64/libpthread.so.0(+0xf370)[0x7f134dadc370]
 
/usr/libexec/mysqld(_ZN11Query_cache12move_by_typeEPPhPP17Query_cache_blockPmS3_+0x730)[0x7f134e305920]
 
/usr/libexec/mysqld(_ZN11Query_cache10pack_cacheEv+0x71)[0x7f134e305b21]
 
/usr/libexec/mysqld(_ZN11Query_cache4packEP3THDmj+0x4e)[0x7f134e305e0e]
 
/usr/libexec/mysqld(_Z20reload_acl_and_cacheP3THDyP10TABLE_LISTPi+0x690)[0x7f134e428970]
 
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x1400)[0x7f134e33b6a0]
 
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x332)[0x7f134e343682]
 
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x243c)[0x7f134e34699c]
 
/usr/libexec/mysqld(_Z10do_commandP3THD+0x14a)[0x7f134e3471fa]
 
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f134e40e45a]
 
/usr/libexec/mysqld(handle_one_connection+0x40)[0x7f134e40e600]
 
/lib64/libpthread.so.0(+0x7dc5)[0x7f134dad4dc5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f134c34d73d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f0f2c00bd20): FLUSH QUERY CACHE
 
Connection ID (thread ID): 20616
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off

Comment by Daniel Black [ 2017-03-13 ]

The obvious workaround is to not FLUSH QUERY CACHE. It can't be giving you any benefit compared to the crashing.

Comment by Elena Stepanova [ 2021-04-28 ]

MDEV-25365 has a test case for what appears to be an identical failure (although not a great test case, but better than nothing). So, while it's generally against common sense to close an older report as a duplicate of a new one, in this case we'll continue tracking this problem in MDEV-25365.

Generated at Thu Feb 08 07:55:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.