Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Won't Fix
-
10.0.34-galera
-
ubuntu-trusty-14.04-amd64-server-20180201
Description
Our entire Galera cluster crashed, each one with the same stacktrace here:
Jan 4 03:57:25 localhost mysqld: 190104 3:57:25 [ERROR] mysqld got signal 11 ;
|
Jan 4 03:57:25 localhost mysqld: This could be because you hit a bug. It is also possible that this binary
|
Jan 4 03:57:25 localhost mysqld: or one of the libraries it was linked against is corrupt, improperly built,
|
Jan 4 03:57:25 localhost mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Jan 4 03:57:25 localhost mysqld:
|
Jan 4 03:57:25 localhost mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Jan 4 03:57:25 localhost mysqld:
|
Jan 4 03:57:25 localhost mysqld: We will try our best to scrape up some info that will hopefully help
|
Jan 4 03:57:25 localhost mysqld: diagnose the problem, but since we have already crashed,
|
Jan 4 03:57:25 localhost mysqld: something is definitely wrong and this may fail.
|
Jan 4 03:57:25 localhost mysqld:
|
Jan 4 03:57:25 localhost mysqld: Server version: 10.0.34-MariaDB-1~trusty-wsrep
|
Jan 4 03:57:25 localhost mysqld: key_buffer_size=209715200
|
Jan 4 03:57:25 localhost mysqld: read_buffer_size=131072
|
Jan 4 03:57:25 localhost mysqld: max_used_connections=362
|
Jan 4 03:57:25 localhost mysqld: max_threads=2002
|
Jan 4 03:57:25 localhost mysqld: thread_count=363
|
Jan 4 03:57:25 localhost mysqld: It is possible that mysqld could use up to
|
Jan 4 03:57:25 localhost mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 16902085 K bytes of memory
|
Jan 4 03:57:25 localhost mysqld: Hope that's ok; if not, decrease some variables in the equation.
|
Jan 4 03:57:25 localhost mysqld:
|
Jan 4 03:57:25 localhost mysqld: Thread pointer: 0x7f7520457008
|
Jan 4 03:57:25 localhost mysqld: Attempting backtrace. You can use the following information to find out
|
Jan 4 03:57:25 localhost mysqld: where mysqld died. If you see no messages after this, something went
|
Jan 4 03:57:25 localhost mysqld: terribly wrong...
|
Jan 4 03:57:25 localhost mysqld: stack_bottom = 0x7f755106fe10 thread_stack 0x40000
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xc095fe]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x767de6]
|
Jan 4 03:57:26 localhost mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f928964a330]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT11get_explainEP11st_mem_root+0x6a)[0x86297a]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN24save_explain_data_internEP13Explain_querybbbPKc+0x7d3)[0x634873]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x6df)[0x6390bf]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x76)[0x63a446]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_Z19mysql_explain_unionP3THDP18st_select_lex_unitP13select_result+0x117)[0x63adf7]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld[0x518dc2]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x6d6a)[0x5e7bea]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld[0x5eb52e]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1d77)[0x5ed887]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x2c2)[0x5ee582]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x30b)[0x6bb3db]
|
Jan 4 03:57:26 localhost mysqld: /usr/sbin/mysqld(handle_one_connection+0x39)[0x6bb4b9]
|
Jan 4 03:57:27 localhost mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f9289642184]
|
Jan 4 03:57:27 localhost mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f9288d6503d]
|
Jan 4 03:57:27 localhost mysqld:
|
Jan 4 03:57:27 localhost mysqld: Trying to get some variables.
|
Jan 4 03:57:27 localhost mysqld: Some pointers may be invalid and cause the dump to abort.
|
Jan 4 03:57:27 localhost mysqld: Query (0x7f74cbaf6020): is an invalid pointer
|
Jan 4 03:57:27 localhost mysqld: Connection ID (thread ID): 5691574
|
Jan 4 03:57:27 localhost mysqld: Status: NOT_KILLED
|
Jan 4 03:57:27 localhost mysqld:
|
Jan 4 03:57:27 localhost mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
|
Jan 4 03:57:27 localhost mysqld:
|
Jan 4 03:57:27 localhost mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Jan 4 03:57:27 localhost mysqld: information that should help you find out what is causing the crash.
|
Jan 4 03:57:27 localhost mysqld:
|
Jan 4 03:57:27 localhost mysqld: We think the query pointer is invalid, but we will try to print it anyway.
|
Jan 4 03:57:27 localhost mysqld: Query: EXPLAIN SELECT `companies`.`id` FROM `companies` WHERE (companies.is_flagged_for_deletion <> 1) AND (companies.account_id = 161335) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615459 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 4765516 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 4873672 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 2328136))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615441 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328088))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615435 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328055 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 2328058))) AND (com
|
Jan 4 03:57:27 localhost mysqld: panies.next_entry_name IS NULL) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615408 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328019))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615411 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328049 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 3269848))) ORDER BY `companies`.`updated_at` DESC, `companies`.`id` DESC
|
Jan 4 03:57:27 localhost mysqld:
|
Jan 4 03:57:28 localhost kernel: [27243144.333350] TCP: TCP: Possible SYN flooding on port 3306. Sending cookies. Check SNMP counters.
|
Jan 4 03:57:28 localhost mysqld_safe: Number of processes running now: 0
|
Jan 4 03:57:28 localhost mysqld_safe: WSREP: not restarting wsrep node automatically
|
Jan 4 03:57:28 localhost mysqld_safe: mysqld from pid file /var/lib/mysql/mysqld.pid ended
|
Attachments
Activity
We have been able to reproduce the crash on 10.0.34 and subsequently found the trigger to be related to using latin1 encoding on this particular dataset. Testing has shown that an upgrade to 10.0.37 fixes the issue. If you know of any related bugs that were fixed in either .35, .36, or .37 could you add the links? We would like to further understand root cause as we have not been able to identify the exact data that causes the crash, and it also seems to be related to the connection state (it doesn't happen at first, but then happens continuously until the connection is reset)
I am not able to reproduce it right away. I suspect it to be related to MDEV-15247, but it might be only a theory.
Would you be able to upload datadump of those tables to ftp.askmonty.org/private, then I will try to get the crash?
Unfortunately we cannot provide a data dump due to customer data privacy. It doesn't appear to be related to 15247 because that bug actually is about utf8 causing the crash. Ours is about utf8 preventing the crash when we are defaulted to latin1
Jan 9 21:28:57 localhost mysqld: 190109 21:28:57 [ERROR] mysqld got signal 11 ;
Jan 9 21:28:57 localhost mysqld: This could be because you hit a bug. It is also possible that this binary
Jan 9 21:28:57 localhost mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Jan 9 21:28:57 localhost mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: We will try our best to scrape up some info that will hopefully help
Jan 9 21:28:57 localhost mysqld: diagnose the problem, but since we have already crashed,
Jan 9 21:28:57 localhost mysqld: something is definitely wrong and this may fail.
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: Server version: 10.0.34-MariaDB-1~trusty-wsrep
Jan 9 21:28:57 localhost mysqld: key_buffer_size=209715200
Jan 9 21:28:57 localhost mysqld: read_buffer_size=131072
Jan 9 21:28:57 localhost mysqld: max_used_connections=324
Jan 9 21:28:57 localhost mysqld: max_threads=2002
Jan 9 21:28:57 localhost mysqld: thread_count=325
Jan 9 21:28:57 localhost mysqld: It is possible that mysqld could use up to
Jan 9 21:28:57 localhost mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 16902085 K bytes of memory
Jan 9 21:28:57 localhost mysqld: Hope that's ok; if not, decrease some variables in the equation.
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: Thread pointer: 0x7fc6e88b3008
Jan 9 21:28:57 localhost mysqld: Attempting backtrace. You can use the following information to find out
Jan 9 21:28:57 localhost mysqld: where mysqld died. If you see no messages after this, something went
Jan 9 21:28:57 localhost mysqld: terribly wrong...
Jan 9 21:28:57 localhost mysqld: stack_bottom = 0x7fc6ffd79e10 thread_stack 0x40000
Jan 9 21:29:00 localhost mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xc095fe]
Jan 9 21:29:00 localhost mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x767de6]
Jan 9 21:29:03 localhost mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fe453a47330]
Jan 9 21:29:04 localhost mysqld: 190109 21:29:04 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.60395S), skipping check
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT11get_explainEP11st_mem_root+0x6a)[0x86297a]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN24save_explain_data_internEP13Explain_querybbbPKc+0x7d3)[0x634873]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x6df)[0x6390bf]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x76)[0x63a446]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z19mysql_explain_unionP3THDP18st_select_lex_unitP13select_result+0x117)[0x63adf7]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld[0x518dc2]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x6d6a)[0x5e7bea]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld[0x5eb52e]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1d77)[0x5ed887]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x2c2)[0x5ee582]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x30b)[0x6bb3db]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(handle_one_connection+0x39)[0x6bb4b9]
Jan 9 21:29:07 localhost mysqld: 190109 21:29:07 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.65851S), skipping check
Jan 9 21:29:08 localhost mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7fe453a3f184]
Jan 9 21:29:09 localhost mysqld: 190109 21:29:09 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.6081S), skipping check
Jan 9 21:29:11 localhost mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe45316203d]
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: Trying to get some variables.
Jan 9 21:29:11 localhost mysqld: Some pointers may be invalid and cause the dump to abort.
Jan 9 21:29:11 localhost mysqld: Query (0x7fc6ea9d8020): is an invalid pointer
Jan 9 21:29:11 localhost mysqld: Connection ID (thread ID): 255594
Jan 9 21:29:11 localhost mysqld: Status: NOT_KILLED
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Jan 9 21:29:11 localhost mysqld: information that should help you find out what is causing the crash.
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: We think the query pointer is invalid, but we will try to print it anyway.
Jan 9 21:29:11 localhost mysqld: Query: EXPLAIN SELECT `companies`.`id` FROM `companies` WHERE (companies.is_flagged_for_deletion <> 1) AND (companies.account_id = 161335) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615459 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 4765516 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 4873672 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 2328136))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615441 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328088))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615435 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328055 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 2328058))) AND (com
Jan 9 21:29:11 localhost mysqld: panies.next_entry_name IS NULL) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615408 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328019))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615411 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328049 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 3269848))) ORDER BY `companies`.`updated_at` DESC, `companies`.`id` DESC