ubuntu-trusty-14.04-amd64-server-20180201

Invalid query pointer is not the reason of the problem, it's a nuisance which happens on certain non-debug builds while generating crash reports.

Could you please add the output of

SHOW CREATE TABLE companies;

SHOW CREATE TABLE custom_field_values;

Is it possible for you to try the same query and see if the crash is reproducible?
Please upload also your .cnf file(s)

Here is the my.cnf file from one of the servers. They are all the same. We tried to reproduce on our staging server by executing the query and it worked fine. We do not want to execute in production due to risk. my.cnf

companies CREATE TABLE `companies` (↵ `id` int(11) NOT NULL AUTO_INCREMENT,↵ `name` varchar(255) DEFAULT NULL,↵ `address_1` varchar(255) DEFAULT NULL,↵ `city` varchar(255) DEFAULT NULL,↵ `postal_code` varchar(255) DEFAULT NULL,↵ `state` varchar(255) DEFAULT NULL,↵ `phone1` varchar(255) DEFAULT NULL,↵ `fax` varchar(255) DEFAULT NULL,↵ `email` varchar(255) DEFAULT NULL,↵ `web` varchar(255) DEFAULT NULL,↵ `account_id` int(11) DEFAULT NULL,↵ `phone1_desc` varchar(255) DEFAULT NULL,↵ `phone2` varchar(255) DEFAULT NULL,↵ `phone2_desc` varchar(255) DEFAULT NULL,↵ `phone3` varchar(255) DEFAULT NULL,↵ `phone3_desc` varchar(255) DEFAULT NULL,↵ `phone4` varchar(255) DEFAULT NULL,↵ `phone4_desc` varchar(255) DEFAULT NULL,↵ `country` varchar(255) DEFAULT NULL,↵ `description` text,↵ `image_name` varchar(255) DEFAULT NULL,↵ `image_file` blob,↵ `image_content_type` varchar(255) DEFAULT NULL,↵ `address_2` varchar(255) DEFAULT NULL,↵ `delta` tinyint(1) NOT NULL DEFAULT '1',↵ `updated_at` datetime DEFAULT NULL,↵ `created_at` datetime DEFAULT NULL,↵ `image_file_name` varchar(255) DEFAULT NULL,↵ `import_id` int(11) DEFAULT NULL,↵ `is_flagged_for_deletion` tinyint(1) DEFAULT '0',↵ `created_by_user_id` int(11) DEFAULT NULL,↵ `next_entry_name` varchar(255) DEFAULT NULL,↵ `next_entry_due` datetime DEFAULT NULL,↵ `next_entry_all_day` tinyint(1) DEFAULT NULL,↵ `next_entry_id` int(11) DEFAULT NULL,↵ `latest_activity_id` int(11) DEFAULT NULL,↵ `owner_id` int(11) DEFAULT NULL,↵ `is_sample` tinyint(1) DEFAULT '0',↵ PRIMARY KEY (`id`),↵ KEY `index_companies_on_account_id` (`account_id`),↵ KEY `index_companies_on_account_id_and_name` (`account_id`,`name`),↵ KEY `index_companies_on_is_flagged_for_deletion` (`is_flagged_for_deletion`),↵ KEY `index_companies_on_import_id` (`import_id`),↵ KEY `index_companies_on_created_by_user_id` (`created_by_user_id`),↵ KEY `index_companies_on_latest_activity_id` (`latest_activity_id`),↵ KEY `index_companies_on_owner_id` (`owner_id`),↵ KEY `index_companies_on_account_id_and_email` (`account_id`,`email`),↵ CONSTRAINT `__companies_accounts_fk` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`)↵) ENGINE=InnoDB AUTO_INCREMENT=104848088 DEFAULT CHARSET=latin1

custom_field_values CREATE TABLE `custom_field_values` (↵ `id` int(11) NOT NULL AUTO_INCREMENT,↵ `custom_field_label_id` int(11) DEFAULT NULL,↵ `custom_field_label_dropdown_entry_id` int(11) DEFAULT NULL,↵ `association_id` int(11) DEFAULT NULL,↵ `association_type` varchar(255) DEFAULT NULL,↵ `account_id` int(11) DEFAULT NULL,↵ `deal_id` int(11) DEFAULT NULL,↵ `company_id` int(11) DEFAULT NULL,↵ `person_id` int(11) DEFAULT NULL,↵ `string_value` varchar(5000) DEFAULT NULL,↵ `numeric_value` decimal(38,8) DEFAULT NULL,↵ `date_value` date DEFAULT NULL,↵ `created_at` datetime DEFAULT NULL,↵ `updated_at` datetime DEFAULT NULL,↵ PRIMARY KEY (`id`),↵ KEY `index_custom_field_values_on_custom_field_label_id` (`custom_field_label_id`),↵ KEY `index_custom_field_values_on_deal_id_and_custom_field_label_id` (`deal_id`,`custom_field_label_id`),↵ KEY `index_custom_field_values_on_person_id_and_custom_field_label_id` (`person_id`,`custom_field_label_id`),↵ KEY `index_cfv_on_cfl_id_and_string_value` (`custom_field_label_id`,`string_value`(767)),↵ KEY `index_cfv_on_cfl_id_and_numeric_value` (`custom_field_label_id`,`numeric_value`),↵ KEY `index_cfv_on_cfl_id_and_date_value` (`custom_field_label_id`,`date_value`),↵ KEY `index_cfv_on_cfl_id_and_association_id` (`custom_field_label_id`,`association_id`),↵ KEY `index_cfv_on_cfl_id_and_cfl_dropdown_entry_id` (`custom_field_label_id`,`custom_field_label_dropdown_entry_id`),↵ KEY `index_custom_field_values_on_company_id` (`company_id`),↵ KEY `index_custom_field_values_on_deal_id` (`deal_id`),↵ KEY `index_custom_field_values_on_person_id` (`person_id`),↵ KEY `index_custom_field_values_on_account_id` (`account_id`)↵) ENGINE=InnoDB AUTO_INCREMENT=1188538541 DEFAULT CHARSET=latin1

Any new info here? We had another production crash across the cluster today. Similar but different. Will post the stack traces.

Jan 9 21:29:17 localhost mysqld: 190109 21:29:17 [ERROR] mysqld got signal 11 ;
Jan 9 21:29:17 localhost mysqld: This could be because you hit a bug. It is also possible that this binary
Jan 9 21:29:17 localhost mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Jan 9 21:29:17 localhost mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Jan 9 21:29:17 localhost mysqld:
Jan 9 21:29:17 localhost mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jan 9 21:29:17 localhost mysqld:
Jan 9 21:29:17 localhost mysqld: We will try our best to scrape up some info that will hopefully help
Jan 9 21:29:17 localhost mysqld: diagnose the problem, but since we have already crashed,
Jan 9 21:29:17 localhost mysqld: something is definitely wrong and this may fail.
Jan 9 21:29:17 localhost mysqld:
Jan 9 21:29:17 localhost mysqld: Server version: 10.0.34-MariaDB-1~trusty-wsrep
Jan 9 21:29:17 localhost mysqld: key_buffer_size=209715200
Jan 9 21:29:17 localhost mysqld: read_buffer_size=131072
Jan 9 21:29:17 localhost mysqld: max_used_connections=330
Jan 9 21:29:17 localhost mysqld: max_threads=2002
Jan 9 21:29:17 localhost mysqld: thread_count=332
Jan 9 21:29:17 localhost mysqld: It is possible that mysqld could use up to
Jan 9 21:29:17 localhost mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 16902085 K bytes of memory
Jan 9 21:29:17 localhost mysqld: Hope that's ok; if not, decrease some variables in the equation.
Jan 9 21:29:17 localhost mysqld:
Jan 9 21:29:17 localhost mysqld: Thread pointer: 0x7fdce2f00008
Jan 9 21:29:17 localhost mysqld: Attempting backtrace. You can use the following information to find out
Jan 9 21:29:17 localhost mysqld: where mysqld died. If you see no messages after this, something went
Jan 9 21:29:17 localhost mysqld: terribly wrong...
Jan 9 21:29:17 localhost mysqld: stack_bottom = 0x7fe01e0d2e10 thread_stack 0x40000
Jan 9 21:29:19 localhost mysqld: 190109 21:29:19 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.84925S), skipping check
Jan 9 21:29:19 localhost mysqld: 190109 21:29:19 [Warning] Aborted connection 251905 to db: 'pipelinedeals_production' user: 'pipelinedeals' host: 'ip-172-16-15-138.ec2.internal' (Got an error reading communication packets)
Jan 9 21:29:19 localhost mysqld: 190109 21:29:19 [Warning] Aborted connection 251966 to db: 'pipelinedeals_production' user: 'pipelinedeals' host: 'ip-172-16-15-138.ec2.internal' (Got an error reading communication packets)
Jan 9 21:29:20 localhost mysqld: 190109 21:29:20 [Note] WSREP: cleaning up cb737d2a (tcp://172.16.15.226:4567)
Jan 9 21:29:20 localhost mysqld: 190109 21:29:20 [Warning] Aborted connection 248929 to db: 'pipelinedeals_production' user: 'pipelinedeals' host: 'ip-172-16-15-183.ec2.internal' (Got an error writing communication packets)
Jan 9 21:29:20 localhost mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xc095fe]
Jan 9 21:29:20 localhost mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x767de6]
Jan 9 21:29:21 localhost mysqld: 190109 21:29:21 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.50937S), skipping check
Jan 9 21:29:22 localhost mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7ffa11a5d330]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT11get_explainEP11st_mem_root+0x6a)[0x86297a]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN24save_explain_data_internEP13Explain_querybbbPKc+0x7d3)[0x634873]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x6df)[0x6390bf]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x76)[0x63a446]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_Z19mysql_explain_unionP3THDP18st_select_lex_unitP13select_result+0x117)[0x63adf7]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld[0x518dc2]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x6d6a)[0x5e7bea]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld[0x5eb52e]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1d77)[0x5ed887]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x2c2)[0x5ee582]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x30b)[0x6bb3db]
Jan 9 21:29:25 localhost mysqld: /usr/sbin/mysqld(handle_one_connection+0x39)[0x6bb4b9]
Jan 9 21:29:26 localhost mysqld: 190109 21:29:26 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.57302S), skipping check
Jan 9 21:29:28 localhost kernel: [27738264.320016] do_general_protection: 15 callbacks suppressed
Jan 9 21:29:28 localhost kernel: [27738264.320021] traps: mysqld[1192] general protection ip:86297a sp:7fdc8ec6dca0 error:0 in mysqld[400000+d94000]

Co

Jan 9 21:28:57 localhost mysqld: 190109 21:28:57 [ERROR] mysqld got signal 11 ;
Jan 9 21:28:57 localhost mysqld: This could be because you hit a bug. It is also possible that this binary
Jan 9 21:28:57 localhost mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Jan 9 21:28:57 localhost mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: We will try our best to scrape up some info that will hopefully help
Jan 9 21:28:57 localhost mysqld: diagnose the problem, but since we have already crashed,
Jan 9 21:28:57 localhost mysqld: something is definitely wrong and this may fail.
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: Server version: 10.0.34-MariaDB-1~trusty-wsrep
Jan 9 21:28:57 localhost mysqld: key_buffer_size=209715200
Jan 9 21:28:57 localhost mysqld: read_buffer_size=131072
Jan 9 21:28:57 localhost mysqld: max_used_connections=324
Jan 9 21:28:57 localhost mysqld: max_threads=2002
Jan 9 21:28:57 localhost mysqld: thread_count=325
Jan 9 21:28:57 localhost mysqld: It is possible that mysqld could use up to
Jan 9 21:28:57 localhost mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 16902085 K bytes of memory
Jan 9 21:28:57 localhost mysqld: Hope that's ok; if not, decrease some variables in the equation.
Jan 9 21:28:57 localhost mysqld:
Jan 9 21:28:57 localhost mysqld: Thread pointer: 0x7fc6e88b3008
Jan 9 21:28:57 localhost mysqld: Attempting backtrace. You can use the following information to find out
Jan 9 21:28:57 localhost mysqld: where mysqld died. If you see no messages after this, something went
Jan 9 21:28:57 localhost mysqld: terribly wrong...
Jan 9 21:28:57 localhost mysqld: stack_bottom = 0x7fc6ffd79e10 thread_stack 0x40000
Jan 9 21:29:00 localhost mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xc095fe]
Jan 9 21:29:00 localhost mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x3b6)[0x767de6]
Jan 9 21:29:03 localhost mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fe453a47330]
Jan 9 21:29:04 localhost mysqld: 190109 21:29:04 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.60395S), skipping check
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT11get_explainEP11st_mem_root+0x6a)[0x86297a]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN24save_explain_data_internEP13Explain_querybbbPKc+0x7d3)[0x634873]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x6df)[0x6390bf]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x76)[0x63a446]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z19mysql_explain_unionP3THDP18st_select_lex_unitP13select_result+0x117)[0x63adf7]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld[0x518dc2]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x6d6a)[0x5e7bea]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld[0x5eb52e]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1d77)[0x5ed887]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x2c2)[0x5ee582]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x30b)[0x6bb3db]
Jan 9 21:29:05 localhost mysqld: /usr/sbin/mysqld(handle_one_connection+0x39)[0x6bb4b9]
Jan 9 21:29:07 localhost mysqld: 190109 21:29:07 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.65851S), skipping check
Jan 9 21:29:08 localhost mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7fe453a3f184]
Jan 9 21:29:09 localhost mysqld: 190109 21:29:09 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.6081S), skipping check
Jan 9 21:29:11 localhost mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe45316203d]
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: Trying to get some variables.
Jan 9 21:29:11 localhost mysqld: Some pointers may be invalid and cause the dump to abort.
Jan 9 21:29:11 localhost mysqld: Query (0x7fc6ea9d8020): is an invalid pointer
Jan 9 21:29:11 localhost mysqld: Connection ID (thread ID): 255594
Jan 9 21:29:11 localhost mysqld: Status: NOT_KILLED
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Jan 9 21:29:11 localhost mysqld: information that should help you find out what is causing the crash.
Jan 9 21:29:11 localhost mysqld:
Jan 9 21:29:11 localhost mysqld: We think the query pointer is invalid, but we will try to print it anyway.
Jan 9 21:29:11 localhost mysqld: Query: EXPLAIN SELECT `companies`.`id` FROM `companies` WHERE (companies.is_flagged_for_deletion <> 1) AND (companies.account_id = 161335) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615459 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 4765516 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 4873672 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 2328136))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615441 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328088))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615435 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328055 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 2328058))) AND (com
Jan 9 21:29:11 localhost mysqld: panies.next_entry_name IS NULL) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615408 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328019))) AND (EXISTS (SELECT 1 FROM `custom_field_values` WHERE `custom_field_values`.`custom_field_label_id` = 1615411 AND (company_id = companies.id) AND (`custom_field_values`.custom_field_label_dropdown_entry_id = 2328049 OR `custom_field_values`.custom_field_label_dropdown_entry_id = 3269848))) ORDER BY `companies`.`updated_at` DESC, `companies`.`id` DESC

We have been able to reproduce the crash on 10.0.34 and subsequently found the trigger to be related to using latin1 encoding on this particular dataset. Testing has shown that an upgrade to 10.0.37 fixes the issue. If you know of any related bugs that were fixed in either .35, .36, or .37 could you add the links? We would like to further understand root cause as we have not been able to identify the exact data that causes the crash, and it also seems to be related to the connection state (it doesn't happen at first, but then happens continuously until the connection is reset)

I am not able to reproduce it right away. I suspect it to be related to MDEV-15247, but it might be only a theory.
Would you be able to upload datadump of those tables to ftp.askmonty.org/private, then I will try to get the crash?

Unfortunately we cannot provide a data dump due to customer data privacy. It doesn't appear to be related to 15247 because that bug actually is about utf8 causing the crash. Ours is about utf8 preventing the crash when we are defaulted to latin1

Support for 10.0-galera has ended.