[MDEV-7401] Query optimizer crashes while optimizing a join Created: 2015-01-02  Updated: 2015-01-30  Resolved: 2015-01-23

Status: Closed
Project: MariaDB Server
Component/s: Optimizer, Storage Engine - InnoDB
Affects Version/s: 10.0.14
Fix Version/s: 10.0.16

Type: Bug Priority: Major
Reporter: Thomas Mischke Assignee: Unassigned
Resolution: Fixed Votes: 2
Labels: optimizer
Environment:

Linux 2.6.32-504.el6.x86_64


Issue Links:
Blocks
is blocked by MDEV-7453 10.0.16 merge Closed

 Description   

In our production database the following statement crashes the query optimizer:

explain select b.info1
from
(
select distinct info1
from state s1
where s1.status='FR' and s1.timestamp>'20141101' and s1.timestamp<'20141105'
) b
left outer join state s
  on s.timestamp>='20141201' and s.timestamp < '20141201230000' and s.info1 = b.info1
where s.`status`='FR'

This statement is a reduced one, that is the real statement is much larger, but this seems to be the part that crashes the optimizer. When we select slightly different date values, the query can be optimized with the following execution plan:

'id','select_type','table','type','possible_keys','key','key_len','ref','rows','Extra','1','PRIMARY','<derived2>','ALL',\N,\N,\N,\N,'5650','Using where','1','PRIMARY','s','ref','timestamp,iStatus,iStatusNext,info1','info1','63','b.info1','27','Using index condition; Using where','2','DERIVED','s1','range','timestamp,iStatus,iStatusNext','iStatus','312',\N,'5650','Using index condition; Using temporary',

If we replace the left outer join by inner join, the statement works. If we drop the subselect and replace it with something static, it works, too. If we remove the "where" clause and write this condition into the "on" clause (which of course changes the statement), it works, too.

In the log file of the server we find the following error report:

150102 12:51:22 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.0.14-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=13
max_threads=502
thread_count=11
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1233792 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0x7ffa59f0b008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7ffc93195d00 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb6b85b]
/usr/sbin/mysqld(handle_fatal_signal+0x398)[0x723f08]
/lib64/libpthread.so.0[0x36d1a0f710]
/usr/sbin/mysqld(_Z16best_access_pathP4JOINP13st_join_tableyjbdP11st_positionS4_+0x158b)[0x5ff87b]
/usr/sbin/mysqld[0x6012a8]
/usr/sbin/mysqld[0x601663]
/usr/sbin/mysqld[0x60198b]
/usr/sbin/mysqld(_Z11choose_planP4JOINy+0xf5)[0x601ef5]
/usr/sbin/mysqld[0x52a437]
/usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x6de)[0x61d3de]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xd8)[0x61fab8]
/usr/sbin/mysqld(_Z19mysql_explain_unionP3THDP18st_select_lex_unitP13select_result+0x117)[0x6205c7]
/usr/sbin/mysqld[0x5cad47]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4c97)[0x5d58d7]
/usr/sbin/mysqld[0x5d7402]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1b20)[0x5d95c0]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x453)[0x6946a3]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x694772]
/usr/sbin/mysqld[0xa6137d]
/lib64/libpthread.so.0[0x36d1a079d1]
/lib64/libc.so.6(clone+0x6d)[0x36d16e89dd]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7ffa41c22020): is an invalid pointer
Connection ID (thread ID): 25
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

Now I hope, that this is enough information for you to find out, what is happening here. If not, please feel free to ask for anything missing.

Best regards,

Thomas Mischke



 Comments   
Comment by sbester1 [ 2015-01-02 ]

What is output of: SHOW CREATE TABLE `state` ; ?

Comment by Elena Stepanova [ 2015-01-04 ]

It can be MDEV-6830 – or not, it's hard to say without knowing the structure of `state`. So, as sbester1 asked above, please paste the output of SHOW CREATE TABLE `state`.

Comment by Thomas Mischke [ 2015-01-05 ]

This is the output of

SHOW CREATE TABLE `state`

:

CREATE TABLE `state` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `jobnr` varchar(10) DEFAULT NULL,
  `ordernr` varchar(10) DEFAULT NULL,
  `techid` varchar(10) DEFAULT NULL,
  `Class` varchar(45) DEFAULT NULL,
  `timestamp` datetime DEFAULT NULL,
  `station` varchar(45) DEFAULT NULL,
  `app` varchar(45) DEFAULT NULL,
  `op` varchar(45) DEFAULT NULL,
  `info1` varchar(1024) DEFAULT NULL,
  `info2` varchar(1024) DEFAULT NULL,
  `info3` varchar(1024) DEFAULT NULL,
  `info4` varchar(1024) DEFAULT NULL,
  `info5` varchar(1024) DEFAULT NULL,
  `status` varchar(100) DEFAULT NULL,
  `side` varchar(2) DEFAULT NULL,
  `user` varchar(45) DEFAULT NULL,
  `origindate` datetime DEFAULT NULL,
  `nextstatus` varchar(100) DEFAULT NULL,
  `nexttimestamp` datetime DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `JobNumber` (`jobnr`,`timestamp`),
  KEY `timestamp` (`timestamp`),
  KEY `station` (`station`,`timestamp`),
  KEY `techid` (`techid`,`timestamp`),
  KEY `iStatus` (`status`,`timestamp`),
  KEY `iStatusNext` (`status`,`nextstatus`,`timestamp`),
  KEY `iNextStatus` (`nextstatus`,`status`,`nexttimestamp`),
  KEY `info1` (`info1`(20),`timestamp`),
  KEY `info2` (`info2`(20),`timestamp`),
  KEY `info3` (`info3`(20),`timestamp`),
  KEY `info4` (`info4`(20),`timestamp`),
  KEY `info5` (`info5`(20),`timestamp`),
  KEY `iOrderid` (`ordernr`,`timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=116453204 DEFAULT CHARSET=utf8

Since nearly never lines are deleted in this table, AUTO_INCREMENT shows, how many rows are in the table at the moment.

Comment by sbester1 [ 2015-01-05 ]

I could crash 10.0.14 and 10.0.15 within seconds

Version: '10.0.15-MariaDB'  socket: ''  port: 3306  mariadb.org binary distribution
150105 13:55:22 [ERROR] mysqld got exception 0xc0000005 ;
mysqld.exe!best_access_path()[sql_select.cc:6044]
mysqld.exe!best_extension_by_limited_search()[sql_select.cc:7705]
mysqld.exe!best_extension_by_limited_search()[sql_select.cc:7777]
mysqld.exe!greedy_search()[sql_select.cc:6944]
mysqld.exe!choose_plan()[sql_select.cc:6520]
mysqld.exe!make_join_statistics()[sql_select.cc:4038]
mysqld.exe!JOIN::optimize_inner()[sql_select.cc:1340]
mysqld.exe!JOIN::optimize()[sql_select.cc:1024]
mysqld.exe!mysql_select()[sql_select.cc:3294]
mysqld.exe!handle_select()[sql_select.cc:373]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5261]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2545]
mysqld.exe!mysql_parse()[sql_parse.cc:6412]
mysqld.exe!dispatch_command()[sql_parse.cc:1302]
mysqld.exe!do_command()[sql_parse.cc:1003]
mysqld.exe!threadpool_process_request()[threadpool_common.cc:233]
mysqld.exe!io_completion_callback()[threadpool_win.cc:568]

Comment by Elena Stepanova [ 2015-01-05 ]

To sbester1:

Hi Shane.
It's not difficult to crash 10.0.14/15 or the current tree in best_access_path, I can do it too, that's why MDEV-6830 exists (it should be fixed in 10.0.16 but the fix is not in the tree yet).
The question is whether your crash is the same as MDEV-6830 or a different one; and the same about the initially reported crash, which might well be the same or different from yours.
Can you paste the whole test case, or are you doing a stress test?
If you have the test case, I can check it against MDEV-6830 patch to see if the crash goes away.

thomas.mischke,

Thanks. I'll try to get your exact crash with your table structure and query. It might also help if you paste the output of SHOW TABLE STATUS LIKE 'state', SHOW INDEX IN state and SHOW VARIABLES.

Comment by Thomas Mischke [ 2015-01-05 ]

Here we go.

SHOW TABLE STATUS LIKE 'state':

'Name','Engine','Version','Row_format','Rows','Avg_row_length','Data_length','Max_data_length','Index_length','Data_free','Auto_increment','Create_time','Update_time','Check_time','Collation','Checksum','Create_options','Comment'
'state','InnoDB','10','Compact','97584790','222','21688745984','0','40238940160','2354053120','116466657','2014-11-05 15:47:10',\N,\N,'utf8_general_ci',\N,'','',

SHOW INDEX IN state

'Table','Non_unique','Key_name','Seq_in_index','Column_name','Collation','Cardinality','Sub_part','Packed','Null','Index_type','Comment','Index_comment'
'state','0','PRIMARY','1','id','A','97584986',\N,\N,'','BTREE','',''
'state','1','JobNumber','1','jobnr','A','1626416',\N,\N,'YES','BTREE','',''
'state','1','JobNumber','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','timestamp','1','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','station','1','station','A','30562',\N,\N,'YES','BTREE','',''
'state','1','station','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','techid','1','techid','A','8132082',\N,\N,'YES','BTREE','',''
'state','1','techid','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','iStatus','1','status','A','52948',\N,\N,'YES','BTREE','',''
'state','1','iStatus','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','iStatusNext','1','status','A','35995',\N,\N,'YES','BTREE','',''
'state','1','iStatusNext','2','nextstatus','A','91801',\N,\N,'YES','BTREE','',''
'state','1','iStatusNext','3','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','iNextStatus','1','nextstatus','A','36535',\N,\N,'YES','BTREE','',''
'state','1','iNextStatus','2','status','A','107236',\N,\N,'YES','BTREE','',''
'state','1','iNextStatus','3','nexttimestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','info1','1','info1','A','3614258','20',\N,'YES','BTREE','',''
'state','1','info1','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','info2','1','info2','A','2033020','20',\N,'YES','BTREE','',''
'state','1','info2','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','info3','1','info3','A','33696','20',\N,'YES','BTREE','',''
'state','1','info3','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','info4','1','info4','A','2','20',\N,'YES','BTREE','',''
'state','1','info4','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','info5','1','info5','A','1148','20',\N,'YES','BTREE','',''
'state','1','info5','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''
'state','1','iOrderid','1','ordernr','A','1394071',\N,\N,'YES','BTREE','',''
'state','1','iOrderid','2','timestamp','A','97584986',\N,\N,'YES','BTREE','',''

SHOW VARIABLES

'Variable_name','Value'
'aria_block_size','8192'
'aria_checkpoint_interval','30'
'aria_checkpoint_log_activity','1048576'
'aria_force_start_after_recovery_failures','0'
'aria_group_commit','none'
'aria_group_commit_interval','0'
'aria_log_file_size','1073741824'
'aria_log_purge_type','immediate'
'aria_max_sort_file_size','9223372036853727232'
'aria_page_checksum','ON'
'aria_pagecache_age_threshold','300'
'aria_pagecache_buffer_size','134217728'
'aria_pagecache_division_limit','100'
'aria_pagecache_file_hash_size','512'
'aria_recover','NORMAL'
'aria_repair_threads','1'
'aria_sort_buffer_size','268434432'
'aria_stats_method','nulls_unequal'
'aria_sync_log_dir','NEWFILE'
'aria_used_for_temp_tables','ON'
'auto_increment_increment','1'
'auto_increment_offset','1'
'autocommit','ON'
'automatic_sp_privileges','ON'
'back_log','150'
'basedir','/usr'
'big_tables','ON'
'binlog_annotate_row_events','OFF'
'binlog_cache_size','32768'
'binlog_checksum','NONE'
'binlog_commit_wait_count','0'
'binlog_commit_wait_usec','100000'
'binlog_direct_non_transactional_updates','OFF'
'binlog_format','MIXED'
'binlog_optimize_thread_scheduling','ON'
'binlog_stmt_cache_size','32768'
'bulk_insert_buffer_size','8388608'
'character_set_client','utf8'
'character_set_connection','utf8'
'character_set_database','utf8'
'character_set_filesystem','binary'
'character_set_results','utf8'
'character_set_server','utf8'
'character_set_system','utf8'
'character_sets_dir','/usr/share/mysql/charsets/'
'collation_connection','utf8_general_ci'
'collation_database','utf8_general_ci'
'collation_server','utf8_unicode_ci'
'completion_type','NO_CHAIN'
'concurrent_insert','AUTO'
'connect_timeout','10'
'datadir','/var/lib/mysql/'
'date_format','%Y-%m-%d'
'datetime_format','%Y-%m-%d %H:%i:%s'
'deadlock_search_depth_long','15'
'deadlock_search_depth_short','4'
'deadlock_timeout_long','50000000'
'deadlock_timeout_short','10000'
'debug_no_thread_alarm','OFF'
'default_master_connection',''
'default_regex_flags',''
'default_storage_engine','InnoDB'
'default_week_format','0'
'delay_key_write','ON'
'delayed_insert_limit','100'
'delayed_insert_timeout','300'
'delayed_queue_size','1000'
'div_precision_increment','4'
'error_count','0'
'event_scheduler','OFF'
'expensive_subquery_limit','100'
'expire_logs_days','5'
'external_user',''
'extra_max_connections','1'
'extra_port','0'
'flush','OFF'
'flush_time','0'
'foreign_key_checks','ON'
'ft_boolean_syntax','+ -><()~*:""&|'
'ft_max_word_len','84'
'ft_min_word_len','4'
'ft_query_expansion_limit','20'
'ft_stopword_file','(built-in)'
'general_log','OFF'
'general_log_file','OptoSQL3.log'
'group_concat_max_len','1024'
'gtid_binlog_pos','0-3-86688840'
'gtid_binlog_state','0-4-81723160,0-3-86688840'
'gtid_current_pos','0-3-86688840'
'gtid_domain_id','0'
'gtid_ignore_duplicates','OFF'
'gtid_seq_no','0'
'gtid_slave_pos','0-3-86688840'
'gtid_strict_mode','OFF'
'have_compress','YES'
'have_crypt','YES'
'have_dynamic_loading','YES'
'have_geometry','YES'
'have_openssl','YES'
'have_profiling','YES'
'have_query_cache','YES'
'have_rtree_keys','YES'
'have_ssl','DISABLED'
'have_symlink','DISABLED'
'histogram_size','0'
'histogram_type','SINGLE_PREC_HB'
'host_cache_size','128'
'hostname','OptoSQL3'
'identity','0'
'ignore_builtin_innodb','OFF'
'ignore_db_dirs',''
'in_transaction','0'
'init_connect',''
'init_file',''
'init_slave',''
'innodb_adaptive_flushing','ON'
'innodb_adaptive_flushing_lwm','10'
'innodb_adaptive_hash_index','ON'
'innodb_adaptive_hash_index_partitions','1'
'innodb_adaptive_max_sleep_delay','150000'
'innodb_additional_mem_pool_size','8388608'
'innodb_api_bk_commit_interval','5'
'innodb_api_disable_rowlock','OFF'
'innodb_api_enable_binlog','OFF'
'innodb_api_enable_mdl','OFF'
'innodb_api_trx_level','0'
'innodb_autoextend_increment','64'
'innodb_autoinc_lock_mode','1'
'innodb_buffer_pool_dump_at_shutdown','OFF'
'innodb_buffer_pool_dump_now','OFF'
'innodb_buffer_pool_filename','ib_buffer_pool'
'innodb_buffer_pool_instances','8'
'innodb_buffer_pool_load_abort','OFF'
'innodb_buffer_pool_load_at_startup','OFF'
'innodb_buffer_pool_load_now','OFF'
'innodb_buffer_pool_populate','OFF'
'innodb_buffer_pool_size','8589934592'
'innodb_change_buffer_max_size','25'
'innodb_change_buffering','all'
'innodb_checksum_algorithm','innodb'
'innodb_checksums','ON'
'innodb_cleaner_lsn_age_factor','high_checkpoint'
'innodb_cmp_per_index_enabled','OFF'
'innodb_commit_concurrency','0'
'innodb_compression_failure_threshold_pct','5'
'innodb_compression_level','6'
'innodb_compression_pad_pct_max','50'
'innodb_concurrency_tickets','5000'
'innodb_corrupt_table_action','assert'
'innodb_data_file_path','ibdata1:12M:autoextend'
'innodb_data_home_dir',''
'innodb_disable_sort_file_cache','OFF'
'innodb_doublewrite','ON'
'innodb_empty_free_list_algorithm','backoff'
'innodb_fake_changes','OFF'
'innodb_fast_shutdown','1'
'innodb_file_format','Antelope'
'innodb_file_format_check','ON'
'innodb_file_format_max','Antelope'
'innodb_file_per_table','ON'
'innodb_flush_log_at_timeout','1'
'innodb_flush_log_at_trx_commit','1'
'innodb_flush_method',''
'innodb_flush_neighbors','1'
'innodb_flushing_avg_loops','30'
'innodb_force_load_corrupted','OFF'
'innodb_force_recovery','0'
'innodb_foreground_preflush','exponential_backoff'
'innodb_ft_aux_table',''
'innodb_ft_cache_size','8000000'
'innodb_ft_enable_diag_print','OFF'
'innodb_ft_enable_stopword','ON'
'innodb_ft_max_token_size','84'
'innodb_ft_min_token_size','3'
'innodb_ft_num_word_optimize','2000'
'innodb_ft_result_cache_limit','2000000000'
'innodb_ft_server_stopword_table',''
'innodb_ft_sort_pll_degree','2'
'innodb_ft_total_cache_size','640000000'
'innodb_ft_user_stopword_table',''
'innodb_io_capacity','200'
'innodb_io_capacity_max','2000'
'innodb_kill_idle_transaction','0'
'innodb_large_prefix','OFF'
'innodb_lock_wait_timeout','50'
'innodb_locking_fake_changes','ON'
'innodb_locks_unsafe_for_binlog','OFF'
'innodb_log_arch_dir','./'
'innodb_log_arch_expire_sec','0'
'innodb_log_archive','OFF'
'innodb_log_block_size','512'
'innodb_log_buffer_size','8388608'
'innodb_log_checksum_algorithm','innodb'
'innodb_log_compressed_pages','ON'
'innodb_log_file_size','104857600'
'innodb_log_files_in_group','2'
'innodb_log_group_home_dir','./'
'innodb_lru_scan_depth','1024'
'innodb_max_bitmap_file_size','104857600'
'innodb_max_changed_pages','1000000'
'innodb_max_dirty_pages_pct','75'
'innodb_max_dirty_pages_pct_lwm','0'
'innodb_max_purge_lag','0'
'innodb_max_purge_lag_delay','0'
'innodb_mirrored_log_groups','1'
'innodb_monitor_disable',''
'innodb_monitor_enable',''
'innodb_monitor_reset',''
'innodb_monitor_reset_all',''
'innodb_old_blocks_pct','37'
'innodb_old_blocks_time','1000'
'innodb_online_alter_log_max_size','134217728'
'innodb_open_files','512'
'innodb_optimize_fulltext_only','OFF'
'innodb_page_size','16384'
'innodb_print_all_deadlocks','OFF'
'innodb_purge_batch_size','300'
'innodb_purge_threads','1'
'innodb_random_read_ahead','OFF'
'innodb_read_ahead_threshold','56'
'innodb_read_io_threads','4'
'innodb_read_only','OFF'
'innodb_replication_delay','0'
'innodb_rollback_on_timeout','OFF'
'innodb_rollback_segments','128'
'innodb_sched_priority_cleaner','19'
'innodb_show_locks_held','10'
'innodb_show_verbose_locks','0'
'innodb_simulate_comp_failures','0'
'innodb_sort_buffer_size','1048576'
'innodb_spin_wait_delay','6'
'innodb_stats_auto_recalc','ON'
'innodb_stats_method','nulls_equal'
'innodb_stats_on_metadata','OFF'
'innodb_stats_persistent','ON'
'innodb_stats_persistent_sample_pages','20'
'innodb_stats_sample_pages','8'
'innodb_stats_transient_sample_pages','8'
'innodb_status_output','OFF'
'innodb_status_output_locks','OFF'
'innodb_strict_mode','OFF'
'innodb_support_xa','ON'
'innodb_sync_array_size','1'
'innodb_sync_spin_loops','30'
'innodb_table_locks','ON'
'innodb_thread_concurrency','0'
'innodb_thread_sleep_delay','10000'
'innodb_track_changed_pages','OFF'
'innodb_undo_directory','.'
'innodb_undo_logs','128'
'innodb_undo_tablespaces','0'
'innodb_use_atomic_writes','OFF'
'innodb_use_fallocate','OFF'
'innodb_use_global_flush_log_at_trx_commit','ON'
'innodb_use_native_aio','ON'
'innodb_use_stacktrace','OFF'
'innodb_use_sys_malloc','ON'
'innodb_version','5.6.20-68.0'
'innodb_write_io_threads','4'
'insert_id','0'
'interactive_timeout','28800'
'join_buffer_size','131072'
'join_buffer_space_limit','2097152'
'join_cache_level','2'
'keep_files_on_create','OFF'
'key_buffer_size','134217728'
'key_cache_age_threshold','300'
'key_cache_block_size','1024'
'key_cache_division_limit','100'
'key_cache_file_hash_size','512'
'key_cache_segments','0'
'large_files_support','ON'
'large_page_size','0'
'large_pages','OFF'
'last_gtid',''
'last_insert_id','0'
'lc_messages','en_US'
'lc_messages_dir',''
'lc_time_names','en_US'
'license','GPL'
'local_infile','ON'
'lock_wait_timeout','31536000'
'locked_in_memory','OFF'
'log_bin','ON'
'log_bin_trust_function_creators','OFF'
'log_error','/var/log/mysqld.log'
'log_output','FILE'
'log_queries_not_using_indexes','OFF'
'log_slave_updates','ON'
'log_slow_filter','admin,filesort,filesort_on_disk,full_join,full_scan,query_cache,query_cache_miss,tmp_table,tmp_table_on_disk'
'log_slow_rate_limit','1'
'log_slow_verbosity',''
'log_warnings','1'
'long_query_time','10.000000'
'low_priority_updates','OFF'
'lower_case_file_system','OFF'
'lower_case_table_names','1'
'master_verify_checksum','OFF'
'max_allowed_packet','33554432'
'max_binlog_cache_size','18446744073709547520'
'max_binlog_size','1073741824'
'max_binlog_stmt_cache_size','18446744073709547520'
'max_connect_errors','100'
'max_connections','500'
'max_delayed_threads','20'
'max_error_count','64'
'max_heap_table_size','67108864'
'max_insert_delayed_threads','20'
'max_join_size','18446744073709551615'
'max_length_for_sort_data','1024'
'max_long_data_size','33554432'
'max_prepared_stmt_count','16382'
'max_relay_log_size','1073741824'
'max_seeks_for_key','4294967295'
'max_sort_length','1024'
'max_sp_recursion_depth','0'
'max_tmp_tables','32'
'max_user_connections','0'
'max_write_lock_count','4294967295'
'metadata_locks_cache_size','1024'
'metadata_locks_hash_instances','8'
'min_examined_row_limit','0'
'mrr_buffer_size','262144'
'multi_range_count','256'
'myisam_block_size','1024'
'myisam_data_pointer_size','6'
'myisam_max_sort_file_size','9223372036853727232'
'myisam_mmap_size','18446744073709551615'
'myisam_recover_options','DEFAULT'
'myisam_repair_threads','1'
'myisam_sort_buffer_size','134216704'
'myisam_stats_method','nulls_unequal'
'myisam_use_mmap','OFF'
'net_buffer_length','16384'
'net_read_timeout','30'
'net_retry_count','10'
'net_write_timeout','60'
'old','OFF'
'old_alter_table','OFF'
'old_mode',''
'old_passwords','OFF'
'open_files_limit','2505'
'optimizer_prune_level','1'
'optimizer_search_depth','62'
'optimizer_selectivity_sampling_limit','100'
'optimizer_switch','index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on'
'optimizer_use_condition_selectivity','1'
'performance_schema','ON'
'performance_schema_accounts_size','100'
'performance_schema_digests_size','10000'
'performance_schema_events_stages_history_long_size','10000'
'performance_schema_events_stages_history_size','10'
'performance_schema_events_statements_history_long_size','10000'
'performance_schema_events_statements_history_size','10'
'performance_schema_events_waits_history_long_size','10000'
'performance_schema_events_waits_history_size','10'
'performance_schema_hosts_size','100'
'performance_schema_max_cond_classes','80'
'performance_schema_max_cond_instances','2900'
'performance_schema_max_file_classes','50'
'performance_schema_max_file_handles','32768'
'performance_schema_max_file_instances','2154'
'performance_schema_max_mutex_classes','200'
'performance_schema_max_mutex_instances','8000'
'performance_schema_max_rwlock_classes','40'
'performance_schema_max_rwlock_instances','3800'
'performance_schema_max_socket_classes','10'
'performance_schema_max_socket_instances','1020'
'performance_schema_max_stage_classes','150'
'performance_schema_max_statement_classes','180'
'performance_schema_max_table_handles','800'
'performance_schema_max_table_instances','12500'
'performance_schema_max_thread_classes','50'
'performance_schema_max_thread_instances','1100'
'performance_schema_session_connect_attrs_size','512'
'performance_schema_setup_actors_size','100'
'performance_schema_setup_objects_size','100'
'performance_schema_users_size','100'
'pid_file','/var/lib/mysql/OptoSQL3.pid'
'plugin_dir','/usr/lib64/mysql/plugin/'
'plugin_maturity','unknown'
'port','3306'
'preload_buffer_size','32768'
'profiling','OFF'
'profiling_history_size','15'
'progress_report_time','5'
'protocol_version','10'
'proxy_user',''
'pseudo_slave_mode','OFF'
'pseudo_thread_id','10793'
'query_alloc_block_size','8192'
'query_cache_limit','1048576'
'query_cache_min_res_unit','4096'
'query_cache_size','67108864'
'query_cache_strip_comments','OFF'
'query_cache_type','ON'
'query_cache_wlock_invalidate','OFF'
'query_prealloc_size','8192'
'rand_seed1','0'
'rand_seed2','0'
'range_alloc_block_size','4096'
'read_buffer_size','131072'
'read_only','ON'
'read_rnd_buffer_size','262144'
'relay_log',''
'relay_log_index',''
'relay_log_info_file','relay-log.info'
'relay_log_purge','ON'
'relay_log_recovery','OFF'
'relay_log_space_limit','0'
'replicate_annotate_row_events','OFF'
'replicate_do_db',''
'replicate_do_table',''
'replicate_events_marked_for_skip','replicate'
'replicate_ignore_db',''
'replicate_ignore_table',''
'replicate_wild_do_table',''
'replicate_wild_ignore_table',''
'report_host',''
'report_password',''
'report_port','3306'
'report_user',''
'rowid_merge_buff_size','8388608'
'rpl_recovery_rank','0'
'secure_auth','OFF'
'secure_file_priv',''
'server_id','4'
'skip_external_locking','ON'
'skip_name_resolve','ON'
'skip_networking','OFF'
'skip_replication','OFF'
'skip_show_database','OFF'
'slave_compressed_protocol','OFF'
'slave_ddl_exec_mode','IDEMPOTENT'
'slave_domain_parallel_threads','0'
'slave_exec_mode','STRICT'
'slave_load_tmpdir','/tmp'
'slave_max_allowed_packet','1073741824'
'slave_net_timeout','3600'
'slave_parallel_max_queued','131072'
'slave_parallel_threads','0'
'slave_skip_errors','OFF'
'slave_sql_verify_checksum','ON'
'slave_transaction_retries','10'
'slave_type_conversions',''
'slow_launch_time','2'
'slow_query_log','ON'
'slow_query_log_file','OptoSQL3-slow.log'
'socket','/var/lib/mysql/mysql.sock'
'sort_buffer_size','2097152'
'sql_auto_is_null','OFF'
'sql_big_selects','ON'
'sql_buffer_result','OFF'
'sql_log_bin','ON'
'sql_log_off','OFF'
'sql_mode',''
'sql_notes','ON'
'sql_quote_show_create','ON'
'sql_safe_updates','OFF'
'sql_select_limit','18446744073709551615'
'sql_slave_skip_counter','0'
'sql_warnings','OFF'
'ssl_ca',''
'ssl_capath',''
'ssl_cert',''
'ssl_cipher',''
'ssl_crl',''
'ssl_crlpath',''
'ssl_key',''
'storage_engine','InnoDB'
'stored_program_cache','256'
'sync_binlog','0'
'sync_frm','ON'
'sync_master_info','0'
'sync_relay_log','0'
'sync_relay_log_info','0'
'system_time_zone','CET'
'table_definition_cache','400'
'table_open_cache','512'
'thread_cache_size','4'
'thread_concurrency','10'
'thread_handling','one-thread-per-connection'
'thread_pool_idle_timeout','60'
'thread_pool_max_threads','500'
'thread_pool_oversubscribe','3'
'thread_pool_size','8'
'thread_pool_stall_limit','500'
'thread_stack','294912'
'time_format','%H:%i:%s'
'time_zone','SYSTEM'
'timed_mutexes','OFF'
'timestamp','1420462569.240073'
'tmp_table_size','134217728'
'tmpdir','/tmp'
'transaction_alloc_block_size','8192'
'transaction_prealloc_size','4096'
'tx_isolation','REPEATABLE-READ'
'tx_read_only','OFF'
'unique_checks','ON'
'updatable_views_with_limit','YES'
'use_stat_tables','NEVER'
'userstat','OFF'
'version','10.0.14-MariaDB-log'
'version_comment','MariaDB Server'
'version_compile_machine','x86_64'
'version_compile_os','Linux'
'version_malloc_library','bundled jemalloc'
'wait_timeout','28800'
'warning_count','0'

Comment by sbester1 [ 2015-01-05 ]

I repeated this with a six line testcase with the reporters table + query. But what crashes win64 build doesn't crash win32 nor linux build. The testcase is sporadic, at best, and build dependant. A 10.0.15 debug build on win32 crashed in line st_key::actual_rec_per_key()[table.cc:7176].

Comment by Elena Stepanova [ 2015-01-09 ]

It once again points at MDEV-6830 – both the sporadic part (it was crashing on some builds, not crashing on others, depended on the sequence of queries, etc.); and the st_key::actual_rec_per_key part – I had a stack trace there too (it's in MDEV-6830 comments).

Unless we get more data from thomas.mischke (SHOW TABLE STATUS LIKE 'state', SHOW INDEX IN state and SHOW VARIABLES) and it helps to reproduce the exact crash he had, we'll have to assume it's the same problem as MDEV-6830, too.

Comment by Thomas Mischke [ 2015-01-12 ]

I posted the output of those three statements about one week ago. Do you need anything else, or the output in a different format?

Is 10.0.16 stable enough so that I can install it on our machine to test, if the error still appears (I still see that 10.0.15 is the latest stable version for download)?

Comment by Elena Stepanova [ 2015-01-12 ]

Sorry, I missed that comment. I will look at the data shortly.
Regarding 10.0.16, it's too early to try anyway, the bugfix we are talking about was pushed into 5.5, but it hasn't yet been merged into 10.0 tree.

Comment by Elena Stepanova [ 2015-01-12 ]

Thanks again for the query and the data, I am now able to reproduce the crash.
There is no point trying the latest builds, the bug is still present in the current tree. I applied MDEV-6830 bugfix to the tree manually, the crash went away; but I'll keep this report open until the proper merge 5.5->10.0 and will re-check it again then.

As a workaround, you can try either
SET optimizer_switch = 'derived_with_keys=off';
or
SET big_tables = 0;
Hopefully at least one will help. In my test each of them helps, but it can depend on the data.

Test case reduced farther (although it makes sense to also try the initial one after the bug is fixed, just to make sure. For the initial one, take the table from the comment and the query from the description, no need to populate the table; make sure that InnoDB is present and big_tables = ON).

Test case

--source include/have_innodb.inc
 
CREATE TABLE `state` (
  `id` int(11) NOT NULL,
  `info1` varchar(1024) DEFAULT NULL,
  `status` varchar(100) DEFAULT NULL,
  PRIMARY KEY (id),
  KEY (status)
) ENGINE=InnoDB;
 
INSERT INTO state VALUES (1,'foo','FR'),(2,'bar','NN');
 
SET big_tables=1;
 
select * from
(
select distinct info1
from state s1
) b
left join state s
  on s.info1 = b.info1
where s.`status`='FR';

Stack trace from 10.0 revno 4550

#3  <signal handler called>
#4  0x0000000000715f9b in Index_statistics::get_avg_frequency (this=0xa5a5a5a5a5a5a5a5, i=0) at 10.0/sql/sql_statistics.h:418
#5  0x0000000000760688 in st_key::actual_rec_per_key (this=0x7f1b7442a858, i=0) at 10.0/sql/table.cc:7176
#6  0x00000000006bf5c3 in best_access_path (join=0x7f1b7445b088, s=0x7f1b74629b60, remaining_tables=1, idx=1, disable_jbuf=false, record_count=1, pos=0x7f1b7462a4e0, loose_scan_pos=0x7f1b8957db30) at 10.0/sql/sql_select.cc:6001
#7  0x00000000006c2f14 in best_extension_by_limited_search (join=0x7f1b7445b088, remaining_tables=1, idx=1, record_count=1, read_time=2.2000000000000002, search_depth=61, prune_level=1, use_cond_selectivity=1) at 10.0/sql/sql_select.cc:7701
#8  0x00000000006c337a in best_extension_by_limited_search (join=0x7f1b7445b088, remaining_tables=3, idx=0, record_count=1, read_time=0, search_depth=62, prune_level=1, use_cond_selectivity=1) at 10.0/sql/sql_select.cc:7770
#9  0x00000000006c1787 in greedy_search (join=0x7f1b7445b088, remaining_tables=3, search_depth=62, prune_level=1, use_cond_selectivity=1) at 10.0/sql/sql_select.cc:6942
#10 0x00000000006c0d6f in choose_plan (join=0x7f1b7445b088, join_tables=3) at 10.0/sql/sql_select.cc:6519
#11 0x00000000006ba73d in make_join_statistics (join=0x7f1b7445b088, tables_list=..., conds=0x7f1b74629420, keyuse_array=0x7f1b7445b390) at 10.0/sql/sql_select.cc:4038
#12 0x00000000006b13a3 in JOIN::optimize_inner (this=0x7f1b7445b088) at 10.0/sql/sql_select.cc:1339
#13 0x00000000006b033a in JOIN::optimize (this=0x7f1b7445b088) at 10.0/sql/sql_select.cc:1024
#14 0x00000000006b7ebf in mysql_select (thd=0x7f1b80dee070, rref_pointer_array=0x7f1b80df26e0, tables=0x7f1b744196c8, wild_num=1, fields=..., conds=0x7f1b7441aa48, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7f1b74743020, unit=0x7f1b80df1d80, select_lex=0x7f1b80df2468) at 10.0/sql/sql_select.cc:3294
#15 0x00000000006ae549 in handle_select (thd=0x7f1b80dee070, lex=0x7f1b80df1cb8, result=0x7f1b74743020, setup_tables_done_option=0) at 10.0/sql/sql_select.cc:373
#16 0x0000000000683244 in execute_sqlcom_select (thd=0x7f1b80dee070, all_tables=0x7f1b744196c8) at 10.0/sql/sql_parse.cc:5261
#17 0x000000000067b542 in mysql_execute_command (thd=0x7f1b80dee070) at 10.0/sql/sql_parse.cc:2545
#18 0x00000000006859bd in mysql_parse (thd=0x7f1b80dee070, rawbuf=0x7f1b74741088 "select * from\n(\nselect distinct info1\nfrom state s1\n) b\nleft join state s\non s.info1 = b.info1\nwhere s.`status`='FR'", length=116, parser_state=0x7f1b8957f610) at 10.0/sql/sql_parse.cc:6407
#19 0x00000000006787b2 in dispatch_command (command=COM_QUERY, thd=0x7f1b80dee070, packet=0x7f1b7bab9071 "select * from\n(\nselect distinct info1\nfrom state s1\n) b\nleft join state s\non s.info1 = b.info1\nwhere s.`status`='FR'", packet_length=116) at 10.0/sql/sql_parse.cc:1299
#20 0x0000000000677b57 in do_command (thd=0x7f1b80dee070) at 10.0/sql/sql_parse.cc:996
#21 0x000000000079489b in do_handle_one_connection (thd_arg=0x7f1b80dee070) at 10.0/sql/sql_connect.cc:1375
#22 0x00000000007945ee in handle_one_connection (arg=0x7f1b80dee070) at 10.0/sql/sql_connect.cc:1289
#23 0x0000000000ccdc56 in pfs_spawn_thread (arg=0x7f1b7bab10f0) at 10.0/storage/perfschema/pfs.cc:1860
#24 0x00007f1b891b9b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#25 0x00007f1b8747020d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

Comment by Thomas Mischke [ 2015-01-12 ]

Both workarounds prevent out initial statement from crashing the server.

Thanks for the workaround, and looking forward to the next release without this bug!

Comment by Elena Stepanova [ 2015-01-23 ]

Checked both testcases on 10.0 revno 4587 (after the merge), no crashes anymore. Closing as fixed.

Comment by Thomas Mischke [ 2015-01-30 ]

Today we updated to version 10.0.16. The above mentioned statements are now working without workaround. Thanks!

Generated at Thu Feb 08 07:19:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.