[MDEV-8779] mysqld got signal 11 in sql/opt_range_mrr.cc:100(step_down_to) Created: 2015-09-09  Updated: 2015-10-17  Resolved: 2015-09-21

Status: Closed
Project: MariaDB Server
Component/s: Optimizer
Affects Version/s: 10.0, 10.1
Fix Version/s: 10.0.22, 10.1.8

Type: Bug Priority: Critical
Reporter: Dan Vande More Assignee: Sergei Petrunia
Resolution: Fixed Votes: 0
Labels: eits
Environment:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"

Issue Links:
Relates
relates to MDEV-8634 abort in calculate_cond_selectivity_f... Closed
relates to MDEV-8903 Buildbot valgrind failure: Invalid re... Closed
Sprint: 10.1.8-2

 Description   

Config

 
histogram_size=255
 
optimizer_use_condition_selectivity=5
 
use_stat_tables='preferably'
 
optimizer_selectivity_sampling_limit=97
 
query_cache_type=1
 
optimizer_switch='index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=on,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=off,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on'
 
mrr_buffer_size=32M
 
join_buffer_space_limit=32M
 
join_cache_level=5
 
tmp_table_size=448M
 
max_heap_table_size=448M
 
sort_buffer_size=32M
 
join_buffer_size=32M

Using either tokudb or inndob, I have at query that can reliably trigger:

stack_bottom = 0x7f91b7f27e10 thread_stack 0x48000
 
mysys/stacktrace.c:247(my_print_stacktrace)[0x7f91b7997efb]
 
sql/signal_handler.cc:153(handle_fatal_signal)[0x7f91b752ff05]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f91b60c1340]
 
sql/opt_range_mrr.cc:100(step_down_to)[0x7f91b731fb49]
 
sql/opt_range_mrr.cc:196(sel_arg_range_seq_next(void*, st_key_multi_range*))[0x7f91b732303b]
 
sql/opt_range.cc:3426(records_in_column_ranges)[0x7f91b732405d]

I have rebuilt the table several times, including rounds of optimize & analyze table, anayzle table persistent for all, etc.

CREATE TABLE `f` (
 
  `i` int(10) unsigned NOT NULL AUTO_INCREMENT,
 
  `u` varchar(2048) NOT NULL,
 
  `n` varchar(2048) NOT NULL,
 
  `e` tinyint(1) unsigned NOT NULL DEFAULT '1',
 
  `d` tinyint(1) unsigned NOT NULL,
 
  `c` datetime DEFAULT NULL,
 
  `p` int(10) unsigned NOT NULL,
 
  `g` tinyint(1) unsigned NOT NULL DEFAULT '0',
 
  `ur` varchar(2048) NOT NULL,
 
  `r` bigint(20) unsigned DEFAULT NULL,
 
  `cl` tinyint(1) unsigned NOT NULL DEFAULT '0',
 
  PRIMARY KEY (`i`),
 
  UNIQUE KEY (`i`,`d`,`p`),
 
  KEY (`p`),
 
  KEY (`d`,`p`),
 
  KEY (`e`,`d`,`p`),
 
  KEY (`r`)
 
) ENGINE=tokudb DEFAULT CHARSET=utf8 /* `compression`=tokudb_quicklz */;
 
 
 
MariaDB [test]> SELECT   f.i,   f.u,   f.n,   f.e,   f.d,   f.c,   f.p,   f.g,   f.ur,   f.r,   f.cl FROM   f WHERE   (f.d = 0 AND     f.p = '1' AND     f.i != '-1' AND     f.n = 'some text' );
 
ERROR 2013 (HY000): Lost connection to MySQL server during query



 Comments   
Comment by Elena Stepanova [ 2015-09-09 ]

fimbulvetr
Thanks for the report and the test case, and also for providing your mysqld options right away, it helped to save time.

The crash is reproducible on 10.0 and 10.1 with InnoDB or TokuDB; not reproducible with MyISAM.

MTR test case

 
--source include/have_innodb.inc
 
 
 
set optimizer_use_condition_selectivity=5;
 
CREATE TABLE `f` (
 
`i` int(10) unsigned NOT NULL AUTO_INCREMENT,
 
`n` varchar(2048) NOT NULL,
 
`d` tinyint(1) unsigned NOT NULL,
 
`p` int(10) unsigned NOT NULL,
 
PRIMARY KEY (`i`)
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
 
SELECT * FROM f WHERE f.d = 0 AND f.p = '1' AND f.i != '-1' AND f.n = 'some text';

I am getting a broken (or incomplete) stack trace both on 10.0 and 10.1, on two different machines, although it looks somewhat different. As I understand, it was broken for fimbulvetr as well.

Examples:

Wheezy, 10.0 commit 0ce0b88080fd39f6841206b64d8723af9779e849

 
Thread 1 (Thread 0x7fe9a6c3e700 (LWP 28276)):
 
#0  __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63
 
#1  0x0000000000e594ab in my_write_core (sig=11) at 10.0/mysys/stacktrace.c:457
 
#2  0x000000000086d33d in handle_fatal_signal (sig=11) at 10.0/sql/signal_handler.cc:262
 
#3  <signal handler called>
 
#4  0x0000000000981e1b in calculate_cond_selectivity_for_table (thd=0x7fe99941a070, table=0x7fe991c9e470, cond=0x7fe991f44f50) at 10.0/sql/opt_range.cc:3658
 
#5  0x0000000000000000 in ?? ()

Wheezy, 10.1 commit 4cb6edba780085cbb82ef5fce2298f7df5fa0169

 
Thread 1 (Thread 0x7ff247469700 (LWP 28362)):
 
#0  __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63
 
#1  0x00007ff24825d026 in my_write_core (sig=11) at 10.1/mysys/stacktrace.c:456
 
#2  0x00007ff247c02bbe in handle_fatal_signal (sig=11) at 10.1/sql/signal_handler.cc:273
 
#3  <signal handler called>
 
#4  0x00007ff247d1ea9a in step_down_to (arg=0x7ff2474640a0, key_tree=0x7ff22f10e8f0) at 10.1/sql/opt_range_mrr.cc:100
 
#5  0x00007ff247d1ef29 in sel_arg_range_seq_next (rseq=0x7ff2474640a0, range=0x7ff247464050) at 10.1/sql/opt_range_mrr.cc:236
 
#6  0x00007ff247d244f1 in records_in_column_ranges (param=0x7ff247464750, idx=2, tree=0x7ff22f10e8f0) at 10.1/sql/opt_range.cc:2818
 
#7  0x00007ff247d24e17 in calculate_cond_selectivity_for_table (thd=0x7ff23b9dd070, table=0x7ff22f0ad870, cond=0x7ff22f197b68) at 10.1/sql/opt_range.cc:3061
 
#8  0x0000000000000000 in ?? ()

Trusty 10.0 commit 0ce0b88080fd39f6841206b64d8723af9779e849

 
Thread 1 (Thread 0x7fe1b332f700 (LWP 14135)):
 
#0  0x00007fe1b24dcf8c in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
 
#1  0x0000000000e6c6ec in my_write_core (sig=11) at 10.0/mysys/stacktrace.c:457
 
#2  0x000000000085bd6b in handle_fatal_signal (sig=11) at 10.0/sql/signal_handler.cc:262
 
#3  <signal handler called>
 
#4  0x0000000000970092 in step_down_to (arg=0x7fe1b332a530, key_tree=0x7fe19e7fe8f0) at 10.0/sql/opt_range_mrr.cc:100
 
#5  0x0000000000970506 in sel_arg_range_seq_next (rseq=0x7fe1b332a530, range=0x7fe1b332a4e0) at 10.0/sql/opt_range_mrr.cc:236
 
#6  0x0000000000975958 in records_in_column_ranges (param=0x7fe1b332abd0, idx=2, tree=0x7fe19e7fe8f0) at 10.0/sql/opt_range.cc:3415
 
#7  0x000000000097624a in calculate_cond_selectivity_for_table (thd=0x7fe1ab1ac070, table=0x7fe19e49e470, cond=0x7fe19e5a8f50) at 10.0/sql/opt_range.cc:3656
 
#8  0x0000000000000000 in ?? ()

Apart from fixing the crash, it is also very important to investigate what's going on with the stack trace. As discussed, assigning to svoj for a preliminary analysis.

Comment by Sergey Vojtovich [ 2015-09-11 ]

In create_key_parts_for_pseudo_indexes() we remember field->key_length() for every used field. For "`n` varchar(2048) NOT NULL" we get 6144 (2048 * mbmaxlen of UTF-8, which is 3).

Later we do step_down_to()/SEL_ARG::store_max(), which does memcpy(*max_key,max_value,length). IIUC max_key was allocated as: RANGE_OPT_PARAM::max_key[MAX_KEY_LENGTH+MAX_FIELD_WIDTH], in other words 3072 + 255 * 3 + 1= 3838.

So we actually do memcpy(max_key[3838], max_value, 6144). This statement effectively breaks stack trace, crash happens later as an outcome of this overwrite.

Comment by Sergei Petrunia [ 2015-09-20 ]

I was wondering if partition pruning is affected by this problem. It turns out, no. partition_info::check_partition_field_length() disallows creation of partitioning schemes which have partitioning columns occupy more than MAX_KEY_LENGTH when encoded with KeyTupleFormat.

Comment by Sergei Petrunia [ 2015-09-20 ]

Looking through opt_range.cc code, RANGE_OPT_PARAM::min_key and max-key seem to be the only places where the code implies that key images do not exceed MAX_KEY_LENGTH in size.

The steps are:

  • get_mm_leaf() stores field value in table->record[0] (check value->save_in_field_no_warnings() calls)
  • Then, SEL_ARG object is created. It has key part values, for which buffer of appropriate size (key_part->store_length) is allocated on MEM_ROOT.
  • Then, we call sel_arg_range_seq_init/sel_arg_range_seq_next and this is where it attempts to store the key value in RANGE_OPT_PARAM::min_key/max_key.

Buffer overrun seems to only be possible in #3.

Comment by Sergei Petrunia [ 2015-09-20 ]

http://lists.askmonty.org/pipermail/commits/2015-September/008419.html

Comment by Sergei Petrunia [ 2015-09-21 ]

Following svoj 's advice, debugged a few queries. new alloc_root calls do not cause calls to malloc()

Comment by Sergei Petrunia [ 2015-09-22 ]

The patch had a typo which caused MDEV-8826. This is now fixed and pushed in both 10.0 and 10.1

Generated at Thu Feb 08 07:29:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.