[MDEV-26190] Stack smashing SIGSEGV in optimized builds starting in table_cond_selectivity Created: 2021-07-20  Updated: 2021-09-06  Resolved: 2021-09-06

Status: Closed
Project: MariaDB Server
Component/s: Optimizer, Storage Engine - InnoDB
Affects Version/s: 10.4, 10.5, 10.6, 10.7
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Roel Van de Paar Assignee: Roel Van de Paar
Resolution: Fixed Votes: 0
Labels: not-10.2, not-10.3, regression, stack-smashing

Issue Links:
Relates
relates to MDEV-17783 AddressSanitizer: stack-buffer-overf... Closed
relates to MDEV-23937 SIGSEGV in looped best_extension_by_l... Closed

 Description   

This may prove to be a duplicate of MDEV-17783 and/or MDEV-23937, however there is a different situation here (stack smashing). It is also unclear if the MDEV-17783 patch already made it to 10.7 trunk. A 10.7 trunk build from today still shows the issue. Finally, the versions affected look to be different.

# mysqld options required for replay: --innodb_strict_mode=OFF
 
SET SESSION join_cache_level=5;
 
CREATE TABLE t3 (c INT) ENGINE=InnoDB;
 
CREATE TABLE t2 (c INT) ENGINE=InnoDB PARTITION BY RANGE (c)(PARTITION p0 VALUES LESS THAN (1),PARTITION p VALUES LESS THAN (2), PARTITION p2 VALUES LESS THAN (3));
 
CREATE TABLE t (c1 INT,c2 INT,c3 INT,c4 INT,c5 INT,c6 INT,c7 INT,c8 INT,c9 INT,c10 INT,c11 INT,c12 INT,c13 INT,c14 INT,c15 INT,c16 INT,c17 INT,c18 INT,c19 INT,c20 INT,c21 INT,c22 INT,c23 INT,c24 INT,c25 INT,c26 INT,c27 INT,c28 INT,c29 INT,c30 INT,c31 INT,c32 INT,c33 INT,c34 INT,c35 INT,c36 INT,c37 INT,c38 INT,c39 INT,c40 INT,c41 INT,c42 INT,c43 INT,c44 INT,c45 INT,c46 INT,c47 INT,c48 INT,c49 INT,c50 INT,c51 INT,c52 INT,c53 INT,c54 INT,c55 INT,c56 INT,c57 INT,c58 INT,c59 INT,c60 INT,c61 INT,c62 INT,c63 INT,c64 INT,c65 INT) ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
 
SELECT * FROM (SELECT * FROM t NATURAL JOIN t3) as t NATURAL RIGHT JOIN (SELECT * FROM t NATURAL JOIN t2) AS t2;
 
DROP FUNCTION IF EXISTS f1;

Leads to:

10.7.0 71ed8c136fa203b9b3a678a6d5cc72235ef73ef7 (Debug)

 
Core was generated by `/test/MD200721-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGSEGV, Segmentation fault.
 
#0  0x00005568eba98a05 in table_cond_selectivity (join=<optimized out>, 
    idx=<optimized out>, s=<optimized out>, rem_tables=<optimized out>)
 
    at /test/10.7_dbg/sql/sql_select.cc:9669
 
9669	  return sel;
 
[Current thread is 1 (Thread 0x153375946700 (LWP 325379))]
 
(gdb) bt
 
#0  0x00005568eba98a05 in table_cond_selectivity (join=<optimized out>, idx=<optimized out>, s=<optimized out>, rem_tables=<optimized out>) at /test/10.7_dbg/sql/sql_select.cc:9669
 
#1  0x0001000100010001 in ?? ()
 
#2  0x0000000000000006 in ?? ()
 
#3  0x0000000000000390 in ?? ()
 
#4  0x401999999999999a in ?? ()
 
#5  0x4008000000000000 in ?? ()
 
#6  0x0000000000000010 in ?? ()
 
#7  0x0000000400000003 in ?? ()
 
#8  0x000000010000003c in ?? ()
 
#9  0x0000000000000260 in ?? ()
 
#10 0x4008000000000000 in ?? ()
 
#11 0x4013333333333333 in ?? ()
 
#12 0x0000000000000000 in ?? ()

10.6.3 891a927e804c5a3a582f6137c2f316ef7abb25ca (Optimized)

 
Core was generated by `/test/MD280621-mariadb-10.6.3-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGSEGV, Segmentation fault.
 
#0  0x0000559a0f8c4dc9 in table_cond_selectivity (join=<optimized out>, 
    idx=<optimized out>, s=<optimized out>, rem_tables=<optimized out>)
 
    at /test/10.6_opt/sql/sql_select.cc:9354
 
9354	    table_map used_tables= item_equal->used_tables();
 
[Current thread is 1 (Thread 0x14cabc0b4700 (LWP 341412))]
 
(gdb) bt
 
#0  0x0000559a0f8c4dc9 in table_cond_selectivity (join=<optimized out>, idx=<optimized out>, s=<optimized out>, rem_tables=<optimized out>) at /test/10.6_opt/sql/sql_select.cc:9354
 
#1  0x0001000100010001 in ?? ()
 
#2  0x0000000000000390 in ?? ()
 
#3  0x401999999999999a in ?? ()
 
#4  0x4008000000000000 in ?? ()
 
#5  0x0000000000000010 in ?? ()
 
#6  0x4008000000000000 in ?? ()
 
#7  0x0000550000000003 in ?? ()
 
#8  0x0000003c00000004 in ?? ()
 
#9  0x4013333333333333 in ?? ()
 
#10 0x0000000000000260 in ?? ()
 
#11 0x000014cabc0b1f30 in ?? ()
 
#12 0x4008000000000000 in ?? ()
 
#13 0x0000000000000000 in ?? ()

The stack smashing may be observed only x in y runs. Ref first comment below for other related stack.

Bug confirmed present in:
MariaDB: 10.4.20 (dbg), 10.4.20 (opt), 10.5.11 (dbg), 10.5.11 (opt), 10.6.3 (dbg), 10.6.3 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.2.39 (dbg), 10.2.39 (opt), 10.3.30 (dbg), 10.3.30 (opt)
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.34 (dbg), 5.7.34 (opt), 8.0.24 (dbg), 8.0.24 (opt)

 Comments   
Comment by Roel Van de Paar [ 2021-07-20 ]

A somewhat shortened testcase will produce a crash somewhat similar to MDEV-17783, though again on different versions.

SET SESSION join_cache_level=5;
 
CREATE TABLE t3 (c INT) ENGINE=InnoDB;
 
CREATE TABLE t2 (c INT) ENGINE=InnoDB PARTITION BY RANGE (c)(PARTITION p0 VALUES LESS THAN (1),PARTITION p VALUES LESS THAN (2), PARTITION p2 VALUES LESS THAN (3));
 
CREATE TABLE t1 (c1 INT,c2 INT,c3 INT,c4 INT,c5 INT,c6 INT,c7 INT,c8 INT,c9 INT,c10 INT,c11 INT,c12 INT,c13 INT,c14 INT,c15 INT,c16 INT,c17 INT,c18 INT,c19 INT,c20 INT,c21 INT,c22 INT,c23 INT,c24 INT,c25 INT,c26 INT,c27 INT,c28 INT,c29 INT,c30 INT,c31 INT,c32 INT,c33 INT,c34 INT,c35 INT,c36 INT,c37 INT,c38 INT,c39 INT,c40 INT) ENGINE=InnoDB;
 
SELECT * FROM (SELECT * FROM t1 JOIN t3) AS t NATURAL JOIN (SELECT * FROM t1 JOIN t2) AS t2;
 
DROP FUNCTION IF EXISTS f;

Leads to:

10.7.0 71ed8c136fa203b9b3a678a6d5cc72235ef73ef7 (Debug)

 
Core was generated by `/test/MD200721-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGSEGV, Segmentation fault.
 
#0  restore_prev_nj_state (last=last@entry=0x100010001)
 
    at /test/10.7_dbg/sql/sql_select.cc:17231
 
17231	  TABLE_LIST *last_emb= last->table->pos_in_table_list->embedding;
 
[Current thread is 1 (Thread 0x14b8a80d4700 (LWP 833151))]
 
(gdb) bt
 
#0  restore_prev_nj_state (last=last@entry=0x100010001) at /test/10.7_dbg/sql/sql_select.cc:17231
 
#1  0x0000556e128cbb9d in best_extension_by_limited_search (join=join@entry=0x14b87003a3e0, remaining_tables=remaining_tables@entry=10, idx=idx@entry=2, record_count=record_count@entry=1, read_time=2.3999999999999999, search_depth=search_depth@entry=60, prune_level=prune_level@entry=1, use_cond_selectivity=use_cond_selectivity@entry=4) at /test/10.7_dbg/sql/sql_select.cc:9989
 
#2  0x0000556e128cba30 in best_extension_by_limited_search (join=join@entry=0x14b87003a3e0, remaining_tables=remaining_tables@entry=14, idx=idx@entry=1, record_count=record_count@entry=1, read_time=1.2, search_depth=search_depth@entry=61, prune_level=prune_level@entry=1, use_cond_selectivity=use_cond_selectivity@entry=4) at /test/10.7_dbg/sql/sql_select.cc:9946
 
#3  0x0000556e128cba30 in best_extension_by_limited_search (join=join@entry=0x14b87003a3e0, remaining_tables=remaining_tables@entry=15, idx=idx@entry=0, record_count=record_count@entry=1, read_time=read_time@entry=0, search_depth=search_depth@entry=62, prune_level=prune_level@entry=1, use_cond_selectivity=use_cond_selectivity@entry=4) at /test/10.7_dbg/sql/sql_select.cc:9946
 
#4  0x0000556e128cccef in greedy_search (use_cond_selectivity=<optimized out>, prune_level=<optimized out>, search_depth=62, remaining_tables=15, join=0x14b87003a3e0) at /test/10.7_dbg/sql/sql_select.cc:9063
 
#5  choose_plan (join=join@entry=0x14b87003a3e0, join_tables=<optimized out>) at /test/10.7_dbg/sql/sql_select.cc:8628
 
#6  0x0000556e128fbe76 in make_join_statistics (join=join@entry=0x14b87003a3e0, tables_list=@0x14b870013fc0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x14b8700b4cc8, last = 0x14b8700a7d58, elements = 4}, <No data fields>}, keyuse_array=keyuse_array@entry=0x14b87003a700) at /test/10.7_dbg/sql/sql_select.cc:5868
 
#7  0x0000556e1290300f in JOIN::optimize_inner (this=this@entry=0x14b87003a3e0) at /test/10.7_dbg/sql/sql_select.cc:2452
 
#8  0x0000556e129032ba in JOIN::optimize (this=this@entry=0x14b87003a3e0) at /test/10.7_dbg/sql/sql_select.cc:1808
 
#9  0x0000556e12903933 in mysql_select (thd=thd@entry=0x14b870000db8, tables=0x14b870016710, fields=@0x14b870014048: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x14b870014338, last = 0x14b8700cfa38, elements = 41}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x14b87003a3b8, unit=0x14b870005130, select_lex=0x14b870013da8) at /test/10.7_dbg/sql/sql_select.cc:4967
 
#10 0x0000556e12903c37 in handle_select (thd=thd@entry=0x14b870000db8, lex=lex@entry=0x14b870005068, result=result@entry=0x14b87003a3b8, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.7_dbg/sql/sql_select.cc:545
 
#11 0x0000556e12865a5e in execute_sqlcom_select (thd=thd@entry=0x14b870000db8, all_tables=0x14b870016710) at /test/10.7_dbg/sql/sql_parse.cc:6252
 
#12 0x0000556e128728ed in mysql_execute_command (thd=thd@entry=0x14b870000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.7_dbg/sql/sql_parse.cc:3947
 
#13 0x0000556e1285eead in mysql_parse (thd=thd@entry=0x14b870000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14b8a80d3400) at /test/10.7_dbg/sql/sql_parse.cc:8026
 
#14 0x0000556e1286d9f8 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14b870000db8, packet=packet@entry=0x14b87000b749 "SELECT * FROM (SELECT * FROM t1 JOIN t3) AS t NATURAL JOIN (SELECT * FROM t1 JOIN t2) AS t2", packet_length=packet_length@entry=91, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_class.h:1340
 
#15 0x0000556e12870e05 in do_command (thd=0x14b870000db8, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_parse.cc:1404
 
#16 0x0000556e129e6f48 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x556e15c85a28, put_in_cache=put_in_cache@entry=true) at /test/10.7_dbg/sql/sql_connect.cc:1410
 
#17 0x0000556e129e754d in handle_one_connection (arg=arg@entry=0x556e15c85a28) at /test/10.7_dbg/sql/sql_connect.cc:1312
 
#18 0x0000556e12e4f5b2 in pfs_spawn_thread (arg=0x556e15b6bd28) at /test/10.7_dbg/storage/perfschema/pfs.cc:2201
 
#19 0x000014b8be66a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#20 0x000014b8be258293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.7.0 71ed8c136fa203b9b3a678a6d5cc72235ef73ef7 (Optimized)

 
Core was generated by `/test/MD200721-mariadb-10.7.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
 
Program terminated with signal SIGSEGV, Segmentation fault.
 
#0  best_extension_by_limited_search (join=0x15140802da70, 
    remaining_tables=10, idx=2, record_count=1, read_time=2.3999999999999999, 
    search_depth=60, prune_level=1, use_cond_selectivity=4)
 
    at /test/10.7_opt/sql/sql_select.cc:9944
 
9944	        swap_variables(JOIN_TAB*, join->best_ref[idx], *pos);
 
[Current thread is 1 (Thread 0x151448174700 (LWP 850082))]
 
(gdb) bt
 
#0  best_extension_by_limited_search (join=0x15140802da70, remaining_tables=10, idx=2, record_count=1, read_time=2.3999999999999999, search_depth=60, prune_level=1, use_cond_selectivity=4) at /test/10.7_opt/sql/sql_select.cc:9944
 
#1  0x0000555f58ecbdf0 in best_extension_by_limited_search (join=0x15140802da70, remaining_tables=14, idx=1, record_count=1, read_time=1.2, search_depth=61, prune_level=1, use_cond_selectivity=4) at /test/10.7_opt/sql/sql_select.cc:9946
 
#2  0x0000555f58ecbdf0 in best_extension_by_limited_search (join=0x15140802da70, remaining_tables=15, idx=0, record_count=1, read_time=0, search_depth=62, prune_level=1, use_cond_selectivity=4) at /test/10.7_opt/sql/sql_select.cc:9946
 
#3  0x0000555f58ecc26c in greedy_search (use_cond_selectivity=<optimized out>, prune_level=<optimized out>, search_depth=62, remaining_tables=15, join=0x15140802da70) at /test/10.7_opt/sql/sql_select.cc:9063
 
#4  choose_plan (join=0x15140802da70, join_tables=<optimized out>) at /test/10.7_opt/sql/sql_select.cc:8628
 
#5  0x0000555f58ef896f in make_join_statistics (keyuse_array=0x15140802dd90, tables_list=@0x151408010ae0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x151408083c28, last = 0x151408077328, elements = 4}, <No data fields>}, join=0x15140802da70) at /test/10.7_opt/sql/sql_select.cc:5868
 
#6  JOIN::optimize_inner (this=0x15140802da70) at /test/10.7_opt/sql/sql_select.cc:2452
 
#7  0x0000555f58ef8e73 in JOIN::optimize (this=this@entry=0x15140802da70) at /test/10.7_opt/sql/sql_select.cc:1808
 
#8  0x0000555f58ef8f37 in mysql_select (thd=0x151408000c58, tables=0x151408013230, fields=<optimized out>, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x15140802da48, unit=0x151408004e10, select_lex=0x1514080108c8) at /test/10.7_opt/sql/sql_select.cc:4967
 
#9  0x0000555f58ef9757 in handle_select (thd=thd@entry=0x151408000c58, lex=lex@entry=0x151408004d48, result=result@entry=0x15140802da48, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.7_opt/sql/sql_select.cc:545
 
#10 0x0000555f58e7cd31 in execute_sqlcom_select (thd=0x151408000c58, all_tables=0x151408013230) at /test/10.7_opt/sql/sql_parse.cc:6252
 
#11 0x0000555f58e8aa96 in mysql_execute_command (thd=0x151408000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:3947
 
#12 0x0000555f58e77d46 in mysql_parse (thd=0x151408000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:8026
 
#13 0x0000555f58e83c35 in dispatch_command (command=COM_QUERY, thd=0x151408000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.7_opt/sql/sql_class.h:1340
 
#14 0x0000555f58e85b57 in do_command (thd=0x151408000c58, blocking=blocking@entry=true) at /test/10.7_opt/sql/sql_parse.cc:1404
 
#15 0x0000555f58fa0fe7 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.7_opt/sql/sql_connect.cc:1410
 
#16 0x0000555f58fa134d in handle_one_connection (arg=arg@entry=0x555f5b581028) at /test/10.7_opt/sql/sql_connect.cc:1312
 
#17 0x0000555f592f32d8 in pfs_spawn_thread (arg=0x555f5b131c88) at /test/10.7_opt/storage/perfschema/pfs.cc:2201
 
#18 0x000015145d529609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 
#19 0x000015145d117293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.4.20 (dbg), 10.4.20 (opt), 10.5.11 (dbg), 10.5.11 (opt), 10.6.3 (dbg), 10.6.3 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.2.39 (dbg), 10.2.39 (opt), 10.3.30 (dbg), 10.3.30 (opt)
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.34 (dbg), 5.7.34 (opt), 8.0.24 (dbg), 8.0.24 (opt)

Comment by Roel Van de Paar [ 2021-07-26 ]

FYI, Issue still present in trunk 10.6.4 42b9daaea7ece6155558f7c3bf638e8e175a7ff7 (Debug) build today. Both testcases tested.

Comment by Sergei Petrunia [ 2021-07-27 ]

Note: I can see that the fix for MDEV-17783: 

commit cdb29960d2ed4dd8c51c3ee1f95c1ab0ff953142
 
Author: Sergei Petrunia <psergey@askmonty.org>
 
Date:   Thu May 20 18:03:35 2021 +0300
 
 
 
    MDEV-17783: AddressSanitizer: stack-buffer-overflow in table_cond_selectivity

is in 10.2 and 10.3, but it is not merged into 10.4 yet.

Comment by Sergei Petrunia [ 2021-07-27 ]

so, if I take 10.5:

commit a42c80bd480c2060eca30101dab20ea6f6418bc8 (HEAD -> 10.5)
 
Merge: bcedb4200f2 baf0ef9a18f
 
Author: Marko Mäkelä <marko.makela@mariadb.com>
 
Date:   Mon Jun 21 14:22:22 2021 +0300
 
 
 
    Merge 10.4 into 10.5

the testcase for this bug ( from the first comment) is reproducible.
Then, if I apply the fix for MDEV-17783, the bug goes away.

Comment by Roel Van de Paar [ 2021-07-28 ]

psergei Thank you for the analysis. I will assign this to myself and recheck later to make sure.

Comment by Roel Van de Paar [ 2021-09-06 ]

Confirmed all fixed.

Generated at Thu Feb 08 09:43:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.