Don Lindsay added a comment - 2017-04-08 01:20 Also segfaults MariaDB-server-10.2.5 (regardless of the setting of max_recursive_iterations, ie the same bug)

Daniel Black added a comment - 2017-04-08 08:42 max_recursive_iterations is for CTE - its not a setting that affects regexes Ubuntu-17.04: libpcre3:amd64 2:8.39-3 .... #536 0x00007fad28da9aee in match (eptr=<optimized out>, ecode=0x7facbc0160a2 "y", mstart=0x7facbc0160c8 "Alpha,Bravo,Charlie,Delta,Echo,Foxtrot,StrataCentral,Golf,Hotel,India,Juliet,Kilo,Lima,Mike,StrataL3,November,Oscar,StrataL2,Sand,P3,P4SwitchTest,Arsys,Poppa,ExtensionMgr,Arp,Quebec,Romeo,StrataApiV2,"..., offset_top=2, md=0x7fad251ef4c0, eptrb=0x0, rdepth=<optimized out>) at pcre_exec.c:2061 #537 0x00007fad28d9cdd6 in match (eptr=<optimized out>, ecode=0x7facbc01604c "\222\205", mstart=0x7facbc0160c8 "Alpha,Bravo,Charlie,Delta,Echo,Foxtrot,StrataCentral,Golf,Hotel,India,Juliet,Kilo,Lima,Mike,StrataL3,November,Oscar,StrataL2,Sand,P3,P4SwitchTest,Arsys,Poppa,ExtensionMgr,Arp,Quebec,Romeo,StrataApiV2,"..., offset_top=2, md=0x7fad251ef4c0, eptrb=0x0, rdepth=<optimized out>) at pcre_exec.c:1878 #538 0x00007fad28dad6a1 in pcre_exec (argument_re=0x7facbc016008, extra_data=<optimized out>, subject=<optimized out>, length=1288, start_offset=0, options=0, offsets=0x7facbc0103d4, offsetcount=0) at pcre_exec.c:6936 ---Type <return> to continue, or q <return> to quit--- #539 0x000000000078f269 in Regexp_processor_pcre::pcre_exec_with_warn (code=<optimized out>, extra=<optimized out>, subject=0x7facbc0160c8 "Alpha,Bravo,Charlie,Delta,Echo,Foxtrot,StrataCentral,Golf,Hotel,India,Juliet,Kilo,Lima,Mike,StrataL3,November,Oscar,StrataL2,Sand,P3,P4SwitchTest,Arsys,Poppa,ExtensionMgr,Arp,Quebec,Romeo,StrataApiV2,"..., length=2, startoffset=0, options=0, ovector=0x7facbc0103d4, ovecsize=0, this=<optimized out>) at /home/build/sql/item_cmpfunc.cc:5486 #540 Regexp_processor_pcre::exec (this=0x7facbc010390, str=<optimized out>, offset=0, n_result_offsets_to_convert=0) at /home/build/sql/item_cmpfunc.cc:5508 #541 0x000000000078f3f5 in Regexp_processor_pcre::exec (this=0x7facbc010390, item=0x7facbc010010, offset=0, n_result_offsets_to_convert=0) at /home/build/sql/item_cmpfunc.cc:5538 #542 0x000000000078f686 in Item_func_regex::val_int (this=0x7facbc0102d0) at /home/build/sql/item_cmpfunc.cc:5580 #543 0x00000000007ad85a in eval_const_cond (cond=0x7facbc0162df) at /home/build/sql/item_func.cc:79 #544 0x000000000061bc34 in Item::remove_eq_conds (this=0x7facbc0102d0, thd=<optimized out>, cond_value=0x7facbc010878, top_level_arg=<optimized out>) at /home/build/sql/sql_select.cc:15532 #545 0x00000000005fd5e7 in optimize_cond (join=0x7facbc010570, conds=0x7facbc0102d0, join_list=<optimized out>, ignore_on_conds=<error reading variable: access outside bounds of object referenced via synthetic pointer>, cond_value=0x7facbc010878, cond_equal=<optimized out>, flags=1) at /home/build/sql/sql_select.cc:15088 #546 JOIN::optimize_inner (this=0x7facbc010570) at /home/build/sql/sql_select.cc:1309 #547 0x00000000005fcdff in JOIN::optimize (this=0x7facbc010570) at /home/build/sql/sql_select.cc:1084 #548 0x00000000005faed5 in mysql_select (thd=<optimized out>, tables=<optimized out>, wild_num=<optimized out>, fields=..., conds=<optimized out>, og_num=<optimized out>, order=<optimized out>, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=0x7fad251ef4c0, unit=<optimized out>, select_lex=0x7facbc004a78) at /home/build/sql/sql_select.cc:3644   #549 0x00000000005fad0f in handle_select (thd=0x7facbc0009a8, lex=0x7facbc004280, result=0x7facbc010550, setup_tables_done_option=<optimized out>) at /home/build/sql/sql_select.cc:361 #550 0x00000000005dd810 in execute_sqlcom_select (thd=0x7facbc0009a8, all_tables=<optimized out>) at /home/build/sql/sql_parse.cc:6431 #551 0x00000000005d68ad in mysql_execute_command (thd=<optimized out>) at /home/build/sql/sql_parse.cc:3448 #552 0x00000000005d44d8 in mysql_parse (thd=0x7facbc0009a8, rawbuf=0x7facbc00ef30 "SELECT 1 FROM dual WHERE ('Alpha,Bravo,Charlie,Delta,Echo,Foxtrot,StrataCentral,Golf,Hotel,India,Juliet,Kilo,Lima,Mike,StrataL3,November,Oscar,StrataL2,Sand,P3,P4SwitchTest,Arsys,Poppa,ExtensionMgr,Ar"..., length=<optimized out>, parser_state=0x7fad251f1400, ---Type <return> to continue, or q <return> to quit--- is_com_multi=<optimized out>, is_next_command=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /home/build/sql/sql_parse.cc:7874 #553 0x00000000005d23c7 in dispatch_command (command=<optimized out>, thd=0x7facbc0009a8, packet=0x7facbc00ef30 "SELECT 1 FROM dual WHERE ('Alpha,Bravo,Charlie,Delta,Echo,Foxtrot,StrataCentral,Golf,Hotel,India,Juliet,Kilo,Lima,Mike,StrataL3,November,Oscar,StrataL2,Sand,P3,P4SwitchTest,Arsys,Poppa,ExtensionMgr,Ar"..., packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /home/build/sql/sql_parse.cc:1811 #554 0x00000000005d3471 in do_command (thd=0x7facbc0009a8) at /home/build/sql/sql_parse.cc:1361 #555 0x000000000069e75e in do_handle_one_connection (connect=<optimized out>) at /home/build/sql/sql_connect.cc:1354 #556 0x000000000069e3d3 in handle_one_connection (arg=0x3ea5a58) at /home/build/sql/sql_connect.cc:1260 #557 0x00007fad29b566da in start_thread (arg=0x7fad251f2700) at pthread_create.c:456 #558 0x00007fad27a3f17f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105

Daniel Black added a comment - 2017-04-08 13:14 - edited proof of concept fix in PR with extended warnings 100 for depth? Above test case shows 537 match functions before the stack overflowed. http://pcre.org/original/doc/html/pcrestack.html shows about 500-656 bytes per stack frame https://mariadb.com/kb/en/mariadb/server-system-variables/#thread_stack a default of 297984 seems to correspond to about 600 frames. The rest of the server needs a few stack frames too. If we where to make depth dynamic in Item_func_regex::val_int and use (like check_stack_overrun) the concepts below: m_pcre_extra.match_limit_recursion= (my_thread_stack_size - margin - used_stack(thd->thread_stack,( char *) &stack_used)) ) / 656; 656 could be determined programmatic (from pcretest) and perhaps an extra margin.

Daniel Black added a comment - 2017-04-11 03:03 serg do you want to go down the path of dynamically determining the pcre recursion limit based on available stack?

Sergei Golubchik added a comment - 2017-04-17 13:41 We've added stack size check to pcre (first to our copy, then the patch was merged upstream). May be pcre needs to call it in more places? I'll take a look. Dynamically determining the pcre recursion limit is an interesting idea, thanks! I'll give it a try.

Daniel Black added a comment - 2017-04-17 22:20 Had a bit of a look. How far of using pcre2 (version 10.x). Has a more featured api

Daniel Black added a comment - 2017-04-18 07:23 > May be pcre needs to call it in more places? I think you are right. pcre_stack_guard is only called from pcre_compile currently and not from match. There isn't an upstream bug for it. The version 2 api is more explicit calling it pcre2_set_compile_recursion_guard (which has a recursion value as an argument) however there isn't a match callback. Added https://github.com/MariaDB/server/pull/356/commits/91908884347e69fe8cbc8a9a78c0f77bf5b4e1bc , the pcre_exec function wasn't returning the right value (returned ~20) however the non-exposed match function was. So its in an incomplete state now.