[MDEV-28714] JSON functions still don't respect KILL on centos73-ppc64, main.func_json_notembedded fails Created: 2022-05-31  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: JSON, Platform Power
Affects Version/s: 10.6, 10.7, 10.8, 10.9
Fix Version/s: 10.6

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Rucha Deodhar
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Duplicate
is duplicated by MDEV-28251 too small max_statement_time is ignored Closed
Relates
relates to MDEV-27955 main.func_json_notembedded test fails... Closed
relates to MDEV-24909 JSON functions don't respect KILL QUE... Closed

 Description   

main.func_json_notembedded fails reliably on centos73-ppc64 in buildbot and when run on the same VM manually.
It is not a test race condition. max_statement_time acts as planned and kills json_merge_patch, but it remains in Killed state.

10.6 a6160356

 
MariaDB [(none)]> show processlist;
 
+----+------+-----------------+------+---------+------+-----------+-------------------------------------+----------+
 
| Id | User | Host            | db   | Command | Time | State     | Info                                | Progress |
 
+----+------+-----------------+------+---------+------+-----------+-------------------------------------+----------+
 
|  4 | root | localhost       | test | Sleep   |   33 |           | NULL                                |    0.000 |
 
|  5 | root | localhost       | test | Killed  |   32 | Executing | select json_merge_patch(@obj, @obj) |    0.000 |
 
|  6 | root | localhost:47644 | NULL | Query   |    0 | starting  | show processlist                    |    0.000 |
 
+----+------+-----------------+------+---------+------+-----------+-------------------------------------+----------+

A couple of consequently taken stack traces:

Thread 3 (Thread 0x3fff9ae58030 (LWP 3587)):
 
#0  0x0000000046f2c928 in json_scan_next (j=0x3fff9ae54dc0) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:977
 
#1  0x0000000046f2d064 in v_string (j=0x3fff9ae54dc0) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:404
 
#2  0x0000000046f2c57c in skip_key (j=0x3fff9ae54dc0) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:857
 
#3  0x0000000046f2c928 in json_scan_next (j=0x3fff9ae54dc0) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:977
 
#4  0x0000000046f2d6a4 in json_skip_to_level (j=<optimized out>, level=<optimized out>) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:1218
 
#5  0x0000000046682a80 in do_merge_patch (str=str@entry=0x3fff9ae55280, je1=je1@entry=0x3fff9ae54dc0, je2=je2@entry=0x3fff9ae54eb0, empty_result=empty_result@entry=0x3fff9ae54db0) at /usr/src/debug/MariaDB-/src_0/sql/item_jsonfunc.cc:2448
 
#6  0x0000000046683a40 in Item_func_json_merge_patch::val_str (this=0x3fff88010d30, str=0x3fff9ae55280) at /usr/src/debug/MariaDB-/src_0/sql/item_jsonfunc.cc:2548
 
#7  0x00000000466cad34 in Type_handler::Item_send_str (this=<optimized out>, item=<optimized out>, protocol=<optimized out>, buf=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_type.cc:7467
 
#8  0x000000004660c4f8 in Type_handler_string_result::Item_send (this=<optimized out>, item=<optimized out>, protocol=<optimized out>, buf=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_type.h:5454
 
#9  0x00000000463fb7dc in Item::send (this=0x3fff88010d30, protocol=0x3fff88000f38, buffer=0x3fff9ae55250) at /usr/src/debug/MariaDB-/src_0/sql/item.h:1227
 
#10 0x00000000463f8dec in Protocol::send_result_set_row (this=0x3fff88000f38, row_items=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/protocol.cc:1328
 
#11 0x000000004648f450 in select_send::send_data (this=0x3fff88011738, items=...) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.cc:3116
 
#12 0x000000004651fdc4 in select_result_sink::send_data_with_check (this=<optimized out>, items=..., sent=0, u=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.h:5680
 
#13 0x000000004655fbd0 in send_data_with_check (sent=0, u=<optimized out>, items=..., this=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.h:5677
 
#14 JOIN::exec_inner (this=0x3fff88011760) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:4629
 
#15 0x0000000046560234 in JOIN::exec (this=0x3fff88011760) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:4541
 
#16 0x000000004655e198 in mysql_select (thd=0x3fff880009b8, tables=0x0, fields=..., conds=0x0, og_num=<optimized out>, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x3fff88011738, unit=0x3fff88004b60, select_lex=0x3fff88010698) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:5020
 
#17 0x000000004655e8f4 in handle_select (thd=0x3fff880009b8, lex=0x3fff88004a98, result=0x3fff88011738, setup_tables_done_option=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:565
 
#18 0x00000000463570b8 in execute_sqlcom_select (thd=thd@entry=0x3fff880009b8, all_tables=0x0) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:6255
 
#19 0x00000000464f2ba0 in mysql_execute_command (thd=0x3fff880009b8, is_called_from_prepared_stmt=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:3945
 
#20 0x00000000464f5218 in mysql_parse (thd=0x3fff880009b8, rawbuf=<optimized out>, length=<optimized out>, parser_state=0x3fff9ae569d0) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:8029
 
#21 0x00000000464f7794 in dispatch_command (command=<optimized out>, thd=0x3fff880009b8, packet=0x3fff88008219 "", packet_length=<optimized out>, blocking=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1896
 
#22 0x00000000464f8bb4 in do_command (thd=0x3fff880009b8, blocking=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1410
 
#23 0x0000000046620ab8 in do_handle_one_connection (connect=<optimized out>, put_in_cache=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1418
 
#24 0x0000000046620e74 in handle_one_connection (arg=0x1000ebb91d8) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1312
 
#25 0x0000000046aad524 in pfs_spawn_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
 
#26 0x00003fffa5a9c2bc in .start_thread () from /lib64/libpthread.so.0
 
#27 0x00003fffa56db304 in .__clone () from /lib64/libc.so.6


Thread 3 (Thread 0x3fff9ae58030 (LWP 3587)):
 
#0  0x0000000046f2c21c in json_read_keyname_chr (j=0x3fff9ae54dc0) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:908
 
#1  0x0000000046f2c438 in skip_key (j=0x3fff9ae54dc0) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:851
 
#2  0x0000000046f2c928 in json_scan_next (j=0x3fff9ae54dc0) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:977
 
#3  0x0000000046f2d6a4 in json_skip_to_level (j=<optimized out>, level=<optimized out>) at /usr/src/debug/MariaDB-/src_0/strings/json_lib.c:1218
 
#4  0x0000000046682a80 in do_merge_patch (str=str@entry=0x3fff9ae55280, je1=je1@entry=0x3fff9ae54dc0, je2=je2@entry=0x3fff9ae54eb0, empty_result=empty_result@entry=0x3fff9ae54db0) at /usr/src/debug/MariaDB-/src_0/sql/item_jsonfunc.cc:2448
 
#5  0x0000000046683a40 in Item_func_json_merge_patch::val_str (this=0x3fff88010d30, str=0x3fff9ae55280) at /usr/src/debug/MariaDB-/src_0/sql/item_jsonfunc.cc:2548
 
#6  0x00000000466cad34 in Type_handler::Item_send_str (this=<optimized out>, item=<optimized out>, protocol=<optimized out>, buf=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_type.cc:7467
 
#7  0x000000004660c4f8 in Type_handler_string_result::Item_send (this=<optimized out>, item=<optimized out>, protocol=<optimized out>, buf=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_type.h:5454
 
#8  0x00000000463fb7dc in Item::send (this=0x3fff88010d30, protocol=0x3fff88000f38, buffer=0x3fff9ae55250) at /usr/src/debug/MariaDB-/src_0/sql/item.h:1227
 
#9  0x00000000463f8dec in Protocol::send_result_set_row (this=0x3fff88000f38, row_items=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/protocol.cc:1328
 
#10 0x000000004648f450 in select_send::send_data (this=0x3fff88011738, items=...) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.cc:3116
 
#11 0x000000004651fdc4 in select_result_sink::send_data_with_check (this=<optimized out>, items=..., sent=0, u=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.h:5680
 
#12 0x000000004655fbd0 in send_data_with_check (sent=0, u=<optimized out>, items=..., this=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_class.h:5677
 
#13 JOIN::exec_inner (this=0x3fff88011760) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:4629
 
#14 0x0000000046560234 in JOIN::exec (this=0x3fff88011760) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:4541
 
#15 0x000000004655e198 in mysql_select (thd=0x3fff880009b8, tables=0x0, fields=..., conds=0x0, og_num=<optimized out>, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x3fff88011738, unit=0x3fff88004b60, select_lex=0x3fff88010698) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:5020
 
#16 0x000000004655e8f4 in handle_select (thd=0x3fff880009b8, lex=0x3fff88004a98, result=0x3fff88011738, setup_tables_done_option=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_select.cc:565
 
#17 0x00000000463570b8 in execute_sqlcom_select (thd=thd@entry=0x3fff880009b8, all_tables=0x0) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:6255
 
#18 0x00000000464f2ba0 in mysql_execute_command (thd=0x3fff880009b8, is_called_from_prepared_stmt=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:3945
 
#19 0x00000000464f5218 in mysql_parse (thd=0x3fff880009b8, rawbuf=<optimized out>, length=<optimized out>, parser_state=0x3fff9ae569d0) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:8029
 
#20 0x00000000464f7794 in dispatch_command (command=<optimized out>, thd=0x3fff880009b8, packet=0x3fff88008219 "", packet_length=<optimized out>, blocking=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1896
 
#21 0x00000000464f8bb4 in do_command (thd=0x3fff880009b8, blocking=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_parse.cc:1410
 
#22 0x0000000046620ab8 in do_handle_one_connection (connect=<optimized out>, put_in_cache=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1418
 
#23 0x0000000046620e74 in handle_one_connection (arg=0x1000ebb91d8) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1312
 
#24 0x0000000046aad524 in pfs_spawn_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
 
#25 0x00003fffa5a9c2bc in .start_thread () from /lib64/libpthread.so.0
 
#26 0x00003fffa56db304 in .__clone () from /lib64/libc.so.6
 
 
 
Thread 2 (Thread 0x3fff9adf8030 (LWP 3593)):
 
#0  0x00003fffa5aa1bd4 in .pthread_cond_timedwait () from /lib64/libpthread.so.0
 
#1  0x00000000463852ac in psi_cond_timedwait (that=0x47886dc8 <thread_cache>, mutex=0x47886e38 <thread_cache+112>, abstime=0x3fff9adf73e0, file=<optimized out>, line=<optimized out>) at /usr/src/debug/MariaDB-/src_0/mysys/my_thr_init.c:609
 
#2  0x0000000046621164 in inline_mysql_cond_timedwait (src_file=0x46f8a310 "/home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.9/sql/thread_cache.h", src_line=177, abstime=<optimized out>, mutex=<optimized out>, that=<optimized out>) at /usr/src/debug/MariaDB-/src_0/include/mysql/psi/mysql_thread.h:1086
 
#3  Thread_cache::park (this=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/thread_cache.h:176
 
#4  0x0000000046620b6c in do_handle_one_connection (connect=<optimized out>, put_in_cache=<optimized out>) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1430
 
#5  0x0000000046620e74 in handle_one_connection (arg=0x1000ebb91d8) at /usr/src/debug/MariaDB-/src_0/sql/sql_connect.cc:1312
 
#6  0x0000000046aad524 in pfs_spawn_thread (arg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/perfschema/pfs.cc:2201
 
#7  0x00003fffa5a9c2bc in .start_thread () from /lib64/libpthread.so.0
 
#8  0x00003fffa56db304 in .__clone () from /lib64/libc.so.6



 Comments   
Comment by Elena Stepanova [ 2022-05-31 ]

The test will be skipped on this builder in the old buildbot. When/if it's fixed, please create a TODO for re-enabling it.

Comment by Daniel Black [ 2022-06-01 ]

advice from [~serg]

 
I'd try to disable the test for builders with little memory. Something like
 
 
 
if ($mysql_error = ER_no_memory) {
 
  skip Too little memory;
 
}
 
after the select that causes OOM
 
 
 
 
 
this way it won't waste time on a separate mem size check
 
 
 
 
 
to avoid putting this after every select, I'd put the one that needs most memory first, and only add this if() after it
 
 
 
 
 
no, I don't know what select needs most memory, but it's easy to find out

Comment by Elena Stepanova [ 2022-06-01 ]

kvm-based package builders run with the standard options, 8G memory as of now.
The test doesn't OOM and doesn't fail with a wrong result as in the Windows case which was discussed, it just reliably exhibits behavior which was originally present universally and described in MDEV-24909. Now it is apparently limited to a specific platform.

Generated at Thu Feb 08 10:02:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.