[MDEV-26582] SIGSEGV in spider_db_bulk_insert and spider_db_connect and spider_db_before_query, and hang in "End of update loop" / "Reset for next command" query states Created: 2021-09-10  Updated: 2021-12-12  Resolved: 2021-10-18

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider, Virtual Columns
Affects Version/s: 10.5, 10.6, 10.7
Fix Version/s: 10.5.13, 10.6.5

Type: Bug Priority: Critical
Reporter: Roel Van de Paar Assignee: Nayuta Yanagisawa (Inactive)
Resolution: Fixed Votes: 0
Labels: affects-tests, hang, not-10.2, not-10.3, not-10.4, regression

Issue Links:
Problem/Incident
is caused by MDEV-19002 Partition performance optimization Stalled
Relates
relates to MDEV-24769 Spider crash when selecting all rows ... Closed
relates to MDEV-26587 Assertion `mutex->__data.__owner == 0... Closed
relates to MDEV-26546 SIGSEGV's in spider_db_connect on SHO... Closed
relates to MDEV-27231 Spider: SIGSEGV in spider_db_bulk_ins... Closed

 Description   

Repeat the following testcase till you get a crash or semi-hang. Sporadic. For all situations described below, usually about 2 repeats are sufficient, but at times it may be more.

DROP DATABASE test;
CREATE DATABASE test;
USE test;
SET sql_mode='';
INSTALL PLUGIN spider SONAME 'ha_spider.so';
CREATE TABLE t (i CHAR,n CHAR GENERATED ALWAYS AS (MD5 (i)) VIRTUAL) ENGINE=SPIDER;
INSERT INTO t VALUES (0,0);
CREATE TABLE t2 (c INT);
INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1;

Leads to:

10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Optimized)

Core was generated by `/test/MD040921-mariadb-10.7.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  spider_db_bulk_insert (spider=spider@entry=0x152970039c30, 
    table=0x15297005d5d8, copy_info=copy_info@entry=0x15297003a020, 
    bulk_end=bulk_end@entry=true)
    at /test/10.7_opt/storage/spider/spd_db_conn.cc:6710
[Current thread is 1 (Thread 0x1529c04c2700 (LWP 341906))]
(gdb) bt
#0  spider_db_bulk_insert (spider=spider@entry=0x152970039c30, table=0x15297005d5d8, copy_info=copy_info@entry=0x15297003a020, bulk_end=bulk_end@entry=true) at /test/10.7_opt/storage/spider/spd_db_conn.cc:6710
#1  0x00001529c00db474 in ha_spider::end_bulk_insert (this=0x152970039c30) at /test/10.7_opt/storage/spider/ha_spider.cc:9896
#2  ha_spider::end_bulk_insert (this=0x152970039c30) at /test/10.7_opt/storage/spider/ha_spider.cc:9887
#3  0x000055d9d390297d in select_insert::abort_result_set (this=0x152970011ec0) at /test/10.7_opt/sql/sql_insert.cc:4294
#4  0x000055d9d39b7152 in handle_select (thd=thd@entry=0x152970000c58, lex=lex@entry=0x152970004d38, result=result@entry=0x152970011ec0, setup_tables_done_option=setup_tables_done_option@entry=1073741824) at /test/10.7_opt/sql/sql_select.cc:563
#5  0x000055d9d3949001 in mysql_execute_command (thd=0x152970000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:4711
#6  0x000055d9d3935456 in mysql_parse (thd=0x152970000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:8036
#7  0x000055d9d3941345 in dispatch_command (command=COM_QUERY, thd=0x152970000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.7_opt/sql/sql_class.h:1358
#8  0x000055d9d3943217 in do_command (thd=0x152970000c58, blocking=blocking@entry=true) at /test/10.7_opt/sql/sql_parse.cc:1404
#9  0x000055d9d3a5eae7 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.7_opt/sql/sql_connect.cc:1418
#10 0x000055d9d3a5ee2d in handle_one_connection (arg=arg@entry=0x55d9d6d2e548) at /test/10.7_opt/sql/sql_connect.cc:1312
#11 0x000055d9d3db2298 in pfs_spawn_thread (arg=0x55d9d71d0ed8) at /test/10.7_opt/storage/perfschema/pfs.cc:2201
#12 0x00001529c3555609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00001529c3143293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Debug)

Core was generated by `/test/MD040921-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000150f47cc9248 in spider_db_bulk_insert (
    spider=spider@entry=0x150f3004de80, table=0x150f3004d5a8, 
    copy_info=copy_info@entry=0x150f3004e270, bulk_end=bulk_end@entry=true)
    at /test/10.7_dbg/storage/spider/spd_db_conn.cc:6708
[Current thread is 1 (Thread 0x150f7c12c700 (LWP 347996))]
(gdb) bt
#0  0x0000150f47cc9248 in spider_db_bulk_insert (spider=spider@entry=0x150f3004de80, table=0x150f3004d5a8, copy_info=copy_info@entry=0x150f3004e270, bulk_end=bulk_end@entry=true) at /test/10.7_dbg/storage/spider/spd_db_conn.cc:6708
#1  0x0000150f47d3a77a in ha_spider::end_bulk_insert (this=0x150f3004de80) at /test/10.7_dbg/storage/spider/ha_spider.cc:9896
#2  0x000055d75821bd66 in handler::ha_end_bulk_insert (this=0x150f3004de80) at /test/10.7_dbg/sql/handler.cc:4990
#3  0x000055d757ec42fe in select_insert::abort_result_set (this=0x150f300153a0) at /test/10.7_dbg/sql/sql_insert.cc:4294
#4  0x000055d757fa7d7d in handle_select (thd=thd@entry=0x150f30000db8, lex=lex@entry=0x150f30005058, result=result@entry=0x150f300153a0, setup_tables_done_option=setup_tables_done_option@entry=1073741824) at /test/10.7_dbg/sql/sql_select.cc:563
#5  0x000055d757f1889b in mysql_execute_command (thd=thd@entry=0x150f30000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.7_dbg/sql/sql_parse.cc:4711
#6  0x000055d757f02ccb in mysql_parse (thd=thd@entry=0x150f30000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x150f7c12b400) at /test/10.7_dbg/sql/sql_parse.cc:8036
#7  0x000055d757f118d0 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x150f30000db8, packet=packet@entry=0x150f3000b739 "INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1", packet_length=packet_length@entry=45, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_class.h:1358
#8  0x000055d757f14cd6 in do_command (thd=0x150f30000db8, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_parse.cc:1404
#9  0x000055d75808b0c8 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55d75bcc3608, put_in_cache=put_in_cache@entry=true) at /test/10.7_dbg/sql/sql_connect.cc:1418
#10 0x000055d75808b6cd in handle_one_connection (arg=arg@entry=0x55d75bcc3608) at /test/10.7_dbg/sql/sql_connect.cc:1312
#11 0x000055d7584f4ade in pfs_spawn_thread (arg=0x55d75bbc39c8) at /test/10.7_dbg/storage/perfschema/pfs.cc:2201
#12 0x0000150f803fa609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x0000150f7ffe8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Note the different stack on 10.6 optimized:

10.6.5 9ac1ac006197c8979db1dc73f4e983f623e831e8 (Optimized)

Core was generated by `/test/MD160821-mariadb-10.6.5-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00001506700e8cbd in spider_db_connect (share=0x150634087d48, 
    conn=conn@entry=0x150634076748, link_idx=0)
    at /test/10.6_opt/storage/spider/spd_db_conn.cc:119
[Current thread is 1 (Thread 0x150684658700 (LWP 347695))]
(gdb) bt
#0  0x00001506700e8cbd in spider_db_connect (share=0x150634087d48, conn=conn@entry=0x150634076748, link_idx=0) at /test/10.6_opt/storage/spider/spd_db_conn.cc:119
#1  0x00001506700e9a78 in spider_db_conn_queue_action (conn=0x150634076748) at /test/10.6_opt/storage/spider/spd_db_conn.cc:293
#2  0x00001506700efc90 in spider_db_before_query (conn=0x150634076748, need_mon=<optimized out>) at /test/10.6_opt/storage/spider/spd_db_conn.cc:629
#3  0x00001506700efea2 in spider_db_set_names_internal (trx=0x150634055128, share=0x15063406e6b8, conn=conn@entry=0x150634076748, all_link_idx=0, need_mon=0x150634075088) at /test/10.6_opt/storage/spider/spd_db_conn.cc:930
#4  0x00001506700f0059 in spider_db_set_names (spider=spider@entry=0x15063405d9a0, conn=conn@entry=0x150634076748, link_idx=link_idx@entry=0) at /test/10.6_opt/storage/spider/spd_db_conn.cc:977
#5  0x00001506700f46a9 in spider_db_bulk_insert (spider=spider@entry=0x15063405d9a0, table=0x1506340400f8, copy_info=copy_info@entry=0x15063405dd90, bulk_end=bulk_end@entry=true) at /test/10.6_opt/storage/spider/spd_db_conn.cc:6745
#6  0x000015067013e4a4 in ha_spider::end_bulk_insert (this=0x15063405d9a0) at /test/10.6_opt/storage/spider/ha_spider.cc:9896
#7  ha_spider::end_bulk_insert (this=0x15063405d9a0) at /test/10.6_opt/storage/spider/ha_spider.cc:9887
#8  0x00005574e2988aed in select_insert::abort_result_set (this=0x150634011f60) at /test/10.6_opt/sql/sql_insert.cc:4294
#9  0x00005574e2a301f2 in handle_select (thd=thd@entry=0x150634000c58, lex=lex@entry=0x150634004d38, result=result@entry=0x150634011f60, setup_tables_done_option=setup_tables_done_option@entry=1073741824) at /test/10.6_opt/sql/sql_select.cc:563
#10 0x00005574e29cf155 in mysql_execute_command (thd=0x150634000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.6_opt/sql/sql_parse.cc:4711
#11 0x00005574e29bb5c6 in mysql_parse (thd=0x150634000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.6_opt/sql/sql_parse.cc:8030
#12 0x00005574e29c74c5 in dispatch_command (command=COM_QUERY, thd=0x150634000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.6_opt/sql/sql_class.h:1357
#13 0x00005574e29c93e7 in do_command (thd=0x150634000c58, blocking=blocking@entry=true) at /test/10.6_opt/sql/sql_parse.cc:1404
#14 0x00005574e2ad7ef7 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.6_opt/sql/sql_connect.cc:1418
#15 0x00005574e2ad823d in handle_one_connection (arg=arg@entry=0x5574e5df0698) at /test/10.6_opt/sql/sql_connect.cc:1312
#16 0x00005574e2e62778 in pfs_spawn_thread (arg=0x5574e5d67568) at /test/10.6_opt/storage/perfschema/pfs.cc:2201
#17 0x0000150685f91609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x0000150685b7f293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

This crash is already listed in MDEV-26546 and seems to be consistently generated in 10.6.5 optimized only as a result of this testcase (though the testcase overall is still sporadic).

Furthermore, on 10.5.13 (optimized) we see this possible semi-hang:

10.5.13 0268b8712288d46fbd8a43fdef6bada399b68dff (Optimized)

10.5.13-opt>show processlist;
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------------+----------+
| Id | User        | Host      | db   | Command | Time | State                                              | Info                                          | Progress |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------------+----------+
|  4 | root        | localhost | test | Query   | 1199 | End of update loop                                 | INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1 |    0.000 |
|  5 | system user |           | NULL | Sleep   | 1199 | Reset for next command                             | NULL                                          |    0.000 |
|  6 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
|  7 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
|  8 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
|  9 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 10 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 11 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 12 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 13 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 14 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 16 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 15 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 17 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 18 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 20 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 19 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 22 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 21 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 24 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 23 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 25 | root        | localhost | test | Query   |    0 | starting                                           | show processlist                              |    0.000 |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------------+----------+
22 rows in set (0.000 sec)

Even CTRL+C (query killed) does not seem to help as the state becomes;

10.5.13 0268b8712288d46fbd8a43fdef6bada399b68dff (Optimized)

10.5.13-opt>show processlist;
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------------+----------+
| Id | User        | Host      | db   | Command | Time | State                                              | Info                                          | Progress |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------------+----------+
|  4 | root        | localhost | test | Killed  |   88 | End of update loop                                 | INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1 |    0.000 |
|  5 | system user |           | NULL | Sleep   |   88 | Reset for next command                             | NULL                                          |    0.000 |
|  6 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
|  7 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
|  8 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
|  9 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 11 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 10 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 12 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 13 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 14 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                          |    0.000 |
| 15 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 16 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 17 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 19 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 18 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 21 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 20 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 22 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 23 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 24 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                          |    0.000 |
| 25 | root        | localhost | test | Query   |    0 | starting                                           | show processlist                              |    0.000 |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------------+----------+
22 rows in set (0.000 sec)

This state too is consistently reproducible with the given test-case, in a sporadic way. For the hang, I broke in with gdb and took a snapshot of all threads:

10.5.13 0268b8712288d46fbd8a43fdef6bada399b68dff (Optimized)

(gdb) thread apply all bt
 
Thread 31 (Thread 0x14615fdfe700 (LWP 1416971)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x14615fdfdd40, clockid=<optimized out>, expected=0, futex_word=0x563b020cf188) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x14615fdfdd40, clockid=<optimized out>, mutex=0x563b020bf8d0, cond=0x563b020cf160) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x563b020cf160, mutex=0x563b020bf8d0, abstime=0x14615fdfdd40) at pthread_cond_wait.c:656
#3  0x0000563b00464c82 in __gthread_cond_timedwait (__abs_timeout=0x14615fdfdd40, __mutex=<optimized out>, __cond=0x563b020cf160) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#4  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>..., __lock=..., this=0x563b020cf160) at /usr/include/c++/9/condition_variable:188
#5  std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x563b020cf160) at /usr/include/c++/9/condition_variable:121
#6  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x563b020cf160) at /usr/include/c++/9/condition_variable:152
#7  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x563b020bf7c0, lk=..., thread_data=thread_data@entry=0x563b020cf160) at /test/10.5_opt/tpool/tpool_generic.cc:446
#8  0x0000563b00464fc1 in tpool::thread_pool_generic::get_task (this=0x563b020bf7c0, thread_var=0x563b020cf160, t=0x14615fdfdde8) at /test/10.5_opt/tpool/tpool_generic.cc:499
#9  0x0000563b00466625 in tpool::thread_pool_generic::worker_main (this=0x563b020bf7c0, thread_var=0x563b020cf160) at /test/10.5_opt/tpool/tpool_generic.cc:544
#10 0x00001461945bdde4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 30 (Thread 0x1461905fc700 (LWP 1372719)):
#0  0x00001461942bbaff in __GI___poll (fds=fds@entry=0x1461905fb9b8, nfds=nfds@entry=1, timeout=timeout@entry=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x0000563b000b2aee in poll (__timeout=28800000, __nfds=1, __fds=0x1461905fb9b8) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  vio_io_wait (vio=vio@entry=0x563b025b0348, event=<optimized out>, timeout=28800000) at /test/10.5_opt/vio/viosocket.c:1000
#3  0x0000563b000b2c64 in vio_socket_io_wait (vio=0x563b025b0348, event=<optimized out>) at /test/10.5_opt/vio/viosocket.c:118
#4  0x0000563b000b2d43 in vio_read (vio=0x563b025b0348, buf=0x146150008058 "\001", size=4) at /test/10.5_opt/vio/viosocket.c:199
#5  0x0000563b00031d65 in my_real_read (net=0x146150000ed8, complen=complen@entry=0x1461905fbc38, header=header@entry=1 '\001') at /test/10.5_opt/sql/net_serv.cc:991
#6  0x0000563b00033118 in my_net_read_packet_reallen (net=net@entry=0x146150000ed8, read_from_server=read_from_server@entry=1 '\001', reallen=reallen@entry=0x1461905fbc88) at /test/10.5_opt/sql/net_serv.cc:1272
#7  0x0000563b00033221 in my_net_read_packet (net=net@entry=0x146150000ed8, read_from_server=read_from_server@entry=1 '\001') at /test/10.5_opt/sql/net_serv.cc:1256
#8  0x0000563affce9e12 in do_command (thd=0x146150000c58) at /test/10.5_opt/sql/sql_parse.cc:1234
#9  0x0000563affdf02e1 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x563b027804b8, put_in_cache=put_in_cache@entry=true) at /test/10.5_opt/sql/sql_connect.cc:1418
#10 0x0000563affdf075d in handle_one_connection (arg=arg@entry=0x563b027804b8) at /test/10.5_opt/sql/sql_connect.cc:1312
#11 0x0000563b0017f5c9 in pfs_spawn_thread (arg=0x563b025b0348) at /test/10.5_opt/storage/perfschema/pfs.cc:2201
#12 0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 29 (Thread 0x14615d4b3700 (LWP 359400)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04ef80) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04ef90, cond=0x14613c04ef58) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04ef58, mutex=mutex@entry=0x14613c04ef90) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04ef90, that=0x14613c04ef58) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04ef38) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 28 (Thread 0x14615d6b4700 (LWP 359399)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04eeb0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04eec0, cond=0x14613c04ee88) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04ee88, mutex=mutex@entry=0x14613c04eec0) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04eec0, that=0x14613c04ee88) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04ee68) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 27 (Thread 0x14615d8b5700 (LWP 359398)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04ede0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04edf0, cond=0x14613c04edb8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04edb8, mutex=mutex@entry=0x14613c04edf0) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04edf0, that=0x14613c04edb8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04ed98) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 26 (Thread 0x14615dab6700 (LWP 359397)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04ed10) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04ed20, cond=0x14613c04ece8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04ece8, mutex=mutex@entry=0x14613c04ed20) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04ed20, that=0x14613c04ece8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04ecc8) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 25 (Thread 0x14615dcb7700 (LWP 359396)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04ec40) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04ec50, cond=0x14613c04ec18) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04ec18, mutex=mutex@entry=0x14613c04ec50) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04ec50, that=0x14613c04ec18) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04ebf8) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 24 (Thread 0x14615deb8700 (LWP 359395)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04eb70) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04eb80, cond=0x14613c04eb48) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04eb48, mutex=mutex@entry=0x14613c04eb80) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04eb80, that=0x14613c04eb48) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04eb28) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 23 (Thread 0x14615e0b9700 (LWP 359394)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04eaa0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04eab0, cond=0x14613c04ea78) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04ea78, mutex=mutex@entry=0x14613c04eab0) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04eab0, that=0x14613c04ea78) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04ea58) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 22 (Thread 0x14615e2ba700 (LWP 359393)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e9d0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e9e0, cond=0x14613c04e9a8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e9a8, mutex=mutex@entry=0x14613c04e9e0) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04e9e0, that=0x14613c04e9a8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04e988) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 21 (Thread 0x14615e4bb700 (LWP 359392)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e900) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e910, cond=0x14613c04e8d8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e8d8, mutex=mutex@entry=0x14613c04e910) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04e910, that=0x14613c04e8d8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04e8b8) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 20 (Thread 0x14615e6bc700 (LWP 359391)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e830) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e840, cond=0x14613c04e808) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e808, mutex=mutex@entry=0x14613c04e840) at pthread_cond_wait.c:638
#3  0x000014615fb5d4b8 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10543, mutex=0x14613c04e840, that=0x14613c04e808) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_crd_action (arg=0x14613c04e7e8) at /test/10.5_opt/storage/spider/spd_table.cc:10543
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 19 (Thread 0x14615e8bd700 (LWP 359390)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e760) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e770, cond=0x14613c04e738) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e738, mutex=mutex@entry=0x14613c04e770) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e770, that=0x14613c04e738) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e718) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 18 (Thread 0x14615eabe700 (LWP 359389)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e690) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e6a0, cond=0x14613c04e668) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e668, mutex=mutex@entry=0x14613c04e6a0) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e6a0, that=0x14613c04e668) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e648) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 17 (Thread 0x14615ecbf700 (LWP 359388)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e5c0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e5d0, cond=0x14613c04e598) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e598, mutex=mutex@entry=0x14613c04e5d0) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e5d0, that=0x14613c04e598) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e578) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 16 (Thread 0x14615eec0700 (LWP 359387)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e4f0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e500, cond=0x14613c04e4c8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e4c8, mutex=mutex@entry=0x14613c04e500) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e500, that=0x14613c04e4c8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e4a8) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 15 (Thread 0x14615f0c1700 (LWP 359386)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e420) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e430, cond=0x14613c04e3f8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e3f8, mutex=mutex@entry=0x14613c04e430) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e430, that=0x14613c04e3f8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e3d8) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 14 (Thread 0x14615f2c2700 (LWP 359385)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e350) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e360, cond=0x14613c04e328) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e328, mutex=mutex@entry=0x14613c04e360) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e360, that=0x14613c04e328) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e308) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 13 (Thread 0x14615f4c3700 (LWP 359383)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e280) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e290, cond=0x14613c04e258) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e258, mutex=mutex@entry=0x14613c04e290) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e290, that=0x14613c04e258) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e238) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 12 (Thread 0x14615f6c4700 (LWP 359382)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e1b0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e1c0, cond=0x14613c04e188) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e188, mutex=mutex@entry=0x14613c04e1c0) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e1c0, that=0x14613c04e188) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e168) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 11 (Thread 0x14615f8c5700 (LWP 359381)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e0e0) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e0f0, cond=0x14613c04e0b8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04e0b8, mutex=mutex@entry=0x14613c04e0f0) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e0f0, that=0x14613c04e0b8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04e098) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 10 (Thread 0x14615fac6700 (LWP 359380)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x14613c04e010) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14613c04e020, cond=0x14613c04dfe8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x14613c04dfe8, mutex=mutex@entry=0x14613c04e020) at pthread_cond_wait.c:638
#3  0x000014615fb5c388 in inline_mysql_cond_wait (src_file=0x14615fbcb640 "/test/10.5_opt/storage/spider/spd_table.cc", src_line=10391, mutex=0x14613c04e020, that=0x14613c04dfe8) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  spider_table_bg_sts_action (arg=0x14613c04dfc8) at /test/10.5_opt/storage/spider/spd_table.cc:10391
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 9 (Thread 0x146190647700 (LWP 346951)):
#0  __lll_lock_wait (futex=futex@entry=0x14613c071a60, private=0) at lowlevellock.c:52
#1  0x00001461946dd131 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x14613c071a60) at ../nptl/pthread_mutex_lock.c:115
#2  0x000014615fb32a11 in inline_mysql_mutex_lock (src_file=0x14615fbcb2a0 "/test/10.5_opt/storage/spider/spd_db_conn.cc", src_line=6727, that=0x14613c071a60) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:767
#3  spider_db_bulk_insert (spider=spider@entry=0x14613c0590c0, table=0x14613c039568, copy_info=copy_info@entry=0x14613c0594b0, bulk_end=bulk_end@entry=true) at /test/10.5_opt/storage/spider/spd_db_conn.cc:6727
#4  0x000014615fb80424 in ha_spider::end_bulk_insert (this=0x14613c0590c0) at /test/10.5_opt/storage/spider/ha_spider.cc:9896
#5  ha_spider::end_bulk_insert (this=0x14613c0590c0) at /test/10.5_opt/storage/spider/ha_spider.cc:9887
#6  0x0000563affca91cd in select_insert::abort_result_set (this=0x14613c011b98) at /test/10.5_opt/sql/sql_insert.cc:4282
#7  0x0000563affd522c2 in handle_select (thd=thd@entry=0x14613c000c58, lex=lex@entry=0x14613c004b78, result=result@entry=0x14613c011b98, setup_tables_done_option=setup_tables_done_option@entry=1073741824) at /test/10.5_opt/sql/sql_select.cc:462
#8  0x0000563affcf001d in mysql_execute_command (thd=0x14613c000c58) at /test/10.5_opt/sql/sql_parse.cc:4770
#9  0x0000563affcdb143 in mysql_parse (thd=0x14613c000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_parse.cc:8100
#10 0x0000563affce7925 in dispatch_command (command=COM_QUERY, thd=0x14613c000c58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>, is_next_command=<optimized out>) at /test/10.5_opt/sql/sql_class.h:1290
#11 0x0000563affce9eb2 in do_command (thd=0x14613c000c58) at /test/10.5_opt/sql/sql_parse.cc:1370
#12 0x0000563affdf02e1 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x563b0273a008, put_in_cache=put_in_cache@entry=true) at /test/10.5_opt/sql/sql_connect.cc:1418
#13 0x0000563affdf075d in handle_one_connection (arg=arg@entry=0x563b0273a008) at /test/10.5_opt/sql/sql_connect.cc:1312
#14 0x0000563b0017f5c9 in pfs_spawn_thread (arg=0x563b026ef738) at /test/10.5_opt/storage/perfschema/pfs.cc:2201
#15 0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#16 0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 8 (Thread 0x146190692700 (LWP 346912)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x563b00e521a8 <COND_manager+40>) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x563b00e521c0 <LOCK_manager>, cond=0x563b00e52180 <COND_manager>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x563b00e52180 <COND_manager>, mutex=mutex@entry=0x563b00e521c0 <LOCK_manager>) at pthread_cond_wait.c:638
#3  0x0000563affcd9463 in inline_mysql_cond_wait (that=0x563b00e52180 <COND_manager>, mutex=0x563b00e521c0 <LOCK_manager>, src_file=0x563b0065e9a0 "/test/10.5_opt/sql/sql_manager.cc", src_line=103) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1233
#4  handle_manager (arg=arg@entry=0x0) at /test/10.5_opt/sql/sql_manager.cc:103
#5  0x0000563b0017f5c9 in pfs_spawn_thread (arg=0x563b02747b78) at /test/10.5_opt/storage/perfschema/pfs.cc:2201
#6  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 7 (Thread 0x1461906dd700 (LWP 346909)):
#0  0x00001461941ed322 in __GI___sigtimedwait (set=set@entry=0x1461906dccb0, info=info@entry=0x1461906dcd30, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:29
#1  0x00001461941ed38b in __GI___sigwaitinfo (set=set@entry=0x1461906dccb0, info=info@entry=0x1461906dcd30) at ../sysdeps/unix/sysv/linux/sigwaitinfo.c:25
#2  0x0000563affc12cd4 in my_sigwait (code=<synthetic pointer>, sig=0x1461906dcca8, set=0x1461906dccb0) at /test/10.5_opt/include/my_pthread.h:196
#3  signal_hand (arg=arg@entry=0x0) at /test/10.5_opt/sql/mysqld.cc:2960
#4  0x0000563b0017f5c9 in pfs_spawn_thread (arg=0x563b020095c8) at /test/10.5_opt/storage/perfschema/pfs.cc:2201
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 6 (Thread 0x14615ffff700 (LWP 346906)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x14615fffed40, clockid=<optimized out>, expected=0, futex_word=0x563b020cf20c) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x14615fffed40, clockid=<optimized out>, mutex=0x563b020bf8d0, cond=0x563b020cf1e0) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x563b020cf1e0, mutex=0x563b020bf8d0, abstime=0x14615fffed40) at pthread_cond_wait.c:656
#3  0x0000563b00464c82 in __gthread_cond_timedwait (__abs_timeout=0x14615fffed40, __mutex=<optimized out>, __cond=0x563b020cf1e0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872
#4  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>..., __lock=..., this=0x563b020cf1e0) at /usr/include/c++/9/condition_variable:188
#5  std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x563b020cf1e0) at /usr/include/c++/9/condition_variable:121
#6  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x563b020cf1e0) at /usr/include/c++/9/condition_variable:152
#7  tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x563b020bf7c0, lk=..., thread_data=thread_data@entry=0x563b020cf1e0) at /test/10.5_opt/tpool/tpool_generic.cc:446
#8  0x0000563b00464fc1 in tpool::thread_pool_generic::get_task (this=0x563b020bf7c0, thread_var=0x563b020cf1e0, t=0x14615fffede8) at /test/10.5_opt/tpool/tpool_generic.cc:499
#9  0x0000563b00466625 in tpool::thread_pool_generic::worker_main (this=0x563b020bf7c0, thread_var=0x563b020cf1e0) at /test/10.5_opt/tpool/tpool_generic.cc:544
#10 0x00001461945bdde4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 5 (Thread 0x1461904dd700 (LWP 346899)):
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x563b00dcd118 <buf_pool+856>) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x563b00dcd080 <buf_pool+704>, cond=0x563b00dcd0f0 <buf_pool+816>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=cond@entry=0x563b00dcd0f0 <buf_pool+816>, mutex=mutex@entry=0x563b00dcd080 <buf_pool+704>) at pthread_cond_wait.c:638
#3  0x0000563b003ada68 in buf_flush_page_cleaner () at /test/10.5_opt/storage/innobase/buf/buf0flu.cc:2238
#4  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 4 (Thread 0x1461918de700 (LWP 346896)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000563b00468358 in my_getevents (min_nr=1, nr=256, ev=0x1461918dbde0, ctx=<optimized out>) at /test/10.5_opt/tpool/aio_linux.cc:107
#2  tpool::aio_linux::getevent_thread_routine (aio=0x563b02107600) at /test/10.5_opt/tpool/aio_linux.cc:107
#3  0x00001461945bdde4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 3 (Thread 0x146191f01700 (LWP 346792)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x146191f00d30, clockid=<optimized out>, expected=0, futex_word=0x563b01684ea8 <COND_checkpoint+40>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x146191f00d30, clockid=<optimized out>, mutex=0x563b01684ec0 <LOCK_checkpoint>, cond=0x563b01684e80 <COND_checkpoint>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x563b01684e80 <COND_checkpoint>, mutex=mutex@entry=0x563b01684ec0 <LOCK_checkpoint>, abstime=abstime@entry=0x146191f00d30) at pthread_cond_wait.c:656
#3  0x0000563b00115621 in inline_mysql_cond_timedwait (src_file=0x563b007d1030 "/test/10.5_opt/storage/maria/ma_servicethread.c", src_line=115, abstime=0x146191f00d30, mutex=0x563b01684ec0 <LOCK_checkpoint>, that=0x563b01684e80 <COND_checkpoint>) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1270
#4  my_service_thread_sleep (control=control@entry=0x563b00dbf640 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /test/10.5_opt/storage/maria/ma_servicethread.c:115
#5  0x0000563b0010d6ff in ma_checkpoint_background (arg=0x1e) at /test/10.5_opt/storage/maria/ma_checkpoint.c:707
#6  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#7  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 2 (Thread 0x146193e94700 (LWP 346778)):
#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x146193e93db0, clockid=<optimized out>, expected=0, futex_word=0x563b01727588 <COND_timer+40>) at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x146193e93db0, clockid=<optimized out>, mutex=0x563b017275a0 <LOCK_timer>, cond=0x563b01727560 <COND_timer>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=cond@entry=0x563b01727560 <COND_timer>, mutex=mutex@entry=0x563b017275a0 <LOCK_timer>, abstime=abstime@entry=0x146193e93db0) at pthread_cond_wait.c:656
#3  0x0000563b004d5dd4 in inline_mysql_cond_timedwait (that=0x563b01727560 <COND_timer>, mutex=0x563b017275a0 <LOCK_timer>, src_file=0x563b0082ac68 "/test/10.5_opt/mysys/thr_timer.c", src_line=321, abstime=0x146193e93db0) at /test/10.5_opt/include/mysql/psi/mysql_thread.h:1270
#4  timer_handler (arg=<optimized out>) at /test/10.5_opt/mysys/thr_timer.c:321
#5  0x00001461946da609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00001461942c8293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
Thread 1 (Thread 0x14619400f800 (LWP 346773)):
#0  0x00001461942bbaff in __GI___poll (fds=fds@entry=0x7ffc946064f0, nfds=nfds@entry=2, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00001461942d8aa2 in __poll_chk (fds=fds@entry=0x7ffc946064f0, nfds=nfds@entry=2, timeout=timeout@entry=-1, fdslen=fdslen@entry=24) at poll_chk.c:27
#2  0x0000563affc1b08f in poll (__timeout=-1, __nfds=2, __fds=0x7ffc946064f0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
#3  handle_connections_sockets () at /test/10.5_opt/sql/mysqld.cc:6207
#4  0x0000563affc1beb5 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/10.5_opt/sql/mysqld.cc:5656
#5  0x00001461941cd0b3 in __libc_start_main (main=0x563affbdf970 <main(int, char**)>, argc=10, argv=0x7ffc94606958, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc94606948) at ../csu/libc-start.c:308
#6  0x0000563affc1071e in _start () at /test/10.5_opt/sql/mysqld.cc:4321

Bug confirmed present in:
MariaDB: 10.5.13 (dbg), 10.5.13 (opt), 10.6.5 (dbg), 10.6.5 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.2.41 (dbg), 10.2.41 (opt), 10.3.32 (dbg), 10.3.32 (opt), 10.4.22 (dbg), 10.4.22 (opt)
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.35 (dbg), 5.7.35 (opt), 8.0.26 (dbg), 8.0.26 (opt)



 Comments   
Comment by Roel Van de Paar [ 2021-09-10 ]

Unique ID's filtered for this bug

SIGSEGV|spider_db_bulk_insert|ha_spider::end_bulk_insert|ha_spider::end_bulk_insert|select_insert::abort_result_set
SIGSEGV|spider_db_bulk_insert|ha_spider::end_bulk_insert|handler::ha_end_bulk_insert|select_insert::abort_result_set

And the following was filtered already due to MDEV-26546

SIGSEGV|spider_db_connect|spider_db_conn_queue_action|spider_db_before_query|spider_db_set_names_internal

Comment by Roel Van de Paar [ 2021-09-10 ]

INSTALL PLUGIN spider SONAME 'ha_spider.so';
CREATE TABLE t (c INT) ENGINE=SPIDER;
CREATE TABLE t0 (a INT) ENGINE=SPIDER;
INSERT INTO t0 VALUES (0);
SELECT MATCH (a) AGAINST (0x0) FROM t;
INSERT INTO t0 SELECT * FROM t0;

Leads to:

10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Debug)

Core was generated by `/test/MD040921-mariadb-10.7.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000014ea764c5248 in spider_db_bulk_insert (
    spider=spider@entry=0x14ea54079d20, table=0x14ea5405fac8, 
    copy_info=copy_info@entry=0x14ea5407a110, bulk_end=bulk_end@entry=true)
    at /test/10.7_dbg/storage/spider/spd_db_conn.cc:6708
[Current thread is 1 (Thread 0x14ea848bd700 (LWP 3898423))]
(gdb) bt
#0  0x000014ea764c5248 in spider_db_bulk_insert (spider=spider@entry=0x14ea54079d20, table=0x14ea5405fac8, copy_info=copy_info@entry=0x14ea5407a110, bulk_end=bulk_end@entry=true) at /test/10.7_dbg/storage/spider/spd_db_conn.cc:6708
#1  0x000014ea7653677a in ha_spider::end_bulk_insert (this=0x14ea54079d20) at /test/10.7_dbg/storage/spider/ha_spider.cc:9896
#2  0x0000564de0b98d66 in handler::ha_end_bulk_insert (this=0x14ea54079d20) at /test/10.7_dbg/sql/handler.cc:4990
#3  0x0000564de08412fe in select_insert::abort_result_set (this=0x14ea540159c0) at /test/10.7_dbg/sql/sql_insert.cc:4294
#4  0x0000564de0924d7d in handle_select (thd=thd@entry=0x14ea54000db8, lex=lex@entry=0x14ea54005058, result=result@entry=0x14ea540159c0, setup_tables_done_option=setup_tables_done_option@entry=1073741824) at /test/10.7_dbg/sql/sql_select.cc:563
#5  0x0000564de089589b in mysql_execute_command (thd=thd@entry=0x14ea54000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.7_dbg/sql/sql_parse.cc:4711
#6  0x0000564de087fccb in mysql_parse (thd=thd@entry=0x14ea54000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14ea848bc400) at /test/10.7_dbg/sql/sql_parse.cc:8036
#7  0x0000564de088e8d0 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14ea54000db8, packet=packet@entry=0x14ea5400b739 "INSERT INTO t0 SELECT * FROM t0", packet_length=packet_length@entry=31, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_class.h:1358
#8  0x0000564de0891cd6 in do_command (thd=0x14ea54000db8, blocking=blocking@entry=true) at /test/10.7_dbg/sql/sql_parse.cc:1404
#9  0x0000564de0a080c8 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x564de3d9b548, put_in_cache=put_in_cache@entry=true) at /test/10.7_dbg/sql/sql_connect.cc:1418
#10 0x0000564de0a086cd in handle_one_connection (arg=arg@entry=0x564de3d9b548) at /test/10.7_dbg/sql/sql_connect.cc:1312
#11 0x0000564de0e71ade in pfs_spawn_thread (arg=0x564de3c9a348) at /test/10.7_dbg/storage/perfschema/pfs.cc:2201
#12 0x000014ea9ac56609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x000014ea9a844293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

And this new stack in 10.7 optimized:

10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Optimized)

Core was generated by `/test/MD040921-mariadb-10.7.0-linux-x86_64-opt/bin/mysqld --no-defaults --core-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000014e09152fcc5 in spider_db_before_query (need_mon=<optimized out>, 
    conn=0x14e06805f198) at /test/10.7_opt/storage/spider/spd_db_conn.cc:672
[Current thread is 1 (Thread 0x14e0a0c59700 (LWP 1891291))]
(gdb) bt
#0  0x000014e09152fcc5 in spider_db_before_query (need_mon=<optimized out>, conn=0x14e06805f198) at /test/10.7_opt/storage/spider/spd_db_conn.cc:672
#1  spider_db_before_query (conn=0x14e06805f198, need_mon=<optimized out>) at /test/10.7_opt/storage/spider/spd_db_conn.cc:615
#2  0x000014e09152fea2 in spider_db_set_names_internal (trx=0x14e06803f078, share=0x14e0680572b8, conn=conn@entry=0x14e06805f198, all_link_idx=0, need_mon=0x14e06805dad8) at /test/10.7_opt/storage/spider/spd_db_conn.cc:930
#3  0x000014e091530059 in spider_db_set_names (spider=spider@entry=0x14e068023830, conn=conn@entry=0x14e06805f198, link_idx=link_idx@entry=0) at /test/10.7_opt/storage/spider/spd_db_conn.cc:977
#4  0x000014e0915346a9 in spider_db_bulk_insert (spider=spider@entry=0x14e068023830, table=0x14e068027b68, copy_info=copy_info@entry=0x14e068023c20, bulk_end=bulk_end@entry=true) at /test/10.7_opt/storage/spider/spd_db_conn.cc:6745
#5  0x000014e09157e474 in ha_spider::end_bulk_insert (this=0x14e068023830) at /test/10.7_opt/storage/spider/ha_spider.cc:9896
#6  ha_spider::end_bulk_insert (this=0x14e068023830) at /test/10.7_opt/storage/spider/ha_spider.cc:9887
#7  0x00005607d689897d in select_insert::abort_result_set (this=0x14e0680124e0) at /test/10.7_opt/sql/sql_insert.cc:4294
#8  0x00005607d694d152 in handle_select (thd=thd@entry=0x14e068000c58, lex=lex@entry=0x14e068004d38, result=result@entry=0x14e0680124e0, setup_tables_done_option=setup_tables_done_option@entry=1073741824) at /test/10.7_opt/sql/sql_select.cc:563
#9  0x00005607d68df001 in mysql_execute_command (thd=0x14e068000c58, is_called_from_prepared_stmt=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:4711
#10 0x00005607d68cb456 in mysql_parse (thd=0x14e068000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/10.7_opt/sql/sql_parse.cc:8036
#11 0x00005607d68d7345 in dispatch_command (command=COM_QUERY, thd=0x14e068000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/10.7_opt/sql/sql_class.h:1358
#12 0x00005607d68d9217 in do_command (thd=0x14e068000c58, blocking=blocking@entry=true) at /test/10.7_opt/sql/sql_parse.cc:1404
#13 0x00005607d69f4ae7 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/10.7_opt/sql/sql_connect.cc:1418
#14 0x00005607d69f4e2d in handle_one_connection (arg=arg@entry=0x5607d8770548) at /test/10.7_opt/sql/sql_connect.cc:1312
#15 0x00005607d6d48298 in pfs_spawn_thread (arg=0x5607d8c13048) at /test/10.7_opt/storage/perfschema/pfs.cc:2201
#16 0x000014e0b5fd9609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#17 0x000014e0b5bc7293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.5.13 (dbg), 10.5.13 (opt), 10.6.5 (dbg), 10.6.5 (opt), 10.7.0 (dbg), 10.7.0 (opt)

Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.2.41 (dbg), 10.2.41 (opt), 10.3.32 (dbg), 10.3.32 (opt), 10.4.22 (dbg), 10.4.22 (opt)
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.35 (dbg), 5.7.35 (opt), 8.0.26 (dbg), 8.0.26 (opt)

Adding filter

SIGSEGV|spider_db_before_query|spider_db_before_query|spider_db_set_names_internal|spider_db_set_names

Comment by Roel Van de Paar [ 2021-09-10 ]

It may be that the spider_db_before_query and spider_db_connect are different bugs. TBD.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-13 ]

I could slightly reduce the test case in the issue description. sql_mode and INSERT INTO t VALUES(0) seems to be fundamental for the test.

USE test;
INSTALL PLUGIN spider SONAME 'ha_spider.so';
CREATE TABLE t (i CHAR) ENGINE=SPIDER;
INSERT INTO t VALUES (0);
INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1;

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-13 ]

For the crash case, the problem here is that an invalid value is set to conn->dbton_id.

Thread 14 "mariadbd" hit Breakpoint 1, spider_db_bulk_insert (spider=0x7fc83408e390, table=0x7fc834075c48, copy_info=0x7fc83408e780, bulk_end=true)
    at /home/vagrant/repo/mariadb-server/storage/spider/spd_db_conn.cc:6708
6708              dbton_handler = spider->dbton_handler[conn->dbton_id];
...
(gdb) p *(spider->dbton_handler)@2
$12 = {0x7fc83409c5a0, 0x0}
(gdb) p conn->dbton_id
$13 = 2408550287

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-13 ]

I cannot reproduce the bug with MTR if I write a test according to the Spider test convention. However, the following one works (crashes):

CREATE DATABASE IF NOT EXISTS auto_test_local;
USE auto_test_local;
 
INSTALL PLUGIN spider SONAME 'ha_spider.so';
 
CREATE TABLE t (i CHAR) ENGINE=SPIDER;
--error 1429
INSERT INTO t VALUES (0);
--error 1054
INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1;
 
DROP DATABASE IF EXISTS auto_test_local;
 
--disable_query_log
--disable_result_log
--source ../../include/deinit_spider.inc
--enable_result_log
--enable_query_log

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-15 ]

I checked where the value of conn->dbton_id is changed. According to rr trace, it is changed by my_free.

It seems that Spider once frees a connection at the first insert, INSERT INTO t VALUES (0), and then reuses the connection for the second insert. As a result, almost all attributes, including dbton_id, of conn are invalid.

Thread 2 hit Breakpoint 1, spider_db_bulk_insert (spider=0x149f94049030, table=0x149f941a3ed8, copy_info=0x149f94049420, bulk_end=false)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_db_conn.cc:6708
6708              dbton_handler = spider->dbton_handler[conn->dbton_id];
(rr) watch -l conn->dbton_id
Hardware watchpoint 2: -location conn->dbton_id
(rr) p conn
$1 = (SPIDER_CONN *) 0x149f940715e8
(rr) bt
#0  spider_db_bulk_insert (spider=0x149f94049030, table=0x149f941a3ed8, copy_info=0x149f94049420, bulk_end=false) at /home/nayuta/repo/mariadb-server/storage/spider/spd_db_conn.cc:6708
#1  0x0000149f9bf1fab6 in ha_spider::write_row (this=0x149f94049030, buf=0x149f94048bc8 "\375\060", '\245' <repeats 14 times>, "\350\213\004\224\237\024")
    at /home/nayuta/repo/mariadb-server/storage/spider/ha_spider.cc:10039
#2  0x0000558a8d7ccb17 in handler::ha_write_row (this=0x149f94049030, buf=0x149f94048bc8 "\375\060", '\245' <repeats 14 times>, "\350\213\004\224\237\024")
    at /home/nayuta/repo/mariadb-server/sql/handler.cc:7168
#3  0x0000558a8d3fab47 in write_record (thd=0x149f94000db8, table=0x149f941a3ed8, info=0x14a00c05bbf0, sink=0x0) at /home/nayuta/repo/mariadb-server/sql/sql_insert.cc:2129
#4  0x0000558a8d3f783e in mysql_insert (thd=0x149f94000db8, table_list=0x149f940152c8, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR,
    ignore=false, result=0x0) at /home/nayuta/repo/mariadb-server/sql/sql_insert.cc:1118
#5  0x0000558a8d44b617 in mysql_execute_command (thd=0x149f94000db8) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:4624
#6  0x0000558a8d457112 in mysql_parse (thd=0x149f94000db8, rawbuf=0x149f940151e0 "INSERT INTO t VALUES (0)", length=24, parser_state=0x14a00c05c490, is_com_multi=false,
    is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:8100
#7  0x0000558a8d443085 in dispatch_command (command=COM_QUERY, thd=0x149f94000db8, packet=0x149f94093b79 "INSERT INTO t VALUES (0)", packet_length=24, is_com_multi=false,
    is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1891
#8  0x0000558a8d441877 in do_command (thd=0x149f94000db8) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1370
#9  0x0000558a8d5f2774 in do_handle_one_connection (connect=0x558a91405ab8, put_in_cache=true) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1418
#10 0x0000558a8d5f2427 in handle_one_connection (arg=0x558a913eccf8) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1312
#11 0x0000558a8db17c65 in pfs_spawn_thread (arg=0x558a9135d808) at /home/nayuta/repo/mariadb-server/storage/perfschema/pfs.cc:2201
#12 0x000014a0352b0609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x000014a034e84293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(rr) c
Continuing.
 
Thread 2 hit Hardware watchpoint 2: -location conn->dbton_id
 
Old value = 0
New value = 143
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151     ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(rr)
Continuing.
 
Thread 2 hit Hardware watchpoint 2: -location conn->dbton_id
 
Old value = 143
New value = 36751
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151     in ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
(rr)
Continuing.
 
Thread 2 hit Hardware watchpoint 2: -location conn->dbton_id
 
Old value = 36751
New value = 9408399
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151     in ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
(rr)
Continuing.
 
Thread 2 hit Hardware watchpoint 2: -location conn->dbton_id
 
Old value = 9408399
New value = 2408550287
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151     in ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
(rr) bt
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
#1  0x0000558a8e0ea906 in free_memory (ptr=0x149f940715c0) at /home/nayuta/repo/mariadb-server/mysys/safemalloc.c:279
#2  0x0000558a8e0ea5bb in sf_free (ptr=0x149f940715c0) at /home/nayuta/repo/mariadb-server/mysys/safemalloc.c:198
#3  0x0000558a8e0d7896 in my_free (ptr=0x149f940715d8) at /home/nayuta/repo/mariadb-server/mysys/my_malloc.c:211
#4  0x0000149f9beebccf in spider_free_mem (trx=0x149f9403f988, ptr=0x149f940715e8, my_flags=0) at /home/nayuta/repo/mariadb-server/storage/spider/spd_malloc.cc:188
#5  0x0000149f9be8d539 in spider_free_conn (conn=0x149f940715e8) at /home/nayuta/repo/mariadb-server/storage/spider/spd_conn.cc:1404
#6  0x0000149f9be8b1b7 in spider_free_conn_from_trx (trx=0x149f9403f988, conn=0x149f940715e8, another=false, trx_free=false, roop_count=0x14a00c05bba4)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_conn.cc:420
#7  0x0000149f9be422a5 in spider_free_trx_conn (trx=0x149f9403f988, trx_free=false) at /home/nayuta/repo/mariadb-server/storage/spider/spd_trx.cc:117
#8  0x0000149f9be4d445 in spider_rollback (hton=0x149f941a2c78, thd=0x149f94000db8, all=false) at /home/nayuta/repo/mariadb-server/storage/spider/spd_trx.cc:3559
#9  0x0000558a8d7bb45a in ha_rollback_trans (thd=0x149f94000db8, all=false) at /home/nayuta/repo/mariadb-server/sql/handler.cc:2068
#10 0x0000558a8d60ef09 in trans_rollback_stmt (thd=0x149f94000db8) at /home/nayuta/repo/mariadb-server/sql/transaction.cc:535
#11 0x0000558a8d451526 in mysql_execute_command (thd=0x149f94000db8) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:6109
#12 0x0000558a8d457112 in mysql_parse (thd=0x149f94000db8, rawbuf=0x149f940151e0 "INSERT INTO t VALUES (0)", length=24, parser_state=0x14a00c05c490, is_com_multi=false,
    is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:8100
#13 0x0000558a8d443085 in dispatch_command (command=COM_QUERY, thd=0x149f94000db8, packet=0x149f94093b79 "INSERT INTO t VALUES (0)", packet_length=24, is_com_multi=false,
    is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1891
#14 0x0000558a8d441877 in do_command (thd=0x149f94000db8) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1370
#15 0x0000558a8d5f2774 in do_handle_one_connection (connect=0x558a91405ab8, put_in_cache=true) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1418
#16 0x0000558a8d5f2427 in handle_one_connection (arg=0x558a913eccf8) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1312
#17 0x0000558a8db17c65 in pfs_spawn_thread (arg=0x558a9135d808) at /home/nayuta/repo/mariadb-server/storage/perfschema/pfs.cc:2201
#18 0x000014a0352b0609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#19 0x000014a034e84293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(rr) c
Continuing.
 
Thread 2 hit Breakpoint 1, spider_db_bulk_insert (spider=0x149f94049030, table=0x149f941a3ed8, copy_info=0x149f94049420, bulk_end=true)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_db_conn.cc:6708
6708              dbton_handler = spider->dbton_handler[conn->dbton_id];
(rr) p conn
$2 = (SPIDER_CONN *) 0x149f940715e8
(rr) bt
#0  spider_db_bulk_insert (spider=0x149f94049030, table=0x149f941a3ed8, copy_info=0x149f94049420, bulk_end=true) at /home/nayuta/repo/mariadb-server/storage/spider/spd_db_conn.cc:6708
#1  0x0000149f9bf1f3f4 in ha_spider::end_bulk_insert (this=0x149f94049030) at /home/nayuta/repo/mariadb-server/storage/spider/ha_spider.cc:9896
#2  0x0000558a8d7c4c85 in handler::ha_end_bulk_insert (this=0x149f94049030) at /home/nayuta/repo/mariadb-server/sql/handler.cc:4670
#3  0x0000558a8d401338 in select_insert::abort_result_set (this=0x149f94016998) at /home/nayuta/repo/mariadb-server/sql/sql_insert.cc:4282
#4  0x0000558a8d4900b1 in handle_select (thd=0x149f94000db8, lex=0x149f94004e98, result=0x149f94016998, setup_tables_done_option=1073741824)
    at /home/nayuta/repo/mariadb-server/sql/sql_select.cc:462
#5  0x0000558a8d44c02f in mysql_execute_command (thd=0x149f94000db8) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:4770
#6  0x0000558a8d457112 in mysql_parse (thd=0x149f94000db8, rawbuf=0x149f940151e0 "INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1", length=45, parser_state=0x14a00c05c490,
    is_com_multi=false, is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:8100
#7  0x0000558a8d443085 in dispatch_command (command=COM_QUERY, thd=0x149f94000db8, packet=0x149f94093b79 "INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1", packet_length=45,
    is_com_multi=false, is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1891
#8  0x0000558a8d441877 in do_command (thd=0x149f94000db8) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1370
#9  0x0000558a8d5f2774 in do_handle_one_connection (connect=0x558a91405ab8, put_in_cache=true) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1418
#10 0x0000558a8d5f2427 in handle_one_connection (arg=0x558a913eccf8) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1312
#11 0x0000558a8db17c65 in pfs_spawn_thread (arg=0x558a9135d808) at /home/nayuta/repo/mariadb-server/storage/perfschema/pfs.cc:2201
#12 0x000014a0352b0609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x000014a034e84293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-15 ]

A possible workaround for the bug is setting spider_conn_recycle_mode to 2 (do recycle).
https://mariadb.com/kb/en/spider-server-system-variables/#spider_conn_recycle_mode

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-15 ]

A possible fix would be to remove a freed connection from ha_spider::conns. Another possible fix would be to check if a connection is freed and to allocate a new one if the existing one is freed. In this case, we also need to mark the freed connection in some way.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-16 ]

I tested 10.4 HEAD. In 10.4, the connection is freed after the first INSERT, but it is reallocated before the second insert.

Thread 2 hit Hardware watchpoint 2: -location conn->dbton_id
 
Old value = 2768240640
New value = 0
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151     in ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S
(rr) bt
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
#1  0x00005651682bce32 in my_malloc (size=2560, my_flags=48) at /home/nayuta/repo/mariadb-server/mysys/my_malloc.c:127
#2  0x0000145980df5798 in spider_bulk_alloc_mem (trx=0x14595403eb30, id=18, func_name=0x145980ea3e4d "<unknown>",
    file_name=0x145980ea3ce0 "/home/nayuta/repo/mariadb-server/storage/spider/spd_conn.cc", line_no=478, my_flags=48)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_malloc.cc:236
#3  0x0000145980d9a5ef in spider_create_conn (share=0x145954067d90, spider=0x145954066058, link_idx=0, base_link_idx=0, conn_kind=1, error_num=0x1459ac05575c)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_conn.cc:478
#4  0x0000145980d9bb99 in spider_get_conn (share=0x145954067d90, link_idx=0, conn_key=0x145954074ca8 "0mysql", trx=0x14595403eb30, spider=0x145954066058,
    another=false, thd_chg=true, conn_kind=1, error_num=0x1459ac05575c) at /home/nayuta/repo/mariadb-server/storage/spider/spd_conn.cc:1080
#5  0x0000145980d60683 in spider_check_trx_and_get_conn (thd=0x145954000d90, spider=0x145954066058, use_conn_kind=true)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_trx.cc:3903
#6  0x0000145980e065ad in ha_spider::check_access_kind (this=0x145954066058, thd=0x145954000d90, write_request=true)
    at /home/nayuta/repo/mariadb-server/storage/spider/ha_spider.cc:952
#7  0x0000145980e06740 in ha_spider::store_lock (this=0x145954066058, thd=0x145954000d90, to=0x145954015db8, lock_type=TL_WRITE_CONCURRENT_INSERT)
    at /home/nayuta/repo/mariadb-server/storage/spider/ha_spider.cc:1020
#8  0x0000565167b5acc5 in get_lock_data (thd=0x145954000d90, table_ptr=0x145954015d90, count=1, flags=3) at /home/nayuta/repo/mariadb-server/sql/lock.cc:791
#9  0x0000565167b5957d in mysql_lock_tables (thd=0x145954000d90, tables=0x145954015d90, count=1, flags=0) at /home/nayuta/repo/mariadb-server/sql/lock.cc:298
#10 0x00005651675f72e0 in lock_tables (thd=0x145954000d90, tables=0x145954014790, count=1, flags=0) at /home/nayuta/repo/mariadb-server/sql/sql_base.cc:5588
#11 0x00005651675f6872 in open_and_lock_tables (thd=0x145954000d90, options=..., tables=0x145954014790, derived=true, flags=0, prelocking_strategy=0x1459ac055bb8)
    at /home/nayuta/repo/mariadb-server/sql/sql_base.cc:5326
#12 0x00005651675ae3b1 in open_and_lock_tables (thd=0x145954000d90, tables=0x145954014790, derived=true, flags=0)
    at /home/nayuta/repo/mariadb-server/sql/sql_base.h:503
#13 0x000056516769aa07 in mysql_execute_command (thd=0x145954000d90) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:4677
#14 0x00005651676a5bab in mysql_parse (thd=0x145954000d90, rawbuf=0x145954014688 "INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1", length=45,
    parser_state=0x1459ac056520, is_com_multi=false, is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:7995
#15 0x0000565167691eb3 in dispatch_command (command=COM_QUERY, thd=0x145954000d90, packet=0x14595400acf1 "INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1",
    packet_length=45, is_com_multi=false, is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1857
#16 0x0000565167690719 in do_command (thd=0x145954000d90) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1373
#17 0x00005651678227dc in do_handle_one_connection (connect=0x56516a2a4eb0) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1420
#18 0x0000565167822475 in handle_one_connection (arg=0x56516a2a4eb0) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1316
#19 0x00001459b7297609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x00001459b6e6d293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(rr) c
Continuing.
 
Thread 2 hit Breakpoint 1, spider_db_bulk_insert (spider=0x145954066058, table=0x1459540651f0, copy_info=0x145954066438, bulk_end=true)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_db_conn.cc:6237
6237              dbton_handler = spider->dbton_handler[conn->dbton_id];
(rr) bt
#0  spider_db_bulk_insert (spider=0x145954066058, table=0x1459540651f0, copy_info=0x145954066438, bulk_end=true)
    at /home/nayuta/repo/mariadb-server/storage/spider/spd_db_conn.cc:6237
#1  0x0000145980e28f6a in ha_spider::end_bulk_insert (this=0x145954066058) at /home/nayuta/repo/mariadb-server/storage/spider/ha_spider.cc:10042
#2  0x0000565167a14d25 in handler::ha_end_bulk_insert (this=0x145954066058) at /home/nayuta/repo/mariadb-server/sql/handler.cc:4441
#3  0x00005651676548ab in select_insert::abort_result_set (this=0x145954015de8) at /home/nayuta/repo/mariadb-server/sql/sql_insert.cc:4153
#4  0x00005651676dd044 in handle_select (thd=0x145954000d90, lex=0x145954004c00, result=0x145954015de8, setup_tables_done_option=1073741824)
    at /home/nayuta/repo/mariadb-server/sql/sql_select.cc:454
#5  0x000056516769abda in mysql_execute_command (thd=0x145954000d90) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:4712
#6  0x00005651676a5bab in mysql_parse (thd=0x145954000d90, rawbuf=0x145954014688 "INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1", length=45,
    parser_state=0x1459ac056520, is_com_multi=false, is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:7995
#7  0x0000565167691eb3 in dispatch_command (command=COM_QUERY, thd=0x145954000d90, packet=0x14595400acf1 "INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1",
    packet_length=45, is_com_multi=false, is_next_command=false) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1857
#8  0x0000565167690719 in do_command (thd=0x145954000d90) at /home/nayuta/repo/mariadb-server/sql/sql_parse.cc:1373
#9  0x00005651678227dc in do_handle_one_connection (connect=0x56516a2a4eb0) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1420
#10 0x0000565167822475 in handle_one_connection (arg=0x56516a2a4eb0) at /home/nayuta/repo/mariadb-server/sql/sql_connect.cc:1316
#11 0x00001459b7297609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00001459b6e6d293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-16 ]

The bug is introduced by the following commit:

commit e954d9de886aebc68c39240304fe97ae88276dbb
Author: Kentoku SHIBA <kentokushiba@gmail.com>
Date:   Tue Mar 3 02:50:40 2020 +0900
 
    MDEV-19002 Spider performance optimization with partition
 
    Change the following function for batch call instead of each partition
    - store_lock
    - external_lock
    - start_stmt
    - extra
    - cond_push
    - info_push
    - top_table

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-22 ]

The following patch fixes the bug but I'm not sure this is the right fix. https://github.com/MariaDB/server/commit/9ba6402b31caa8fcf3d1bbc658f792eb0898b504

Comment by Daniel Black [ 2021-09-23 ]

no, it look like your patch triggers an ASAN as it adds check_access_kind_for_connection to store_lock. A deeper fix around the memory management is needed.

from: https://buildbot.mariadb.org/#/builders/168/builds/5791/steps/8/logs/stdio

spider.auto_increment                    w11 [ retry-fail ]
        Test ended at 2021-09-22 23:17:48
CURRENT_TEST: spider.auto_increment
mysqltest: At line 100: query 'ALTER TABLE tbl_a MODIFY col_c MEDIUMINT NOT NULL DEFAULT 100' failed: 2013: Lost connection to MySQL server during query
The result from queries just before the failure was:
< snip >
) MASTER_1_ENGINE MASTER_1_AUTO_INCREMENT_2_1 MASTER_1_COMMENT_2_1
SHOW CREATE TABLE tbl_a;
Table	Create Table
tbl_a	CREATE TABLE `tbl_a` (
  `col_a` int(11) NOT NULL AUTO_INCREMENT,
  `col_b` varchar(20) DEFAULT 'defg',
  `col_c` int(11) NOT NULL DEFAULT 100,
  PRIMARY KEY (`col_a`)
) ENGINE=SPIDER AUTO_INCREMENT=20 DEFAULT CHARSET=latin1 COMMENT='database "auto_test_remote", table "tbl_a", srv "s_2_1", aim "0"'
INSERT INTO tbl_a () VALUES ();
INSERT INTO tbl_a () VALUES ();
SHOW CREATE TABLE tbl_a;
Table	Create Table
tbl_a	CREATE TABLE `tbl_a` (
  `col_a` int(11) NOT NULL AUTO_INCREMENT,
  `col_b` varchar(20) DEFAULT 'defg',
  `col_c` int(11) NOT NULL DEFAULT 100,
  PRIMARY KEY (`col_a`)
) ENGINE=SPIDER DEFAULT CHARSET=latin1 COMMENT='database "auto_test_remote", table "tbl_a", srv "s_2_1", aim "0"'
ALTER TABLE tbl_a MODIFY col_c MEDIUMINT NOT NULL DEFAULT 100;
More results from queries before failure can be found in /buildbot/amd64-ubuntu-1804-clang10-asan/build/mysql-test/var/11/log/auto_increment.log
CURRENT_TEST: spider.auto_increment
mysqltest: At line 100: query 'ALTER TABLE tbl_a MODIFY col_c MEDIUMINT NOT NULL DEFAULT 100' failed: 2013: Lost connection to MySQL server during query
The result from queries just before the failure was:
< snip >
) MASTER_1_ENGINE MASTER_1_AUTO_INCREMENT_2_1 MASTER_1_COMMENT_2_1
SHOW CREATE TABLE tbl_a;
Table	Create Table
tbl_a	CREATE TABLE `tbl_a` (
  `col_a` int(11) NOT NULL AUTO_INCREMENT,
  `col_b` varchar(20) DEFAULT 'defg',
  `col_c` int(11) NOT NULL DEFAULT 100,
  PRIMARY KEY (`col_a`)
) ENGINE=SPIDER AUTO_INCREMENT=20 DEFAULT CHARSET=latin1 COMMENT='database "auto_test_remote", table "tbl_a", srv "s_2_1", aim "0"'
INSERT INTO tbl_a () VALUES ();
INSERT INTO tbl_a () VALUES ();
SHOW CREATE TABLE tbl_a;
Table	Create Table
tbl_a	CREATE TABLE `tbl_a` (
  `col_a` int(11) NOT NULL AUTO_INCREMENT,
  `col_b` varchar(20) DEFAULT 'defg',
  `col_c` int(11) NOT NULL DEFAULT 100,
  PRIMARY KEY (`col_a`)
) ENGINE=SPIDER DEFAULT CHARSET=latin1 COMMENT='database "auto_test_remote", table "tbl_a", srv "s_2_1", aim "0"'
ALTER TABLE tbl_a MODIFY col_c MEDIUMINT NOT NULL DEFAULT 100;
More results from queries before failure can be found in /buildbot/amd64-ubuntu-1804-clang10-asan/build/mysql-test/var/11/log/auto_increment.log
Server [mysqld.1.1 - pid: 141624, winpid: 141624, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
$ /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd --defaults-group-suffix=.1.1 --defaults-file=/buildbot/amd64-ubuntu-1804-clang10-asan/build/mysql-test/var/11/my.cnf --log-output=file --loose-innodb --loose-skip-performance-schema --core-file --loose-debug-sync-timeout=300
2021-09-22 23:17:43 0 [Note] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd (mysqld 10.5.13-MariaDB-debug-log) starting as process 141625 ...
2021-09-22 23:17:43 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32246)
2021-09-22 23:17:43 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
2021-09-22 23:17:43 0 [Note] Plugin 'SEQUENCE' is disabled.
2021-09-22 23:17:43 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2021-09-22 23:17:43 0 [Note] InnoDB: Uses event mutexes
2021-09-22 23:17:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-09-22 23:17:43 0 [Note] InnoDB: Number of pools: 1
2021-09-22 23:17:43 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-09-22 23:17:43 0 [Note] mariadbd: O_TMPFILE is not supported on /buildbot/amd64-ubuntu-1804-clang10-asan/build/mysql-test/var/tmp/11/mysqld.1.1 (disabling future attempts)
2021-09-22 23:17:43 0 [Note] InnoDB: Using Linux native AIO
2021-09-22 23:17:43 0 [Note] InnoDB: Initializing buffer pool, total size = 8388608, chunk size = 8388608
2021-09-22 23:17:43 0 [Note] InnoDB: Completed initialization of buffer pool
2021-09-22 23:17:43 0 [Note] InnoDB: 128 rollback segments are active.
2021-09-22 23:17:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-09-22 23:17:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-09-22 23:17:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-09-22 23:17:43 0 [Note] InnoDB: 10.5.13 started; log sequence number 47834; transaction id 27
2021-09-22 23:17:43 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_K7Jd/11/mysqld.1.1/data/ib_buffer_pool
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_BUFFER_PAGE' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_TRX' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_CMP_PER_INDEX' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_METRICS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_CMP' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'THREAD_POOL_WAITS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'THREAD_POOL_QUEUES' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_FIELDS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_BUFFER_PAGE_LRU' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_LOCKS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_CMPMEM' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'THREAD_POOL_GROUPS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_FOREIGN_COLS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2021-09-22 23:17:43 0 [Note] InnoDB: Buffer pool(s) load completed at 210922 23:17:43
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_BUFFER_POOL_STATS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_FOREIGN' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_TABLES' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_COLUMNS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_VIRTUAL' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_INDEXES' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'user_variables' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'THREAD_POOL_STATS' is disabled.
2021-09-22 23:17:43 0 [Note] Plugin 'unix_socket' is disabled.
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown option '--loose-pam-debug'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown option '--loose-aria'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_port=20000'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_port_wr=20001'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_threads=2'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_threads_wr=1'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_support_merge_table=0'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_direct_update_mode=2'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_unlimited_boundary=65536'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_bulk_insert=0'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_bulk_insert_timeout=0'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_general_log=1'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_timeout=30'
2021-09-22 23:17:43 0 [Warning] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: unknown variable 'loose_handlersocket_close_table_interval=2'
2021-09-22 23:17:44 0 [Note] Server socket created on IP: '127.0.0.1'.
2021-09-22 23:17:44 0 [Note] Reading of all Master_info entries succeeded
2021-09-22 23:17:44 0 [Note] Added new Master_info '' to hash table
2021-09-22 23:17:44 0 [Note] /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/mariadbd: ready for connections.
Version: '10.5.13-MariaDB-debug-log'  socket: '/buildbot/amd64-ubuntu-1804-clang10-asan/build/mysql-test/var/tmp/11/mysqld.1.1.sock'  port: 16200  Source distribution
=================================================================
==141625==ERROR: AddressSanitizer: heap-use-after-free on address 0x61f0000517e8 at pc 0x7ff3d61390eb bp 0x7ff3d66e5910 sp 0x7ff3d66e5908
WRITE of size 1 at 0x61f0000517e8 thread T16
    #0 0x7ff3d61390ea in ha_spider::check_access_kind_for_connection(THD*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/ha_spider.cc:973:5
    #1 0x7ff3d6139907 in ha_spider::store_lock(THD*, st_thr_lock_data**, thr_lock_type) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/ha_spider.cc:1066:7
    #2 0x1622222 in get_lock_data(THD*, TABLE**, unsigned int, unsigned int) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/lock.cc:809:25
    #3 0x1621673 in mysql_lock_tables(THD*, TABLE**, unsigned int, unsigned int) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/lock.cc:298:20
    #4 0x94880d in lock_tables(THD*, TABLE_LIST*, unsigned int, unsigned int) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_base.cc:5512:23
    #5 0xd5322a in mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_table.cc:10786:7
    #6 0xed8f55 in Sql_cmd_alter_table::execute(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_alter.cc:539:11
    #7 0xad6da1 in mysql_execute_command(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:6056:26
    #8 0xac6aaf in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:8100:18
    #9 0xabf535 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:1891:7
    #10 0xac8020 in do_command(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:1370:17
    #11 0xec1083 in do_handle_one_connection(CONNECT*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_connect.cc:1418:11
    #12 0xec0ad2 in handle_one_connection /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_connect.cc:1312:5
    #13 0x1b3746e in pfs_spawn_thread /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/perfschema/pfs.cc:2201:3
    #14 0x7ff3e34f86da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #15 0x7ff3e287471e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)
0x61f0000517e8 is located 360 bytes inside of 3264-byte region [0x61f000051680,0x61f000052340)
freed by thread T16 here:
    #0 0x78c4ed in free /home/brian/src/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:123:3
    #1 0x283310b in my_free /buildbot/amd64-ubuntu-1804-clang10-asan/build/mysys/my_malloc.c:211:3
    #2 0x7ff3d610cd26 in spider_free_mem(st_spider_transaction*, void*, unsigned long) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_malloc.cc:188:3
    #3 0x7ff3d600d1cc in spider_free_conn(st_spider_conn*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_conn.cc:1404:3
    #4 0x7ff3d600c537 in spider_free_conn_from_trx(st_spider_transaction*, st_spider_conn*, bool, bool, int*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_conn.cc:420:9
    #5 0x7ff3d5f73b63 in spider_free_trx_conn(st_spider_transaction*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_trx.cc:117:9
    #6 0x7ff3d5f8faac in spider_commit(handlerton*, THD*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_trx.cc:3486:5
    #7 0x126d93b in commit_one_phase_2(THD*, bool, THD_TRANS*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/handler.cc:1956:17
    #8 0x126ea8f in ha_commit_one_phase(THD*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/handler.cc:1935:8
    #9 0x126b858 in ha_commit_trans(THD*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/handler.cc:1729:12
    #10 0xeffa3e in trans_commit_stmt(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/transaction.cc:472:10
    #11 0xade3a4 in mysql_execute_command(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:6116:7
    #12 0xac6aaf in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:8100:18
    #13 0xabf535 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:1891:7
    #14 0xac8020 in do_command(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:1370:17
    #15 0xec1083 in do_handle_one_connection(CONNECT*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_connect.cc:1418:11
    #16 0xec0ad2 in handle_one_connection /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_connect.cc:1312:5
previously allocated by thread T16 here:
    #0 0x78c76d in malloc /home/brian/src/final/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:145:3
    #1 0x28327ea in my_malloc /buildbot/amd64-ubuntu-1804-clang10-asan/build/mysys/my_malloc.c:90:29
    #2 0x7ff3d610d477 in spider_bulk_alloc_mem(st_spider_transaction*, unsigned int, char const*, char const*, unsigned long, unsigned long, ...) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_malloc.cc:236:29
    #3 0x7ff3d600df30 in spider_create_conn(st_spider_share*, ha_spider*, int, int, unsigned int, int*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_conn.cc:593:7
    #4 0x7ff3d601298a in spider_get_conn(st_spider_share*, int, char*, st_spider_transaction*, ha_spider*, bool, bool, unsigned int, int*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_conn.cc:1218:20
    #5 0x7ff3d5f927ca in spider_check_trx_and_get_conn(THD*, ha_spider*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/spd_trx.cc:3898:15
    #6 0x7ff3d6138b18 in ha_spider::check_access_kind_for_connection(THD*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/ha_spider.cc:954:20
    #7 0x7ff3d6139907 in ha_spider::store_lock(THD*, st_thr_lock_data**, thr_lock_type) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/ha_spider.cc:1066:7
    #8 0x1622222 in get_lock_data(THD*, TABLE**, unsigned int, unsigned int) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/lock.cc:809:25
    #9 0x1621673 in mysql_lock_tables(THD*, TABLE**, unsigned int, unsigned int) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/lock.cc:298:20
    #10 0x94880d in lock_tables(THD*, TABLE_LIST*, unsigned int, unsigned int) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_base.cc:5512:23
    #11 0x94b522 in open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_base.cc:5224:7
    #12 0xa02033 in open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_base.h:507:10
    #13 0xa02033 in mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool, select_result*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_insert.cc:757:9
    #14 0xae23bc in mysql_execute_command(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:4624:10
    #15 0xac6aaf in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:8100:18
    #16 0xabf535 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:1891:7
    #17 0xac8020 in do_command(THD*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_parse.cc:1370:17
    #18 0xec1083 in do_handle_one_connection(CONNECT*, bool) /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_connect.cc:1418:11
    #19 0xec0ad2 in handle_one_connection /buildbot/amd64-ubuntu-1804-clang10-asan/build/sql/sql_connect.cc:1312:5
Thread T16 created by T0 here:
    #0 0x77751a in pthread_create /home/brian/src/final/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:214:3
    #1 0x1b37770 in my_thread_create(unsigned long*, pthread_attr_t const*, void* (*)(void*), void*) /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/perfschema/my_thread.h:48:10
    #2 0x1b37770 in pfs_spawn_thread_v1 /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/perfschema/pfs.cc:2252:15
SUMMARY: AddressSanitizer: heap-use-after-free /buildbot/amd64-ubuntu-1804-clang10-asan/build/storage/spider/ha_spider.cc:973:5 in ha_spider::check_access_kind_for_connection(THD*, bool)

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-23 ]

danblack True. Thank you very much for your notice. I didn't check ASAN build. I will look at it deeper.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-28 ]

The following MTR test case does not fail on 10.5 HEAD.

--echo
--echo MDEV-26582 SIGSEGV in spider_db_bulk_insert and spider_db_connect and spider_db_before_query, and hang in "End of update loop" / "Reset for next command" query states
--echo
 
--disable_query_log
--disable_result_log
--source ../../include/init_spider.inc
--enable_result_log
--enable_query_log
 
CREATE SERVER s FOREIGN DATA WRAPPER mysql OPTIONS (USER 'root', HOST '127.0.0.1', PORT 16000, DATABASE 'auto_test_remote');
CREATE DATABASE auto_test_remote;
USE auto_test_remote;
CREATE TABLE t (i CHAR) ENGINE=InnoDB;
SET @@session.spider_same_server_link = ON;
 
CREATE DATABASE auto_test_local;
USE auto_test_local;
 
CREATE TABLE t (i CHAR) ENGINE=SPIDER COMMENT='srv "s", table "t"';
INSERT INTO t VALUES (0);
--error 1054
INSERT t SELECT 1 ON DUPLICATE KEY UPDATE c=1;
 
DROP DATABASE auto_test_remote;
DROP DATABASE auto_test_local;
 
--disable_query_log
--disable_result_log
--source ../../include/deinit_spider.inc
--enable_result_log
--enable_query_log

In the above test case, ha_spider::end_bulk_insert() does not access conn->dbton_id. The behavior seems to be correct because, at the call of ha_spider::end_bulk_insert(), the second INSERT is already appeared to be invalid (i.e. JOIN::prepare() returns an error) and there is no need to access the data nodes.

On the other hand, in the problematic case, ha_spider::end_bulk_insert() DOES access to conn->dbton_id. This is ultimately because spider_mbase_handler::is_bulk_insert_exec_period() returns TRUE.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-28 ]

This is just a guess but it seems to be assumed that the connection is allocated in ha_spider::dml_init(), which is called in the depth of JOIN::exec() and thus Spider should not touch the connection before the call of the function.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-28 ]

In the above test case with the proper connection string, (int) insert_sql.length() > insert_pos is FALSE and thus the function spider_mbase_handler::is_bulk_insert_exec_period() returns FALSE too.

In the problematic (crash) case, (int) insert_sql.length() > insert_pos is TRUE and the function returns TRUE too.

bool spider_mbase_handler::is_bulk_insert_exec_period(
  bool bulk_end
) {
  ...
  if (
    (bulk_end || (int) insert_sql.length() >= spider->bulk_size) &&
    (int) insert_sql.length() > insert_pos
  ) {
    DBUG_RETURN(TRUE);
  }
  DBUG_RETURN(FALSE);
}

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-29 ]

Spider does not reset insert_sql and insert_pos at the start of the second INSERT (or at the end of the first INSERT). Thus, spider_mbase_handler::is_bulk_insert_exec_period(), which is called by the second INSERT, refers the values set by the first INSERT. This is the reason why the function returns TRUE wrongly.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-29 ]

Another fix here: https://github.com/MariaDB/server/commit/d14adfb362d9dad2344c6cf522f0d427b38766b1

I am almost convinced that the fix is OK but let me check further.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-29 ]

serg Please review https://github.com/MariaDB/server/commit/34c7535df3048d4d0928dda04d42d7ce832967e3

Comment by Nayuta Yanagisawa (Inactive) [ 2021-10-18 ]

https://github.com/MariaDB/server/commit/5fcde24091c01ce0f5941d434bd5761c07e51478

Comment by Sergei Golubchik [ 2021-10-18 ]

5fcde24091c01ce0f5941d434bd5761c07e51478 is ok to push, thanks!

Generated at Thu Feb 08 09:46:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.