[MDEV-32492] SIGSEGV in spider_conn_first_link_idx and others on DELETE, INSERT and SELECT Created: 2023-10-17  Updated: 2023-11-28

Status: Confirmed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.4, 10.5, 10.6, 10.9, 10.10, 10.11, 11.0, 11.1, 11.2, 11.3
Fix Version/s: 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2

Type: Bug Priority: Major
Reporter: Roel Van de Paar Assignee: Yuchen Pei
Resolution: Unresolved Votes: 0
Labels: affects-tests, corruption, memory_corruption

Issue Links:
Duplicate
is duplicated by MDEV-32558 ERROR 1429 (base) versus crash [SIGSE... Closed
Relates
relates to MDEV-28856 Spider: Implement more engine-defined... Closed
relates to MDEV-29962 SIGSEGV in ha_spider::lock_tables on ... Stalled
relates to MDEV-32558 ERROR 1429 (base) versus crash [SIGSE... Closed

 Description   

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
--error ER_NO_SUCH_TABLE
SHOW CREATE TABLE t;
CREATE TABLE t (c INT, PRIMARY KEY(c)) ENGINE=Spider;
SHOW CREATE TABLE t;
DROP TABLE t;
CREATE TABLE t (c INT) ENGINE=Spider REMOTE_PORT="1 1";
DELETE FROM t;

11.3.0 a4031e4c051d4fc69f466a5678ceb5d3598bf525

Core was generated by `/test/MDEV28856_MD141023-mariadb-11.3.0-linux-x86_64-dbg/bin/mariadbd --default'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000149a923b2764 in spider_conn_first_link_idx (
    thd=thd@entry=0x149a80000d58, link_statuses=0x149a80074468, 
    access_balances=0x149a80128b18, conn_link_idx=0x149a80166a48, 
    link_count=2, link_status=link_status@entry=1)
    at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/storage/spider/spd_conn.cc:3746
[Current thread is 1 (Thread 0x149a924b4640 (LWP 3283577))]
(gdb) bt
#0  0x0000149a923b2764 in spider_conn_first_link_idx (thd=thd@entry=0x149a80000d58, link_statuses=0x149a80074468, access_balances=0x149a80128b18, conn_link_idx=0x149a80166a48, link_count=2, link_status=link_status@entry=1) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/storage/spider/spd_conn.cc:3746
#1  0x0000149a923905fe in spider_check_trx_and_get_conn (thd=0x149a80000d58, spider=spider@entry=0x149a80249a60) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/storage/spider/spd_trx.cc:3523
#2  0x0000149a9240198d in ha_spider::info (this=0x149a80249a60, flag=18) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/storage/spider/ha_spider.cc:6653
#3  0x000055c5cb7f5003 in Sql_cmd_delete::delete_from_single_table (this=this@entry=0x149a80015698, thd=thd@entry=0x149a80000d58) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_delete.cc:430
#4  0x000055c5cb7f7510 in Sql_cmd_delete::execute_inner (this=0x149a80015698, thd=0x149a80000d58) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_delete.cc:1800
#5  0x000055c5cb88ba57 in Sql_cmd_dml::execute (this=0x149a80015698, thd=0x149a80000d58) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_select.cc:33350
#6  0x000055c5cb84902a in mysql_execute_command (thd=thd@entry=0x149a80000d58, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_parse.cc:4360
#7  0x000055c5cb84e2a5 in mysql_parse (thd=thd@entry=0x149a80000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x149a924b31e0) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_parse.cc:7734
#8  0x000055c5cb85061b in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x149a80000d58, packet=packet@entry=0x149a801111d9 "DELETE FROM t", packet_length=packet_length@entry=13, blocking=blocking@entry=true) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_class.h:251
#9  0x000055c5cb85273b in do_command (thd=0x149a80000d58, blocking=blocking@entry=true) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_parse.cc:1406
#10 0x000055c5cb9b198d in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55c5ce277188, put_in_cache=put_in_cache@entry=true) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_connect.cc:1445
#11 0x000055c5cb9b1c82 in handle_one_connection (arg=arg@entry=0x55c5ce277188) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/sql/sql_connect.cc:1347
#12 0x000055c5cbdf45bc in pfs_spawn_thread (arg=0x55c5ce1ef218) at /test/git-bisect/preview-11.3-preview_MDEV-28856_patched_dbg/storage/perfschema/pfs.cc:2201
#13 0x0000149a97a94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x0000149a97b26a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



 Comments   
Comment by Yuchen Pei [ 2023-10-18 ]

This is an existing bug independent of MDEV-28856. Here's a case
that does not use the new options failing at 11.0
5e2d08b5e89ec600f46021d99beeb2635eef4f45:

--echo #
--echo # MDEV-32492 SIGSEGV in spider_conn_first_link_idx on DELETE
--echo #
--disable_query_log
--disable_result_log
--source ../../t/test_init.inc
--enable_result_log
--enable_query_log
 
--error ER_NO_SUCH_TABLE
SHOW CREATE TABLE t;
CREATE TABLE t (c INT, PRIMARY KEY(c)) ENGINE=Spider;
SHOW CREATE TABLE t;
DROP TABLE t;
CREATE TABLE t (c INT) ENGINE=Spider COMMENT='port "1 1"';
DELETE FROM t;
 
--disable_query_log
--disable_result_log
--source ../../t/test_deinit.inc
--enable_result_log
--enable_query_log
--echo #
--echo # end of test mdev_32492
--echo #

Also, with ASAN, we get assertion failure, which could be due to
different random numbers generated between with asan and without
asan:

mariadbd: /home/ycp/source/mariadb-server/11.0/src/storage/spider/spd_conn.cc:3732: int spider_conn_first_link_idx(THD*, long int*, long int*, uint*, int, int): Assertion `(conn_link_idx[link_idx] - link_idx) % link_count == 0' failed.

Comment by Roel Van de Paar [ 2023-10-18 ]

Confirmed versions affected to be 10.4 to 11.3 based on new testcase.
Confirmed bug is thus not caused by MDEV-28856 nor blocks it.

Comment by Yuchen Pei [ 2023-10-18 ]

Also, it seems like the initial

--error ER_NO_SUCH_TABLE
SHOW CREATE TABLE t;

is not needed.

Comment by Roel Van de Paar [ 2023-10-19 ]

Another testcase (CLI) that produces a slightly different stack

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE TABLE t (a INT,PRIMARY KEY(a)) ENGINE=Spider;
SHOW CREATE TABLE t;
DROP TABLE t;
CREATE TABLE t (a INT) ENGINE=Spider COMMENT='PORT "1 1"';
INSERT INTO t VALUES (1),(1);

Leads to

11.3.0 905c3d61e18ae6222d0d195c43d335046eec65d9 (Debug)

Core was generated by `/test/MD270923-mariadb-11.3.0-linux-x86_64-dbg/bin/mariadbd --no-defaults --cor'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00001539780b36b0 in spider_conn_first_link_idx (
    thd=thd@entry=0x153914000d58, link_statuses=0x1539140ab938, 
    access_balances=0x1539140aba48, conn_link_idx=0x1539140b6b48, 
    link_count=2, link_status=link_status@entry=1)
    at /test/11.3_dbg/storage/spider/spd_conn.cc:3733
[Current thread is 1 (Thread 0x15398c064640 (LWP 2713145))]
(gdb) bt
#0  0x00001539780b36b0 in spider_conn_first_link_idx (thd=thd@entry=0x153914000d58, link_statuses=0x1539140ab938, access_balances=0x1539140aba48, conn_link_idx=0x1539140b6b48, link_count=2, link_status=link_status@entry=1) at /test/11.3_dbg/storage/spider/spd_conn.cc:3733
#1  0x000015397809158c in spider_check_trx_and_get_conn (thd=thd@entry=0x153914000d58, spider=spider@entry=0x1539140ad460) at /test/11.3_dbg/storage/spider/spd_trx.cc:3458
#2  0x00001539780ef3a5 in ha_spider::check_access_kind_for_connection (this=this@entry=0x1539140ad460, thd=thd@entry=0x153914000d58, write_request=<optimized out>) at /test/11.3_dbg/storage/spider/ha_spider.cc:592
#3  0x0000153978103be4 in ha_spider::dml_init (this=this@entry=0x1539140ad460) at /test/11.3_dbg/storage/spider/ha_spider.cc:12207
#4  0x00001539781066ed in ha_spider::write_row (this=0x1539140ad460, buf=0x1539140ad048 "\375\001") at /test/11.3_dbg/storage/spider/ha_spider.cc:7889
#5  0x0000555f1852463f in handler::ha_write_row (this=0x1539140ad460, buf=0x1539140ad048 "\375\001") at /test/11.3_dbg/sql/handler.cc:7851
#6  0x0000555f181d145e in write_record (thd=thd@entry=0x153914000d58, table=table@entry=0x153914072c58, info=info@entry=0x15398c0629f0, sink=sink@entry=0x0) at /test/11.3_dbg/sql/sql_insert.cc:2205
#7  0x0000555f181dbffa in mysql_insert (thd=thd@entry=0x153914000d58, table_list=<optimized out>, fields=@0x1539140061a0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x555f1987d2a0 <end_of_list>, last = 0x1539140061a0, elements = 0}, <No data fields>}, values_list=@0x1539140061e8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x153914014220, last = 0x1539140142e0, elements = 2}, <No data fields>}, update_fields=@0x1539140061d0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x555f1987d2a0 <end_of_list>, last = 0x1539140061d0, elements = 0}, <No data fields>}, update_values=@0x1539140061b8: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x555f1987d2a0 <end_of_list>, last = 0x1539140061b8, elements = 0}, <No data fields>}, duplic=DUP_ERROR, ignore=false, result=0x0) at /test/11.3_dbg/sql/sql_insert.cc:1154
#8  0x0000555f1821663c in mysql_execute_command (thd=thd@entry=0x153914000d58, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/11.3_dbg/sql/sql_parse.cc:4426
#9  0x0000555f1821b5f7 in mysql_parse (thd=thd@entry=0x153914000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x15398c0631e0) at /test/11.3_dbg/sql/sql_parse.cc:7744
#10 0x0000555f1821d96d in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x153914000d58, packet=packet@entry=0x15391400b189 "INSERT INTO t VALUES (1),(1)", packet_length=packet_length@entry=28, blocking=blocking@entry=true) at /test/11.3_dbg/sql/sql_class.h:247
#11 0x0000555f1821fa8d in do_command (thd=0x153914000d58, blocking=blocking@entry=true) at /test/11.3_dbg/sql/sql_parse.cc:1406
#12 0x0000555f1837fcb5 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x555f1aff1a88, put_in_cache=put_in_cache@entry=true) at /test/11.3_dbg/sql/sql_connect.cc:1445
#13 0x0000555f1837ffaa in handle_one_connection (arg=arg@entry=0x555f1aff1a88) at /test/11.3_dbg/sql/sql_connect.cc:1347
#14 0x0000555f187c220a in pfs_spawn_thread (arg=0x555f1af5eac8) at /test/11.3_dbg/storage/perfschema/pfs.cc:2201
#15 0x000015398e694ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#16 0x000015398e726a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Comment by Roel Van de Paar [ 2023-10-24 ]

--source include/have_innodb.inc
--let $SOCKET= `SELECT @@global.socket`
INSTALL PLUGIN Spider SONAME 'ha_spider.so';
eval CREATE SERVER srv FOREIGN DATA WRAPPER mysql OPTIONS (SOCKET "$SOCKET",DATABASE 'test',USER 'spider',PASSWORD 'pwd');
CREATE TABLE t1 (a INT,b VARCHAR(255),PRIMARY KEY(a)) ENGINE=Spider COMMENT="srv 'srv', table 't1', read_only_mode '1'";
# SET SESSION SPIDER_IGNORE_COMMENTS=1;
--error ER_CONNECT_TO_FOREIGN_DATA_SOURCE
INSERT INTO t1 VALUES (1,'aaa'),(2,'bbb'),(3,'ccc'),(4,'ddd');
SHOW CREATE TABLE t1;
DROP TABLE t1;
CREATE TABLE t1 (a INT) ENGINE=Spider comment="port '123 456'";
INSERT IGNORE INTO t1 VALUES (42),(42);

Will generate another SIGSEGV on INSERT:

SIGSEGV|spider_create_conn|spider_get_conn|spider_check_trx_and_get_conn|ha_spider::check_access_kind_for_connection

On both BASE and PATCH for MDEV-28856. However, at the CLI, and on optimized builds only, this testcase:

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE SERVER srv FOREIGN DATA WRAPPER mysql OPTIONS (SOCKET '../socket.sock',DATABASE 'test',user 'Spider',PASSWORD '');
CREATE TABLE t1 (a INT,b VARCHAR(255),PRIMARY KEY(a)) ENGINE=Spider COMMENT="srv 'srv', table 't1', read_only_mode '1'";
INSERT INTO t1 VALUES (1,'aaa'),(2,'bbb'),(3,'ccc'),(4,'ddd');
SHOW CREATE TABLE t1;
DROP TABLE t1;
CREATE TABLE t1 (a INT) ENGINE=Spider COMMENT='port "123 456"';
INSERT IGNORE INTO t1 VALUES (42),(42);

Will create a different outcome. This is discussed in MDEV-32558 as this particular issues is not present before MDEV-28556.
Also, when this testcase is executed against optimized UB+ASAN builds, the same differences hold (crash versus no crash, loop versus read only). When executed against UB+ASAN debug buids, we see:

preview-11.3.0-preview 465f9beea1c43a1dad74330aa2dc30927bc224f5 (Debug)

mariadbd: /test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/spd_conn.cc:3732: int spider_conn_first_link_idx(THD*, long int*, long int*, uint*, int, int): Assertion `(conn_link_idx[link_idx] - link_idx) % link_count == 0' failed.

preview-11.3.0-preview 465f9beea1c43a1dad74330aa2dc30927bc224f5 (Debug)

Core was generated by `/test/BASE_MDEV28856_UBASAN_MD161023-mariadb-11.3.0-linux-x86_64-dbg/bin/mariad'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=23122991380032)
    at ./nptl/pthread_kill.c:44
[Current thread is 1 (Thread 0x1507bdaff640 (LWP 595797))]
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=23122991380032) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=23122991380032) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=23122991380032, signo=6) at ./nptl/pthread_kill.c:89
#3  0x000056417ead15f7 in my_write_core (sig=6) at /test/git-bisect/preview-11.3-preview_dbg_san/mysys/stacktrace.c:424
#4  0x000056417c3a4379 in handle_fatal_signal (sig=<optimized out>) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/signal_handler.cc:360
#5  <signal handler called>
#6  __pthread_kill_implementation (no_tid=0, signo=6, threadid=23122991380032) at ./nptl/pthread_kill.c:44
#7  __pthread_kill_internal (signo=6, threadid=23122991380032) at ./nptl/pthread_kill.c:78
#8  __GI___pthread_kill (threadid=23122991380032, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#9  0x00001507e0442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#10 0x00001507e04287f3 in __GI_abort () at ./stdlib/abort.c:79
#11 0x00001507e042871b in __assert_fail_base (fmt=0x1507e05dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x1507bd2c35e0 "(conn_link_idx[link_idx] - link_idx) % link_count == 0", file=0x1507bd2c2c00 "/test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/spd_conn.cc", line=3732, function=<optimized out>) at ./assert/assert.c:92
#12 0x00001507e0439e96 in __GI___assert_fail (assertion=0x1507bd2c35e0 "(conn_link_idx[link_idx] - link_idx) % link_count == 0", file=0x1507bd2c2c00 "/test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/spd_conn.cc", line=3732, function=0x1507bd2c3640 "int spider_conn_first_link_idx(THD*, long int*, long int*, uint*, int, int)") at ./assert/assert.c:101
#13 0x00001507bcf721d3 in spider_conn_first_link_idx (thd=thd@entry=0x62b00015e218, link_statuses=<optimized out>, access_balances=<optimized out>, conn_link_idx=conn_link_idx@entry=0x6170000513b8, link_count=link_count@entry=2, link_status=link_status@entry=1) at /test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/spd_conn.cc:3732
#14 0x00001507bcea4123 in spider_check_trx_and_get_conn (thd=thd@entry=0x62b00015e218, spider=spider@entry=0x61f000048ab8) at /test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/spd_trx.cc:3523
#15 0x00001507bd0877e1 in ha_spider::check_access_kind_for_connection (this=this@entry=0x61f000048ab8, thd=thd@entry=0x62b00015e218, write_request=write_request@entry=true) at /test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/ha_spider.cc:592
#16 0x00001507bd13a298 in ha_spider::dml_init (this=this@entry=0x61f000048ab8) at /test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/ha_spider.cc:12207
#17 0x00001507bd153c52 in ha_spider::write_row (this=0x61f000048ab8, buf=<optimized out>) at /test/git-bisect/preview-11.3-preview_dbg_san/storage/spider/ha_spider.cc:7889
#18 0x000056417c42ca75 in handler::ha_write_row (this=this@entry=0x61f000048ab8, buf=0x619000283cc8 "\375*") at /test/git-bisect/preview-11.3-preview_dbg_san/sql/handler.cc:7840
#19 0x000056417a952288 in write_record (thd=thd@entry=0x62b00015e218, table=table@entry=0x619000283798, info=info@entry=0x1507bdafbf90, sink=sink@entry=0x0) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_insert.cc:2204
#20 0x000056417a9b7525 in mysql_insert (thd=thd@entry=0x62b00015e218, table_list=<optimized out>, fields=@0x62b000163458: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x56418954e1c0 <end_of_list>, last = 0x62b000163458, elements = 0}, <No data fields>}, values_list=@0x62b0001634a0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x629000222018, last = 0x629000222100, elements = 2}, <No data fields>}, update_fields=@0x62b000163488: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x56418954e1c0 <end_of_list>, last = 0x62b000163488, elements = 0}, <No data fields>}, update_values=@0x62b000163470: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x56418954e1c0 <end_of_list>, last = 0x62b000163470, elements = 0}, <No data fields>}, duplic=<optimized out>, ignore=<optimized out>, result=<optimized out>) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_insert.cc:1154
#21 0x000056417abb905c in mysql_execute_command (thd=thd@entry=0x62b00015e218, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_parse.cc:4416
#22 0x000056417abda3a1 in mysql_parse (thd=thd@entry=0x62b00015e218, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x1507bdafd790) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_parse.cc:7734
#23 0x000056417abea117 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x62b00015e218, packet=packet@entry=0x62900021c219 "INSERT IGNORE INTO t1 VALUES (42),(42)", packet_length=packet_length@entry=38, blocking=blocking@entry=true) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_parse.cc:1893
#24 0x000056417abf805e in do_command (thd=0x62b00015e218, blocking=blocking@entry=true) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_parse.cc:1406
#25 0x000056417b5e4bf4 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x608000002138, put_in_cache=put_in_cache@entry=true) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_connect.cc:1445
#26 0x000056417b5e610f in handle_one_connection (arg=0x608000002138) at /test/git-bisect/preview-11.3-preview_dbg_san/sql/sql_connect.cc:1347
#27 0x00001507e0494ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#28 0x00001507e0526a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

And for the feature tree only, optimized, we see:

bb-11.3-mdev-28856-and-fixes cc08a83ef4225960dccb46bd68fc549160d21841 (Optimized, UBASAN)

SIGSEGV|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::check_access_kind_for_connection|ha_spider::dml_init

Comment by Roel Van de Paar [ 2023-10-24 ]

Additional testcase

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE SERVER srv FOREIGN DATA WRAPPER mysql OPTIONS (SOCKET '../socket.sock', DATABASE 'test', USER 'Spider', PASSWORD '');
CREATE TABLE t1 (c INT, KEY(c)) ENGINE=Spider COMMENT='WRAPPER "mysql", SRV "srv",TABLE "t2", PK_NAME "f"';
SET GLOBAL general_log=1;
INSERT INTO t1 VALUES (1, "aaa"),(2, "bbb"),(3, "ccc"),(4, "ddd");
SHOW CREATE TABLE t1;
DROP TABLE t1;
CREATE TABLE t1 (a INT) ENGINE=Spider COMMENT='port "123 456"';
SELECT * FROM t1;
INSERT IGNORE INTO t1 VALUES (42),(42);

Which produces these additional stacks on debug/optimized:

SIGSEGV|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::info|TABLE_LIST::fetch_number_of_rows
SIGSEGV|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::info|make_join_statistics

Note: these two SIGSEGV's are seen both before (base) and after MDEV-32558 (patch) with this testcase, no difference for this one.

Comment by Yuchen Pei [ 2023-10-25 ]

The problem is again some SPIDER_TRX overstaying its welcome. So let
us group it with other similar tickets e.g. MDEV-29962 to fix
together.

At 11.0 be24e75229a4496e525677149c8a116a3ff4b372.

At SHOW CREATE TABLE (A), the trx_ha is created with only one
conn_link_idx (3), because the table has only one remote link

spider_create_trx_ha > spider_check_trx_and_get_conn > ha_spider::update_create_info > add_table_options > show_create_table > mysqld_show_create > mysql_parse > dispatch_command > do_command > handle_one_connection > pfs_spawn_thread

At CREATE TABLE (B), the spider->conn_link_idx is allocated with two
conn_link_idx when initiating the SPIDER_SHARE

spider_init_share > spider_get_share > ha_spider::open > open_table_from_share > open_table > open_tables > open_and_lock_tables > mysql_delete > mysql_execute_command > dispatch_command > do_command > do_handle_one_connection > pfs_spawn_thread

At DELETE FROM (C), spider->conn_link_idx is overwritten with
trx_ha->conn_link_idx (1), thereby missing one link

spider_check_trx_ha > spider_check_trx_and_get_conn > ha_spider::info > mysql_execute_command > mysql_parse > do_command > do_handle_one_connection > pfs_spawn_thread

Then it fails the assertion because spider->conn_link_idx[1] is a
random number when looking for the first link_idx (2)

spider_conn_first_link_idx > spider_check_trx_and_get_conn > mysql_delete > mysql_execute_command > mysql_parse > do_command > do_handle_one_connection > pfs_spawn_thread

CREATE TABLE t (c INT, PRIMARY KEY(c)) ENGINE=Spider;
SHOW CREATE TABLE t;            # (A)
DROP TABLE t;
CREATE TABLE t (c INT) ENGINE=Spider COMMENT='port "1 1"'; # (B)
DELETE FROM t;                                             # (C)

int spider_check_trx_and_get_conn(THD *thd, ha_spider *spider)
{
//  [... 15 lines elided]
  if (spider->wide_handler->sql_command != SQLCOM_DROP_TABLE)
  {
    SPIDER_TRX_HA *trx_ha = spider_check_trx_ha(trx, spider); // (1)
//  [... 13 lines elided]
    if (
//  [... 7 lines elided]
    ) {
//  [... 5 lines elided]
      if (
//  [... 2 lines elided]
      ) {
        search_link_idx = spider_conn_first_link_idx(thd,
          share->link_statuses, share->access_balances, spider->conn_link_idx,
          share->link_count, SPIDER_LINK_STATUS_OK); // (2)
//  [... 28 lines elided]
      }
//  [... 81 lines elided]
    } else {
//  [... 76 lines elided]
    }
    spider->set_first_link_idx();
    DBUG_RETURN(spider_create_trx_ha(trx, spider, trx_ha)); // (3)
  }
  spider->spider_thread_id = trx->spider_thread_id;
  DBUG_RETURN(0);
}

Comment by Roel Van de Paar [ 2023-10-26 ]

More information/analysis in MDEV-32558 as well as additional testcases & stacks.

Comment by Roel Van de Paar [ 2023-10-26 ]

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET '../socket.sock',DATABASE 'a',USER 'a',PASSWORD '');
CREATE TABLE t1 (c INT) ENGINE=Spider COMMENT='WRAPPER "mysql",SRV "srv",TABLE "t2",query_cache_sync "3"';
SHOW CREATE TABLE t1;
DROP TABLE t1;
CREATE TABLE t1 (a INT,b VARCHAR(1),c1 INT,c2 INT,PRIMARY KEY(a),UNIQUE KEY(c1),KEY(c2)) ENGINE=Spider AUTO_INCREMENT_mode=1 bgs_mode=1 bulk_size=41 bulk_update_size=42 connect_timeout="43 44" remote_database=foo63 default_file=foo44 default_group=foo45 delete_all_rows_type=0 DRIVER=foo47 DSN=foo48 FILEDSN=foo49 force_bulk_delete=1 force_bulk_update=NO remote_host=foo52 IDX="f c1 ig PRIMARY u c2" multi_split_read=54 net_read_timeout=" 55 56" net_write_timeout=" 56 " remote_password=foo57 REMOTE_PORT="234 837 " PRIORITY=59 query_cache=2 query_cache_sync=3 read_only=1 REMOTE_SERVER="srv" skip_parallel_search=2 remote_socket=foo67 ssl_capath=foo68 ssl_ca=foo69 ssl_cert=foo70 ssl_cipher=foo71 ssl_key=foo72 ssl_vsc=1 REMOTE_TABLE=foo65 table_count_mode=3 remote_username=foo75 use_pushdown_udf=YES WRAPPER=mysql;
SELECT MAX(a) FROM t1;

SIGSEGV|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::info|opt_sum_query

Comment by Roel Van de Paar [ 2023-10-26 ]

On UB+ASAN builds:

INSTALL PLUGIN spider SONAME 'ha_spider.so';
SET SESSION spider_ignore_comments=1;
CREATE TABLE t1 (c int) ENGINE=Spider COMMENT='WRAPPER "mysql",srv "srv",TABLE "t2", MONITORING_KIND "2"';
SHOW CREATE TABLE t1;
DROP TABLE t1;
CREATE TABLE t1 (a INT) ENGINE=Spider REMOTE_PORT="123 456";
DELETE FROM t1;

Leads to:

(conn_link_idx[link_idx] - link_idx) % link_count == 0|SIGABRT|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::info|Sql_cmd_delete::delete_from_single_table

Same bug, on legacy:

INSTALL PLUGIN spider SONAME 'ha_spider.so';
SET SESSION;
CREATE TABLE t1 (c int) ENGINE=Spider;
SHOW CREATE TABLE t1;
DROP TABLE t1;
CREATE TABLE t1 (a INT) ENGINE=Spider COMMENT="port '123 456'";
DELETE FROM t1;

Comment by Roel Van de Paar [ 2023-10-26 ]

On UB+ASAN builds:

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
SET SESSION SPIDER_IGNORE_COMMENTS=1;
CREATE TABLE t1 (c INT, d INT, e INT, PRIMARY KEY(c), KEY(d), UNIQUE KEY(e)) ENGINE=Spider COMMENT='WRAPPER "mysql", SRV "srv",TABLE "t2", idx000 "f PRIMARY", idx001 "u d", idx002 "ig e"';
SHOW CREATE TABLE t1;
DROP TABLE t1, t2;
SELECT * FROM t1;
CREATE TABLE t1 (a INT) ENGINE=Spider REMOTE_PORT="123 456";
SELECT * FROM t1;

Leads to:

(conn_link_idx[link_idx] - link_idx) % link_count == 0|SIGABRT|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::info|TABLE_LIST::fetch_number_of_rows

Same bug, on legacy:

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE TABLE t1 (c INT, d INT, e INT, PRIMARY KEY(c), KEY(d), UNIQUE KEY(e)) ENGINE=Spider;
SHOW CREATE TABLE t1;
DROP TABLE t1, t2;
SELECT * FROM t1;
CREATE TABLE t1 (a INT) ENGINE=Spider COMMENT="port '123 456'";
SELECT * FROM t1;

Comment by Roel Van de Paar [ 2023-10-26 ]

Testcase reductions for this UniqueID/stack specifically:

(conn_link_idx[link_idx] - link_idx) % link_count == 0|SIGABRT|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::info|ha_spider::update_create_info

Have not worked. However, given the above and MDEV-32558 it is deemed to be the same bug.

Comment by Roel Van de Paar [ 2023-10-26 ]

Testcase reductions for this UniqueID/stack specifically:

SIGSEGV|is_prefix|build_table_filename|spider_conn_queue_loop_check|spider_get_conn

Have not worked. Discussed with Yuchen who analyzed that Spider could not access share->tgt_table_names[conn_link_idx], where conn_link_idx = spider->conn_link_idx[link_idx] and link_idx == 1. Concluded to be the same issue. Full stack:

11.3.0 cc08a83ef4225960dccb46bd68fc549160d21841 (Optimized)

Core was generated by `/test/28856_P2_MD211023-mariadb-11.3.0-linux-x86_64-opt/bin/mariadbd --no-defau'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  is_prefix (
    s=0x40f5180000000001 <error: Cannot access memory at address 0x40f5180000000001>, t=0x5571e4e5e350 "#sql")
    at /test/bb-11.3-mdev-28856-and-fixes_opt/strings/is_prefix.c:42
[Current thread is 1 (Thread 0x14d6ec051640 (LWP 1935713))]
(gdb) bt
#0  is_prefix (s=0x40f5180000000001 <error: Cannot access memory at address 0x40f5180000000001>, t=0x5571e4e5e350 "#sql") at /test/bb-11.3-mdev-28856-and-fixes_opt/strings/is_prefix.c:42
#1  0x00005571e4386616 in build_table_filename (buff=buff@entry=0x14d6ec04e830 "\300", bufflen=bufflen@entry=512, db=<optimized out>, table_name=0x40f5180000000000 <error: Cannot access memory at address 0x40f5180000000000>, ext=ext@entry=0x14d6d8196c6b "", flags=flags@entry=0) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_table.cc:571
#2  0x000014d6d81196a0 in spider_conn_queue_loop_check (conn=0x14d5dc0e5d48, spider=<optimized out>, link_idx=<optimized out>) at /test/bb-11.3-mdev-28856-and-fixes_opt/storage/spider/spd_conn.cc:1322
#3  0x000014d6d811f8dd in spider_get_conn (share=share@entry=0x14d5dc0dca38, link_idx=<optimized out>, link_idx@entry=1, conn_key=<optimized out>, trx=trx@entry=0x14d5dc016b08, spider=spider@entry=0x14d5dc0c2240, another=another@entry=false, thd_chg=<optimized out>, error_num=<optimized out>) at /test/bb-11.3-mdev-28856-and-fixes_opt/storage/spider/spd_conn.cc:832
#4  0x000014d6d8103984 in spider_check_trx_and_get_conn (thd=<optimized out>, spider=spider@entry=0x14d5dc0c2240) at /test/bb-11.3-mdev-28856-and-fixes_opt/storage/spider/spd_trx.cc:3578
#5  0x000014d6d8162603 in ha_spider::info (this=0x14d5dc0c2240, flag=18) at /test/bb-11.3-mdev-28856-and-fixes_opt/storage/spider/ha_spider.cc:6653
#6  0x00005571e434fa82 in make_join_statistics (join=0x14d5dc012338, tables_list=@0x14d5dc010da0: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x14d5dc012af0, last = 0x14d5dc012af0, elements = 1}, <No data fields>}, keyuse_array=0x14d5dc012698) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_select.cc:5499
#7  0x00005571e4356852 in JOIN::optimize_inner (this=0x14d5dc012338) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_select.cc:2624
#8  0x00005571e4356eaa in JOIN::optimize (this=this@entry=0x14d5dc012338) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_select.cc:1944
#9  0x00005571e4356fa1 in mysql_select (thd=0x14d5dc000c68, tables=0x14d5dc0111a8, fields=<optimized out>, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x14d5dc012310, unit=0x14d5dc004fb8, select_lex=0x14d5dc010b88) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_select.cc:5235
#10 0x00005571e43577f4 in handle_select (thd=thd@entry=0x14d5dc000c68, lex=lex@entry=0x14d5dc004ed8, result=result@entry=0x14d5dc012310, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_select.cc:628
#11 0x00005571e42cb685 in execute_sqlcom_select (thd=0x14d5dc000c68, all_tables=0x14d5dc0111a8) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_parse.cc:6012
#12 0x00005571e42da792 in mysql_execute_command (thd=0x14d5dc000c68, is_called_from_prepared_stmt=<optimized out>) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_parse.cc:3911
#13 0x00005571e42dbb66 in mysql_parse (thd=0x14d5dc000c68, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_parse.cc:7734
#14 0x00005571e42de2fd in dispatch_command (command=COM_QUERY, thd=0x14d5dc000c68, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_parse.cc:1990
#15 0x00005571e42e00a0 in do_command (thd=0x14d5dc000c68, blocking=blocking@entry=true) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_parse.cc:1406
#16 0x00005571e44080ff in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_connect.cc:1445
#17 0x00005571e440844d in handle_one_connection (arg=arg@entry=0x5571e6e657c8) at /test/bb-11.3-mdev-28856-and-fixes_opt/sql/sql_connect.cc:1347
#18 0x00005571e47b24f1 in pfs_spawn_thread (arg=0x5571e6e65838) at /test/bb-11.3-mdev-28856-and-fixes_opt/storage/perfschema/pfs.cc:2201
#19 0x000014d6edc94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014d6edd26a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Comment by Roel Van de Paar [ 2023-10-26 ]

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE TABLE t1 (a INT) ENGINE=Spider REMOTE_TABLE=t CONNECTION="srv s_2_1";
SHOW CREATE TABLE t1;
SET SESSION SPIDER_SUPPRESS_COMMENT_IGNORED_WARNING=1;
SELECT table_name, SERVER, tgt_table_name FROM mysql.spider_tables;
DROP TABLE t1;
CREATE TABLE t1 (a INT) ENGINE=Spider REMOTE_PORT="123 456";
DELETE FROM t1;

Leads to:

(conn_link_idx[link_idx] - link_idx) % link_count == 0|SIGABRT|spider_conn_first_link_idx|spider_check_trx_and_get_conn|ha_spider::info|Sql_cmd_delete::delete_from_single_table

Generated at Thu Feb 08 10:31:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.