[MDEV-29854] SIGSEGV in spider_string::length on LOCK TABLES Created: 2022-10-22  Updated: 2023-10-14  Resolved: 2023-10-12

Status: Closed
Project: MariaDB Server
Component/s: Locking, Storage Engine - Spider
Affects Version/s: 10.4, 10.11
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Roel Van de Paar Assignee: Yuchen Pei
Resolution: Duplicate Votes: 0
Labels: locking, thread_hang

Issue Links:
Duplicate
duplicates MDEV-29963 SIGSEGV in spider_db_mbase::append_lo... Closed
duplicates MDEV-31357 ASAN heap-use-after-free in spider_li... Closed
Issue split
split to MDEV-32451 Hang when locking both spider table a... Confirmed
Relates
relates to MDEV-29583 Disallow Spider self/dual/multi/cross... Closed
relates to MDEV-30580 Investigate whether self-/multi- refe... Open
relates to MDEV-32451 Hang when locking both spider table a... Confirmed

 Description   

Interestingly, this issue seems to exist only in 10.4 and 10.11.

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET'',DATABASE'',USER'',PASSWORD '');
CREATE TABLE t (c INT KEY,b INT UNIQUE) ENGINE=Spider;
CREATE TABLE t2 (c INT,d INT,CONSTRAINT fk FOREIGN KEY(d,c) REFERENCES t (c,d)) ENGINE=Spider COMMENT='WRAPPER "mysql",srv "srv",TABLE "t"';
LOCK TABLES t WRITE,t2 WRITE;
TRUNCATE t2;
LOCK TABLES t2 AS a WRITE;

Leads to:

10.11.1 50c5743adc87e1cdec1431a02558f6540fe5a6d5 (Debug)

Core was generated by `/test/MD221022-mariadb-10.11.1-linux-x86_64-dbg/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  spider_string::length (this=0x2) at /test/10.11_dbg/sql/sql_string.h:278
278	  inline uint32 length() const { return str_length;}
[Current thread is 1 (Thread 0x14f9d0050700 (LWP 449653))]
(gdb) bt
#0  spider_string::length (this=0x2) at /test/10.11_dbg/sql/sql_string.h:278
#1  0x000014f9a774492a in spider_link_get_key (link_for_hash=0x14f9700c5338, length=0x14f9d004e8f8, not_used=<optimized out>) at /test/10.11_dbg/storage/spider/spd_table.cc:385
#2  0x000055843f6a937c in my_hash_key (first=1 '\001', length=0x14f9d004e8f8, record=<optimized out>, hash=0x14f9700c6008) at /test/10.11_dbg/mysys/hash.c:196
#3  hashcmp (hash=hash@entry=0x14f9700c6008, pos=pos@entry=0x14f9700c60c8, key=key@entry=0x14f970089498 "`test`.`t`", length=length@entry=10) at /test/10.11_dbg/mysys/hash.c:379
#4  0x000055843f6a95aa in my_hash_first_from_hash_value (hash=hash@entry=0x14f9700c6008, hash_value=<optimized out>, key=0x14f970089498 "`test`.`t`", length=length@entry=10, current_record=current_record@entry=0x14f9d004e99c) at /test/10.11_dbg/mysys/hash.c:290
#5  0x000055843f6a95fb in my_hash_search_using_hash_value (hash=hash@entry=0x14f9700c6008, hash_value=<optimized out>, key=<optimized out>, length=length@entry=10) at /test/10.11_dbg/mysys/hash.c:244
#6  0x000014f9a77a2324 in spider_mbase_handler::append_lock_tables_list (this=0x14f970083090, conn=0x14f9700c5398, link_idx=0, appended=0x14f9d004ea1c) at /test/10.11_dbg/storage/spider/spd_db_mysql.cc:13130
#7  0x000014f9a778fa6e in ha_spider::append_lock_tables_list (this=this@entry=0x14f9700b6a80) at /test/10.11_dbg/storage/spider/ha_spider.cc:12081
#8  0x000014f9a778feac in ha_spider::store_lock (this=0x14f9700b6a80, thd=0x14f970000d48, to=0x14f9700b9528, lock_type=TL_WRITE) at /test/10.11_dbg/storage/spider/ha_spider.cc:775
#9  0x000055843f115d40 in get_lock_data (thd=thd@entry=0x14f970000d48, table_ptr=table_ptr@entry=0x14f970013be0, count=count@entry=1, flags=flags@entry=1) at /test/10.11_dbg/sql/sql_string.h:655
#10 0x000055843f116160 in mysql_lock_tables (thd=thd@entry=0x14f970000d48, tables=tables@entry=0x14f970013be0, count=count@entry=1, flags=flags@entry=0) at /test/10.11_dbg/sql/lock.cc:301
#11 0x000055843ec46af1 in lock_tables (thd=thd@entry=0x14f970000d48, tables=0x14f970013308, count=<optimized out>, flags=flags@entry=0) at /test/10.11_dbg/sql/sql_base.cc:5821
#12 0x000055843ecc2240 in lock_tables_open_and_lock_tables (thd=thd@entry=0x14f970000d48, tables=<optimized out>) at /test/10.11_dbg/sql/sql_parse.cc:2958
#13 0x000055843eccdd2e in mysql_execute_command (thd=thd@entry=0x14f970000d48, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.11_dbg/sql/sql_parse.cc:5124
#14 0x000055843ecb8f90 in mysql_parse (thd=thd@entry=0x14f970000d48, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14f9d004f300) at /test/10.11_dbg/sql/sql_parse.cc:8023
#15 0x000055843ecc64ac in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14f970000d48, packet=packet@entry=0x14f97000af09 "LOCK TABLES t2 AS a WRITE", packet_length=packet_length@entry=25, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_class.h:1346
#16 0x000055843ecc88f4 in do_command (thd=0x14f970000d48, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_parse.cc:1407
#17 0x000055843ee25067 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x5584412ffdd8, put_in_cache=put_in_cache@entry=true) at /test/10.11_dbg/sql/sql_connect.cc:1416
#18 0x000055843ee25536 in handle_one_connection (arg=0x5584412ffdd8) at /test/10.11_dbg/sql/sql_connect.cc:1318
#19 0x000014f9e910a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x000014f9e8cf6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Bug confirmed present in:
MariaDB: 10.4.27 (dbg), 10.11.1 (dbg)

Bug (or feature/syntax) confirmed not present in:
MariaDB: 10.3.37 (dbg), 10.3.37 (opt), 10.4.27 (opt), 10.5.18 (dbg), 10.5.18 (opt), 10.6.10 (dbg), 10.6.10 (opt), 10.7.6 (dbg), 10.7.6 (opt), 10.8.5 (dbg), 10.8.5 (opt), 10.9.3 (dbg), 10.9.3 (opt), 10.10.2 (dbg), 10.10.2 (opt), 10.11.1 (opt)
MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.38 (dbg), 5.7.38 (opt), 8.0.29 (dbg), 8.0.29 (opt)



 Comments   
Comment by Roel Van de Paar [ 2022-10-22 ]

This related testcase locks up the thread it is replayed in:

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET'../socket.sock',DATABASE'',USER'',PASSWORD '');  # Socket location must be valid. Other values not.
CREATE TABLE t (a INT KEY,b INT UNIQUE) ENGINE=InnoDB;
CREATE TABLE t2 (id INT,id2 INT,UNIQUE (id,id2),CONSTRAINT t_id_fk FOREIGN KEY(id2,id) REFERENCES t (id,id2)) ENGINE=Spider COMMENT='WRAPPER "mysql",srv "srv",TABLE "t"';
LOCK TABLES t WRITE,t2 WRITE;

Note table t should not be reachable by Spider (no user etc.).

10.11.1 50c5743adc87e1cdec1431a02558f6540fe5a6d5 (Debug)

10.11.1-dbg>show processlist;
+----+-------------+-----------+------+---------+------+----------------------------------------------------+------------------------------+----------+
| Id | User        | Host      | db   | Command | Time | State                                              | Info                         | Progress |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+------------------------------+----------+
|  4 | root        | localhost | test | Query   |  129 | System lock                                        | LOCK TABLES t WRITE,t2 WRITE |    0.000 |
|  5 | system user |           | NULL | Sleep   |  129 | Reset for next command                             | NULL                         |    0.000 |
|  6 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
|  7 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
|  8 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
|  9 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
| 10 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
| 11 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
| 12 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
| 13 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
| 14 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                         |    0.000 |
| 15 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 16 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 17 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 18 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 19 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 20 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 21 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 22 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 23 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 24 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                         |    0.000 |
| 25 |             | localhost | test | Query   |  129 | Waiting for table metadata lock                    | lock tables `test`.`t` write |    0.000 |
| 26 | root        | localhost | test | Query   |    0 | starting                                           | show processlist             |    0.000 |
+----+-------------+-----------+------+---------+------+----------------------------------------------------+------------------------------+----------+
23 rows in set (0.000 sec)

Comment by Roel Van de Paar [ 2022-10-24 ]

I tried running git bisect for this bug, but it comes up with a list of 75 possible bad git commits.

Comment by Roel Van de Paar [ 2022-11-07 ]

Found an additional testcase:

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET'',DATABASE'',USER'',PASSWORD '');
CREATE TABLE t (a INT) ENGINE=Spider;
CREATE TABLE t2 (b INT) ENGINE=Spider COMMENT='WRAPPER "mysql",srv "srv",TABLE "t"';
LOCK TABLES t AS a READ,t2 AS b WRITE;
DROP TABLE t2;
CREATE TABLE t2 (c INT) ENGINE=Spider COMMENT='WRAPPER "mysql",srv "srv",TABLE "t"';
LOCK TABLES t2 WRITE;

Leads to:

10.11.1 50c5743adc87e1cdec1431a02558f6540fe5a6d5 (Debug)

Core was generated by `/test/MD221022-mariadb-10.11.1-linux-x86_64-dbg/bin/mysqld --no-defaults --core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  spider_string::length (this=0x2) at /test/10.11_dbg/sql/sql_string.h:278
278	  inline uint32 length() const { return str_length;}
[Current thread is 1 (Thread 0x14ebf40e5700 (LWP 2669643))]
(gdb) bt
#0  spider_string::length (this=0x2) at /test/10.11_dbg/sql/sql_string.h:278
#1  0x000014ebe814292a in spider_link_get_key (link_for_hash=0x14eb840c4638, length=0x14ebf40e38f8, not_used=<optimized out>) at /test/10.11_dbg/storage/spider/spd_table.cc:385
#2  0x000055fdabd7937c in my_hash_key (first=1 '\001', length=0x14ebf40e38f8, record=<optimized out>, hash=0x14eb840c5308) at /test/10.11_dbg/mysys/hash.c:196
#3  hashcmp (hash=hash@entry=0x14eb840c5308, pos=pos@entry=0x14eb840c53c8, key=key@entry=0x14eb84057d38 "`test`.`t`", length=length@entry=10) at /test/10.11_dbg/mysys/hash.c:379
#4  0x000055fdabd795aa in my_hash_first_from_hash_value (hash=hash@entry=0x14eb840c5308, hash_value=<optimized out>, key=0x14eb84057d38 "`test`.`t`", length=length@entry=10, current_record=current_record@entry=0x14ebf40e399c) at /test/10.11_dbg/mysys/hash.c:290
#5  0x000055fdabd795fb in my_hash_search_using_hash_value (hash=hash@entry=0x14eb840c5308, hash_value=<optimized out>, key=<optimized out>, length=length@entry=10) at /test/10.11_dbg/mysys/hash.c:244
#6  0x000014ebe81a0324 in spider_mbase_handler::append_lock_tables_list (this=0x14eb840c4190, conn=0x14eb840c4698, link_idx=0, appended=0x14ebf40e3a1c) at /test/10.11_dbg/storage/spider/spd_db_mysql.cc:13130
#7  0x000014ebe818da6e in ha_spider::append_lock_tables_list (this=this@entry=0x14eb840c8080) at /test/10.11_dbg/storage/spider/ha_spider.cc:12081
#8  0x000014ebe818deac in ha_spider::store_lock (this=0x14eb840c8080, thd=0x14eb84000d48, to=0x14eb840c2b88, lock_type=TL_WRITE) at /test/10.11_dbg/storage/spider/ha_spider.cc:775
#9  0x000055fdab7e5d40 in get_lock_data (thd=thd@entry=0x14eb84000d48, table_ptr=table_ptr@entry=0x14eb84013bc8, count=count@entry=1, flags=flags@entry=1) at /test/10.11_dbg/sql/sql_string.h:655
#10 0x000055fdab7e6160 in mysql_lock_tables (thd=thd@entry=0x14eb84000d48, tables=tables@entry=0x14eb84013bc8, count=count@entry=1, flags=flags@entry=0) at /test/10.11_dbg/sql/lock.cc:301
#11 0x000055fdab316af1 in lock_tables (thd=thd@entry=0x14eb84000d48, tables=0x14eb840132f0, count=<optimized out>, flags=flags@entry=0) at /test/10.11_dbg/sql/sql_base.cc:5821
#12 0x000055fdab392240 in lock_tables_open_and_lock_tables (thd=thd@entry=0x14eb84000d48, tables=<optimized out>) at /test/10.11_dbg/sql/sql_parse.cc:2958
#13 0x000055fdab39dd2e in mysql_execute_command (thd=thd@entry=0x14eb84000d48, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.11_dbg/sql/sql_parse.cc:5124
#14 0x000055fdab388f90 in mysql_parse (thd=thd@entry=0x14eb84000d48, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14ebf40e4300) at /test/10.11_dbg/sql/sql_parse.cc:8023
#15 0x000055fdab3964ac in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14eb84000d48, packet=packet@entry=0x14eb8400af09 "LOCK TABLES t2 WRITE", packet_length=packet_length@entry=20, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_class.h:1346
#16 0x000055fdab3988f4 in do_command (thd=0x14eb84000d48, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_parse.cc:1407
#17 0x000055fdab4f5067 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55fdadcf08d8, put_in_cache=put_in_cache@entry=true) at /test/10.11_dbg/sql/sql_connect.cc:1416
#18 0x000055fdab4f5536 in handle_one_connection (arg=0x55fdadcf08d8) at /test/10.11_dbg/sql/sql_connect.cc:1318
#19 0x000014ec0df61609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x000014ec0db4d133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

This testcase only reproduces in 10.11

Comment by Roel Van de Paar [ 2022-11-07 ]

Reconfirmed that the original testcase crashes 10.4 debug with the same stack, and does not crash 10.5 to 10.10 debug.

Comment by Yuchen Pei [ 2023-10-12 ]

I've been testing these cases at 11.0 with a fix for MDEV-29963
0199a5d0f634051f23d4c7f7ce7651c7f93c818f.

The case in the description is almost identical to that in MDEV-31357,
and fixable with the patch for MDEV-29963.

The hang case in comment[1] is a different bug - opened MDEV-32451.

The case in comment[2] is almost identical to that of MDEV-29963, and
fixable by the patch for that ticket.

[1] https://jira.mariadb.org/browse/MDEV-29854?focusedCommentId=238531&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-238531
[2] https://jira.mariadb.org/browse/MDEV-29854?focusedCommentId=240914&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-240914

Generated at Thu Feb 08 10:11:48 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.