Roel Van de Paar added a comment - 2024-09-24 08:35 ASAN sees a heap-upse-after-free : CS 11.7.0 5bbda9711131845ae6b4315a268b4d1710943a85 (Debug, UBASAN) ==798838==ERROR: AddressSanitizer: heap-use-after-free on address 0x51f0000ba828 at pc 0x146ee333e7dd bp 0x146ee54fe300 sp 0x146ee54fe2f0 READ of size 8 at 0x51f0000ba828 thread T12 #0 0x146ee333e7dc in spider_tree_first(st_spider_conn*) /test/11.7_dbg_san/storage/spider/spd_conn.cc:1605 #1 0x146ee326b80b in spider_rollback(handlerton*, THD*, bool) /test/11.7_dbg_san/storage/spider/spd_trx.cc:3188 #2 0x55de60e31f74 in ha_rollback_trans(THD*, bool) /test/11.7_dbg_san/sql/handler.cc:2336 #3 0x55de6029bff4 in trans_rollback_stmt(THD*) /test/11.7_dbg_san/sql/transaction.cc:566 #4 0x55de5f8feae3 in mysql_execute_command(THD*, bool) /test/11.7_dbg_san/sql/sql_parse.cc:5927 #5 0x55de5f903194 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/11.7_dbg_san/sql/sql_parse.cc:7876 #6 0x55de5f911290 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/11.7_dbg_san/sql/sql_parse.cc:1892 #7 0x55de5f91e013 in do_command(THD*, bool) /test/11.7_dbg_san/sql/sql_parse.cc:1405 #8 0x55de60206ab3 in do_handle_one_connection(CONNECT*, bool) /test/11.7_dbg_san/sql/sql_connect.cc:1448 #9 0x55de602080e7 in handle_one_connection /test/11.7_dbg_san/sql/sql_connect.cc:1350 #10 0x55de5efa21e9 in asan_thread_start(void*) (/test/UBASAN_MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd+0x7d1e1e9) (BuildId: 4b8c1741cf0d7a713ea5d94dcb62211d213957ae) #11 0x146f2629ca93 in start_thread nptl/pthread_create.c:447 #12 0x146f26329c3b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78   0x51f0000ba828 is located 424 bytes inside of 3184-byte region [0x51f0000ba680,0x51f0000bb2f0) freed by thread T12 here: #0 0x55de5f03e718 in free.part.0 (/test/UBASAN_MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd+0x7dba718) (BuildId: 4b8c1741cf0d7a713ea5d94dcb62211d213957ae) #1 0x55de632c7708 in my_free /test/11.7_dbg_san/mysys/my_malloc.c:221 #2 0x146ee342b18b in spider_free_mem(st_spider_transaction*, void*, unsigned long) /test/11.7_dbg_san/storage/spider/spd_malloc.cc:183 #3 0x146ee33412a4 in spider_free_conn(st_spider_conn*) /test/11.7_dbg_san/storage/spider/spd_conn.cc:825 #4 0x146ee3360155 in spider_free_conn_from_trx(st_spider_transaction*, st_spider_conn*, bool, bool, int*) /test/11.7_dbg_san/storage/spider/spd_conn.cc:372 #5 0x146ee325484f in spider_free_trx_conn(st_spider_transaction*, bool) /test/11.7_dbg_san/storage/spider/spd_trx.cc:114 #6 0x146ee3270865 in spider_internal_xa_rollback_by_xid(THD*, st_spider_transaction*, xid_t*) /test/11.7_dbg_san/storage/spider/spd_trx.cc:2921 #7 0x146ee327f993 in spider_xa_rollback_by_xid(handlerton*, xid_t*) /test/11.7_dbg_san/storage/spider/spd_trx.cc:3312 #8 0x55de60e186f0 in xarollback_handlerton /test/11.7_dbg_san/sql/handler.cc:2440 #9 0x55de5f99713b in plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int**, void*), int, unsigned int, void*) /test/11.7_dbg_san/sql/sql_plugin.cc:2548 #10 0x55de60e3ab0f in ha_commit_or_rollback_by_xid(xid_t*, bool) /test/11.7_dbg_san/sql/handler.cc:2462 #11 0x55de609473b7 in trans_xa_commit(THD*) /test/11.7_dbg_san/sql/xa.cc:676 #12 0x55de5f8fc0f8 in mysql_execute_command(THD*, bool) /test/11.7_dbg_san/sql/sql_parse.cc:5716 #13 0x55de5f903194 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/11.7_dbg_san/sql/sql_parse.cc:7876 #14 0x55de5f911290 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/11.7_dbg_san/sql/sql_parse.cc:1892 #15 0x55de5f91e013 in do_command(THD*, bool) /test/11.7_dbg_san/sql/sql_parse.cc:1405 #16 0x55de60206ab3 in do_handle_one_connection(CONNECT*, bool) /test/11.7_dbg_san/sql/sql_connect.cc:1448 #17 0x55de602080e7 in handle_one_connection /test/11.7_dbg_san/sql/sql_connect.cc:1350 #18 0x55de5efa21e9 in asan_thread_start(void*) (/test/UBASAN_MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd+0x7d1e1e9) (BuildId: 4b8c1741cf0d7a713ea5d94dcb62211d213957ae)   previously allocated by thread T12 here: #0 0x55de5f03fbd7 in malloc (/test/UBASAN_MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd+0x7dbbbd7) (BuildId: 4b8c1741cf0d7a713ea5d94dcb62211d213957ae) #1 0x55de632c7305 in my_malloc /test/11.7_dbg_san/mysys/my_malloc.c:93 #2 0x146ee342b5b1 in spider_bulk_alloc_mem(st_spider_transaction*, unsigned int, char const*, char const*, unsigned long, unsigned long, ...) /test/11.7_dbg_san/storage/spider/spd_malloc.cc:231 #3 0x146ee3354fb1 in spider_create_conn(st_spider_share*, ha_spider*, int, int, int*) /test/11.7_dbg_san/storage/spider/spd_conn.cc:425 #4 0x146ee335c44c in spider_get_conn(st_spider_share*, int, char*, st_spider_transaction*, ha_spider*, bool, bool, int*) /test/11.7_dbg_san/storage/spider/spd_conn.cc:730 #5 0x146ee3281bbc in spider_check_trx_and_get_conn(THD*, ha_spider*) /test/11.7_dbg_san/storage/spider/spd_trx.cc:3495 #6 0x146ee3458b6e in ha_spider::check_access_kind_for_connection(THD*, bool) /test/11.7_dbg_san/storage/spider/ha_spider.cc:621 #7 0x146ee350762d in ha_spider::dml_init() /test/11.7_dbg_san/storage/spider/ha_spider.cc:12069 #8 0x146ee352360a in ha_spider::rnd_init(bool) /test/11.7_dbg_san/storage/spider/ha_spider.cc:5375 #9 0x55de5f22cef7 in handler::ha_rnd_init(bool) /test/11.7_dbg_san/sql/handler.h:3579 #10 0x55de61f44b45 in ha_partition::check_misplaced_rows(unsigned int, bool) /test/11.7_dbg_san/sql/ha_partition.cc:11405 #11 0x55de61f48113 in ha_partition::handle_opt_part(THD*, st_ha_check_opt*, unsigned int, unsigned int) /test/11.7_dbg_san/sql/ha_partition.cc:1436 #12 0x55de61f48afd in ha_partition::handle_opt_partitions(THD*, st_ha_check_opt*, unsigned int) /test/11.7_dbg_san/sql/ha_partition.cc:1596 #13 0x55de61f4a3ee in ha_partition::repair(THD*, st_ha_check_opt*) /test/11.7_dbg_san/sql/ha_partition.cc:1348 #14 0x55de60e63c8b in handler::ha_repair(THD*, st_ha_check_opt*) /test/11.7_dbg_san/sql/handler.cc:5406 #15 0x55de602800c1 in mysql_admin_table /test/11.7_dbg_san/sql/sql_admin.cc:937 #16 0x55de60290337 in Sql_cmd_repair_table::execute(THD*) /test/11.7_dbg_san/sql/sql_admin.cc:1698 #17 0x55de60209fab in Sql_cmd_alter_table_repair_partition::execute(THD*) /test/11.7_dbg_san/sql/sql_partition_admin.cc:839 #18 0x55de5f8fd710 in mysql_execute_command(THD*, bool) /test/11.7_dbg_san/sql/sql_parse.cc:5849 #19 0x55de5f903194 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/11.7_dbg_san/sql/sql_parse.cc:7876 #20 0x55de5f911290 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/11.7_dbg_san/sql/sql_parse.cc:1892 #21 0x55de5f91e013 in do_command(THD*, bool) /test/11.7_dbg_san/sql/sql_parse.cc:1405 #22 0x55de60206ab3 in do_handle_one_connection(CONNECT*, bool) /test/11.7_dbg_san/sql/sql_connect.cc:1448 #23 0x55de602080e7 in handle_one_connection /test/11.7_dbg_san/sql/sql_connect.cc:1350 #24 0x55de5efa21e9 in asan_thread_start(void*) (/test/UBASAN_MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd+0x7d1e1e9) (BuildId: 4b8c1741cf0d7a713ea5d94dcb62211d213957ae)   Thread T12 created by T0 here: #0 0x55de5f037549 in pthread_create (/test/UBASAN_MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd+0x7db3549) (BuildId: 4b8c1741cf0d7a713ea5d94dcb62211d213957ae) #1 0x55de5f0aca96 in create_thread_to_handle_connection(CONNECT*) /test/11.7_dbg_san/sql/mysqld.cc:6270 #2 0x55de5f0bea4d in create_new_thread(CONNECT*) /test/11.7_dbg_san/sql/mysqld.cc:6332 #3 0x55de5f0bf27f in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /test/11.7_dbg_san/sql/mysqld.cc:6394 #4 0x55de5f0c0567 in handle_connections_sockets() /test/11.7_dbg_san/sql/mysqld.cc:6507 #5 0x55de5f0c524e in mysqld_main(int, char**) /test/11.7_dbg_san/sql/mysqld.cc:6165 #6 0x55de5f09a917 in main /test/11.7_dbg_san/sql/main.cc:34 #7 0x146f2622a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 #8 0x146f2622a28a in __libc_start_main_impl ../csu/libc-start.c:360 #9 0x55de5ef7f7f4 in _start (/test/UBASAN_MD090924-mariadb-11.7.0-linux-x86_64-dbg/bin/mariadbd+0x7cfb7f4) (BuildId: 4b8c1741cf0d7a713ea5d94dcb62211d213957ae)   SUMMARY: AddressSanitizer: heap-use-after-free /test/11.7_dbg_san/storage/spider/spd_conn.cc:1605 in spider_tree_first(st_spider_conn*) Shadow bytes around the buggy address: 0x51f0000ba580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x51f0000ba600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x51f0000ba680: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51f0000ba700: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51f0000ba780: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd =>0x51f0000ba800: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd 0x51f0000ba880: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51f0000ba900: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51f0000ba980: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51f0000baa00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x51f0000baa80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==798838==ABORTING 240924 18:32:52 [ERROR] mysqld got signal 6 ;

Roel Van de Paar added a comment - 2024-09-24 08:46 - edited I have been able to repeat the SIGSEGV in spider_tree_first in a debug build, and that three times, with this testcase: . /mtr --repeat 200 test There will be lots of output due to offset tables diffs, but when it crashes, MTR will stop. Once it crashed immediately, the other two times it took some time (minutes). --source include/have_log_bin.inc --source include/have_partition.inc INSTALL PLUGIN Spider SONAME 'ha_spider.so' ; CREATE TABLE t (c CHAR (1) KEY ,c2 CHAR (1) BINARY CHARACTER SET 'utf8' COLLATE 'utf8_bin' ,c3 CHAR (1) BINARY ,c4 VARCHAR (1) CHARACTER SET 'utf8' COLLATE 'utf8_bin' ) ENGINE=Spider ROW_FORMAT=COMPACT; SET SESSION pseudo_slave_mode= ON ; --error ER_TABLE_EXISTS_ERROR CREATE TABLE t (pk INT , KEY (pk)) ENGINE=Spider; XA START 'xa1' ; --error ER_CONNECT_TO_FOREIGN_DATA_SOURCE INSERT INTO t (c) VALUES (1); XA END 'xa1' ; XA PREPARE 'xa1' ; CREATE TABLE t3 (id INT ) ENGINE=Spider PARTITION BY RANGE (id) (PARTITION p0 VALUES LESS THAN (0),PARTITION p VALUES LESS THAN (1),PARTITION p2 VALUES LESS THAN (200)); ALTER TABLE t3 REPAIR PARTITION p; --error ER_XA_RBROLLBACK XA COMMIT 'xa1' ; --error ER_CONNECT_TO_FOREIGN_DATA_SOURCE SELECT * FROM t; The have_log_bin should not be required, but the issue did not replay without it When the same MTR testcase is run against an optimized build in the same way, it produces the spider_db_rollback stack.

Yuchen Pei added a comment - 2024-10-02 03:25 - edited I can reproduce this locally without exception at the current 11.2 f1b4d36cc39981a2abd211993fcb5cce003e6940, using the testcase provided in the previous comment. A quick look tells me that it is a use-after-free of a SPIDER_CONN . At crash: (rr) p trx->join_trx_top $11 = (SPIDER_CONN *) 0x7f2f2c327d48 (rr) p trx->join_trx_top->c_small $12 = (st_spider_conn *) 0x8f8f8f8f8f8f8f8f # stack # 1 in spider_rollback of /home/ycp/source/mariadb-server/11.2/src/storage/spider/spd_trx.cc:3187 # 2 in ha_rollback_trans of /home/ycp/source/mariadb-server/11.2/src/sql/handler.cc:2324 # 3 in trans_rollback_stmt of /home/ycp/source/mariadb-server/11.2/src/sql/transaction.cc:566 put a watch -l on trx->join_trx_top->c_small and reverse-continue, and we reach a stack containing a spider_free_conn() call with the same SPIDER_CONN : (rr) p conn $14 = (SPIDER_CONN *) 0x7f2f2c327d48 # stack: # 5 in spider_free_conn of /home/ycp/source/mariadb-server/11.2/src/storage/spider/spd_conn.cc:817 # 6 in spider_free_conn_from_trx of /home/ycp/source/mariadb-server/11.2/src/storage/spider/spd_conn.cc:370 # 7 in spider_free_trx_conn of /home/ycp/source/mariadb-server/11.2/src/storage/spider/spd_trx.cc:113 # 8 in spider_internal_xa_rollback_by_xid of /home/ycp/source/mariadb-server/11.2/src/storage/spider/spd_trx.cc:2920 # 9 in spider_xa_rollback_by_xid of /home/ycp/source/mariadb-server/11.2/src/storage/spider/spd_trx.cc:3311 # 10 in xarollback_handlerton of /home/ycp/source/mariadb-server/11.2/src/sql/handler.cc:2428

Yuchen Pei added a comment - 2024-10-04 08:25 10.5 ok, 10.6 bad. Bisecting result: e95bba9c58c0f08bdc6ce031d11fb5b52e814e53 is the first bad commit commit e95bba9c58c0f08bdc6ce031d11fb5b52e814e53 Merge: 686865e112f 2c60d43d7df Author: Sergei Golubchik <serg@mariadb.org> Date: Fri Dec 15 20:00:58 2023 +0100 Merge branch '10.5' into 10.6

Yuchen Pei added a comment - 2024-10-08 04:28 - edited Compare the runs between the first bad commit and its parent 10.6 commit, we see that in both cases, the SPIDER_CONN* trx->join_trx_top , assigned at the ALTER TABLE statement, is freed in the XA COMMIT statement, before spider attempts to access it again in spider_rolloback() in the final SELECT statement. The difference is that in the good commit, the creation of a new SPIDER_CONN in a handler info() call happens to create one at exactly the same address. I am not sure why but this looks more like a coincidence to me, and the bug is hidden in 10.5 as well.