[MDEV-28352] Spider: heap-use-after-free in ha_spider::lock_tables(), heap freed by spider_commit() Created: 2022-04-20  Updated: 2023-12-08  Resolved: 2022-06-27

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.5, 10.6, 10.7, 10.8, 10.9
Fix Version/s: 10.5.17, 10.6.9, 10.7.5, 10.8.4, 10.9.2

Type: Bug Priority: Critical
Reporter: Nayuta Yanagisawa (Inactive) Assignee: Nayuta Yanagisawa (Inactive)
Resolution: Fixed Votes: 0
Labels: not-10.4

Issue Links:
Blocks
blocks MDEV-28676 Spider: Got error 12701 when reading ... Closed
blocks MDEV-28683 Spider: SIGSEGV in spider_db_direct_d... Closed
Relates
relates to MDEV-28775 Implement connection manager in Spider Stalled
relates to MDEV-30014 heap-use-after-free in ha_spider::lo... Closed
relates to MDEV-27239 Spider: Assertion `thd->transaction->... Closed

 Description   

The test for MDEV-27239 is disabled due to the present bug. Enable the test once the bug is fixed.

--disable_query_log
 
--disable_result_log
 
--source ../../t/test_init.inc
 
--enable_result_log
 
--enable_query_log
 
 
 
CREATE DATABASE auto_test_local;
 
USE auto_test_local;
 
 
 
CREATE TABLE tbl_a (a INT) ENGINE=SPIDER;
 
FLUSH TABLE tbl_a WITH READ LOCK;
 
BEGIN;
 
 
 
DROP DATABASE auto_test_local;
 
 
 
--disable_query_log
 
--disable_result_log
 
--source ../../t/test_deinit.inc
 
--enable_result_log
 
--enable_query_log


=================================================================
 
==153765==ERROR: AddressSanitizer: heap-use-after-free on address 0x61f0000541ec at pc 0x7f218e572770 bp 0x7f218ec926b0 sp 0x7f218ec926a0
 
READ of size 4 at 0x61f0000541ec thread T16
 
    #0 0x7f218e57276f in ha_spider::lock_tables() /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/ha_spider.cc:16309
 
    #1 0x7f218e4d6a3b in ha_spider::external_lock(THD*, int) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/ha_spider.cc:1286
 
    #2 0x55d928ec682c in handler::ha_external_lock(THD*, int) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/handler.cc:6718
 
    #3 0x55d92892ebe1 in handler::ha_external_unlock(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/handler.h:3392
 
    #4 0x55d9291dd20b in unlock_external /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/lock.cc:727
 
    #5 0x55d9291da8ec in mysql_unlock_tables(THD*, st_mysql_lock*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/lock.cc:432
 
    #6 0x55d9291da70f in mysql_unlock_tables(THD*, st_mysql_lock*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/lock.cc:415
 
    #7 0x55d9284b5694 in close_thread_tables(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_base.cc:911
 
    #8 0x55d9284be26f in Locked_tables_list::unlock_locked_tables(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_base.cc:2376
 
    #9 0x55d928aebf43 in trans_begin(THD*, unsigned int) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/transaction.cc:115
 
    #10 0x55d928672e8d in mysql_execute_command(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:5674
 
    #11 0x55d928683719 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:8116
 
    #12 0x55d928658e7f in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:1907
 
    #13 0x55d928655583 in do_command(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:1375
 
    #14 0x55d928aaaf38 in do_handle_one_connection(CONNECT*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_connect.cc:1418
 
    #15 0x55d928aaa76f in handle_one_connection /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_connect.cc:1312
 
    #16 0x55d92973bbba in pfs_spawn_thread /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/perfschema/pfs.cc:2201
 
    #17 0x7f219d563946  (/lib/x86_64-linux-gnu/libc.so.6+0x94946)
 
    #18 0x7f219d5f3a43 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x124a43)
 
 
 
0x61f0000541ec is located 364 bytes inside of 3120-byte region [0x61f000054080,0x61f000054cb0)
 
freed by thread T16 here:
 
    #0 0x7f219e090517 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127
 
    #1 0x55d92a3ef560 in my_free /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/mysys/my_malloc.c:211
 
    #2 0x7f218e4a50c7 in spider_free_mem(st_spider_transaction*, void*, unsigned long) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_malloc.cc:188
 
    #3 0x7f218e3bfa38 in spider_free_conn(st_spider_conn*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_conn.cc:1404
 
    #4 0x7f218e3b80ab in spider_free_conn_from_trx(st_spider_transaction*, st_spider_conn*, bool, bool, int*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_conn.cc:420
 
    #5 0x7f218e2ee987 in spider_free_trx_conn(st_spider_transaction*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_trx.cc:117
 
    #6 0x7f218e30c0b8 in spider_commit(handlerton*, THD*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_trx.cc:3486
 
    #7 0x55d928e9bc95 in commit_one_phase_2 /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/handler.cc:1971
 
    #8 0x55d928e9b9a3 in ha_commit_one_phase(THD*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/handler.cc:1950
 
    #9 0x55d928e99c6a in ha_commit_trans(THD*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/handler.cc:1744
 
    #10 0x55d928aeed06 in trans_commit_stmt(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/transaction.cc:472
 
    #11 0x55d92867650e in mysql_execute_command(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:6132
 
    #12 0x55d928683719 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:8116
 
    #13 0x55d928658e7f in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:1907
 
    #14 0x55d928655583 in do_command(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:1375
 
    #15 0x55d928aaaf38 in do_handle_one_connection(CONNECT*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_connect.cc:1418
 
    #16 0x55d928aaa76f in handle_one_connection /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_connect.cc:1312
 
    #17 0x55d92973bbba in pfs_spawn_thread /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/perfschema/pfs.cc:2201
 
    #18 0x7f219d563946  (/lib/x86_64-linux-gnu/libc.so.6+0x94946)
 
 
 
previously allocated by thread T16 here:
 
    #0 0x7f219e090867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
 
    #1 0x55d92a3ee6d9 in my_malloc /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/mysys/my_malloc.c:90
 
    #2 0x7f218e4a5833 in spider_bulk_alloc_mem(st_spider_transaction*, unsigned int, char const*, char const*, unsigned long, unsigned long, ...) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_malloc.cc:236
 
    #3 0x7f218e3b9162 in spider_create_conn(st_spider_share*, ha_spider*, int, int, unsigned int, int*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_conn.cc:593
 
    #4 0x7f218e3be84b in spider_get_conn(st_spider_share*, int, char*, st_spider_transaction*, ha_spider*, bool, bool, unsigned int, int*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_conn.cc:1218
 
    #5 0x7f218e432d07 in spider_get_share(char const*, TABLE*, THD*, ha_spider*, int*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/spd_table.cc:5356
 
    #6 0x7f218e4cd66b in ha_spider::open(char const*, int, unsigned int) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/ha_spider.cc:444
 
    #7 0x55d928ea3f66 in handler::ha_open(TABLE*, char const*, int, unsigned int, st_mem_root*, List<String>*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/handler.cc:3013
 
    #8 0x55d9289c64b1 in open_table_from_share(THD*, TABLE_SHARE*, st_mysql_const_lex_string const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, List<String>*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/table.cc:4252
 
    #9 0x55d9284bb4e8 in open_table(THD*, TABLE_LIST*, Open_table_context*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_base.cc:2001
 
    #10 0x55d9284c51dc in open_and_process_table /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_base.cc:3788
 
    #11 0x55d9284c7e29 in open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_base.cc:4271
 
    #12 0x55d9284cd177 in open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_base.cc:5218
 
    #13 0x55d928b46ea7 in open_and_lock_tables /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_base.h:276
 
    #14 0x55d928b493cb in flush_tables_with_read_lock(THD*, TABLE_LIST*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_reload.cc:584
 
    #15 0x55d928671a2c in mysql_execute_command(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:5481
 
    #16 0x55d928683719 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:8116
 
    #17 0x55d928658e7f in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:1907
 
    #18 0x55d928655583 in do_command(THD*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_parse.cc:1375
 
    #19 0x55d928aaaf38 in do_handle_one_connection(CONNECT*, bool) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_connect.cc:1418
 
    #20 0x55d928aaa76f in handle_one_connection /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/sql_connect.cc:1312
 
    #21 0x55d92973bbba in pfs_spawn_thread /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/perfschema/pfs.cc:2201
 
    #22 0x7f219d563946  (/lib/x86_64-linux-gnu/libc.so.6+0x94946)
 
 
 
Thread T16 created by T0 here:
 
    #0 0x7f219e034685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
 
    #1 0x55d929737644 in my_thread_create /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/perfschema/my_thread.h:52
 
    #2 0x55d92973bfad in pfs_spawn_thread_v1 /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/perfschema/pfs.cc:2252
 
    #3 0x55d92833c702 in inline_mysql_thread_create /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/include/mysql/psi/mysql_thread.h:1323
 
    #4 0x55d928352f90 in create_thread_to_handle_connection(CONNECT*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/mysqld.cc:6018
 
    #5 0x55d928353626 in create_new_thread(CONNECT*) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/mysqld.cc:6077
 
    #6 0x55d928353989 in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/mysqld.cc:6142
 
    #7 0x55d9283545d4 in handle_connections_sockets() /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/mysqld.cc:6269
 
    #8 0x55d928352764 in mysqld_main(int, char**) /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/mysqld.cc:5664
 
    #9 0x55d92833b1cc in main /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/sql/main.cc:25
 
    #10 0x7f219d4f8fcf  (/lib/x86_64-linux-gnu/libc.so.6+0x29fcf)
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /home/nayuta_mariadb/repo/mariadb-server/bb-10.5-MDEV-27239/storage/spider/ha_spider.cc:16309 in ha_spider::lock_tables()
 
Shadow bytes around the buggy address:
 
  0x0c3e800027e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c3e800027f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c3e80002800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c3e80002810: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c3e80002820: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
=>0x0c3e80002830: fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd
 
  0x0c3e80002840: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c3e80002850: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c3e80002860: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c3e80002870: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c3e80002880: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
  Shadow gap:              cc
 
==153765==ABORTING
 
----------SERVER LOG END-------------



 Comments   
Comment by Nayuta Yanagisawa (Inactive) [ 2022-04-20 ]

Reproducible on mariadb-10.5.4 but not on mariadb-10.5.3.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-04-20 ] 

> git bisect good
 
d3a6ed05500afa2bbe4c3756d5aae64310144a3c is the first bad commit
 
commit d3a6ed05500afa2bbe4c3756d5aae64310144a3c
 
Author: Kentoku SHIBA <kentokushiba@gmail.com>
 
Date:   Tue Mar 3 02:52:46 2020 +0900
 
 
 
    fix divided lock table issue of Spider
 
 
 
 storage/spider/ha_spider.cc | 54 +++++++++++++++++++++++++++++++++++++++++++--
 
 storage/spider/ha_spider.h  |  1 +
 
 2 files changed, 53 insertions(+), 2 deletions(-)

Comment by Nayuta Yanagisawa (Inactive) [ 2022-04-20 ]

The purpose of d3a6ed05500afa2bbe4c3756d5aae64310144a3c is not clear. I need to reveals that what the commit is for.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-06-13 ]

holyfoot https://github.com/MariaDB/server/commit/bb0ff8b8101c96c700fc9c31f22ca5bb9f933581

Comment by Nayuta Yanagisawa (Inactive) [ 2022-06-16 ]

Let me withdraw the patch because I'd like to give more thought to the problem.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-06-16 ]

One of the causes of the bug is that Spider makes a wrong assumption about the lifetime of ha_spider. Allocation and destruction of ha_spider is the job of the server, and cannot be controlled from the Spider (storage engine) side.

The server stores an instance of ha_spider to MYSQL_LOCK::table at FLUSH TABLE WITH READ LOCK and keeps it until the table is unlocked. However, Spider releases the connection held by the ha_spider instance before the unlock. This results in the heap-use-after-free.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-06-16 ]

Probably, the fix would be to make Spider not to release connections at commit/rollback when Spider is taking external locks.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-06-16 ]

holyfoot https://github.com/MariaDB/server/commit/a3094571db2ab3f1e8157a6241225248848b065e

Comment by Nayuta Yanagisawa (Inactive) [ 2022-06-19 ]

holyfoot I've updated my patch: https://github.com/MariaDB/server/commit/6d4ed8cb63d959be3807e1dcbcf07ab5750f93a8

Comment by Alexey Botchkov [ 2022-06-27 ]

ok to push.

Comment by Roel Van de Paar [ 2022-11-15 ]

See MDEV-30014

Generated at Thu Feb 08 10:00:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.