[Current thread is 1 (Thread 0x14ce635f6700 (LWP 2425768))]
(gdb) bt
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1 0x000055872a9ca4d7 in my_write_core (sig=sig@entry=11) at /test/10.5_dbg/mysys/stacktrace.c:518
#2 0x000055872a1849ba in handle_fatal_signal (sig=11) at /test/10.5_dbg/sql/signal_handler.cc:330
#3 <signal handler called>
#4 0x0000558729e65bfb in Query_cache::unlink_table (this=this@entry=0x55872b714920 <query_cache>, node=0x14ce3f880c78) at /test/10.5_dbg/sql/sql_cache.cc:3618
#5 0x0000558729e67431 in Query_cache::register_all_tables (this=this@entry=0x55872b714920 <query_cache>, thd=thd@entry=0x14ce3f815088, block=block@entry=0x14ce3f880160, tables_used=tables_used@entry=0x14ce3f8747e8, tables_arg=tables_arg@entry=100) at /test/10.5_dbg/sql/sql_cache.cc:3490
#6 0x0000558729e67ba3 in Query_cache::store_query (this=0x55872b714920 <query_cache>, thd=thd@entry=0x14ce3f815088, tables_used=tables_used@entry=0x14ce3f8747e8) at /test/10.5_dbg/sql/sql_cache.cc:1551
#7 0x0000558729edf7cf in execute_sqlcom_select (thd=thd@entry=0x14ce3f815088, all_tables=0x14ce3f8747e8) at /test/10.5_dbg/sql/sql_parse.cc:6208
#8 0x0000558729ed88b6 in mysql_execute_command (thd=thd@entry=0x14ce3f815088) at /test/10.5_dbg/sql/sql_parse.cc:3931
#9 0x0000558729ee5752 in mysql_parse (thd=thd@entry=0x14ce3f815088, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14ce635f5350, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /test/10.5_dbg/sql/sql_parse.cc:7993
#10 0x0000558729ed2204 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14ce3f815088, packet=packet@entry=0x14ce3f867089 "SELECT COUNT(*) FROM t WHERE c1=2", packet_length=packet_length@entry=33, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /test/10.5_dbg/sql/sql_parse.cc:1866
#11 0x0000558729ed09de in do_command (thd=0x14ce3f815088) at /test/10.5_dbg/sql/sql_parse.cc:1347
#12 0x000055872a02cc3b in do_handle_one_connection (connect=<optimized out>, connect@entry=0x14ce424c7808, put_in_cache=put_in_cache@entry=true) at /test/10.5_dbg/sql/sql_connect.cc:1411
#13 0x000055872a02d357 in handle_one_connection (arg=arg@entry=0x14ce424c7808) at /test/10.5_dbg/sql/sql_connect.cc:1313
#14 0x000055872a490ca8 in pfs_spawn_thread (arg=0x14ce60446508) at /test/10.5_dbg/storage/perfschema/pfs.cc:2201
#15 0x000014ce6256f6db in start_thread (arg=0x14ce635f6700) at pthread_create.c:463
#16 0x000014ce6196da3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
MDEV-34075corruption when query cache cannot allocate block
Closed
relates to
MDEV-23127Server crash in Query_cache::double_linked_list_exclude or Assertion `table_block_data->m_cached_query_count >= 0' failed in Query_cache::unlink_table
Confirmed
MDEV-34075corruption when query cache cannot allocate block
Roel Van de Paar
added a comment - marko Thank you for the quick and effective input. Indeed you were correct. Using the first testcase from this comment, we see:
10.11.0 fe1f8f2c6b6f3b8e3383168225f9ae7853028947 (Optimized, UBASAN)
Version: '10.11.0-MariaDB' socket: '/test/UBASAN_MD010922-mariadb-10.11.0-linux-x86_64-opt/socket.sock' port: 11690 MariaDB Server
=================================================================
==2821247==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x63100009c788 at pc 0x563c18de1d1b bp 0x14d63da584f0 sp 0x14d63da584e0
WRITE of size 8 at 0x63100009c788 thread T15
#0 0x563c18de1d1a in Query_cache_block::init(unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:824
#1 0x563c18de83a8 in Query_cache::split_block(Query_cache_block*, unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:3791
#2 0x563c18de9e59 in Query_cache::allocate_block(unsigned long, char, unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:3683
#3 0x563c18de9e59 in Query_cache::allocate_block(unsigned long, char, unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:3659
#4 0x563c18de9f73 in Query_cache::write_block_data(unsigned long, unsigned char*, unsigned long, Query_cache_block::block_type, unsigned int) /test/10.11_opt_san/sql/sql_cache.cc:3038
#5 0x563c18dec309 in Query_cache::insert_table(THD*, unsigned long, char const*, Query_cache_block_table*, unsigned long, unsigned char, unsigned char, char (*)(THD*, char const*, unsigned int, unsigned long long*), unsigned long long, char) /test/10.11_opt_san/sql/sql_cache.cc:3554
#6 0x563c1ba73f0a in ha_partition::reg_query_cache_dependant_table(THD*, char*, unsigned int, char*, unsigned int, unsigned char, Query_cache*, Query_cache_block_table**, handler*, unsigned int*) /test/10.11_opt_san/sql/ha_partition.cc:2618
#7 0x563c1ba753d0 in ha_partition::register_query_cache_dependant_tables(THD*, Query_cache*, Query_cache_block_table**, unsigned int*) /test/10.11_opt_san/sql/ha_partition.cc:2709
#8 0x563c18ded3e8 in Query_cache::register_tables_from_list(THD*, TABLE_LIST*, unsigned int, Query_cache_block_table**) /test/10.11_opt_san/sql/sql_cache.cc:3447
#9 0x563c18dede24 in Query_cache::register_all_tables(THD*, Query_cache_block*, TABLE_LIST*, unsigned int) /test/10.11_opt_san/sql/sql_cache.cc:3482
#10 0x563c18dfd0b7 in Query_cache::store_query(THD*, TABLE_LIST*) /test/10.11_opt_san/sql/sql_cache.cc:1554
#11 0x563c19163c1b in execute_sqlcom_select /test/10.11_opt_san/sql/sql_parse.cc:6260
#12 0x563c191b453b in mysql_execute_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:3945
#13 0x563c19134500 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/10.11_opt_san/sql/sql_parse.cc:8035
#14 0x563c191890ff in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/10.11_opt_san/sql/sql_parse.cc:1894
#15 0x563c191943fd in do_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:1407
#16 0x563c19a7d4cd in do_handle_one_connection(CONNECT*, bool) /test/10.11_opt_san/sql/sql_connect.cc:1418
#17 0x563c19a7fb3c in handle_one_connection /test/10.11_opt_san/sql/sql_connect.cc:1312
#18 0x14d660a69608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
#19 0x14d65fcde132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)
0x63100009c788 is located 416 bytes to the right of 65000-byte region [0x63100008c800,0x63100009c5e8)
allocated by thread T15 here:
#0 0x563c18908a08 in __interceptor_malloc (/test/UBASAN_MD010922-mariadb-10.11.0-linux-x86_64-opt/bin/mariadbd+0x79c3a08)
#1 0x563c1cdf8824 in my_malloc /test/10.11_opt_san/mysys/my_malloc.c:90
#2 0x563c18de4b90 in Query_cache::init_cache() /test/10.11_opt_san/sql/sql_cache.cc:2662
#3 0x563c18de6383 in Query_cache::resize(unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:1333
#4 0x563c19b0a163 in fix_query_cache_size /test/10.11_opt_san/sql/sys_vars.cc:3188
#5 0x563c18b3b31f in sys_var::update(THD*, set_var*) /test/10.11_opt_san/sql/set_var.cc:208
#6 0x563c18b3ed71 in set_var::update(THD*) /test/10.11_opt_san/sql/set_var.cc:863
#7 0x563c18b47479 in sql_set_variables(THD*, List<set_var_base>*, bool) /test/10.11_opt_san/sql/set_var.cc:745
#8 0x563c191a3e64 in mysql_execute_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:5034
#9 0x563c19134500 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/10.11_opt_san/sql/sql_parse.cc:8035
#10 0x563c191890ff in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/10.11_opt_san/sql/sql_parse.cc:1894
#11 0x563c191943fd in do_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:1407
#12 0x563c19a7d4cd in do_handle_one_connection(CONNECT*, bool) /test/10.11_opt_san/sql/sql_connect.cc:1418
#13 0x563c19a7fb3c in handle_one_connection /test/10.11_opt_san/sql/sql_connect.cc:1312
#14 0x14d660a69608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
Thread T15 created by T0 here:
#0 0x563c18835a45 in pthread_create (/test/UBASAN_MD010922-mariadb-10.11.0-linux-x86_64-opt/bin/mariadbd+0x78f0a45)
#1 0x563c18958f83 in create_thread_to_handle_connection(CONNECT*) /test/10.11_opt_san/sql/mysqld.cc:6018
#2 0x563c1896a00f in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /test/10.11_opt_san/sql/mysqld.cc:6139
#3 0x563c1896b037 in handle_connections_sockets() /test/10.11_opt_san/sql/mysqld.cc:6263
#4 0x563c1896df94 in mysqld_main(int, char**) /test/10.11_opt_san/sql/mysqld.cc:5913
#5 0x14d65fbe3082 in __libc_start_main ../csu/libc-start.c:308
SUMMARY: AddressSanitizer: heap-buffer-overflow /test/10.11_opt_san/sql/sql_cache.cc:824 in Query_cache_block::init(unsigned long)
Shadow bytes around the buggy address:
0x0c628000b8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c628000b8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
0x0c628000b8c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b8d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b8e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c628000b8f0: fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b910: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b920: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==2821247==ABORTING
The second testcase in the same comment does not produce an ASAN issue, but possibly due to crash interference. Idem for other testcases in this bug.
Roel Van de Paar
added a comment - - edited The second testcase in the same comment does not produce an ASAN issue, but possibly due to crash interference. Idem for other testcases in this bug.
Set of new stacks (to be expected given the stack smashing) seen with the attached additional testcase MDEV-23256_additional.sql (loop the testcase at the CLI till it crashes)
Given that this can lead to many different stacks, a fix for this bug would be great.
Roel Van de Paar
added a comment - - edited Set of new stacks (to be expected given the stack smashing) seen with the attached additional testcase MDEV-23256_additional.sql (loop the testcase at the CLI till it crashes)
SIGSEGV|_int_malloc|__GI___libc_malloc|my_malloc|root_alloc
SIGSEGV|handler::ha_thd|ha_innobase::info_low|ha_partition::info|make_join_statistics
SIGSEGV|I_P_List<TABLE, I_P_List_adapter<TABLE, &TABLE::global_free_next, &TABLE::global_free_prev>, I_P_List_null_counter, I_P_List_fast_push_back<TABLE> >::remove|tc_acquire_table|tdc_acquire_share|open_table
SIGSEGV|Query_cache::unlink_table|Query_cache::register_all_tables|Query_cache::store_query|execute_sqlcom_select
The last one is already listed above.
Given that this can lead to many different stacks, a fix for this bug would be great.
Roel Van de Paar
added a comment - Another testcase which sporadically gives a different assert:
--source include/have_innodb.inc
--source include/have_partition.inc
SET GLOBAL query_cache_type=DEMAND;
SET GLOBAL query_cache_size=81920;
SET SESSION query_cache_type=1;
CREATE TABLE t1 (c1 INT NOT NULL , c2 CHAR (5)) ENGINE=InnoDB PARTITION BY LINEAR KEY (c1) PARTITIONS 99;
SELECT * FROM t1 WHERE c1 <= '1998-12-29 00:00:00' ORDER BY c1,c2;
--error ER_BAD_FIELD_ERROR
SELECT GROUP_CONCAT(a SEPARATOR '###' ) AS NAMES FROM t1 HAVING LEFT (NAMES, 1)= 'J' ;
SELECT * FROM t1;
SELECT COUNT (*) FROM t1;
--error ER_BAD_FIELD_ERROR
SELECT C.a, c.a FROM t1 c, t1 C;
table_block_data->m_cached_query_count == 0|SIGABRT|Query_cache::unlink_table|Query_cache::register_all_tables|Query_cache::store_query|execute_sqlcom_select
marko Thank you for the quick and effective input. Indeed you were correct. Using the first testcase from this comment, we see:
10.11.0 fe1f8f2c6b6f3b8e3383168225f9ae7853028947 (Optimized, UBASAN)
Version: '10.11.0-MariaDB' socket: '/test/UBASAN_MD010922-mariadb-10.11.0-linux-x86_64-opt/socket.sock' port: 11690 MariaDB Server
=================================================================
==2821247==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x63100009c788 at pc 0x563c18de1d1b bp 0x14d63da584f0 sp 0x14d63da584e0
WRITE of size 8 at 0x63100009c788 thread T15
#0 0x563c18de1d1a in Query_cache_block::init(unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:824
#1 0x563c18de83a8 in Query_cache::split_block(Query_cache_block*, unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:3791
#2 0x563c18de9e59 in Query_cache::allocate_block(unsigned long, char, unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:3683
#3 0x563c18de9e59 in Query_cache::allocate_block(unsigned long, char, unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:3659
#4 0x563c18de9f73 in Query_cache::write_block_data(unsigned long, unsigned char*, unsigned long, Query_cache_block::block_type, unsigned int) /test/10.11_opt_san/sql/sql_cache.cc:3038
#5 0x563c18dec309 in Query_cache::insert_table(THD*, unsigned long, char const*, Query_cache_block_table*, unsigned long, unsigned char, unsigned char, char (*)(THD*, char const*, unsigned int, unsigned long long*), unsigned long long, char) /test/10.11_opt_san/sql/sql_cache.cc:3554
#6 0x563c1ba73f0a in ha_partition::reg_query_cache_dependant_table(THD*, char*, unsigned int, char*, unsigned int, unsigned char, Query_cache*, Query_cache_block_table**, handler*, unsigned int*) /test/10.11_opt_san/sql/ha_partition.cc:2618
#7 0x563c1ba753d0 in ha_partition::register_query_cache_dependant_tables(THD*, Query_cache*, Query_cache_block_table**, unsigned int*) /test/10.11_opt_san/sql/ha_partition.cc:2709
#8 0x563c18ded3e8 in Query_cache::register_tables_from_list(THD*, TABLE_LIST*, unsigned int, Query_cache_block_table**) /test/10.11_opt_san/sql/sql_cache.cc:3447
#9 0x563c18dede24 in Query_cache::register_all_tables(THD*, Query_cache_block*, TABLE_LIST*, unsigned int) /test/10.11_opt_san/sql/sql_cache.cc:3482
#10 0x563c18dfd0b7 in Query_cache::store_query(THD*, TABLE_LIST*) /test/10.11_opt_san/sql/sql_cache.cc:1554
#11 0x563c19163c1b in execute_sqlcom_select /test/10.11_opt_san/sql/sql_parse.cc:6260
#12 0x563c191b453b in mysql_execute_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:3945
#13 0x563c19134500 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/10.11_opt_san/sql/sql_parse.cc:8035
#14 0x563c191890ff in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/10.11_opt_san/sql/sql_parse.cc:1894
#15 0x563c191943fd in do_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:1407
#16 0x563c19a7d4cd in do_handle_one_connection(CONNECT*, bool) /test/10.11_opt_san/sql/sql_connect.cc:1418
#17 0x563c19a7fb3c in handle_one_connection /test/10.11_opt_san/sql/sql_connect.cc:1312
#18 0x14d660a69608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
#19 0x14d65fcde132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)
0x63100009c788 is located 416 bytes to the right of 65000-byte region [0x63100008c800,0x63100009c5e8)
allocated by thread T15 here:
#0 0x563c18908a08 in __interceptor_malloc (/test/UBASAN_MD010922-mariadb-10.11.0-linux-x86_64-opt/bin/mariadbd+0x79c3a08)
#1 0x563c1cdf8824 in my_malloc /test/10.11_opt_san/mysys/my_malloc.c:90
#2 0x563c18de4b90 in Query_cache::init_cache() /test/10.11_opt_san/sql/sql_cache.cc:2662
#3 0x563c18de6383 in Query_cache::resize(unsigned long) /test/10.11_opt_san/sql/sql_cache.cc:1333
#4 0x563c19b0a163 in fix_query_cache_size /test/10.11_opt_san/sql/sys_vars.cc:3188
#5 0x563c18b3b31f in sys_var::update(THD*, set_var*) /test/10.11_opt_san/sql/set_var.cc:208
#6 0x563c18b3ed71 in set_var::update(THD*) /test/10.11_opt_san/sql/set_var.cc:863
#7 0x563c18b47479 in sql_set_variables(THD*, List<set_var_base>*, bool) /test/10.11_opt_san/sql/set_var.cc:745
#8 0x563c191a3e64 in mysql_execute_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:5034
#9 0x563c19134500 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/10.11_opt_san/sql/sql_parse.cc:8035
#10 0x563c191890ff in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/10.11_opt_san/sql/sql_parse.cc:1894
#11 0x563c191943fd in do_command(THD*, bool) /test/10.11_opt_san/sql/sql_parse.cc:1407
#12 0x563c19a7d4cd in do_handle_one_connection(CONNECT*, bool) /test/10.11_opt_san/sql/sql_connect.cc:1418
#13 0x563c19a7fb3c in handle_one_connection /test/10.11_opt_san/sql/sql_connect.cc:1312
#14 0x14d660a69608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
Thread T15 created by T0 here:
#0 0x563c18835a45 in pthread_create (/test/UBASAN_MD010922-mariadb-10.11.0-linux-x86_64-opt/bin/mariadbd+0x78f0a45)
#1 0x563c18958f83 in create_thread_to_handle_connection(CONNECT*) /test/10.11_opt_san/sql/mysqld.cc:6018
#2 0x563c1896a00f in handle_accepted_socket(st_mysql_socket, st_mysql_socket) /test/10.11_opt_san/sql/mysqld.cc:6139
#3 0x563c1896b037 in handle_connections_sockets() /test/10.11_opt_san/sql/mysqld.cc:6263
#4 0x563c1896df94 in mysqld_main(int, char**) /test/10.11_opt_san/sql/mysqld.cc:5913
#5 0x14d65fbe3082 in __libc_start_main ../csu/libc-start.c:308
SUMMARY: AddressSanitizer: heap-buffer-overflow /test/10.11_opt_san/sql/sql_cache.cc:824 in Query_cache_block::init(unsigned long)
Shadow bytes around the buggy address:
0x0c628000b8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c628000b8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
0x0c628000b8c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b8d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b8e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c628000b8f0: fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b910: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b920: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c628000b940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==2821247==ABORTING