[MDEV-23966] btr_search_sys->hash_tables accessed without taking proper ahi latch Created: 2020-10-15  Updated: 2020-10-20  Resolved: 2020-10-19

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.26, 10.2
Fix Version/s: 10.2.35, 10.3.26, 10.4.16, 10.5.7

Type: Bug Priority: Major
Reporter: Matthias Leich Assignee: Thirunarayanan Balathandayuthapani
Resolution: Fixed Votes: 0
Labels: rr-profile


 Description   

Thread 3 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1087300.1092511]
0x00005600907463b4 in btr_get_search_table (index=index@entry=0x6160001a7408) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/include/btr0sea.ic:188
188             return(btr_search_sys->hash_tables[ifold % btr_ahi_parts]);
(rr) bt
#0  0x00005600907463b4 in btr_get_search_table (index=index@entry=0x6160001a7408) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/include/btr0sea.ic:188
#1  0x000056009075b58f in btr_search_update_hash_on_insert (cursor=cursor@entry=0x14d839ffe800, ahi_latch=<optimized out>) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/btr/btr0sea.cc:1846
#2  0x0000560090708d3c in btr_cur_optimistic_insert (flags=flags@entry=0, cursor=cursor@entry=0x14d839ffe800, offsets=offsets@entry=0x14d839ffe6e0, heap=heap@entry=0x14d839ffe6c0, entry=entry@entry=0x616000145908, rec=rec@entry=0x14d839ffe700, 
    big_rec=<optimized out>, n_ext=0, thr=<optimized out>, mtr=<optimized out>) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/btr/btr0cur.cc:3513
#3  0x0000560090389d3b in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=<optimized out>, mode@entry=2, index=index@entry=0x6160001a7408, n_uniq=n_uniq@entry=0, entry=entry@entry=0x616000145908, n_ext=n_ext@entry=0, thr=<optimized out>)
    at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/row/row0ins.cc:2776
#4  0x000056009039b8e4 in row_ins_clust_index_entry (index=index@entry=0x6160001a7408, entry=entry@entry=0x616000145908, thr=thr@entry=0x62100041fdd0, n_ext=n_ext@entry=0) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/row/row0ins.cc:3232
#5  0x000056009039cef8 in row_ins_index_entry (index=0x6160001a7408, entry=0x616000145908, thr=thr@entry=0x62100041fdd0) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/row/row0ins.cc:3357
#6  0x000056009039d214 in row_ins_index_entry_step (node=node@entry=0x62100041f870, thr=thr@entry=0x62100041fdd0) at /usr/include/c++/9/bits/stl_iterator.h:819
#7  0x000056009039eb51 in row_ins (node=node@entry=0x62100041f870, thr=thr@entry=0x62100041fdd0) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/row/row0ins.cc:3645
#8  0x000056009039f836 in row_ins_step (thr=thr@entry=0x62100041fdd0) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/row/row0ins.cc:3784
#9  0x00005600903fa62b in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x61a0003a74a8 "\220\377\242", prebuilt=0x62100041f188, ins_mode=<optimized out>) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/row/row0mysql.cc:1421
#10 0x000056009005dab6 in ha_innobase::write_row (this=0x61c0000e58a8, record=<optimized out>) at /home/mleich/Server/bb-10.3-MDEV-23072/storage/innobase/handler/ha_innodb.cc:7989
#11 0x000056008f937a21 in handler::ha_write_row (this=0x61c0000e58a8, buf=<optimized out>) at /home/mleich/Server/bb-10.3-MDEV-23072/sql/handler.cc:6466
#12 0x000056008f084260 in write_record (thd=thd@entry=0x62a00033c208, table=table@entry=0x61f00017a088, info=info@entry=0x14d83a000ba0) at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_insert.cc:2036
#13 0x000056008f0aae65 in mysql_insert (thd=thd@entry=0x62a00033c208, table_list=0x62b00025a5a0, fields=..., values_list=..., update_fields=..., update_values=..., duplic=<optimized out>, ignore=<optimized out>)
    at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_insert.cc:1072
#14 0x000056008f1539ac in mysql_execute_command (thd=thd@entry=0x62a00033c208) at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_parse.cc:4446
#15 0x000056008f16317f in mysql_parse (thd=thd@entry=0x62a00033c208, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14d83a002f40, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false)
    at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_parse.cc:7810
#16 0x000056008f16a8bf in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x62a00033c208, 
    packet=packet@entry=0x6290009dd209 " INSERT INTO t1 (col1,col2, col_int, col_string, col_text) VALUES /* 162 */ (162,162,162,REPEAT(SUBSTR(CAST( 162 AS CHAR),1,1), 10),REPEAT(SUBSTR(CAST( 162 AS CHAR),1,1), @fill_amount) ), (162,162,162"..., 
    packet_length=packet_length@entry=337, is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_class.h:1134
#17 0x000056008f1741d6 in do_command (thd=0x62a00033c208) at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_parse.cc:1393
#18 0x000056008f53d9d0 in do_handle_one_connection (connect=connect@entry=0x608000001d28) at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_connect.cc:1403
#19 0x000056008f53e0de in handle_one_connection (arg=0x608000001d28) at /home/mleich/Server/bb-10.3-MDEV-23072/sql/sql_connect.cc:1308
#20 0x000056d8458df609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#21 0x000050c653a5f103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(rr) quit
 
[rr 1087300 307225]201015 11:19:48 [rr 1087300 307228][ERROR] mysqld got signal 11 ;
...
Query (0x62b00025a228): INSERT INTO t1 (col1,col2, col_int, col_string, col_text) VALUES /* 162 */ (162,162,162,REPEAT(SUBSTR(CAST( 162 AS CHAR),1,1), 10),REPEAT(SUBSTR(CAST( 162 AS CHAR),1,1), @fill_amount) ), (162,162,162,REPEAT(SUBSTR(CAST( 162 AS CHAR),1,1), 10),REPEAT(SUBSTR(CAST( 162 AS CHAR),1,1), @fill_amount) )  /* E_R Thread31 QNO 114 CON_ID 48 */
# 2020-10-15T11:25:13 [1085571]
# 2020-10-15T11:25:13 [1085571] | Connection ID (thread ID): 48
# 2020-10-15T11:25:13 [1085571] | Status: KILL_TIMEOUT
 
origin/bb-10.3-MDEV-23072 8ebfe9811e805a9b508199bc0224a82f7fa69eb3 2020-10-15T13:26:18+05:30  compiled with debug + ASAN + Og
Per Thiru the SEGV cannot be related to MDEV-23072.
 
RQG
====
git clone https://github.com/mleich1/rqg --branch experimental RQG
    origin/experimental e95247cfe20eb07f11c81bfa551e45571dd3637d 
perl rqg.pl \
--grammar=conf/mariadb/table_stress_innodb.yy \
--gendata=conf/mariadb/table_stress.zz \
--gendata_sql=conf/mariadb/table_stress.sql \
--mysqld=--innodb_use_native_aio=1 \
--mysqld=--innodb_lock_schedule_algorithm=fcfs \
--mysqld=--loose-idle_write_transaction_timeout=0 \
--mysqld=--loose-idle_transaction_timeout=0 \
--mysqld=--loose-idle_readonly_transaction_timeout=0 \
--mysqld=--connect_timeout=60 \
--mysqld=--interactive_timeout=28800 \
--mysqld=--slave_net_timeout=60 \
--mysqld=--net_read_timeout=30 \
--mysqld=--net_write_timeout=60 \
--mysqld=--loose-table_lock_wait_timeout=50 \
--mysqld=--wait_timeout=28800 \
--mysqld=--lock-wait-timeout=86400 \
--mysqld=--innodb-lock-wait-timeout=50 \
--no-mask \
--queries=10000000 \
--seed=random \
--reporters=Backtrace \
--reporters=ErrorLog \
--reporters=Deadlock1 \
--validators=None \
--mysqld=--log_output=none \
--mysqld=--log-bin \
--mysqld=--log_bin_trust_function_creators=1 \
--mysqld=--loose-debug_assert_on_not_freed_memory=0 \
--engine=InnoDB \
--restart_timeout=120 \
--duration=400 \
--mysqld=--loose-innodb_fatal_semaphore_wait_threshold=400 \
--mysqld=--loose-innodb-sync-debug \
--mysqld=--innodb_stats_persistent=on \
--mysqld=--loose-max-statement-time=30 \
--threads=33 \
--mysqld=--innodb_page_size=16K \
--mysqld=--innodb-buffer-pool-size=256M \
--duration=400 \
--no_mask \
--workdir=<local settings> \
--vardir=<local settings> \
--mtr-build-thread=<local settings> \
--basedir1=<local settings> \
--script_debug=_nix_ \
--rr=Server \
--rr_options=--chaos
 
 



 Comments   
Comment by Matthias Leich [ 2020-10-15 ]

rr:/home/mleich/RQG/storage/1602758093
000444.log -- RQG log
000444.tgz - remainings of the test including rr trace
 
cd /home/mleich/RQG/storage/1602758093/tmp/dev/shm/vardir/1602758093/36/1/rr
_RR_TRACE_DIR="." rr replay --mark-stdio
 
Error pattern for RQG
[ 'TBR-702' , 'in btr_get_search_table.{1,300}btr0sea.ic.{1,300}in btr_search_update_hash_on_insert.{1,300}btr0sea.cc.{1,300}in btr_cur_optimistic_insert.{1,1000}btr0cur.cc' ],

Comment by Marko Mäkelä [ 2020-10-19 ]

Thank you! Sorry, it turns out that this was caused by my compiler warning fix to btr_search_update_hash_node_on_insert(). Luckily, that change is not present in any release.

Generated at Thu Feb 08 09:26:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.