[MDEV-25257] SEGV in fts_get_next_doc_id upon some INSERT Created: 2021-03-25  Updated: 2022-08-22  Resolved: 2022-05-25

Status: Closed
Project: MariaDB Server
Component/s: Full-text Search, Storage Engine - InnoDB, Virtual Columns
Affects Version/s: 10.5.10, 10.3.35
Fix Version/s: 10.3.36, 10.4.26, 10.5.17, 10.6.9

Type: Bug Priority: Major
Reporter: Matthias Leich Assignee: Thirunarayanan Balathandayuthapani
Resolution: Fixed Votes: 0
Labels: affects-tests


 Description    

Version: '10.3.35-MariaDB-debug-log'  socket: '/data/Server_bin/10.3_asan/mysql-test/var/tmp/mysqld.1.sock'  port: 16000  Source distribution
 
AddressSanitizer:DEADLYSIGNAL
 
=================================================================
 
==786736==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000240 (pc 0x55d96741dd8c bp 0x7fd49bcc01b0 sp 0x7fd49bcc0190 T27)
 
==786736==The signal is caused by a READ memory access.
 
==786736==Hint: address points to the zero page.
 
    #0 0x55d96741dd8b in fts_get_next_doc_id(dict_table_t const*, unsigned long*) /data/Server/10.3/storage/innobase/fts/fts0fts.cc:2557
 
    #1 0x55d966f9be33 in row_mysql_convert_row_to_innobase /data/Server/10.3/storage/innobase/row/row0mysql.cc:663
 
    #2 0x55d966fa08e6 in row_insert_for_mysql(unsigned char const*, row_prebuilt_t*, ins_mode_t) /data/Server/10.3/storage/innobase/row/row0mysql.cc:1395
 
    #3 0x55d966c7d63d in ha_innobase::write_row(unsigned char*) /data/Server/10.3/storage/innobase/handler/ha_innodb.cc:8146
 
    #4 0x55d966785a24 in handler::ha_write_row(unsigned char*) /data/Server/10.3/sql/handler.cc:6479
 
    #5 0x55d965ff48cf in write_record(THD*, TABLE*, st_copy_info*) /data/Server/10.3/sql/sql_insert.cc:2050
 
    #6 0x55d965feda9f in mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) /data/Server/10.3/sql/sql_insert.cc:1072
 
    #7 0x55d966088a30 in mysql_execute_command(THD*) /data/Server/10.3/sql/sql_parse.cc:4504
 
    #8 0x55d96609f664 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/Server/10.3/sql/sql_parse.cc:7870
 
    #9 0x55d966078737 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/Server/10.3/sql/sql_parse.cc:1852
 
    #10 0x55d966075a3a in do_command(THD*) /data/Server/10.3/sql/sql_parse.cc:1398
 
    #11 0x55d9663e4d4e in do_handle_one_connection(CONNECT*) /data/Server/10.3/sql/sql_connect.cc:1403
 
    #12 0x55d9663e4608 in handle_one_connection /data/Server/10.3/sql/sql_connect.cc:1308
 
    #13 0x7fd4b2d5d608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477
 
    #14 0x7fd4b2c84292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292)
 
 
 
AddressSanitizer can not provide additional info.
 
SUMMARY: AddressSanitizer: SEGV /data/Server/10.3/storage/innobase/fts/fts0fts.cc:2557 in fts_get_next_doc_id(dict_table_t const*, unsigned long*)
 
Thread T27 created by T0 here:
 
    #0 0x7fd4b374d805 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)
 
    #1 0x55d965e08d96 in create_thread_to_handle_connection(CONNECT*) /data/Server/10.3/sql/mysqld.cc:6666
 
    #2 0x55d965e0934a in create_new_thread /data/Server/10.3/sql/mysqld.cc:6736
 
    #3 0x55d965e0a3a1 in handle_connections_sockets() /data/Server/10.3/sql/mysqld.cc:6994
 
    #4 0x55d965e0857d in mysqld_main(int, char**) /data/Server/10.3/sql/mysqld.cc:6288
 
    #5 0x55d965df68cc in main /data/Server/10.3/sql/main.cc:25
 
    #6 0x7fd4b2b890b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)


--source include/have_innodb.inc
 
 
 
SET @save = @@global.innodb_file_per_table;
 
# innodb_file_per_table = 1 does not show the problem.
 
SET @@global.innodb_file_per_table = 0;
 
 
 
# Removing col_int_g lets the problem disappear.
 
CREATE TABLE t1 (
 
   col_int INTEGER, col_text TEXT, col_int_g INTEGER GENERATED ALWAYS AS (col_int)
 
) ENGINE = InnoDB ROW_FORMAT = Redundant ;
 
ALTER TABLE t1 ADD FULLTEXT KEY `ftidx` ( col_text ) ;
 
ALTER TABLE t1 DROP KEY `ftidx` ;
 
INSERT INTO t1 (col_int, col_text) VALUES ( 1255, NULL);
 
 
 
DROP TABLE t1;
 
SET @@global.innodb_file_per_table = @save;



 Comments   
Comment by Matthias Leich [ 2021-03-26 ] 

Search patterns
 
[ 'TBR-965', '#0  .{1,20} in fts_get_next_doc_id.+#1  .{1,20} in row_mysql_convert_row_to_innobase.+#2 .{1,20}in row_insert_for_mysql.+#3 .{1,20}in ha_innobase::write_row.+#4  .{1,20}in handler::ha_write_row.+#5 .{1,20}in write_record' ],
 
[ 'TBR-965', '#3  <signal handler called>.+#4  .{1,20} in fts_get_next_doc_id.+#5  .{1,20} in row_mysql_convert_row_to_innobase.+#6 .{1,20}in row_insert_for_mysql.+#7 .{1,20}in ha_innobase::write_row.+#8  .{1,20}in handler::ha_write_row.+#9 .{1,20}in write_record' ],

Comment by Marko Mäkelä [ 2022-05-13 ]

thiru analyzed that this is related to the following code in dict_sys_tables_rec_read():

	/* For tables created before MySQL 4.1, there may be
 
	garbage in SYS_TABLES.MIX_LEN where flags2 are found. Such tables
 
	would always be in ROW_FORMAT=REDUNDANT which do not have the
 
	high bit set in n_cols, and flags would be zero.
 
	MySQL 4.1 was the first version to support innodb_file_per_table,
 
	that is, *space_id != 0. */
 
	if (not_redundant || *space_id != 0 || *n_cols & DICT_N_COLS_COMPACT) {

Starting with MDEV-12026 in 10.4, we should add || fil_system.sys_space.full_crc32() to that condition. We do know that in that format, all unused parts of pages will be zero-initialized.

I would suggest that change in addition to the proposed fix.

Comment by Marko Mäkelä [ 2022-05-13 ]

OK to push after the pending releases are out.

Comment by Matthias Leich [ 2022-05-19 ] 

The RQG testing on the development tree
 
     origin/bb-10.6-MDEV-13542 dca6937a30eeee56f46fc47ac85656243227e4d2 2022-05-16T16:31:15+03:00
 
showed frequent
 
- SEGV's with a backtrace like in the description on top
 
- cases where the server error log contains the message '[ERROR] mysqld got signal 11' but the server process
 
   did not disappear even after more than 300s waiting --> no core file nor rr trace
 
 
 
I applied Thiru's initial patch for MDEV-25257
 
    diff --git a/storage/innobase/dict/dict0load.cc b/storage/innobase/dict/dict0load.cc
 
index 20916c2c96e..2c3d48b9573 100644
 
--- a/storage/innobase/dict/dict0load.cc
 
+++ b/storage/innobase/dict/dict0load.cc
 
@@ -1383,6 +1383,7 @@ dict_load_columns(
 
                        the flag is set before the table is created. */
 
                        if (table->fts == NULL) {
 
                                table->fts = fts_create(table);
 
+                               table->fts->cache = fts_cache_create(table);
 
                        }
 
 
                        ut_a(table->fts->doc_col == ULINT_UNDEFINED);
 
 
 
 to the development tree and both bad effects were no more seen again in testing.

Comment by Marko Mäkelä [ 2022-06-07 ]

I almost forgot to apply the 10.4 fixup for the case that the data directory was initialized with innodb_checksum_algorithm=full_crc32.

Generated at Thu Feb 08 09:36:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.