[MDEV-25257] SEGV in fts_get_next_doc_id upon some INSERT - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.5.10, 10.3.35
Fix Version/s: 10.3.36, 10.4.26, 10.5.17, 10.6.9
Component/s: Full-text Search, Storage Engine - InnoDB, Virtual Columns
Labels:
- affects-tests

Description

Version: '10.3.35-MariaDB-debug-log'  socket: '/data/Server_bin/10.3_asan/mysql-test/var/tmp/mysqld.1.sock'  port: 16000  Source distribution

AddressSanitizer:DEADLYSIGNAL

=================================================================

==786736==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000240 (pc 0x55d96741dd8c bp 0x7fd49bcc01b0 sp 0x7fd49bcc0190 T27)

==786736==The signal is caused by a READ memory access.

==786736==Hint: address points to the zero page.

    #0 0x55d96741dd8b in fts_get_next_doc_id(dict_table_t const*, unsigned long*) /data/Server/10.3/storage/innobase/fts/fts0fts.cc:2557

    #1 0x55d966f9be33 in row_mysql_convert_row_to_innobase /data/Server/10.3/storage/innobase/row/row0mysql.cc:663

    #2 0x55d966fa08e6 in row_insert_for_mysql(unsigned char const*, row_prebuilt_t*, ins_mode_t) /data/Server/10.3/storage/innobase/row/row0mysql.cc:1395

    #3 0x55d966c7d63d in ha_innobase::write_row(unsigned char*) /data/Server/10.3/storage/innobase/handler/ha_innodb.cc:8146

    #4 0x55d966785a24 in handler::ha_write_row(unsigned char*) /data/Server/10.3/sql/handler.cc:6479

    #5 0x55d965ff48cf in write_record(THD*, TABLE*, st_copy_info*) /data/Server/10.3/sql/sql_insert.cc:2050

    #6 0x55d965feda9f in mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) /data/Server/10.3/sql/sql_insert.cc:1072

    #7 0x55d966088a30 in mysql_execute_command(THD*) /data/Server/10.3/sql/sql_parse.cc:4504

    #8 0x55d96609f664 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /data/Server/10.3/sql/sql_parse.cc:7870

    #9 0x55d966078737 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /data/Server/10.3/sql/sql_parse.cc:1852

    #10 0x55d966075a3a in do_command(THD*) /data/Server/10.3/sql/sql_parse.cc:1398

    #11 0x55d9663e4d4e in do_handle_one_connection(CONNECT*) /data/Server/10.3/sql/sql_connect.cc:1403

    #12 0x55d9663e4608 in handle_one_connection /data/Server/10.3/sql/sql_connect.cc:1308

    #13 0x7fd4b2d5d608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477

    #14 0x7fd4b2c84292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292)

AddressSanitizer can not provide additional info.

SUMMARY: AddressSanitizer: SEGV /data/Server/10.3/storage/innobase/fts/fts0fts.cc:2557 in fts_get_next_doc_id(dict_table_t const*, unsigned long*)

Thread T27 created by T0 here:

    #0 0x7fd4b374d805 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x3a805)

    #1 0x55d965e08d96 in create_thread_to_handle_connection(CONNECT*) /data/Server/10.3/sql/mysqld.cc:6666

    #2 0x55d965e0934a in create_new_thread /data/Server/10.3/sql/mysqld.cc:6736

    #3 0x55d965e0a3a1 in handle_connections_sockets() /data/Server/10.3/sql/mysqld.cc:6994

    #4 0x55d965e0857d in mysqld_main(int, char**) /data/Server/10.3/sql/mysqld.cc:6288

    #5 0x55d965df68cc in main /data/Server/10.3/sql/main.cc:25

    #6 0x7fd4b2b890b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)

--source include/have_innodb.inc

SET @save = @@global.innodb_file_per_table;

# innodb_file_per_table = 1 does not show the problem.

SET @@global.innodb_file_per_table = 0;

# Removing col_int_g lets the problem disappear.

CREATE TABLE t1 (

   col_int INTEGER, col_text TEXT, col_int_g INTEGER GENERATED ALWAYS AS (col_int)

) ENGINE = InnoDB ROW_FORMAT = Redundant ;

ALTER TABLE t1 ADD FULLTEXT KEY `ftidx` ( col_text ) ;

ALTER TABLE t1 DROP KEY `ftidx` ;

INSERT INTO t1 (col_int, col_text) VALUES ( 1255, NULL);

DROP TABLE t1;

SET @@global.innodb_file_per_table = @save;

Attachments

Activity

Ascending order - Click to sort in descending order

Matthias Leich added a comment - 2021-03-26 10:46

Search patterns

[ 'TBR-965', '#0  .{1,20} in fts_get_next_doc_id.+#1  .{1,20} in row_mysql_convert_row_to_innobase.+#2 .{1,20}in row_insert_for_mysql.+#3 .{1,20}in ha_innobase::write_row.+#4  .{1,20}in handler::ha_write_row.+#5 .{1,20}in write_record' ],

[ 'TBR-965', '#3  <signal handler called>.+#4  .{1,20} in fts_get_next_doc_id.+#5  .{1,20} in row_mysql_convert_row_to_innobase.+#6 .{1,20}in row_insert_for_mysql.+#7 .{1,20}in ha_innobase::write_row.+#8  .{1,20}in handler::ha_write_row.+#9 .{1,20}in write_record' ],

Matthias Leich added a comment - 2021-03-26 10:46 Search patterns [ 'TBR-965', '#0 .{1,20} in fts_get_next_doc_id.+#1 .{1,20} in row_mysql_convert_row_to_innobase.+#2 .{1,20}in row_insert_for_mysql.+#3 .{1,20}in ha_innobase::write_row.+#4 .{1,20}in handler::ha_write_row.+#5 .{1,20}in write_record' ], [ 'TBR-965', '#3 <signal handler called>.+#4 .{1,20} in fts_get_next_doc_id.+#5 .{1,20} in row_mysql_convert_row_to_innobase.+#6 .{1,20}in row_insert_for_mysql.+#7 .{1,20}in ha_innobase::write_row.+#8 .{1,20}in handler::ha_write_row.+#9 .{1,20}in write_record' ],

Marko Mäkelä added a comment - 2022-05-13 10:09

thiru analyzed that this is related to the following code in dict_sys_tables_rec_read():

	/* For tables created before MySQL 4.1, there may be

	garbage in SYS_TABLES.MIX_LEN where flags2 are found. Such tables

	would always be in ROW_FORMAT=REDUNDANT which do not have the

	high bit set in n_cols, and flags would be zero.

	MySQL 4.1 was the first version to support innodb_file_per_table,

	that is, *space_id != 0. */

	if (not_redundant || *space_id != 0 || *n_cols & DICT_N_COLS_COMPACT) {

Starting with ~~MDEV-12026~~ in 10.4, we should add || fil_system.sys_space.full_crc32() to that condition. We do know that in that format, all unused parts of pages will be zero-initialized.

I would suggest that change in addition to the proposed fix.

Marko Mäkelä added a comment - 2022-05-13 10:09 thiru analyzed that this is related to the following code in dict_sys_tables_rec_read() : /* For tables created before MySQL 4.1, there may be garbage in SYS_TABLES.MIX_LEN where flags2 are found. Such tables would always be in ROW_FORMAT=REDUNDANT which do not have the high bit set in n_cols, and flags would be zero. MySQL 4.1 was the first version to support innodb_file_per_table, that is, *space_id != 0. */ if (not_redundant || *space_id != 0 || *n_cols & DICT_N_COLS_COMPACT) { Starting with MDEV-12026 in 10.4, we should add || fil_system.sys_space.full_crc32() to that condition. We do know that in that format, all unused parts of pages will be zero-initialized. I would suggest that change in addition to the proposed fix.

Marko Mäkelä added a comment - 2022-05-13 10:10

OK to push after the pending releases are out.

Marko Mäkelä added a comment - 2022-05-13 10:10 OK to push after the pending releases are out.

Matthias Leich added a comment - 2022-05-19 10:44

The RQG testing on the development tree

     origin/bb-10.6-MDEV-13542 dca6937a30eeee56f46fc47ac85656243227e4d2 2022-05-16T16:31:15+03:00

showed frequent

- SEGV's with a backtrace like in the description on top

- cases where the server error log contains the message '[ERROR] mysqld got signal 11' but the server process

   did not disappear even after more than 300s waiting --> no core file nor rr trace

I applied Thiru's initial patch for MDEV-25257

    diff --git a/storage/innobase/dict/dict0load.cc b/storage/innobase/dict/dict0load.cc

index 20916c2c96e..2c3d48b9573 100644

--- a/storage/innobase/dict/dict0load.cc

+++ b/storage/innobase/dict/dict0load.cc

@@ -1383,6 +1383,7 @@ dict_load_columns(

                        the flag is set before the table is created. */

                        if (table->fts == NULL) {

                                table->fts = fts_create(table);

+                               table->fts->cache = fts_cache_create(table);

                        ut_a(table->fts->doc_col == ULINT_UNDEFINED);

 to the development tree and both bad effects were no more seen again in testing.

Matthias Leich added a comment - 2022-05-19 10:44 The RQG testing on the development tree origin/bb-10.6-MDEV-13542 dca6937a30eeee56f46fc47ac85656243227e4d2 2022-05-16T16:31:15+03:00 showed frequent - SEGV's with a backtrace like in the description on top - cases where the server error log contains the message '[ERROR] mysqld got signal 11' but the server process did not disappear even after more than 300s waiting --> no core file nor rr trace I applied Thiru's initial patch for MDEV-25257 diff --git a/storage/innobase/dict/dict0load.cc b/storage/innobase/dict/dict0load.cc index 20916c2c96e..2c3d48b9573 100644 --- a/storage/innobase/dict/dict0load.cc +++ b/storage/innobase/dict/dict0load.cc @@ -1383,6 +1383,7 @@ dict_load_columns( the flag is set before the table is created. */ if (table->fts == NULL) { table->fts = fts_create(table); + table->fts->cache = fts_cache_create(table); } ut_a(table->fts->doc_col == ULINT_UNDEFINED); to the development tree and both bad effects were no more seen again in testing.

Marko Mäkelä added a comment - 2022-06-07 13:25

I almost forgot to apply the 10.4 fixup for the case that the data directory was initialized with innodb_checksum_algorithm=full_crc32.

Marko Mäkelä added a comment - 2022-06-07 13:25 I almost forgot to apply the 10.4 fixup for the case that the data directory was initialized with innodb_checksum_algorithm=full_crc32 .

People

Assignee:: Thirunarayanan Balathandayuthapani

Reporter:: Matthias Leich

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2021-03-25 20:31

Updated:: 2022-08-22 09:33

Resolved:: 2022-05-25 10:12

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server