[MDEV-25721] AddressSanitizer: heap-use-after-free in dict_mem_table_free(dict_table_t*) Created: 2021-05-18  Updated: 2021-05-24  Resolved: 2021-05-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.0, 10.1, 10.6.0, 10.2, 10.3, 10.4
Fix Version/s: 10.6.2, 10.2.39, 10.3.30, 10.4.20, 10.5.11

Type: Bug Priority: Major
Reporter: Matthias Leich Assignee: Thirunarayanan Balathandayuthapani
Resolution: Fixed Votes: 0
Labels: rr-profile-analyzed

Issue Links:
Blocks

 Description   

Scenario:
1. Start the server and generate some initial data
2. Two sessions run their usual DDL/DML battle
3. SIGKILL server during 2. is ongoing
4. Restart attempt with success
5. OPTIMIZE TABLE `test`.`t1` harvests
[rr 2766371 21143]2021-05-18 12:27:58 0 [Note] /data/Server_bin/bb-10.6-monty-innodb_asan/bin/mysqld: ready for connections.
Version: '10.6.1-MariaDB-debug-log'  socket: '/dev/shm/vardir/1621339124/40/1/mysql.sock'  port: 25380  Source distribution
[rr 2766371 22337]2021-05-18 12:27:59 5 [ERROR] InnoDB: Operating system error number 17 in a file operation.
[rr 2766371 22341]2021-05-18 12:27:59 5 [ERROR] InnoDB: Error number 17 means 'File exists'
[rr 2766371 22345]2021-05-18 12:27:59 5 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
[rr 2766371 22349]2021-05-18 12:27:59 5 [Note] InnoDB: The file './test/FTS_0000000000000214_BEING_DELETED.ibd' already exists though the corresponding table did not exist in the InnoDB data dictionary. You can resolve the problem by removing the file.
[rr 2766371 22353]2021-05-18 12:27:59 5 [ERROR] InnoDB: Cannot create file './test/FTS_0000000000000214_BEING_DELETED.ibd'
[rr 2766371 22357]=================================================================
[rr 2766371 22364][rr 2766371 22368]==2766371==ERROR: AddressSanitizer: heap-use-after-free on address 0x618000066b88 at pc 0x55615aad9f0f bp 0x7f71c29c7a10 sp 0x7f71c29c7a00
[rr 2766371 22371][rr 2766371 22451]READ of size 8 at 0x618000066b88 thread T12
[rr 2766371 28178]    #0 0x55615aad9f0e in dict_mem_table_free(dict_table_t*) /data/Server/bb-10.6-monty-innodb/storage/innobase/dict/dict0mem.cc:215
[rr 2766371 28192]    #1 0x55615ab95502 in fts_create_one_common_table /data/Server/bb-10.6-monty-innodb/storage/innobase/fts/fts0fts.cc:1803
[rr 2766371 28196]    #2 0x55615ab95c52 in fts_create_common_tables(trx_t*, dict_table_t*, bool) /data/Server/bb-10.6-monty-innodb/storage/innobase/fts/fts0fts.cc:1868
[rr 2766371 28222]    #3 0x55615a4fa90b in prepare_inplace_alter_table_dict /data/Server/bb-10.6-monty-innodb/storage/innobase/handler/handler0alter.cc:7085
[rr 2766371 28224]    #4 0x55615a504e99 in ha_innobase::prepare_inplace_alter_table(TABLE*, Alter_inplace_info*) /data/Server/bb-10.6-monty-innodb/storage/innobase/handler/handler0alter.cc:8261
[rr 2766371 28244]    #5 0x556159b61ead in handler::ha_prepare_inplace_alter_table(TABLE*, Alter_inplace_info*) /data/Server/bb-10.6-monty-innodb/sql/handler.cc:4898
[rr 2766371 28266]    #6 0x556159616d8f in mysql_inplace_alter_table /data/Server/bb-10.6-monty-innodb/sql/sql_table.cc:7309
[rr 2766371 28276]    #7 0x55615962abfe in mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool) /data/Server/bb-10.6-monty-innodb/sql/sql_table.cc:10180
[rr 2766371 28278]    #8 0x556159633001 in mysql_recreate_table(THD*, TABLE_LIST*, bool) /data/Server/bb-10.6-monty-innodb/sql/sql_table.cc:11239
[rr 2766371 28290]    #9 0x5561597cb26f in admin_recreate_table /data/Server/bb-10.6-monty-innodb/sql/sql_admin.cc:66
[rr 2766371 28294]    #10 0x5561597d2893 in mysql_admin_table /data/Server/bb-10.6-monty-innodb/sql/sql_admin.cc:1127
[rr 2766371 28296]    #11 0x5561597d5109 in Sql_cmd_optimize_table::execute(THD*) /data/Server/bb-10.6-monty-innodb/sql/sql_admin.cc:1496
[rr 2766371 28316]    #12 0x5561593bffaf in mysql_execute_command(THD*) /data/Server/bb-10.6-monty-innodb/sql/sql_parse.cc:5987
[rr 2766371 28320]    #13 0x5561593cc550 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /data/Server/bb-10.6-monty-innodb/sql/sql_parse.cc:8019
[rr 2766371 28322]    #14 0x5561593a475b in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /data/Server/bb-10.6-monty-innodb/sql/sql_parse.cc:1897
[rr 2766371 28324]    #15 0x5561593a1b30 in do_command(THD*, bool) /data/Server/bb-10.6-monty-innodb/sql/sql_parse.cc:1406
[rr 2766371 28334]    #16 0x55615979c673 in do_handle_one_connection(CONNECT*, bool) /data/Server/bb-10.6-monty-innodb/sql/sql_connect.cc:1410
[rr 2766371 28338]    #17 0x55615979bfd5 in handle_one_connection /data/Server/bb-10.6-monty-innodb/sql/sql_connect.cc:1312
[rr 2766371 28346]    #18 0x7f71f1d21608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477
[rr 2766371 28350]    #19 0x7f71f18f7292 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x122292)
 
GIT_SHOW: HEAD, origin/bb-10.6-monty-innodb 47df1765bf413f3a92fa6a83df7b887ace8af9fe 2021-04-28T18:22:38+03:00
 
RQG
===
git clone https://github.com/mleich1/rqg --branch experimental RQG
 
perl rqg.pl \
--grammar=conf/runtime/alter_online.yy \
--gendata=conf/runtime/alter_online.zz \
--reporters=CrashRecovery1 \
--mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \
--mysqld=--loose-idle_write_transaction_timeout=0 \
--mysqld=--loose-idle_transaction_timeout=0 \
--mysqld=--loose-idle_readonly_transaction_timeout=0 \
--mysqld=--connect_timeout=60 \
--mysqld=--interactive_timeout=28800 \
--mysqld=--slave_net_timeout=60 \
--mysqld=--net_read_timeout=30 \
--mysqld=--net_write_timeout=60 \
--mysqld=--loose-table_lock_wait_timeout=50 \
--mysqld=--wait_timeout=28800 \
--mysqld=--lock-wait-timeout=86400 \
--mysqld=--innodb-lock-wait-timeout=50 \
--no-mask \
--queries=10000000 \
--seed=random \
--reporters=Backtrace \
--reporters=ErrorLog \
--reporters=Deadlock1 \
--validators=None \
--mysqld=--log_output=none \
--mysqld=--log-bin \
--mysqld=--log_bin_trust_function_creators=1 \
--mysqld=--loose-debug_assert_on_not_freed_memory=0 \
--engine=InnoDB \
--restart_timeout=240 \
--mysqld=--plugin-load-add=file_key_management.so \
--mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
--duration=100 \
--mysqld=--loose-innodb_fatal_semaphore_wait_threshold=600 \
--mysqld=--innodb_stats_persistent=off \
--mysqld=--innodb_adaptive_hash_index=off \
--mysqld=--loose-innodb_evict_tables_on_commit_debug=off \
--mysqld=--loose-max-statement-time=30 \
--threads=2 \
--mysqld=--innodb-use-native-aio=0 \
--rr=Extended \
--rr_options=--wait \
--mysqld=--innodb_page_size=4K \
--mysqld=--innodb-buffer-pool-size=256M \
--no_mask \
--workdir=<local settings> \
--vardir=<local settings> \
--mtr-build-thread=<local settings> \
--basedir1=<local settings> \
--script_debug=_nix_
 
Search pattern for RQG
[ 'TBR-916-CR-MDEV-25721', 'INFO: Reporter \'CrashRecovery1\': Attempting database recovery using the server.+ERROR: OPTIMIZE TABLE .{1,500} harvested 2013: Lost connection to MySQL server during query.+\[Note\] InnoDB: Starting crash recovery from checkpoint LSN.+\[Note\] .{1,200}mysqld: ready for connections.+SUMMARY: AddressSanitizer: heap-use-after-free.{1,200}dict0mem.cc:.{1,20} in dict_mem_table_free\(dict_table_t\*\).+RESULT: The RQG run ended with status STATUS_RECOVERY_FAILURE' ],
 
There are similar bug reports in JIRA but here the problems occurs after some crash recovery.
And in the current case  sequences/versioned tables/generated columns/BACKUP LOCKS/FOREIGN KEYs/Partitioning are not involved.



 Comments   
Comment by Matthias Leich [ 2021-05-18 ]

pluto:/data/Results/1621339124/TBR-916-CR-MDEV-25721/dev/shm/vardir/1621339124/40/1/rr
_RR_TRACE_DIR="." rr replay --mark-stdio mysqld-0 # Fate of the server till SIGKILL
_RR_TRACE_DIR="." rr replay --mark-stdio mysqld-1 # Fate of the server beginning with restart invoking crash recovery

Comment by Marko Mäkelä [ 2021-05-19 ]

The root cause of the failure was fixed in MDEV-18518 or the second part of MDEV-25506, which caused an orphan file to exist in the data directory after the server was killed. But the double-free in the error handling seems to affect all versions. If row_create_table_for_mysql() fails, it will already have invoked dict_mem_table_free().

Comment by Matthias Leich [ 2021-05-21 ]

bb-10.2-thiru commit 14e5d04e187fbc96806357f8d82dd09e8cf74041
behaved well in RQG testing. No new bad effects.

Generated at Thu Feb 08 09:39:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.