[MDEV-27476] heap-use-after-free in buf_pool_t::is_block_field() Created: 2022-01-12  Updated: 2022-02-21  Resolved: 2022-01-12

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.5, 10.6, 10.7, 10.8
Fix Version/s: 10.5.14, 10.6.6, 10.7.2, 10.8.1

Type: Bug Priority: Major
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: ASAN, debug, rr-profile-analyzed

Issue Links:
Problem/Incident
is caused by MDEV-22110 InnoDB unnecessarily writes unmodifie... Closed
is caused by MDEV-27058 Buffer page descriptors are too large Closed
Relates
relates to MDEV-23158 False negatives from buf_pool.is_unco... Closed

 Description   

mleich found a race condition between buffer pool resizing and an update of the adaptive hash index, while testing MDEV-14425.

==3752116==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d00000d300 at pc 0x5588ff9718a7 bp 0x5038216570b0 sp 0x5038216570a0
 
READ of size 8 at 0x61d00000d300 thread T26
 
    #0 0x5588ff9718a6 in buf_pool_t::is_block_field(void const*) const /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1308
 
    #1 0x5588ff97190a in buf_pool_t::is_uncompressed(buf_block_t const*) const /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1449
 
    #2 0x5588ffcd648f in btr_search_update_block_hash_info /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/btr/btr0sea.cc:415

The function is being called as part of a debug assertion there, so this particular failure only affects debug builds. The culprit (found by setting a hardware watchpoint on the AddressSanitizer reported poison byte and executing reverse-continue in rr) is buffer pool resizing:

Thread 50 (Thread 3752116.3794577):
 
#0  __memset_avx2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200
 
#1  0x00007ff70bf479d9 in ?? () from /lib/x86_64-linux-gnu/libasan.so.5
 
#2  0x00007ff70c028799 in free () from /lib/x86_64-linux-gnu/libasan.so.5
 
#3  0x00005588ffd17b0b in buf_pool_t::resize (this=0x558901d978c0 <buf_pool>) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/buf/buf0buf.cc:1894
 
#4  0x00005588ffcf9ab8 in buf_resize_callback () at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/buf/buf0buf.cc:1989

During the time of this, the thread that is going to fail is already executing that function:

Thread 3 (Thread 3752116.3757608):
 
#0  0x00005588ff971864 in buf_pool_t::is_block_field (this=0x558901d978c0 <buf_pool>, ptr=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1307
 
#1  0x00005588ff97190b in buf_pool_t::is_uncompressed (this=0x558901d978c0 <buf_pool>, block=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1449
 
#2  0x00005588ffcd6490 in btr_search_update_block_hash_info (info=0x61a000553a68, block=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/btr/btr0sea.cc:415

Most invocations of buf_pool_t::is_uncompressed() are in debug assertions, and this faulty debug assertion was introduced in 10.6.

MDEV-27058 in 10.6 would allow the buf_pool.is_uncompressed(block) assertions to be rewritten simply as block->page.frame. That would be a null pointer for blocks of ROW_FORMAT=COMPRESSED tables whose uncompressed page frame has been discarded.

The calls to buf_pool.is_uncompressed() outside assertions seem to be protected correctly by buf_pool.page_hash latch. Those are in Block_hint::buffer_fix_block_if_still_valid() and buf_page_get_low().

The failing debug assertion was added in MDEV-27058.

A similar assertion had been added to mtr_t::modify() already in MDEV-22110 (MariaDB Server 10.5.5). That one is best removed.

This bug only affects debug builds during buffer pool resizing, possibly only when running with ASAN.

 Comments   
Comment by Matthias Leich [ 2022-01-12 ] 

RQG
 
-------
 
# git clone https://github.com/mleich1/rqg --branch experimental RQG
 
#
 
# GIT_SHOW: HEAD -> experimental, origin/experimental f1cb1e206678662cb17c7e1d948fd5e0a9fd50b7 2021-12-21T17:57:05+01:00
 
# rqg.pl  : Version 4.0.4 (2021-12)
 
#
 
# $RQG_HOME/rqg.pl \
 
# --grammar=conf/mariadb/table_stress_innodb_nocopy1.yy \
 
# --gendata=conf/mariadb/table_stress.zz \
 
# --gendata_sql=conf/mariadb/table_stress.sql \
 
# --reporters=Mariabackup_linux \
 
# --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \
 
# --mysqld=--loose-idle_write_transaction_timeout=0 \
 
# --mysqld=--loose-idle_transaction_timeout=0 \
 
# --mysqld=--loose-idle_readonly_transaction_timeout=0 \
 
# --mysqld=--connect_timeout=60 \
 
# --mysqld=--interactive_timeout=28800 \
 
# --mysqld=--slave_net_timeout=60 \
 
# --mysqld=--net_read_timeout=30 \
 
# --mysqld=--net_write_timeout=60 \
 
# --mysqld=--loose-table_lock_wait_timeout=50 \
 
# --mysqld=--wait_timeout=28800 \
 
# --mysqld=--lock-wait-timeout=86400 \
 
# --mysqld=--innodb-lock-wait-timeout=50 \
 
# --no-mask \
 
# --queries=10000000 \
 
# --seed=random \
 
# --reporters=Backtrace \
 
# --reporters=ErrorLog \
 
# --reporters=Deadlock1 \
 
# --validators=None \
 
# --mysqld=--log_output=none \
 
# --mysqld=--log_bin_trust_function_creators=1 \
 
# --mysqld=--loose-debug_assert_on_not_freed_memory=0 \
 
# --engine=InnoDB \
 
# --restart_timeout=240 \
 
# --mysqld=--plugin-load-add=file_key_management.so \
 
# --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
 
# --duration=300 \
 
# --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \
 
# --mysqld=--loose-innodb_read_only_compressed=OFF \
 
# --mysqld=--loose-innodb-sync-debug \
 
# --mysqld=--innodb_stats_persistent=off \
 
# --mysqld=--innodb_adaptive_hash_index=on \
 
# --mysqld=--log-bin \
 
# --mysqld=--sync-binlog=1 \
 
# --mysqld=--loose-innodb_evict_tables_on_commit_debug=off \
 
# --mysqld=--loose-max-statement-time=30 \
 
# --threads=33 \
 
# --mysqld=--innodb-use-native-aio=0 \
 
# --mysqld=--loose-gdb \
 
# --mysqld=--loose-debug-gdb \
 
# --rr=Extended \
 
# --rr_options=--chaos --wait \
 
# --mysqld=--innodb_rollback_on_timeout=OFF \
 
# --vardir_type=fast \
 
# --mysqld=--innodb_page_size=32K \
 
# --mysqld=--innodb-buffer-pool-size=256M \
 
# --no_mask \
 
# <local settings>

Generated at Thu Feb 08 09:53:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.