Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.5.5, 10.6, 10.7(EOL), 10.8(EOL)
Description
mleich found a race condition between buffer pool resizing and an update of the adaptive hash index, while testing MDEV-14425.
==3752116==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d00000d300 at pc 0x5588ff9718a7 bp 0x5038216570b0 sp 0x5038216570a0
|
READ of size 8 at 0x61d00000d300 thread T26
|
#0 0x5588ff9718a6 in buf_pool_t::is_block_field(void const*) const /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1308
|
#1 0x5588ff97190a in buf_pool_t::is_uncompressed(buf_block_t const*) const /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1449
|
#2 0x5588ffcd648f in btr_search_update_block_hash_info /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/btr/btr0sea.cc:415
|
The function is being called as part of a debug assertion there, so this particular failure only affects debug builds. The culprit (found by setting a hardware watchpoint on the AddressSanitizer reported poison byte and executing reverse-continue in rr) is buffer pool resizing:
Thread 50 (Thread 3752116.3794577):
|
#0 __memset_avx2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200
|
#1 0x00007ff70bf479d9 in ?? () from /lib/x86_64-linux-gnu/libasan.so.5
|
#2 0x00007ff70c028799 in free () from /lib/x86_64-linux-gnu/libasan.so.5
|
#3 0x00005588ffd17b0b in buf_pool_t::resize (this=0x558901d978c0 <buf_pool>) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/buf/buf0buf.cc:1894
|
#4 0x00005588ffcf9ab8 in buf_resize_callback () at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/buf/buf0buf.cc:1989
|
During the time of this, the thread that is going to fail is already executing that function:
Thread 3 (Thread 3752116.3757608):
|
#0 0x00005588ff971864 in buf_pool_t::is_block_field (this=0x558901d978c0 <buf_pool>, ptr=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1307
|
#1 0x00005588ff97190b in buf_pool_t::is_uncompressed (this=0x558901d978c0 <buf_pool>, block=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1449
|
#2 0x00005588ffcd6490 in btr_search_update_block_hash_info (info=0x61a000553a68, block=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/btr/btr0sea.cc:415
|
Most invocations of buf_pool_t::is_uncompressed() are in debug assertions, and this faulty debug assertion was introduced in 10.6.
MDEV-27058 in 10.6 would allow the buf_pool.is_uncompressed(block) assertions to be rewritten simply as block->page.frame. That would be a null pointer for blocks of ROW_FORMAT=COMPRESSED tables whose uncompressed page frame has been discarded.
The calls to buf_pool.is_uncompressed() outside assertions seem to be protected correctly by buf_pool.page_hash latch. Those are in Block_hint::buffer_fix_block_if_still_valid() and buf_page_get_low().
The failing debug assertion was added in MDEV-27058.
A similar assertion had been added to mtr_t::modify() already in MDEV-22110 (MariaDB Server 10.5.5). That one is best removed.
This bug only affects debug builds during buffer pool resizing, possibly only when running with ASAN.
Attachments
Issue Links
- is caused by
-
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
RQG-------# git clone https://github.com/mleich1/rqg --branch experimental RQG## GIT_SHOW: HEAD -> experimental, origin/experimental f1cb1e206678662cb17c7e1d948fd5e0a9fd50b7 2021-12-21T17:57:05+01:00# rqg.pl : Version 4.0.4 (2021-12)## $RQG_HOME/rqg.pl \# --grammar=conf/mariadb/table_stress_innodb_nocopy1.yy \# --gendata=conf/mariadb/table_stress.zz \# --gendata_sql=conf/mariadb/table_stress.sql \# --reporters=Mariabackup_linux \# --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \# --mysqld=--loose-idle_write_transaction_timeout=0 \# --mysqld=--loose-idle_transaction_timeout=0 \# --mysqld=--loose-idle_readonly_transaction_timeout=0 \# --mysqld=--connect_timeout=60 \# --mysqld=--interactive_timeout=28800 \# --mysqld=--slave_net_timeout=60 \# --mysqld=--net_read_timeout=30 \# --mysqld=--net_write_timeout=60 \# --mysqld=--loose-table_lock_wait_timeout=50 \# --mysqld=--wait_timeout=28800 \# --mysqld=--lock-wait-timeout=86400 \# --mysqld=--innodb-lock-wait-timeout=50 \# --no-mask \# --queries=10000000 \# --seed=random \# --reporters=Backtrace \# --reporters=ErrorLog \# --reporters=Deadlock1 \# --validators=None \# --mysqld=--log_output=none \# --mysqld=--log_bin_trust_function_creators=1 \# --mysqld=--loose-debug_assert_on_not_freed_memory=0 \# --engine=InnoDB \# --restart_timeout=240 \# --mysqld=--plugin-load-add=file_key_management.so \# --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \# --duration=300 \# --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \# --mysqld=--loose-innodb_read_only_compressed=OFF \# --mysqld=--loose-innodb-sync-debug \# --mysqld=--innodb_stats_persistent=off \# --mysqld=--innodb_adaptive_hash_index=on \# --mysqld=--log-bin \# --mysqld=--sync-binlog=1 \# --mysqld=--loose-innodb_evict_tables_on_commit_debug=off \# --mysqld=--loose-max-statement-time=30 \# --threads=33 \# --mysqld=--innodb-use-native-aio=0 \# --mysqld=--loose-gdb \# --mysqld=--loose-debug-gdb \# --rr=Extended \# --rr_options=--chaos --wait \# --mysqld=--innodb_rollback_on_timeout=OFF \# --vardir_type=fast \# --mysqld=--innodb_page_size=32K \# --mysqld=--innodb-buffer-pool-size=256M \# --no_mask \# <local settings>