Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27476

heap-use-after-free in buf_pool_t::is_block_field()

Details

    Description

      mleich found a race condition between buffer pool resizing and an update of the adaptive hash index, while testing MDEV-14425.

      ==3752116==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d00000d300 at pc 0x5588ff9718a7 bp 0x5038216570b0 sp 0x5038216570a0
      		 
      READ of size 8 at 0x61d00000d300 thread T26
      		 
          #0 0x5588ff9718a6 in buf_pool_t::is_block_field(void const*) const /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1308
      		 
          #1 0x5588ff97190a in buf_pool_t::is_uncompressed(buf_block_t const*) const /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1449
      		 
          #2 0x5588ffcd648f in btr_search_update_block_hash_info /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/btr/btr0sea.cc:415

      The function is being called as part of a debug assertion there, so this particular failure only affects debug builds. The culprit (found by setting a hardware watchpoint on the AddressSanitizer reported poison byte and executing reverse-continue in rr) is buffer pool resizing:

      Thread 50 (Thread 3752116.3794577):
      		 
      #0  __memset_avx2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:200
      		 
      #1  0x00007ff70bf479d9 in ?? () from /lib/x86_64-linux-gnu/libasan.so.5
      		 
      #2  0x00007ff70c028799 in free () from /lib/x86_64-linux-gnu/libasan.so.5
      		 
      #3  0x00005588ffd17b0b in buf_pool_t::resize (this=0x558901d978c0 <buf_pool>) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/buf/buf0buf.cc:1894
      		 
      #4  0x00005588ffcf9ab8 in buf_resize_callback () at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/buf/buf0buf.cc:1989

      During the time of this, the thread that is going to fail is already executing that function:

      Thread 3 (Thread 3752116.3757608):
      		 
      #0  0x00005588ff971864 in buf_pool_t::is_block_field (this=0x558901d978c0 <buf_pool>, ptr=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1307
      		 
      #1  0x00005588ff97190b in buf_pool_t::is_uncompressed (this=0x558901d978c0 <buf_pool>, block=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/include/buf0buf.h:1449
      		 
      #2  0x00005588ffcd6490 in btr_search_update_block_hash_info (info=0x61a000553a68, block=0x304715439930) at /data/Server/preview-10.8-MDEV-14425-innodbF/storage/innobase/btr/btr0sea.cc:415

      Most invocations of buf_pool_t::is_uncompressed() are in debug assertions, and this faulty debug assertion was introduced in 10.6.

      MDEV-27058 in 10.6 would allow the buf_pool.is_uncompressed(block) assertions to be rewritten simply as block->page.frame. That would be a null pointer for blocks of ROW_FORMAT=COMPRESSED tables whose uncompressed page frame has been discarded.

      The calls to buf_pool.is_uncompressed() outside assertions seem to be protected correctly by buf_pool.page_hash latch. Those are in Block_hint::buffer_fix_block_if_still_valid() and buf_page_get_low().

      The failing debug assertion was added in MDEV-27058.

      A similar assertion had been added to mtr_t::modify() already in MDEV-22110 (MariaDB Server 10.5.5). That one is best removed.

      This bug only affects debug builds during buffer pool resizing, possibly only when running with ASAN.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22110 InnoDB unnecessarily writes unmodified pages

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27058 Buffer page descriptors are too large

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23158 False negatives from buf_pool.is_uncompressed() cause unnecessary lookups

          • Major - Major loss of function.
          • Closed

          Activity

            mleich Matthias Leich added a comment - 

            RQG
            		 
            -------
            		 
            # git clone https://github.com/mleich1/rqg --branch experimental RQG
            		 
            #
            		 
            # GIT_SHOW: HEAD -> experimental, origin/experimental f1cb1e206678662cb17c7e1d948fd5e0a9fd50b7 2021-12-21T17:57:05+01:00
            		 
            # rqg.pl  : Version 4.0.4 (2021-12)
            		 
            #
            		 
            # $RQG_HOME/rqg.pl \
            		 
            # --grammar=conf/mariadb/table_stress_innodb_nocopy1.yy \
            		 
            # --gendata=conf/mariadb/table_stress.zz \
            		 
            # --gendata_sql=conf/mariadb/table_stress.sql \
            		 
            # --reporters=Mariabackup_linux \
            		 
            # --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \
            		 
            # --mysqld=--loose-idle_write_transaction_timeout=0 \
            		 
            # --mysqld=--loose-idle_transaction_timeout=0 \
            		 
            # --mysqld=--loose-idle_readonly_transaction_timeout=0 \
            		 
            # --mysqld=--connect_timeout=60 \
            		 
            # --mysqld=--interactive_timeout=28800 \
            		 
            # --mysqld=--slave_net_timeout=60 \
            		 
            # --mysqld=--net_read_timeout=30 \
            		 
            # --mysqld=--net_write_timeout=60 \
            		 
            # --mysqld=--loose-table_lock_wait_timeout=50 \
            		 
            # --mysqld=--wait_timeout=28800 \
            		 
            # --mysqld=--lock-wait-timeout=86400 \
            		 
            # --mysqld=--innodb-lock-wait-timeout=50 \
            		 
            # --no-mask \
            		 
            # --queries=10000000 \
            		 
            # --seed=random \
            		 
            # --reporters=Backtrace \
            		 
            # --reporters=ErrorLog \
            		 
            # --reporters=Deadlock1 \
            		 
            # --validators=None \
            		 
            # --mysqld=--log_output=none \
            		 
            # --mysqld=--log_bin_trust_function_creators=1 \
            		 
            # --mysqld=--loose-debug_assert_on_not_freed_memory=0 \
            		 
            # --engine=InnoDB \
            		 
            # --restart_timeout=240 \
            		 
            # --mysqld=--plugin-load-add=file_key_management.so \
            		 
            # --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
            		 
            # --duration=300 \
            		 
            # --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \
            		 
            # --mysqld=--loose-innodb_read_only_compressed=OFF \
            		 
            # --mysqld=--loose-innodb-sync-debug \
            		 
            # --mysqld=--innodb_stats_persistent=off \
            		 
            # --mysqld=--innodb_adaptive_hash_index=on \
            		 
            # --mysqld=--log-bin \
            		 
            # --mysqld=--sync-binlog=1 \
            		 
            # --mysqld=--loose-innodb_evict_tables_on_commit_debug=off \
            		 
            # --mysqld=--loose-max-statement-time=30 \
            		 
            # --threads=33 \
            		 
            # --mysqld=--innodb-use-native-aio=0 \
            		 
            # --mysqld=--loose-gdb \
            		 
            # --mysqld=--loose-debug-gdb \
            		 
            # --rr=Extended \
            		 
            # --rr_options=--chaos --wait \
            		 
            # --mysqld=--innodb_rollback_on_timeout=OFF \
            		 
            # --vardir_type=fast \
            		 
            # --mysqld=--innodb_page_size=32K \
            		 
            # --mysqld=--innodb-buffer-pool-size=256M \
            		 
            # --no_mask \
            		 
            # <local settings>

            mleich Matthias Leich added a comment - RQG ------- # git clone https://github.com/mleich1/rqg --branch experimental RQG # # GIT_SHOW: HEAD -> experimental, origin/experimental f1cb1e206678662cb17c7e1d948fd5e0a9fd50b7 2021-12-21T17:57:05+01:00 # rqg.pl : Version 4.0.4 (2021-12) # # $RQG_HOME/rqg.pl \ # --grammar=conf/mariadb/table_stress_innodb_nocopy1.yy \ # --gendata=conf/mariadb/table_stress.zz \ # --gendata_sql=conf/mariadb/table_stress.sql \ # --reporters=Mariabackup_linux \ # --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \ # --mysqld=--loose-idle_write_transaction_timeout=0 \ # --mysqld=--loose-idle_transaction_timeout=0 \ # --mysqld=--loose-idle_readonly_transaction_timeout=0 \ # --mysqld=--connect_timeout=60 \ # --mysqld=--interactive_timeout=28800 \ # --mysqld=--slave_net_timeout=60 \ # --mysqld=--net_read_timeout=30 \ # --mysqld=--net_write_timeout=60 \ # --mysqld=--loose-table_lock_wait_timeout=50 \ # --mysqld=--wait_timeout=28800 \ # --mysqld=--lock-wait-timeout=86400 \ # --mysqld=--innodb-lock-wait-timeout=50 \ # --no-mask \ # --queries=10000000 \ # --seed=random \ # --reporters=Backtrace \ # --reporters=ErrorLog \ # --reporters=Deadlock1 \ # --validators=None \ # --mysqld=--log_output=none \ # --mysqld=--log_bin_trust_function_creators=1 \ # --mysqld=--loose-debug_assert_on_not_freed_memory=0 \ # --engine=InnoDB \ # --restart_timeout=240 \ # --mysqld=--plugin-load-add=file_key_management.so \ # --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \ # --duration=300 \ # --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \ # --mysqld=--loose-innodb_read_only_compressed=OFF \ # --mysqld=--loose-innodb-sync-debug \ # --mysqld=--innodb_stats_persistent=off \ # --mysqld=--innodb_adaptive_hash_index=on \ # --mysqld=--log-bin \ # --mysqld=--sync-binlog=1 \ # --mysqld=--loose-innodb_evict_tables_on_commit_debug=off \ # --mysqld=--loose-max-statement-time=30 \ # --threads=33 \ # --mysqld=--innodb-use-native-aio=0 \ # --mysqld=--loose-gdb \ # --mysqld=--loose-debug-gdb \ # --rr=Extended \ # --rr_options=--chaos --wait \ # --mysqld=--innodb_rollback_on_timeout=OFF \ # --vardir_type=fast \ # --mysqld=--innodb_page_size=32K \ # --mysqld=--innodb-buffer-pool-size=256M \ # --no_mask \ # <local settings>

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.