Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35690

SIGSEGV in __sanitizer_cov_trace_pc_guard from sync_array_get_nth_cell upon installing Spider and stopping/restarting the server

Details

    Description

      Starting the server, installing Spider:

      INSTALL SONAME 'ha_spider';
      

      Followed by a regular shutdown (bin/mariadb-admin shutdown), followed by a regular restart leads to the following crash:

      CS 10.5.28 142851f1205d98270b917a98e1bdd483e1b8af0e (Debug, UBASAN, Clang)

      Core was generated by `/test/UBASAN_MD271124-mariadb-10.5.28-linux-x86_64-dbg/bin/mariadbd --no-defaul'.
      Program terminated with signal SIGSEGV, Segmentation fault.
      #0  0x000055a6ce418885 in __sanitizer_cov_trace_pc_guard ()
      [Current thread is 1 (LWP 773329)]
      (gdb) bt
      #0  0x000055a6ce418885 in __sanitizer_cov_trace_pc_guard ()
      #1  0x000055a6d10e4283 in handle_fatal_signal (sig=0)at /test/10.5_dbg_san/sql/signal_handler.cc:137
      #2  <signal handler called>
      #3  0x000055a6ce418885 in __sanitizer_cov_trace_pc_guard ()
      #4  0x000055a6d3e8282b in sync_array_get_nth_cell (arr=0x511000000680, n=9966)at /test/10.5_dbg_san/storage/innobase/sync/sync0arr.cc:271
      #5  0x000055a6d3e97c42 in sync_array_find_thread (arr=0x511000000680, thread=23042266760896)at /test/10.5_dbg_san/storage/innobase/sync/sync0arr.cc:593
      #6  0x000055a6d3e976b4 in sync_array_deadlock_step (arr=0x511000000680, start=0x14f50c001800, thread=23042266760896, pass=0, depth=1)at /test/10.5_dbg_san/storage/innobase/sync/sync0arr.cc:630
      #7  0x000055a6d3e8c250 in sync_array_detect_deadlock (arr=0x511000000680, start=0x14f50c001800, cell=0x14f50c001800, depth=1)at /test/10.5_dbg_san/storage/innobase/sync/sync0arr.cc:840
      #8  0x000055a6d3e88860 in sync_array_wait_event (arr=0x511000000680, cell=@0x14f4f041fa20: 0x14f50c001800)at /test/10.5_dbg_san/storage/innobase/sync/sync0arr.cc:445
      #9  0x000055a6d3ea5b12 in rw_lock_s_lock_spin (lock=0x14f502a0c7e0, pass=0, file_name=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803) at /test/10.5_dbg_san/storage/innobase/sync/sync0rw.cc:371
      #10 0x000055a6d435d9e0 in rw_lock_s_lock_func (lock=0x14f502a0c7e0, pass=0, file_name=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803) at include/sync0rw.inl:294
      #11 0x000055a6d43238c6 in buf_page_mtr_lock (block=0x14f502a0c760, rw_latch=1, mtr=0x14f4f0a2a150, file=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803) at /test/10.5_dbg_san/storage/innobase/buf/buf0buf.cc:2650
      #12 0x000055a6d431bc3d in buf_page_get_low (page_id={m_id = 10}, zip_size=0, rw_latch=1, guess=0x14f502a0c760, mode=10, file=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803, mtr=0x14f4f0a2a150, err=0x14f4f0a2b0c0, allow_ibuf_merge=false)at /test/10.5_dbg_san/storage/innobase/buf/buf0buf.cc:3216
      #13 0x000055a6d4324b53 in buf_page_get_gen (page_id={m_id = 10}, zip_size=0, rw_latch=1, guess=0x14f502a0c760, mode=10, file=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803, mtr=0x14f4f0a2a150, err=0x14f4f0a2b0c0, allow_ibuf_merge=false)at /test/10.5_dbg_san/storage/innobase/buf/buf0buf.cc:3285
      #14 0x000055a6d41a71cb in btr_cur_search_to_nth_level (index=0x517000001d08, level=0, tuple=0x52c0000502e0, mode=PAGE_CUR_GE, latch_mode=1, cursor=0x14f4f0a2a020, file=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803, mtr=0x14f4f0a2a150, autoinc=0)at /test/10.5_dbg_san/storage/innobase/btr/btr0cur.cc:1609
      #15 0x000055a6d427ca58 in btr_pcur_open_low (index=0x517000001d08, level=0, tuple=0x52c0000502e0, mode=PAGE_CUR_GE, latch_mode=1, cursor=0x14f4f0a2a020, file=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803, autoinc=0, mtr=0x14f4f0a2a150) at include/btr0pcur.inl:441
      #16 0x000055a6d427bd9e in btr_pcur_open_on_user_rec_func (index=0x517000001d08, tuple=0x52c0000502e0, mode=PAGE_CUR_GE, latch_mode=1, cursor=0x14f4f0a2a020, file=0x55a6d5cb77af "/test/10.5_dbg_san/storage/innobase/dict/dict0load.cc", line=1803, mtr=0x14f4f0a2a150)at /test/10.5_dbg_san/storage/innobase/btr/btr0pcur.cc:676
      #17 0x000055a6d4550fbf in dict_load_columns (table=0x518000030508, heap=0x52c000050200)at /test/10.5_dbg_san/storage/innobase/dict/dict0load.cc:1802
      #18 0x000055a6d45292f5 in dict_load_table_one (name=@0x14f4f0764ad0: {m_name = 0x513000080260 "mysql/innodb_table_stats", static part_suffix = "#P#"}, ignore_err=DICT_ERR_IGNORE_FK_NOKEY, fk_tables=std::deque with 0 elements)at /test/10.5_dbg_san/storage/innobase/dict/dict0load.cc:2951
      #19 0x000055a6d452698b in dict_load_table (name=0x513000080260 "mysql/innodb_table_stats", ignore_err=DICT_ERR_IGNORE_FK_NOKEY)at /test/10.5_dbg_san/storage/innobase/dict/dict0load.cc:2758
      #20 0x000055a6d452e67f in dict_load_table_on_id (table_id=16, ignore_err=DICT_ERR_IGNORE_FK_NOKEY)at /test/10.5_dbg_san/storage/innobase/dict/dict0load.cc:3196
      #21 0x000055a6d448ee2a in dict_table_open_on_id_low (table_id=16, ignore_err=DICT_ERR_IGNORE_FK_NOKEY, cached_only=false)at /test/10.5_dbg_san/storage/innobase/dict/dict0dict.cc:202
      #22 0x000055a6d449181a in dict_table_open_on_id<true> (table_id=16, dict_locked=false, table_op=DICT_TABLE_OP_NORMAL, thd=0x52b00001c218, mdl=0x51e000001b58)at /test/10.5_dbg_san/storage/innobase/dict/dict0dict.cc:930
      #23 0x000055a6d3c8d0eb in row_purge_parse_undo_rec (node=0x51e0000019b8, undo_rec=0x521000040980 "", thr=0x51e000001908, updated_extern=0x14f4f041d3a0)at /test/10.5_dbg_san/storage/innobase/row/row0purge.cc:1080
      #24 0x000055a6d3c88424 in row_purge (node=0x51e0000019b8, undo_rec=0x521000040980 "", thr=0x51e000001908)at /test/10.5_dbg_san/storage/innobase/row/row0purge.cc:1250
      #25 0x000055a6d3c87844 in row_purge_step (thr=0x51e000001908)at /test/10.5_dbg_san/storage/innobase/row/row0purge.cc:1302
      #26 0x000055a6d39338bd in que_thr_step (thr=0x51e000001908)at /test/10.5_dbg_san/storage/innobase/que/que0que.cc:865
      #27 0x000055a6d392d9ed in que_run_threads_low (thr=0x51e000001908)at /test/10.5_dbg_san/storage/innobase/que/que0que.cc:927
      #28 0x000055a6d392c903 in que_run_threads (thr=0x51e000001908)at /test/10.5_dbg_san/storage/innobase/que/que0que.cc:967
      #29 0x000055a6d3e49914 in srv_task_execute ()at /test/10.5_dbg_san/storage/innobase/srv/srv0srv.cc:1746
      #30 0x000055a6d3e3b13a in purge_worker_callback ()at /test/10.5_dbg_san/storage/innobase/srv/srv0srv.cc:1928
      #31 0x000055a6d4a64ea3 in tpool::task_group::execute (this=0x55a6dade4a40 <purge_task_group>, t=0x55a6dade4b00 <purge_worker_task>)at /test/10.5_dbg_san/tpool/task_group.cc:55
      #32 0x000055a6d4a667f5 in tpool::task::execute (this=0x55a6dade4b00 <purge_worker_task>)at /test/10.5_dbg_san/tpool/task.cc:47
      #33 0x000055a6d4a3beab in tpool::thread_pool_generic::worker_main (this=0x518000000880, thread_var=0x532000018a80)at /test/10.5_dbg_san/tpool/tpool_generic.cc:581
      #34 0x000055a6d4a58347 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*>(__f=@0x504000006068: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x55a6d4a3b7e0 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x504000006060: 0x518000000880, __args=@0x504000006058: 0x532000018a80)at /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:74
      #35 0x000055a6d4a57dc6 in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x504000006068: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x55a6d4a3b7e0 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __args=@0x504000006058: 0x532000018a80, __args=@0x504000006058: 0x532000018a80)at /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96
      #36 0x000055a6d4a57c95 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x504000006058)at /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:292
      #37 0x000055a6d4a57a60 in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x504000006058)at /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:299
      #38 0x000055a6d4a56f88 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x504000006050)at /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_thread.h:244
      #39 0x000014f518eeabb4 in std::execute_native_thread_routine (__p=0x504000006050)at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
      #40 0x000055a6ce3eda8d in asan_thread_start(void*) ()
      #41 0x000014f518a9ca94 in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:447
      #42 0x000014f518b29c3c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
      

      cmake command:

      cmake . -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DWITH_SSL=bundled -DBUILD_CONFIG=mysql_release -DWITH_TOKUDB=0 -DWITH_JEMALLOC=no -DFEATURE_SET=community -DDEBUG_EXTNAME=OFF -DWITH_EMBEDDED_SERVER=0 -DENABLE_DOWNLOADS=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/tmp/boost_392129 -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DWITH_SAFEMALLOC=OFF -DPLUGIN_PERFSCHEMA=NO -DWITH_DBUG_TRACE=OFF -DWITH_ZLIB=bundled -DWITH_ROCKSDB=1 -DWITH_PAM=ON -DWITH_MARIABACKUP=0 -DFORCE_INSOURCE_BUILD=1 -DWITH_ASAN=ON -DWITH_ASAN_SCOPE=ON -DWITH_UBSAN=ON -DWSREP_LIB_WITH_ASAN=ON -DCMAKE_CXX_FLAGS=-fsanitize-coverage=trace-pc-guard -DMYSQL_MAINTAINER_MODE=OFF -DWARNING_AS_ERROR='' -DCMAKE_BUILD_TYPE=Debug
      

      Setup:

      Compiled with a recent version of Clang (I used Clang 18.1.3) with LLVM 18:
           # Note: llvm-17-linker-tools installs /usr/lib/llvm-17/lib/LLVMgold.so, which is needed for compilation, and LLVMgold.so is no longer included in LLVM 18
           sudo apt install clang llvm-18 llvm-18-linker-tools llvm-18-runtime llvm-18-tools llvm-18-dev libstdc++-14-dev llvm-dev llvm-17-linker-tools
           sudo ln -s /usr/lib/llvm-17/lib/LLVMgold.so /usr/lib/llvm-18/lib/LLVMgold.so
      Compiled with: '-DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++' and:
          -DWITH_ASAN=ON -DWITH_ASAN_SCOPE=ON -DWITH_UBSAN=ON -DWSREP_LIB_WITH_ASAN=ON
      Set before execution:
          export UBSAN_OPTIONS=print_stacktrace=1:report_error_type=1
      

      Present in debug builds. Not reproducible in MTR. No immediate issues observed in optimized/release builds.

      Attachments

        Activity

          The function __sanitizer_cov_trace_pc_guard() is specific to clang -fsanitize-coverage=trace-pc-guard. This is not the first issue that involves it, if you really have use for that code coverage data that it would cause to be collected. I think that you should also try a newer version of clang. The current one is 20.

          marko Marko Mäkelä added a comment - The function __sanitizer_cov_trace_pc_guard() is specific to clang -fsanitize-coverage=trace-pc-guard . This is not the first issue that involves it, if you really have use for that code coverage data that it would cause to be collected. I think that you should also try a newer version of clang. The current one is 20.

          People

            ycp Yuchen Pei
            Roel Roel Van de Paar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.