Elena Stepanova added a comment - 2022-06-03 09:08 - edited Note the use of the same server link makes the test case in the description degenerate, at least in the sense that it switches the query cache type both for the parent and the child at once. Possibly it can be dealt with by changing the query_cache_type on the session level in the connection where Spider table is opened, but it doesn't seem very reliable. I would recommend after fixing the issue to add a regression test with proper separation between the parent and the child nodes and with corresponding combinations of enabled/disabled query cache – enabled only on parent, enabled only on child (or, depending on the fix, maybe more than one child), enabled on both.

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-17 15:39 - edited I confirm that the bug is reproducible but it is not reproducible with the --rr option (why?). --echo # --echo # MDEV-28739 Trying to lock uninitialized mutex or hang upon shutdown after using Spider with query_cache --echo #   --disable_query_log --disable_result_log --source ../../t/test_init.inc --enable_result_log --enable_query_log   --connection child2_1 CREATE DATABASE auto_test_remote; USE auto_test_remote;   CREATE TABLE tbl_a (id INT );   --connection master_1 CREATE DATABASE auto_test_local; USE auto_test_local;   set global query_cache_type= on ; set spider_same_server_link = on ;   eval CREATE TABLE tbl_a ( id INT ) $MASTER_1_ENGINE $MASTER_1_CHARSET COMMENT= 'table "tbl_a", srv "s_2_1"' ;   SELECT * FROM tbl_a;   --source include/restart_mysqld.inc   --connection master_1 DROP DATABASE IF EXISTS auto_test_local; --connection child2_1 DROP DATABASE IF EXISTS auto_test_remote;   --disable_query_log --disable_result_log --source ../t/test_deinit.inc --enable_query_log --enable_result_log

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-17 16:00 The workaround would be to disable statistics persistence. However, even with the workaround, the test fails due to MDEV-27912 . set global spider_store_last_crd=0; set global spider_store_last_sts=0;

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-20 08:45 Spider updates system tables too late in the shutdown sequence. I think that updating the tables in handlerton::panic or handlerton::pre_shutdown would fix the problem.

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-20 09:48 A tentative fix. I will refactor this somewhat. https://github.com/MariaDB/server/commit/e26b3932aa536bc71a9dea91058f729f2657a08e

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-20 12:15 spider_panic seems to be called too late for updating the stats tables.

Nayuta Yanagisawa (Inactive) added a comment - 2022-06-20 12:57 The patch above does not work. It prevents the crash but fails to update the stats tables.

Yuchen Pei added a comment - 2023-05-15 04:49 I confirm the testcase at https://jira.mariadb.org/browse/MDEV-28739?focusedCommentId=227057&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-227057 crashes in 11.1 4e5b771e980edfdad5c5414aa62c81d409d585a4. Taking over: safe_mutex: Trying to lock uninitialized mutex at /home/ycp/source/mariadb-server/11.1/src/sql/sql_cache.cc, line 725 230515 14:45:47 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 11.1.0-MariaDB-debug-log source revision: 3ef111610b7f8a6a323975cfdf4a4257feb9dcd9 key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=5 max_threads=153 thread_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63925 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x559ce341ca78 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fffd598bcf8 thread_stack 0x49000 mysys/stacktrace.c:215(my_print_stacktrace)[0x559ce1021efa] sql/signal_handler.cc:238(handle_fatal_signal)[0x559ce07b60cb] ??:0(__restore_rt)[0x7f2440e65140] ??:0(gsignal)[0x7f244067cce1] ??:0(abort)[0x7f2440666537] mysys/thr_mutex.c:247(safe_mutex_lock)[0x559ce10266d9] psi/mysql_thread.h:750(inline_mysql_mutex_lock)[0x559ce0342b1c] sql/sql_cache.cc:726(Query_cache::lock(THD*))[0x559ce03439f0] sql/sql_cache.cc:3319(Query_cache::invalidate_table(THD*, unsigned char*, unsigned long))[0x559ce034a943] sql/sql_cache.cc:2440(Query_cache::invalidate_by_MyISAM_filename(char const*))[0x559ce0348541] sql/sql_cache.cc:1252(query_cache_invalidate_by_MyISAM_filename)[0x559ce0344e21] maria/ha_maria.cc:3092(reset_thd_trn(THD*, st_maria_handler*))[0x559ce09cd467] maria/ha_maria.cc:3630(maria_commit(handlerton*, THD*, bool))[0x559ce09ce981] sql/handler.cc:2126(commit_one_phase_2(THD*, bool, THD_TRANS*, bool))[0x559ce07bd685] sql/handler.cc:2079(ha_commit_one_phase(THD*, bool))[0x559ce07bd442] sql/handler.cc:1873(ha_commit_trans(THD*, bool))[0x559ce07bc562] sql/sql_class.cc:6100(THD::commit_whole_transaction_and_close_tables())[0x559ce03648fc] spider/spd_sys_table.cc:597(spider_sys_close_table(THD*, start_new_trans**))[0x7f243825ac23] spider/spd_sys_table.cc:3236(spider_sys_insert_or_update_table_sts(THD*, char const*, unsigned int, ha_statistics*))[0x7f2438266ba4] spider/spd_table.cc:5623(spider_free_share(st_spider_share*))[0x7f24382e68f5] spider/ha_spider.cc:577(ha_spider::close())[0x7f2438320212] sql/handler.cc:3556(handler::ha_close())[0x559ce07c1aaf] sql/table.cc:4674(closefrm(TABLE*))[0x559ce055b362] sql/table_cache.cc:226(intern_close_table(TABLE*))[0x559ce06d913d] sql/table_cache.cc:317(tc_purge())[0x559ce06d9519] sql/sql_base.cc:330(purge_tables())[0x559ce03270ca] sql/table_cache.cc:641(tdc_start_shutdown())[0x559ce06dab7c] sql/mysqld.cc:1998(clean_up(bool))[0x559ce02491f9] sql/mysqld.cc:6057(mysqld_main(int, char**))[0x559ce0251bdc] sql/main.cc:34(main)[0x559ce0245595] ??:0(__libc_start_main)[0x7f2440667d0a] ??:0(_start)[0x559ce02454ba]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): (null) Connection ID (thread ID): 30 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off   The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash.   We think the query pointer is invalid, but we will try to print it anyway. Query:   Writing a core file... Working directory at /home/ycp/source/mariadb-server/11.1/build/mysql-test/var/mysqld.1.1/data Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10022912 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 124974 124974 processes Max open files 1024 1024 files Max locked memory 4108061184 4108061184 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 124974 124974 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core   Kernel version: Linux version 6.0.0-0.deb11.2-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PREEMPT_DYNAMIC Debian 6.0.3-1~bpo11+1 (2022-10-29)   ----------SERVER LOG END------------- mysqltest failed but provided no output     - found 'core' (0/5) Core generated by '/home/ycp/source/mariadb-server/11.1/build/sql/mariadbd' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). --------------------------   warning: Can't open file /[aio] (deleted) during file-backed mapping note processing [New LWP 3251046] [New LWP 3251060] [New LWP 3251053] [New LWP 3251054] [New LWP 3251055] [New LWP 3251048] [New LWP 3251047] [New LWP 3251056] [New LWP 3251064] [New LWP 3251058] [New LWP 3251059] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/home/ycp/source/mariadb-server/11.1/build/sql/mariadbd --defaults-group-suffix'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f2440e61f44 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0 [Current thread is 1 (Thread 0x7f2440cf5940 (LWP 3251046))]   Thread 11 (Thread 0x7f243234c700 (LWP 3251059)): #0 0x00007f2440e60df8 in pthread_cond_clockwait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce0f80c68 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9dd0, __lock=@0x7f243234bb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f243234bb28: {__d = {__r = 13829482131520213}}) at /usr/include/c++/10/condition_variable:209 #2 0x0000559ce0f80165 in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9dd0, __lock=@0x7f243234bb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f243234bb28: {__d = {__r = 13829482131520213}}) at /usr/include/c++/10/condition_variable:119 #3 0x0000559ce0f7f65a in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x559ce32b9dd0, __lock=@0x7f243234bb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __rtime=@0x559ce32b9a60: {__r = 60000}) at /usr/include/c++/10/condition_variable:172 #4 0x0000559ce0f7cf05 in tpool::thread_pool_generic::wait_for_tasks (this=0x559ce32b9920, lk=@0x7f243234bb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, thread_data=0x559ce32b9dd0) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:480 #5 0x0000559ce0f7d111 in tpool::thread_pool_generic::get_task (this=0x559ce32b9920, thread_var=0x559ce32b9dd0, t=0x7f243234bbd8) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:533 #6 0x0000559ce0f7d397 in tpool::thread_pool_generic::worker_main (this=0x559ce32b9920, thread_var=0x559ce32b9dd0) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:578 #7 0x0000559ce0f82f8a in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f2424000d08: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f2424000d00: 0x559ce32b9920) at /usr/include/c++/10/bits/invoke.h:73 #8 0x0000559ce0f82e7a in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f2424000d08: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/10/bits/invoke.h:95 #9 0x0000559ce0f82dad in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f2424000cf8) at /usr/include/c++/10/thread:264 #10 0x0000559ce0f82d4a in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f2424000cf8) at /usr/include/c++/10/thread:271 #11 0x0000559ce0f82d2e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f2424000cf0) at /usr/include/c++/10/thread:215 #12 0x00007f2440901ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 10 (Thread 0x7f2432cdd700 (LWP 3251058)): #0 0x00007f2440e60df8 in pthread_cond_clockwait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce0f80c68 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9d50, __lock=@0x7f2432cdcb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f2432cdcb28: {__d = {__r = 13829482455995726}}) at /usr/include/c++/10/condition_variable:209 #2 0x0000559ce0f80165 in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9d50, __lock=@0x7f2432cdcb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f2432cdcb28: {__d = {__r = 13829482455995726}}) at /usr/include/c++/10/condition_variable:119 #3 0x0000559ce0f7f65a in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x559ce32b9d50, __lock=@0x7f2432cdcb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __rtime=@0x559ce32b9a60: {__r = 60000}) at /usr/include/c++/10/condition_variable:172 #4 0x0000559ce0f7cf05 in tpool::thread_pool_generic::wait_for_tasks (this=0x559ce32b9920, lk=@0x7f2432cdcb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, thread_data=0x559ce32b9d50) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:480 #5 0x0000559ce0f7d111 in tpool::thread_pool_generic::get_task (this=0x559ce32b9920, thread_var=0x559ce32b9d50, t=0x7f2432cdcbd8) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:533 #6 0x0000559ce0f7d397 in tpool::thread_pool_generic::worker_main (this=0x559ce32b9920, thread_var=0x559ce32b9d50) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:578 #7 0x0000559ce0f82f8a in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f2424000b78: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f2424000b70: 0x559ce32b9920) at /usr/include/c++/10/bits/invoke.h:73 #8 0x0000559ce0f82e7a in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f2424000b78: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/10/bits/invoke.h:95 #9 0x0000559ce0f82dad in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f2424000b68) at /usr/include/c++/10/thread:264 #10 0x0000559ce0f82d4a in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f2424000b68) at /usr/include/c++/10/thread:271 #11 0x0000559ce0f82d2e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f2424000b60) at /usr/include/c++/10/thread:215 #12 0x00007f2440901ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 9 (Thread 0x7f2440050700 (LWP 3251064)): #0 0x00007f244067dba2 in sigtimedwait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000559ce0245746 in my_sigwait (set=0x7f244004fc10, sig=0x7f244004fbd4, code=0x7f244004fbd8) at /home/ycp/source/mariadb-server/11.1/src/include/my_pthread.h:195 #2 0x0000559ce024be9c in signal_hand (arg=0x0) at /home/ycp/source/mariadb-server/11.1/src/sql/mysqld.cc:3263 #3 0x0000559ce0abdfc4 in pfs_spawn_thread (arg=0x559ce3602618) at /home/ycp/source/mariadb-server/11.1/src/storage/perfschema/pfs.cc:2201 #4 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 8 (Thread 0x7f243366e700 (LWP 3251056)): #0 0x00007f2440e60df8 in pthread_cond_clockwait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce0f80c68 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9cd0, __lock=@0x7f243366db90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f243366db28: {__d = {__r = 13829482115437018}}) at /usr/include/c++/10/condition_variable:209 #2 0x0000559ce0f80165 in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9cd0, __lock=@0x7f243366db90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f243366db28: {__d = {__r = 13829482115437018}}) at /usr/include/c++/10/condition_variable:119 #3 0x0000559ce0f7f65a in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x559ce32b9cd0, __lock=@0x7f243366db90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __rtime=@0x559ce32b9a60: {__r = 60000}) at /usr/include/c++/10/condition_variable:172 #4 0x0000559ce0f7cf05 in tpool::thread_pool_generic::wait_for_tasks (this=0x559ce32b9920, lk=@0x7f243366db90: {_M_device = 0x559ce32b9a38, _M_owns = true}, thread_data=0x559ce32b9cd0) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:480 #5 0x0000559ce0f7d111 in tpool::thread_pool_generic::get_task (this=0x559ce32b9920, thread_var=0x559ce32b9cd0, t=0x7f243366dbd8) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:533 #6 0x0000559ce0f7d397 in tpool::thread_pool_generic::worker_main (this=0x559ce32b9920, thread_var=0x559ce32b9cd0) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:578 #7 0x0000559ce0f82f8a in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f243c002278: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f243c002270: 0x559ce32b9920) at /usr/include/c++/10/bits/invoke.h:73 #8 0x0000559ce0f82e7a in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f243c002278: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/10/bits/invoke.h:95 #9 0x0000559ce0f82dad in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f243c002268) at /usr/include/c++/10/thread:264 #10 0x0000559ce0f82d4a in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f243c002268) at /usr/include/c++/10/thread:271 #11 0x0000559ce0f82d2e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f243c002260) at /usr/include/c++/10/thread:215 #12 0x00007f2440901ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 7 (Thread 0x7f24410db700 (LWP 3251047)): #0 0x00007f2440e60ad8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce10274e5 in safe_cond_timedwait (cond=0x559ce2716a00 <COND_timer>, mp=0x559ce2716940 <LOCK_timer>, abstime=0x7f24410dace0, file=0x559ce15c53c0 "/home/ycp/source/mariadb-server/11.1/src/include/mysql/psi/mysql_thread.h", line=1088) at /home/ycp/source/mariadb-server/11.1/src/mysys/thr_mutex.c:548 #2 0x0000559ce1028496 in inline_mysql_cond_timedwait (that=0x559ce2716a00 <COND_timer>, mutex=0x559ce2716940 <LOCK_timer>, abstime=0x7f24410dace0, src_file=0x559ce15c5410 "/home/ycp/source/mariadb-server/11.1/src/mysys/thr_timer.c", src_line=321) at /home/ycp/source/mariadb-server/11.1/src/include/mysql/psi/mysql_thread.h:1088 #3 0x0000559ce1029116 in timer_handler (arg=0x0) at /home/ycp/source/mariadb-server/11.1/src/mysys/thr_timer.c:321 #4 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 6 (Thread 0x7f243b67b700 (LWP 3251048)): #0 0x00007f2440e60ad8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce10274e5 in safe_cond_timedwait (cond=0x559ce2683cc0 <COND_checkpoint>, mp=0x559ce2683c00 <LOCK_checkpoint>, abstime=0x7f243b67ac30, file=0x559ce13d7020 "/home/ycp/source/mariadb-server/11.1/src/include/mysql/psi/mysql_thread.h", line=1088) at /home/ycp/source/mariadb-server/11.1/src/mysys/thr_mutex.c:548 #2 0x0000559ce0a0d713 in inline_mysql_cond_timedwait (that=0x559ce2683cc0 <COND_checkpoint>, mutex=0x559ce2683c00 <LOCK_checkpoint>, abstime=0x7f243b67ac30, src_file=0x559ce13d7070 "/home/ycp/source/mariadb-server/11.1/src/storage/maria/ma_servicethread.c", src_line=115) at /home/ycp/source/mariadb-server/11.1/src/include/mysql/psi/mysql_thread.h:1088 #3 0x0000559ce0a0dbf9 in my_service_thread_sleep (control=0x559ce1c31900 <checkpoint_control>, sleep_time=29000000000) at /home/ycp/source/mariadb-server/11.1/src/storage/maria/ma_servicethread.c:115 #4 0x0000559ce0a01994 in ma_checkpoint_background (arg=0x1e) at /home/ycp/source/mariadb-server/11.1/src/storage/maria/ma_checkpoint.c:725 #5 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 5 (Thread 0x7f2433fff700 (LWP 3251055)): #0 0x00007f2440e60df8 in pthread_cond_clockwait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce0f80c68 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9c50, __lock=@0x7f2433ffeb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f2433ffeb28: {__d = {__r = 13829482131538908}}) at /usr/include/c++/10/condition_variable:209 #2 0x0000559ce0f80165 in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9c50, __lock=@0x7f2433ffeb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f2433ffeb28: {__d = {__r = 13829482131538908}}) at /usr/include/c++/10/condition_variable:119 #3 0x0000559ce0f7f65a in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x559ce32b9c50, __lock=@0x7f2433ffeb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __rtime=@0x559ce32b9a60: {__r = 60000}) at /usr/include/c++/10/condition_variable:172 #4 0x0000559ce0f7cf05 in tpool::thread_pool_generic::wait_for_tasks (this=0x559ce32b9920, lk=@0x7f2433ffeb90: {_M_device = 0x559ce32b9a38, _M_owns = true}, thread_data=0x559ce32b9c50) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:480 #5 0x0000559ce0f7d111 in tpool::thread_pool_generic::get_task (this=0x559ce32b9920, thread_var=0x559ce32b9c50, t=0x7f2433ffebd8) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:533 #6 0x0000559ce0f7d397 in tpool::thread_pool_generic::worker_main (this=0x559ce32b9920, thread_var=0x559ce32b9c50) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:578 #7 0x0000559ce0f82f8a in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x559ce341c9d8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x559ce341c9d0: 0x559ce32b9920) at /usr/include/c++/10/bits/invoke.h:73 #8 0x0000559ce0f82e7a in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x559ce341c9d8: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/10/bits/invoke.h:95 #9 0x0000559ce0f82dad in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x559ce341c9c8) at /usr/include/c++/10/thread:264 #10 0x0000559ce0f82d4a in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x559ce341c9c8) at /usr/include/c++/10/thread:271 #11 0x0000559ce0f82d2e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x559ce341c9c0) at /usr/include/c++/10/thread:215 #12 0x00007f2440901ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 4 (Thread 0x7f2439358700 (LWP 3251054)): #0 0x00007f2440e60ad8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce10274e5 in safe_cond_timedwait (cond=0x559ce1c4fbd0 <buf_pool+17424>, mp=0x559ce1c4fac0 <buf_pool+17152>, abstime=0x7f2439357c20, file=0x559ce1554f68 "/home/ycp/source/mariadb-server/11.1/src/storage/innobase/buf/buf0flu.cc", line=2332) at /home/ycp/source/mariadb-server/11.1/src/mysys/thr_mutex.c:548 #2 0x0000559ce0e83469 in buf_flush_page_cleaner () at /home/ycp/source/mariadb-server/11.1/src/storage/innobase/buf/buf0flu.cc:2332 #3 0x0000559ce0e877df in std::__invoke_impl<void, void (*)()> (__f=@0x559ce33956a8: 0x559ce0e830d1 <buf_flush_page_cleaner()>) at /usr/include/c++/10/bits/invoke.h:60 #4 0x0000559ce0e87789 in std::__invoke<void (*)()> (__fn=@0x559ce33956a8: 0x559ce0e830d1 <buf_flush_page_cleaner()>) at /usr/include/c++/10/bits/invoke.h:95 #5 0x0000559ce0e87736 in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x559ce33956a8) at /usr/include/c++/10/thread:264 #6 0x0000559ce0e8770a in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x559ce33956a8) at /usr/include/c++/10/thread:271 #7 0x0000559ce0e876ee in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x559ce33956a0) at /usr/include/c++/10/thread:215 #8 0x00007f2440901ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #10 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 3 (Thread 0x7f243a8e9700 (LWP 3251053)): #0 0x00007f244073a2e9 in syscall () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000559ce0f83a36 in my_getevents (ctx=0x7f24400e7000, min_nr=1, nr=256, ev=0x7f243a8e6c00) at /home/ycp/source/mariadb-server/11.1/src/tpool/aio_linux.cc:63 #2 0x0000559ce0f83bfe in tpool::aio_linux::getevent_thread_routine (aio=0x559ce32b98f0) at /home/ycp/source/mariadb-server/11.1/src/tpool/aio_linux.cc:105 #3 0x0000559ce0f844d8 in std::__invoke_impl<void, void (*)(tpool::aio_linux*), tpool::aio_linux*> (__f=@0x559ce31de9f0: 0x559ce0f83bbb <tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)>) at /usr/include/c++/10/bits/invoke.h:60 #4 0x0000559ce0f8443f in std::__invoke<void (*)(tpool::aio_linux*), tpool::aio_linux*> (__fn=@0x559ce31de9f0: 0x559ce0f83bbb <tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)>) at /usr/include/c++/10/bits/invoke.h:95 #5 0x0000559ce0f843af in std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> >::_M_invoke<0ul, 1ul> (this=0x559ce31de9e8) at /usr/include/c++/10/thread:264 #6 0x0000559ce0f84368 in std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> >::operator() (this=0x559ce31de9e8) at /usr/include/c++/10/thread:271 #7 0x0000559ce0f8434c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(tpool::aio_linux*), tpool::aio_linux*> > >::_M_run (this=0x559ce31de9e0) at /usr/include/c++/10/thread:215 #8 0x00007f2440901ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #10 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 2 (Thread 0x7f24319bb700 (LWP 3251060)): #0 0x00007f2440e60df8 in pthread_cond_clockwait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce0f80c68 in std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9e50, __lock=@0x7f24319bab90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f24319bab28: {__d = {__r = 13829482131526435}}) at /usr/include/c++/10/condition_variable:209 #2 0x0000559ce0f80165 in std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x559ce32b9e50, __lock=@0x7f24319bab90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __atime=@0x7f24319bab28: {__d = {__r = 13829482131526435}}) at /usr/include/c++/10/condition_variable:119 #3 0x0000559ce0f7f65a in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x559ce32b9e50, __lock=@0x7f24319bab90: {_M_device = 0x559ce32b9a38, _M_owns = true}, __rtime=@0x559ce32b9a60: {__r = 60000}) at /usr/include/c++/10/condition_variable:172 #4 0x0000559ce0f7cf05 in tpool::thread_pool_generic::wait_for_tasks (this=0x559ce32b9920, lk=@0x7f24319bab90: {_M_device = 0x559ce32b9a38, _M_owns = true}, thread_data=0x559ce32b9e50) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:480 #5 0x0000559ce0f7d111 in tpool::thread_pool_generic::get_task (this=0x559ce32b9920, thread_var=0x559ce32b9e50, t=0x7f24319babd8) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:533 #6 0x0000559ce0f7d397 in tpool::thread_pool_generic::worker_main (this=0x559ce32b9920, thread_var=0x559ce32b9e50) at /home/ycp/source/mariadb-server/11.1/src/tpool/tpool_generic.cc:578 #7 0x0000559ce0f82f8a in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__f=@0x7f2424000e98: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>, __t=@0x7f2424000e90: 0x559ce32b9920) at /usr/include/c++/10/bits/invoke.h:73 #8 0x0000559ce0f82e7a in std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=@0x7f2424000e98: (void (tpool::thread_pool_generic::*)(tpool::thread_pool_generic * const, tpool::worker_data *)) 0x559ce0f7d336 <tpool::thread_pool_generic::worker_main(tpool::worker_data*)>) at /usr/include/c++/10/bits/invoke.h:95 #9 0x0000559ce0f82dad in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=0x7f2424000e88) at /usr/include/c++/10/thread:264 #10 0x0000559ce0f82d4a in std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=0x7f2424000e88) at /usr/include/c++/10/thread:271 #11 0x0000559ce0f82d2e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=0x7f2424000e80) at /usr/include/c++/10/thread:215 #12 0x00007f2440901ed0 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007f2440e59ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x00007f2440740a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6   Thread 1 (Thread 0x7f2440cf5940 (LWP 3251046)): #0 0x00007f2440e61f44 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000559ce1021fec in my_write_core (sig=6) at /home/ycp/source/mariadb-server/11.1/src/mysys/stacktrace.c:424 #2 0x0000559ce07b6411 in handle_fatal_signal (sig=6) at /home/ycp/source/mariadb-server/11.1/src/sql/signal_handler.cc:357 #3 <signal handler called> #4 0x00007f244067cce1 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007f2440666537 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000559ce10266d9 in safe_mutex_lock (mp=0x559ce1e45620 <query_cache+160>, my_flags=0, file=0x559ce114f318 "/home/ycp/source/mariadb-server/11.1/src/sql/sql_cache.cc", line=725) at /home/ycp/source/mariadb-server/11.1/src/mysys/thr_mutex.c:245 #7 0x0000559ce0342b1c in inline_mysql_mutex_lock (that=0x559ce1e45620 <query_cache+160>, src_file=0x559ce114f318 "/home/ycp/source/mariadb-server/11.1/src/sql/sql_cache.cc", src_line=725) at /home/ycp/source/mariadb-server/11.1/src/include/mysql/psi/mysql_thread.h:750 #8 0x0000559ce03439f0 in Query_cache::lock (this=0x559ce1e45580 <query_cache>, thd=0x559ce341ca78) at /home/ycp/source/mariadb-server/11.1/src/sql/sql_cache.cc:725 #9 0x0000559ce034a943 in Query_cache::invalidate_table (this=0x559ce1e45580 <query_cache>, thd=0x559ce341ca78, key=0x7fffd598ae90 "mysql", key_length=23) at /home/ycp/source/mariadb-server/11.1/src/sql/sql_cache.cc:3317 #10 0x0000559ce0348541 in Query_cache::invalidate_by_MyISAM_filename (this=0x559ce1e45580 <query_cache>, filename=0x559ce367ce00 "./mysql/spider_table_sts.MAD") at /home/ycp/source/mariadb-server/11.1/src/sql/sql_cache.cc:2439 #11 0x0000559ce0344e21 in query_cache_invalidate_by_MyISAM_filename (filename=0x559ce367ce00 "./mysql/spider_table_sts.MAD") at /home/ycp/source/mariadb-server/11.1/src/sql/sql_cache.cc:1251 #12 0x0000559ce09cd467 in reset_thd_trn (thd=0x559ce341ca78, first_table=0x559ce36bf478) at /home/ycp/source/mariadb-server/11.1/src/storage/maria/ha_maria.cc:3105 #13 0x0000559ce09ce981 in maria_commit (hton=0x559ce3202ca8, thd=0x559ce341ca78, all=false) at /home/ycp/source/mariadb-server/11.1/src/storage/maria/ha_maria.cc:3629 #14 0x0000559ce07bd685 in commit_one_phase_2 (thd=0x559ce341ca78, all=false, trans=0x559ce31399d8, is_real_trans=true) at /home/ycp/source/mariadb-server/11.1/src/sql/handler.cc:2126 #15 0x0000559ce07bd442 in ha_commit_one_phase (thd=0x559ce341ca78, all=false) at /home/ycp/source/mariadb-server/11.1/src/sql/handler.cc:2079 #16 0x0000559ce07bc562 in ha_commit_trans (thd=0x559ce341ca78, all=false) at /home/ycp/source/mariadb-server/11.1/src/sql/handler.cc:1873 #17 0x0000559ce03648fc in THD::commit_whole_transaction_and_close_tables (this=0x559ce341ca78) at /home/ycp/source/mariadb-server/11.1/src/sql/sql_class.cc:6100 #18 0x00007f243825ac23 in spider_sys_close_table (thd=0x559ce341ca78, open_tables_backup=0x7fffd598bcd0) at /home/ycp/source/mariadb-server/11.1/src/storage/spider/spd_sys_table.cc:596 #19 0x00007f2438266ba4 in spider_sys_insert_or_update_table_sts (thd=0x559ce341ca78, name=0x7f240403dc60 "./auto_test_local/tbl_a", name_length=23, stat=0x7f24041ff728) at /home/ycp/source/mariadb-server/11.1/src/storage/spider/spd_sys_table.cc:3235 #20 0x00007f24382e68f5 in spider_free_share (share=0x7f24041feee8) at /home/ycp/source/mariadb-server/11.1/src/storage/spider/spd_table.cc:5615 #21 0x00007f2438320212 in ha_spider::close (this=0x7f24041d3380) at /home/ycp/source/mariadb-server/11.1/src/storage/spider/ha_spider.cc:576 #22 0x0000559ce07c1aaf in handler::ha_close (this=0x7f24041d3380) at /home/ycp/source/mariadb-server/11.1/src/sql/handler.cc:3556 #23 0x0000559ce055b362 in closefrm (table=0x7f24041d2a78) at /home/ycp/source/mariadb-server/11.1/src/sql/table.cc:4674 #24 0x0000559ce06d913d in intern_close_table (table=0x7f24041d2a78) at /home/ycp/source/mariadb-server/11.1/src/sql/table_cache.cc:225 #25 0x0000559ce06d9519 in tc_purge () at /home/ycp/source/mariadb-server/11.1/src/sql/table_cache.cc:317 #26 0x0000559ce03270ca in purge_tables () at /home/ycp/source/mariadb-server/11.1/src/sql/sql_base.cc:328 #27 0x0000559ce06dab7c in tdc_start_shutdown () at /home/ycp/source/mariadb-server/11.1/src/sql/table_cache.cc:639 #28 0x0000559ce02491f9 in clean_up (print_message=true) at /home/ycp/source/mariadb-server/11.1/src/sql/mysqld.cc:1996 #29 0x0000559ce0251bdc in mysqld_main (argc=156, argv=0x559ce3133580) at /home/ycp/source/mariadb-server/11.1/src/sql/mysqld.cc:6056 #30 0x0000559ce0245595 in main (argc=8, argv=0x7fffd598c158) at /home/ycp/source/mariadb-server/11.1/src/sql/main.cc:34

Yuchen Pei added a comment - 2023-11-09 03:16 - edited The crash happens when shutting down the server. For example, the following is a simplified case: --disable_query_log --disable_result_log --source ../../t/test_init.inc --enable_result_log --enable_query_log   set global query_cache_type= on; evalp CREATE SERVER srv FOREIGN DATA WRAPPER mysql OPTIONS (SOCKET "$MASTER_1_MYSOCK", DATABASE 'test',user 'root'); create table t2 (c int); create table t1 (c int) ENGINE=Spider COMMENT='WRAPPER "mysql", srv "srv",TABLE "t2"'; SELECT * FROM t1; shutdown; Removing the shutdown; and we get the same "safe_mutex: Trying to lock uninitialized mutex" failure, as well as crash log in mysqld.1.1.err. The trace: safe_mutex_lock > inline_mysql_mutex_lock > Query_cache::invalidate_table > reset_thd_trn > commit_one_phase_2 > ha_commit_one_phase > THD::commit_whole_transaction_and_close_tables > spider_close_sys_table > spider_free_share > handler::ha_close > closefrm > intern_close_table > purge_tables > tdc_start_shutdown > clean_up > mysqld_main

Yuchen Pei added a comment - 2023-11-09 05:14 - edited The cause is that spider updates its sts system tables during the close() method, which is called a bit too late, after the call to query_cache_destroy() which destroys the structure_guard_mutex . 10.10 // Inside clean_up(): query_cache_destroy(); // [... 4 lines elided] tdc_start_shutdown(); However, to begin with, it is not clear whether the sts and crd tables need to be updated at all when freeing a spider_share. Removing the call to spider_sys_insert_or_update_table_sts() and spider_sys_insert_or_update_table_crd() from spider_free_share() fixes the test, and does not introduce any regressions in existing tests either, see the following commit 6931635fa0f upstream/bb-10.10-mdev-28739 MDEV-28739 [demo] Commenting out spider sts and crd table updates fixes the problem In fact, spider_free_share() is the only place where the spider sts and crd tables are updated. sts tables, for example, are used to populate the spider handler's stat field. If the sts table does not have info about a remote table, spider will make a "show table status" to obtain this piece of information. Apart from failure paths of spider_open_share() etc., spider_free_share() is only called in ha_spider::close(). So the sts and crd tables are only updated at ha_spider::close(). The freshness of the data is thus questionable. It remains to be seen whether spider checks for the freshness before querying these tables. In any case, this ticket is now about the usefulness of the spider sts and crd system tables, so I will group it with other relevant tickets about these tables under the label spider-sts-crd (< https://jira.mariadb.org/issues/?jql=labels%20%3D%20spider-sts-crd >).

Yuchen Pei added a comment - 2023-11-21 07:52 Draft commit, will send to review tomorrow if nothing new occurs: cb8df270bb3 MDEV-28739 MDEV-29421 Remove updating spider sts/crd tables from spider_free_share() We are moving the spider system tables altogether, since the only reference to the update function is removed. See also the commit message.

Yuchen Pei added a comment - 2023-11-22 01:34 Hi holyfoot , ptal thanks (based on 11.0) c29972b314690bc94b890a2aaa6f2898c7880432 MDEV-28739 MDEV-29421 Remove spider persistent table stats   We remove the call to update spider persistent table stats (sts/crd) in spider_free_share(). This prevents spider from opening and closing further tables during close(), which fixes the following issues:   MDEV-28739: ha_spider::close() is called during tdc_start_shutdown(), which is called after query_cache_destroy(). Closing the sts/crd Aria tables will trigger a call to Query_cache::invalidate_table(), which will attempt to use the query cache mutex structure_guard_mutex destroyed previously.   MDEV-29421: during ha_spider::close(), spider_free_share() could trigger another spider_free_share() through updating sts/crd table, because open_table() calls tc_add_table(), which could trigger another ha_spider::close()...   Since spider sts/crd system tables are only updated here, there's no use for these tables any more, and we remove all uses of these tables too.   The removal should not cause any performance issue, as in memory spider table stats are only updated based on a time interval (spider_sts_interval and spider_crd_interval), which defaults to 10 seconds. It should not affect accuracy either, due to the infrequency of server restart. And inaccurate stats are not a problem for optimizer anyway.   To be on the safe side, we defer the removal of the spider sts/crd tables themselves to future. Here's a 10.4 version: 87bf3f002e6 MDEV-28739 MDEV-29421 Remove spider persistent table stats

Alexey Botchkov added a comment - 2023-11-23 21:21 ok to push.

Yuchen Pei added a comment - 2023-11-24 06:11 - edited pushed bdfd93d30c1bba2a8932477f16f6280ee665d818 to 10.4 Just noticed the comment in the description that it was not reproducible on 10.4 at reporting. Just to note that I could reproduce in 10.4 which is why I had a 10.4 fix. Conflict resolution (old commit, make sure to add add_suppression in the test mdev_27575): 10.4->10.5: 1a97c706f072a55e27f9580e7bf6dc8727f4316d 10.5->10.6: ad7620a25cf7f8f517c9f4373d9c1d00853a6141 10.6->10.11: 7ba4f323c6990d1825a2754bf8577f16f5d4cf40 10.11->11.0: e3ce81b1e90c73b1e21c8751c78572893e55b456 ES23.08: 57ab808c0cf34baa34a60df08368b1f4ae9e640e