[MDEV-27026] innodb_fts.concurrent_insert failed in bb, innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch Created: 2021-11-11  Updated: 2022-09-28  Resolved: 2022-05-03

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6, 10.7, 10.8
Fix Version/s: 10.6.6, 10.7.2, 10.8.1

Type: Bug Priority: Blocker
Reporter: Alice Sherepa Assignee: Marko Mäkelä
Resolution: Fixed Votes: 1
Labels: hang

Attachments: HTML File bt    
Issue Links:
Blocks
Relates
relates to MDEV-28289 fts_optimize_sync_table() is acquirin... Closed
relates to MDEV-29660 [ERROR] [FATAL] InnoDB: innodb_fatal_... Closed
relates to MDEV-27983 InnoDB hangs on multiple concurrent r... Closed
relates to MDEV-28289 fts_optimize_sync_table() is acquirin... Closed

 Description   

https://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/31734/steps/mtr_nm/logs/stdio

10.7 a52cd4aedac70676eb100afc8779e

 
nnodb_fts.concurrent_insert 'innodb'    w2 [ fail ]
 
        Test ended at 2021-10-27 18:43:58
 
 
 
CURRENT_TEST: innodb_fts.concurrent_insert
 
mysqltest: At line 48: query 'reap' failed: <Unknown> (2013): Lost connection to server during query
 
 
 
The result from queries just before the failure was:
 
< snip >
 
#
 
# MDEV-19529 InnoDB hang on DROP FULLTEXT INDEX
 
#
 
CREATE TABLE t1(f1 CHAR(100), FULLTEXT(f1))ENGINE=InnoDB;
 
INSERT INTO t1 VALUES('test');
 
CREATE TABLE t2 (f1 char(100), FULLTEXT idx1(f1))ENGINE=InnoDB;
 
INSERT INTO t2 VALUES('mariadb');
 
connection default;
 
SET @saved_dbug = @@GLOBAL.debug_dbug;
 
SET GLOBAL debug_dbug ='+d,fts_instrument_sync_request,ib_optimize_wq_hang';
 
SET DEBUG_SYNC= 'fts_instrument_sync_request
 
		 SIGNAL drop_index_start WAIT_FOR sync_op';
 
INSERT INTO t1 VALUES('Keyword');
 
connect con1,localhost,root,,,;
 
SET DEBUG_SYNC='now WAIT_FOR drop_index_start';
 
SET DEBUG_SYNC= 'norebuild_fts_drop SIGNAL sync_op WAIT_FOR fts_drop_index';
 
ALTER TABLE t2 drop index idx1;
 
connection default;
 
set DEBUG_SYNC= 'now SIGNAL fts_drop_index';
 
connection con1;
 
 
 
More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.7.1/mysql-test/var/2/log/concurrent_insert.log
 
 
 
 
 
Server [mysqld.1 - pid: 5140, winpid: 5140, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
2021-10-27 18:43:56 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
 
211027 18:43:56 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.7.1-MariaDB-debug-log
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=2
 
max_threads=153
 
thread_count=2
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62396 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
mysys/stacktrace.c:212(my_print_stacktrace)[0x81333154]
 
sql/signal_handler.cc:224(handle_fatal_signal)[0x80aa6c58]
 
addr2line: '': No such file
 
[0xb77a1c14]
 
Printing to addr2line failed
 
[0xb77a1c31]
 
??:0(gsignal)[0xb7020e89]
 
??:0(abort)[0xb70223e7]
 
ut/ut0ut.cc:526(ib::error_or_warn::~error_or_warn())[0x81177fb8]
 
srv/srv0srv.cc:1320(srv_monitor_task)[0x8111fd4f]
 
tpool/tpool_generic.cc:313(tpool::thread_pool_generic::timer_generic::run())[0x8139b4a0]
 
tpool/tpool_generic.cc:332(tpool::thread_pool_generic::timer_generic::execute(void*))[0x8139b5ce]
 
tpool/task.cc:37(tpool::task::execute())[0x813a14d2]
 
tpool/tpool_generic.cc:549(tpool::thread_pool_generic::worker_main(tpool::worker_data*))[0x81399993]
 
5/functional:600(void std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const)[0x813a0f17]
 
5/functional:1531(void std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>))[0x813a0ea8]
 
5/functional:1520(std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()())[0x813a0d0d]
 
5/thread:115(std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run())[0x813a0c94]
 
??:0(std::this_thread::__sleep_for(std::chrono::duration<long long, std::ratio<1ll, 1ll> >, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> >))[0xb72bb5de]
 
??:0(start_thread)[0xb73a02b5]
 
??:0(clone)[0xb70dc16e]
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /dev/shm/var_auto_66ir/2/mysqld.1/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             64511                64511                processes 
Max open files            1024                 1024                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       64511                64511                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/share/apport/apport %p %s %c %P
 
 
 
----------SERVER LOG END-------------


[New LWP 5163]
 
[New LWP 5141]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
 
Core was generated by `/mnt/buildbot/build/mariadb-10.7.1/sql/mariadbd --defaults-group-suffix=.1 --de'.
 
Program terminated with signal SIGABRT, Aborted.
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
[Current thread is 1 (Thread 0xac1fbb40 (LWP 5158))]
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb73a6ce1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x8133324e in my_write_core (sig=6) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/stacktrace.c:424
 
#3  0x80aa6fc6 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/signal_handler.cc:345
 
#4  <signal handler called>
 
#5  0xb77a1c31 in __kernel_vsyscall ()
 
#6  0xb7020e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#7  0xb70223e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#8  0x81177fb8 in ib::fatal::~fatal (this=0xac1fb004, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/ut/ut0ut.cc:519
 
#9  0x8111fd4f in srv_monitor_task () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1317
 
#10 0x8139b4a0 in tpool::thread_pool_generic::timer_generic::run (this=0x835abb78) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:313
 
#11 0x8139b5ce in tpool::thread_pool_generic::timer_generic::execute (arg=0x835abb78) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:332
 
#12 0x813a14d2 in tpool::task::execute (this=0x835abba0) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:37
 
#13 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d35e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#14 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600a00, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#15 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf6009f8) at /usr/include/c++/5/functional:1531
 
#16 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf6009f8) at /usr/include/c++/5/functional:1520
 
#17 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf6009ec) at /usr/include/c++/5/thread:115
 
#18 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#19 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#20 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 16 (Thread 0xb6e62800 (LWP 5141)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d1f8f in poll () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x805917fc in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:6152
 
#3  0x80590ded in mysqld_main (argc=155, argv=0x832fbe54) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:5857
 
#4  0x8058473b in main (argc=23, argv=0xbf979b64) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/main.cc:34
 
 
 
Thread 15 (Thread 0xb06f8b40 (LWP 5163)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d1f8f in poll () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x80ca5a3b in vio_io_wait (vio=0x838afd28, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/buildbot/buildbot/build/mariadb-10.7.1/vio/viosocket.c:948
 
#3  0x80ca427b in vio_socket_io_wait (vio=0x838afd28, event=VIO_IO_EVENT_READ) at /home/buildbot/buildbot/build/mariadb-10.7.1/vio/viosocket.c:116
 
#4  0x80ca43ec in vio_read (vio=0x838afd28, buf=0x837ec1c0 "\a", size=4) at /home/buildbot/buildbot/build/mariadb-10.7.1/vio/viosocket.c:182
 
#5  0x80c57c53 in my_real_read (net=0x837e55ac, complen=0xb06f80c8, header=1 '\001') at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/net_serv.cc:1003
 
#6  0x80c5849e in my_net_read_packet_reallen (net=0x837e55ac, read_from_server=1 '\001', reallen=0xb06f813c) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/net_serv.cc:1284
 
#7  0x80c58450 in my_net_read_packet (net=0x837e55ac, read_from_server=1 '\001') at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/net_serv.cc:1268
 
#8  0x80709120 in do_command (thd=0x837e53c8, blocking=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:1264
 
#9  0x808c8f81 in do_handle_one_connection (connect=0x83925018, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1418
 
#10 0x808c8bba in handle_one_connection (arg=0x8374abb8) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1312
 
#11 0x80dabb5d in pfs_spawn_thread (arg=0x837e53c8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
 
#12 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#13 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 14 (Thread 0xac9fcb40 (LWP 5157)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x8112fa7e in srw_mutex_impl<false>::wait (this=0x81dde044 <dict_sys+68>, lk=2147483652) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:348
 
#3  0x8112eb3c in srw_mutex_impl<false>::wait_and_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:461
 
#4  0x80f22fb8 in srw_mutex_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:80
 
#5  0x8112f5f4 in ssux_lock_impl<false>::rd_wait (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:523
 
#6  0x81097f38 in ssux_lock_impl<false>::rd_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:214
 
#7  0x8112f6a7 in srw_lock_impl<false>::psi_rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:581
 
#8  0x81133c1d in srw_lock_impl<false>::rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:481
 
#9  0x8122b433 in dict_sys_t::freeze (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:997
 
#10 0x8122a772 in dict_table_open_on_id (table_id=1, dict_locked=false, table_op=DICT_TABLE_OP_NORMAL, thd=0x8365fc28, mdl=0x8359833c) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:834
 
#11 0x810ddf83 in row_purge_parse_undo_rec (node=0x83598240, undo_rec=0x8359ffe0 "", thr=0x835981e8, updated_extern=0xac9fbff2) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1028
 
#12 0x810de98b in row_purge (node=0x83598240, undo_rec=0x8359ffe0 "", thr=0x835981e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1206
 
#13 0x810deb5e in row_purge_step (thr=0x835981e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1258
 
#14 0x8104dc2f in que_thr_step (thr=0x835981e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:653
 
#15 0x8104deb2 in que_run_threads_low (thr=0x835981e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:709
 
#16 0x8104dfd3 in que_run_threads (thr=0x835981e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:729
 
#17 0x81121af6 in srv_task_execute () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1716
 
#18 0x811221d2 in purge_worker_callback () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1960
 
#19 0x813a1107 in tpool::task_group::execute (this=0x8281b640 <purge_task_group>, t=0x8281b3c0 <purge_worker_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task_group.cc:55
 
#20 0x813a14bb in tpool::task::execute (this=0x8281b3c0 <purge_worker_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:32
 
#21 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d3668) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#22 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600918, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#23 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf600910) at /usr/include/c++/5/functional:1531
 
#24 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf600910) at /usr/include/c++/5/functional:1520
 
#25 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf600904) at /usr/include/c++/5/thread:115
 
#26 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#27 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#28 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 13 (Thread 0xad9feb40 (LWP 5155)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x8112fa7e in srw_mutex_impl<false>::wait (this=0x81dde044 <dict_sys+68>, lk=2147483653) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:348
 
#3  0x8112eb3c in srw_mutex_impl<false>::wait_and_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:461
 
#4  0x80f22fb8 in srw_mutex_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:80
 
#5  0x8112f5f4 in ssux_lock_impl<false>::rd_wait (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:523
 
#6  0x81097f38 in ssux_lock_impl<false>::rd_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:214
 
#7  0x8112f6a7 in srw_lock_impl<false>::psi_rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:581
 
#8  0x81133c1d in srw_lock_impl<false>::rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:481
 
#9  0x8122b433 in dict_sys_t::freeze (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:997
 
#10 0x8122a772 in dict_table_open_on_id (table_id=4, dict_locked=false, table_op=DICT_TABLE_OP_NORMAL, thd=0x83658940, mdl=0x835924fc) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:834
 
#11 0x810ddf83 in row_purge_parse_undo_rec (node=0x83592400, undo_rec=0x835a00a8 "", thr=0x83591b48, updated_extern=0xad9fdec2) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1028
 
#12 0x810de98b in row_purge (node=0x83592400, undo_rec=0x835a00a8 "", thr=0x83591b48) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1206
 
#13 0x810deb5e in row_purge_step (thr=0x83591b48) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1258
 
#14 0x8104dc2f in que_thr_step (thr=0x83591b48) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:653
 
#15 0x8104deb2 in que_run_threads_low (thr=0x83591b48) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:709
 
#16 0x8104dfd3 in que_run_threads (thr=0x83591b48) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:729
 
#17 0x8113a1f1 in trx_purge (n_tasks=4, truncate=false) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/trx/trx0purge.cc:1299
 
#18 0x8112366a in purge_coordinator_state::do_purge (this=0x8281b560 <purge_state>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1815
 
#19 0x811222a8 in purge_coordinator_callback () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1969
 
#20 0x813a1107 in tpool::task_group::execute (this=0x8281b6c0 <purge_coordinator_task_group>, t=0x8281b740 <purge_coordinator_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task_group.cc:55
 
#21 0x813a14bb in tpool::task::execute (this=0x8281b740 <purge_coordinator_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:32
 
#22 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d3768) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#23 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600748, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#24 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf600740) at /usr/include/c++/5/functional:1531
 
#25 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf600740) at /usr/include/c++/5/functional:1520
 
#26 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf600734) at /usr/include/c++/5/thread:115
 
#27 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#28 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#29 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 12 (Thread 0xb05fdb40 (LWP 5149)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb73a5a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x81337a42 in safe_cond_wait (cond=0x81ddd2cc <buf_pool+17100>, mp=0x81ddd240 <buf_pool+16960>, file=0x81853988 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/buf/buf0flu.cc", line=2202) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/thr_mutex.c:494
 
#3  0x81206868 in buf_flush_page_cleaner () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/buf/buf0flu.cc:2202
 
#4  0x8120afc3 in std::_Bind_simple<void (*())()>::_M_invoke<>(std::_Index_tuple<>) (this=0x83567650) at /usr/include/c++/5/functional:1531
 
#5  0x8120af05 in std::_Bind_simple<void (*())()>::operator()() (this=0x83567650) at /usr/include/c++/5/functional:1520
 
#6  0x8120ae8c in std::thread::_Impl<std::_Bind_simple<void (*())()> >::_M_run() (this=0x83567644) at /usr/include/c++/5/thread:115
 
#7  0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 11 (Thread 0xb19feb40 (LWP 5148)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x813a1ad0 in my_getevents (ctx=0xb3329000, min_nr=1, nr=256, ev=0xb19fc1c0) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/aio_linux.cc:63
 
#3  0x813a1c35 in tpool::aio_linux::getevent_thread_routine (aio=0x834574c8) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/aio_linux.cc:104
 
#4  0x813a327c in std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)>::_M_invoke<0u>(std::_Index_tuple<0u>) (this=0x833b3d28) at /usr/include/c++/5/functional:1531
 
#5  0x813a3151 in std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)>::operator()() (this=0x833b3d28) at /usr/include/c++/5/functional:1520
 
#6  0x813a30d8 in std::thread::_Impl<std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)> >::_M_run() (this=0x833b3d1c) at /usr/include/c++/5/thread:115
 
#7  0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 10 (Thread 0xab1f9b40 (LWP 5160)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x8112fa7e in srw_mutex_impl<false>::wait (this=0x81dde044 <dict_sys+68>, lk=2147483650) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:348
 
#3  0x8112eb3c in srw_mutex_impl<false>::wait_and_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:461
 
#4  0x80f22fb8 in srw_mutex_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:80
 
#5  0x8112f5f4 in ssux_lock_impl<false>::rd_wait (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:523
 
#6  0x81097f38 in ssux_lock_impl<false>::rd_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:214
 
#7  0x8112f6a7 in srw_lock_impl<false>::psi_rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=619) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:581
 
#8  0x81133c1d in srw_lock_impl<false>::rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=619) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:481
 
#9  0x8122b433 in dict_sys_t::freeze (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=619) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:997
 
#10 0x81237d52 in dict_table_t::parse_name<false> (this=0x838fd310, db_name=..., tbl_name=..., db_name_len=0xab1f8acc, tbl_name_len=0xab1f8ad0) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:619
 
#11 0x81238ab8 in dict_acquire_mdl_shared<true> (table=0x838fd310, thd=0x835b51e0, mdl=0xab1f8fcc, table_op=DICT_TABLE_OP_NORMAL) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:784
 
#12 0x812be85c in fts_optimize_sync_table (table=0x838fd310, process_message=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/fts/fts0opt.cc:2785
 
#13 0x812bed5c in fts_optimize_callback () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/fts/fts0opt.cc:2894
 
#14 0x813a1107 in tpool::task_group::execute (this=0x82821920 <_ZL10task_group>, t=0x82821988 <task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task_group.cc:55
 
#15 0x813a14bb in tpool::task::execute (this=0x82821988 <task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:32
 
#16 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d34e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#17 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600bd0, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#18 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf600bc8) at /usr/include/c++/5/functional:1531
 
#19 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf600bc8) at /usr/include/c++/5/functional:1520
 
#20 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf600bbc) at /usr/include/c++/5/thread:115
 
#21 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#22 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#23 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 9 (Thread 0xb0742b40 (LWP 5162)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb73a5a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x81337a42 in safe_cond_wait (cond=0x81f88020 <COND_manager>, mp=0x81f87fa0 <LOCK_manager>, file=0x818bc8e4 "/home/buildbot/buildbot/build/mariadb-10.7.1/mysys/my_thr_init.c", line=596) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/thr_mutex.c:494
 
#3  0x813318e9 in psi_cond_wait (that=0x81f88020 <COND_manager>, mutex=0x81f87fa0 <LOCK_manager>, file=0x81443be4 "/home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_manager.cc", line=103) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/my_thr_init.c:596
 
#4  0x80704154 in inline_mysql_cond_wait (that=0x81f88020 <COND_manager>, mutex=0x81f87fa0 <LOCK_manager>, src_file=0x81443be4 "/home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_manager.cc", src_line=103) at /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1070
 
#5  0x80704586 in handle_manager (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_manager.cc:103
 
#6  0x80dabb5d in pfs_spawn_thread (arg=0x836ebf68) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
 
#7  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#8  0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 8 (Thread 0xb078cb40 (LWP 5161)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb7021e8c in sigwaitinfo () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x805847f4 in my_sigwait (set=0xb078c190, sig=0xb078c160, code=0xb078c164) at /home/buildbot/buildbot/build/mariadb-10.7.1/include/my_pthread.h:195
 
#3  0x8058aaa1 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/mysqld.cc:3144
 
#4  0x80dabb5d in pfs_spawn_thread (arg=0x836f08e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
 
#5  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#6  0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 7 (Thread 0xafdfcb40 (LWP 5216)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x8112fb3f in ssux_lock_impl<false>::wait (this=0x81dde044 <dict_sys+68>, lk=2147483649) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:354
 
#3  0x8112f548 in ssux_lock_impl<false>::wr_wait (this=0x81dde044 <dict_sys+68>, lk=2147483649) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:509
 
#4  0x80f1f74f in ssux_lock_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:232
 
#5  0x8112f76d in srw_lock_impl<false>::psi_wr_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=4272) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:598
 
#6  0x810014f9 in srw_lock_impl<false>::wr_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=4272) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:494
 
#7  0x8122adfa in dict_sys_t::lock_wait (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=4272) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:961
 
#8  0x80f19c25 in dict_sys_t::lock (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=4272) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/dict0dict.h:1568
 
#9  0x81235385 in dict_index_set_merge_threshold (index=0x838a2ba8, merge_threshold=50) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:4272
 
#10 0x80f01e21 in innobase_parse_hint_from_comment (thd=0xae3007f8, table=0x83896db0, table_share=0xafdf9d38) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/ha_innodb.cc:11816
 
#11 0x80f4f263 in ha_innobase::commit_inplace_alter_table (this=0x839bd8d8, altered_table=0xafdf9278, ha_alter_info=0xafdf9210, commit=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/handler/handler0alter.cc:11294
 
#12 0x80ab819e in handler::ha_commit_inplace_alter_table (this=0x839bd8d8, altered_table=0xafdf9278, ha_alter_info=0xafdf9210, commit=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/handler.cc:5191
 
#13 0x8081d425 in mysql_inplace_alter_table (thd=0xae3007f8, table_list=0xae310008, table=0x8389f020, altered_table=0xafdf9278, ha_alter_info=0xafdf9210, target_mdl_request=0xafdf9790, ddl_log_state=0xafdf91dc, trigger_param=0xafdf94dc, alter_ctx=0xafdfa0a4) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:7535
 
#14 0x80825c3b in mysql_alter_table (thd=0xae3007f8, new_db=0xae3038d4, new_name=0xae303b2c, create_info=0xafdfaddc, table_list=0xae310008, alter_info=0xafdfad60, order_num=0, order=0x0, ignore=false, if_exists=false) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_table.cc:10302
 
#15 0x808d40b2 in Sql_cmd_alter_table::execute (this=0xae3104f0, thd=0xae3007f8) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_alter.cc:559
 
#16 0x80719b6e in mysql_execute_command (thd=0xae3007f8, is_called_from_prepared_stmt=false) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:5989
 
#17 0x8071fc38 in mysql_parse (thd=0xae3007f8, rawbuf=0xae30ff50 "ALTER TABLE t2 drop index idx1", length=30, parser_state=0xafdfb9b0) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:8028
 
#18 0x8070b0eb in dispatch_command (command=COM_QUERY, thd=0xae3007f8, packet=0xae307911 "ALTER TABLE t2 drop index idx1", packet_length=30, blocking=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:1894
 
#19 0x807097e5 in do_command (thd=0xae3007f8, blocking=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_parse.cc:1403
 
#20 0x808c8f81 in do_handle_one_connection (connect=0x839c16c8, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1418
 
#21 0x808c8bba in handle_one_connection (arg=0x838dc410) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/sql_connect.cc:1312
 
#22 0x80dabb5d in pfs_spawn_thread (arg=0x838b9858) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
 
#23 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#24 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 6 (Thread 0xb2767b40 (LWP 5145)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x81337ced in safe_cond_timedwait (cond=0x827ac5c0 <COND_checkpoint>, mp=0x827ac540 <LOCK_checkpoint>, abstime=0xb276715c, file=0x818bc8e4 "/home/buildbot/buildbot/build/mariadb-10.7.1/mysys/my_thr_init.c", line=609) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/thr_mutex.c:548
 
#3  0x8133197d in psi_cond_timedwait (that=0x827ac5c0 <COND_checkpoint>, mutex=0x827ac540 <LOCK_checkpoint>, abstime=0xb276715c, file=0x816af088 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/maria/ma_servicethread.c", line=116) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/my_thr_init.c:609
 
#4  0x80d025b6 in inline_mysql_cond_timedwait (that=0x827ac5c0 <COND_checkpoint>, mutex=0x827ac540 <LOCK_checkpoint>, abstime=0xb276715c, src_file=0x816af088 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1086
 
#5  0x80d02ae7 in my_service_thread_sleep (control=0x81dc79b8 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/maria/ma_servicethread.c:115
 
#6  0x80cf618b in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/maria/ma_checkpoint.c:725
 
#7  0x80dabb5d in pfs_spawn_thread (arg=0x834b01a0) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
 
#8  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#9  0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 5 (Thread 0xae1ffb40 (LWP 5154)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x8112fa7e in srw_mutex_impl<false>::wait (this=0x81dde044 <dict_sys+68>, lk=2147483655) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:348
 
#3  0x8112eb3c in srw_mutex_impl<false>::wait_and_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:461
 
#4  0x80f22fb8 in srw_mutex_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:80
 
#5  0x8112f5f4 in ssux_lock_impl<false>::rd_wait (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:523
 
#6  0x81097f38 in ssux_lock_impl<false>::rd_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:214
 
#7  0x8112f6a7 in srw_lock_impl<false>::psi_rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:581
 
#8  0x81133c1d in srw_lock_impl<false>::rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:481
 
#9  0x8122b433 in dict_sys_t::freeze (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:997
 
#10 0x8122a772 in dict_table_open_on_id (table_id=3, dict_locked=false, table_op=DICT_TABLE_OP_NORMAL, thd=0x8366e1f8, mdl=0x83591b0c) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:834
 
#11 0x810ddf83 in row_purge_parse_undo_rec (node=0x83591a10, undo_rec=0x835a0020 "", thr=0x835919b8, updated_extern=0xae1feff2) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1028
 
#12 0x810de98b in row_purge (node=0x83591a10, undo_rec=0x835a0020 "", thr=0x835919b8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1206
 
#13 0x810deb5e in row_purge_step (thr=0x835919b8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1258
 
#14 0x8104dc2f in que_thr_step (thr=0x835919b8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:653
 
#15 0x8104deb2 in que_run_threads_low (thr=0x835919b8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:709
 
#16 0x8104dfd3 in que_run_threads (thr=0x835919b8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:729
 
#17 0x81121af6 in srv_task_execute () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1716
 
#18 0x811221d2 in purge_worker_callback () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1960
 
#19 0x813a1107 in tpool::task_group::execute (this=0x8281b640 <purge_task_group>, t=0x8281b3c0 <purge_worker_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task_group.cc:55
 
#20 0x813a14bb in tpool::task::execute (this=0x8281b3c0 <purge_worker_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:32
 
#21 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d37e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#22 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600660, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#23 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf600658) at /usr/include/c++/5/functional:1531
 
#24 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf600658) at /usr/include/c++/5/functional:1520
 
#25 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf60064c) at /usr/include/c++/5/thread:115
 
#26 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#27 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#28 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 4 (Thread 0xaf5ffb40 (LWP 5152)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x8112fa7e in srw_mutex_impl<false>::wait (this=0x81dde044 <dict_sys+68>, lk=2147483654) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:348
 
#3  0x8112eb3c in srw_mutex_impl<false>::wait_and_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:461
 
#4  0x80f22fb8 in srw_mutex_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:80
 
#5  0x8112f5f4 in ssux_lock_impl<false>::rd_wait (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:523
 
#6  0x81097f38 in ssux_lock_impl<false>::rd_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:214
 
#7  0x8112f6a7 in srw_lock_impl<false>::psi_rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:581
 
#8  0x81133c1d in srw_lock_impl<false>::rd_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:481
 
#9  0x8122b433 in dict_sys_t::freeze (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=834) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:997
 
#10 0x8122a772 in dict_table_open_on_id (table_id=2, dict_locked=false, table_op=DICT_TABLE_OP_NORMAL, thd=0x83666f10, mdl=0x8359197c) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:834
 
#11 0x810ddf83 in row_purge_parse_undo_rec (node=0x83591880, undo_rec=0x8359fff0 "", thr=0x83591828, updated_extern=0xaf5feff2) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1028
 
#12 0x810de98b in row_purge (node=0x83591880, undo_rec=0x8359fff0 "", thr=0x83591828) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1206
 
#13 0x810deb5e in row_purge_step (thr=0x83591828) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/row/row0purge.cc:1258
 
#14 0x8104dc2f in que_thr_step (thr=0x83591828) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:653
 
#15 0x8104deb2 in que_run_threads_low (thr=0x83591828) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:709
 
#16 0x8104dfd3 in que_run_threads (thr=0x83591828) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/que/que0que.cc:729
 
#17 0x81121af6 in srv_task_execute () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1716
 
#18 0x811221d2 in purge_worker_callback () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1960
 
#19 0x813a1107 in tpool::task_group::execute (this=0x8281b640 <purge_task_group>, t=0x8281b3c0 <purge_worker_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task_group.cc:55
 
#20 0x813a14bb in tpool::task::execute (this=0x8281b3c0 <purge_worker_task>) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:32
 
#21 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d38e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#22 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600490, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#23 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf600488) at /usr/include/c++/5/functional:1531
 
#24 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf600488) at /usr/include/c++/5/functional:1520
 
#25 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf60047c) at /usr/include/c++/5/thread:115
 
#26 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#27 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#28 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 3 (Thread 0xb33c5b40 (LWP 5144)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb73a5e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x81337ced in safe_cond_timedwait (cond=0x82828900 <COND_timer>, mp=0x82828880 <LOCK_timer>, abstime=0xb33c5210, file=0x818bc8e4 "/home/buildbot/buildbot/build/mariadb-10.7.1/mysys/my_thr_init.c", line=609) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/thr_mutex.c:548
 
#3  0x8133197d in psi_cond_timedwait (that=0x82828900 <COND_timer>, mutex=0x82828880 <LOCK_timer>, abstime=0xb33c5210, file=0x818bd9ec "/home/buildbot/buildbot/build/mariadb-10.7.1/mysys/thr_timer.c", line=321) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/my_thr_init.c:609
 
#4  0x81338c0a in inline_mysql_cond_timedwait (that=0x82828900 <COND_timer>, mutex=0x82828880 <LOCK_timer>, abstime=0xb33c5210, src_file=0x818bd9ec "/home/buildbot/buildbot/build/mariadb-10.7.1/mysys/thr_timer.c", src_line=321) at /home/buildbot/buildbot/build/mariadb-10.7.1/include/mysql/psi/mysql_thread.h:1086
 
#5  0x813398f4 in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/thr_timer.c:321
 
#6  0x80dabb5d in pfs_spawn_thread (arg=0x834253b8) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/perfschema/pfs.cc:2201
 
#7  0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#8  0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 2 (Thread 0xad1fdb40 (LWP 5156)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb70d7bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
 
#2  0x8112fa7e in srw_mutex_impl<false>::wait (this=0x81dde044 <dict_sys+68>, lk=2147483651) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:348
 
#3  0x8112eb3c in srw_mutex_impl<false>::wait_and_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:461
 
#4  0x80f22fb8 in srw_mutex_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:80
 
#5  0x80f1f714 in ssux_lock_impl<false>::wr_lock (this=0x81dde044 <dict_sys+68>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:224
 
#6  0x8112f76d in srw_lock_impl<false>::psi_wr_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=1310) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/sync/srw_lock.cc:598
 
#7  0x810014f9 in srw_lock_impl<false>::wr_lock (this=0x81dde040 <dict_sys+64>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=1310) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/srw_lock.h:494
 
#8  0x8122b178 in dict_sys_t::lock_wait (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=1310) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:980
 
#9  0x80f19c25 in dict_sys_t::lock (this=0x81dde000 <dict_sys>, file=0x818651e0 "/home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc", line=1310) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/include/dict0dict.h:1568
 
#10 0x8122c7bd in dict_sys_t::evict_table_LRU (this=0x81dde000 <dict_sys>, half=false) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/dict/dict0dict.cc:1310
 
#11 0x811211a9 in srv_master_do_idle_tasks (counter_time=4644927963) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1605
 
#12 0x8112171a in srv_master_callback () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1663
 
#13 0x8139b4a0 in tpool::thread_pool_generic::timer_generic::run (this=0x835afc68) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:313
 
#14 0x8139b5ce in tpool::thread_pool_generic::timer_generic::execute (arg=0x835afc68) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:332
 
#15 0x813a14d2 in tpool::task::execute (this=0x835afc90) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:37
 
#16 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d36e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#17 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600830, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#18 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf600828) at /usr/include/c++/5/functional:1531
 
#19 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf600828) at /usr/include/c++/5/functional:1520
 
#20 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf60081c) at /usr/include/c++/5/thread:115
 
#21 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#22 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#23 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
 
 
Thread 1 (Thread 0xac1fbb40 (LWP 5158)):
 
#0  0xb77a1c31 in __kernel_vsyscall ()
 
#1  0xb73a6ce1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0
 
#2  0x8133324e in my_write_core (sig=6) at /home/buildbot/buildbot/build/mariadb-10.7.1/mysys/stacktrace.c:424
 
#3  0x80aa6fc6 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.7.1/sql/signal_handler.cc:345
 
#4  <signal handler called>
 
#5  0xb77a1c31 in __kernel_vsyscall ()
 
#6  0xb7020e89 in raise () from /lib/i386-linux-gnu/libc.so.6
 
#7  0xb70223e7 in abort () from /lib/i386-linux-gnu/libc.so.6
 
#8  0x81177fb8 in ib::fatal::~fatal (this=0xac1fb004, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/ut/ut0ut.cc:519
 
#9  0x8111fd4f in srv_monitor_task () at /home/buildbot/buildbot/build/mariadb-10.7.1/storage/innobase/srv/srv0srv.cc:1317
 
#10 0x8139b4a0 in tpool::thread_pool_generic::timer_generic::run (this=0x835abb78) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:313
 
#11 0x8139b5ce in tpool::thread_pool_generic::timer_generic::execute (arg=0x835abb78) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:332
 
#12 0x813a14d2 in tpool::task::execute (this=0x835abba0) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/task.cc:37
 
#13 0x81399993 in tpool::thread_pool_generic::worker_main (this=0x833f93d8, thread_var=0x834d35e8) at /home/buildbot/buildbot/build/mariadb-10.7.1/tpool/tpool_generic.cc:549
 
#14 0x813a0f17 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaf600a00, __object=0x833f93d8) at /usr/include/c++/5/functional:600
 
#15 0x813a0ea8 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaf6009f8) at /usr/include/c++/5/functional:1531
 
#16 0x813a0d0d in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaf6009f8) at /usr/include/c++/5/functional:1520
 
#17 0x813a0c94 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaf6009ec) at /usr/include/c++/5/thread:115
 
#18 0xb72bb5de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 
#19 0xb73a02b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
 
#20 0xb70dc16e in clone () from /lib/i386-linux-gnu/libc.so.6

10.6 https://buildbot.askmonty.org/buildbot/builders/kvm-asan/builds/6875/steps/mtr_nm/logs/stdio

 Comments   
Comment by Marko Mäkelä [ 2021-11-11 ]

I tried and failed to reproduce this locally by running 20×100+35×100 rounds of the test.

Thread 2 (srv_master_callback) is waiting for an exclusive dict_sys.latch in dict_sys_t::evict_table_LRU().

Thread 7 is waiting for an exclusive dict_sys.latch in dict_index_set_merge_threshold(), as part of ALTER TABLE t2 drop index idx1. I do not think that we actually need dict_sys.latch in that function.

The waiting exclusive lock requests are supposed to block all the dict_sys:freeze() in Thread 4, 5, 10, 13, 14.

I was unable to identify the holder of dict_sys.latch in the stack traces. A core dump would help, but they are thrown away by the CI system.

In any case, dict_index_set_merge_threshold() needs to be rewritten to be part of the DDL transaction. It was missed in MDEV-25919.

Comment by Marko Mäkelä [ 2022-01-05 ]

I wonder if MDEV-27017 fixes this.

Comment by Marko Mäkelä [ 2022-01-24 ]

In bt, alice posted stack traces of all threads of some 10.6 and 10.7 revisions, presumably for the period when the server is hung but the watchdog did not kick in. Unfortunately, we cannot see the table names in the stack traces. For example this thread could be trying to open an FTS_ table:

10.6

 
#8  0x0000560411aee95b in dict_sys_t::freeze (this=0x560414156bc0 <dict_sys>, file=0x560412f6db60 "/10.6/src/storage/innobase/dict/dict0dict.cc", line=834) at /10.6/src/storage/innobase/dict/dict0dict.cc:997
 
#9  0x0000560411aed4ce in dict_table_open_on_id (table_id=1734, dict_locked=false, table_op=DICT_TABLE_OP_NORMAL, thd=0x62b00002a288, mdl=0x61b000002cc0) at /10.6/src/storage/innobase/dict/dict0dict.cc:834
 
#10 0x000056041186c831 in row_purge_parse_undo_rec (node=0x61b000002b20, undo_rec=0x621000044da0 "", thr=0x6170000055d8, updated_extern=0x7ff1c1c15820) at /10.6/src/storage/innobase/row/row0purge.cc:1028

But we could only know that by manually looking up the table from the hash table in the debugger.

alice, are you able to reproduce this after the MDEV-27017 fix was pushed? I’d need your help, because I was never able to reproduce this hang locally.

Comment by Marko Mäkelä [ 2022-01-27 ]

This looks like something potentially caused by MDEV-24167. In a 10.8 core dump (the file names say 10.7) that alice provided to me, I see that two threads are waiting for an exclusive dict_sys.latch:

10.8 db2013787d2f6cb33784834bbb31d1605b4e0af6

 
Thread 12 (Thread 0x7f2b12a02700 (LWP 1730227)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x0000563cd5fb2b99 in srw_mutex_impl<false>::wait (this=0x563cd885bf88 <dict_sys+72>, lk=2147483651) at /home/alice/git/10.7/storage/innobase/sync/srw_lock.cc:348
 
#2  0x0000563cd5fb0a4a in srw_mutex_impl<false>::wait_and_lock (this=0x563cd885bf88 <dict_sys+72>) at /home/alice/git/10.7/storage/innobase/sync/srw_lock.cc:461
 
#3  0x0000563cd5b75405 in srw_mutex_impl<false>::wr_lock (this=0x563cd885bf88 <dict_sys+72>) at /home/alice/git/10.7/storage/innobase/include/srw_lock.h:80
 
#4  0x0000563cd5b6b74e in ssux_lock_impl<false>::wr_lock (this=0x563cd885bf88 <dict_sys+72>) at /home/alice/git/10.7/storage/innobase/include/srw_lock.h:226
 
#5  0x0000563cd5fb2379 in srw_lock_impl<false>::psi_wr_lock (this=0x563cd885bf80 <dict_sys+64>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=1310) at /home/alice/git/10.7/storage/innobase/sync/srw_lock.cc:598
 
#6  0x0000563cd5d5e98d in srw_lock_impl<false>::wr_lock (this=0x563cd885bf80 <dict_sys+64>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=1310) at /home/alice/git/10.7/storage/innobase/include/srw_lock.h:496
 
#7  0x0000563cd6195d0a in dict_sys_t::lock_wait (this=0x563cd885bf40 <dict_sys>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=1310) at /home/alice/git/10.7/storage/innobase/dict/dict0dict.cc:980
 
#8  0x0000563cd5b5ea40 in dict_sys_t::lock (this=0x563cd885bf40 <dict_sys>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=1310) at /home/alice/git/10.7/storage/innobase/include/dict0dict.h:1568
 
#9  0x0000563cd61988c1 in dict_sys_t::evict_table_LRU (this=0x563cd885bf40 <dict_sys>, half=false) at /home/alice/git/10.7/storage/innobase/dict/dict0dict.cc:1310
 
#10 0x0000563cd5f9635f in srv_master_do_idle_tasks (counter_time=475141818237) at /home/alice/git/10.7/storage/innobase/srv/srv0srv.cc:1528
 
#11 0x0000563cd5f96a41 in srv_master_callback () at /home/alice/git/10.7/storage/innobase/srv/srv0srv.cc:1585
 
…
 
Thread 4 (Thread 0x7f2b1861a700 (LWP 1729225)):
 
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
 
#1  0x0000563cd5fb2ca8 in ssux_lock_impl<false>::wait (this=0x563cd885bf88 <dict_sys+72>, lk=2147483649) at /home/alice/git/10.7/storage/innobase/sync/srw_lock.cc:354
 
#2  0x0000563cd5fb1dfd in ssux_lock_impl<false>::wr_wait (this=0x563cd885bf88 <dict_sys+72>, lk=2147483649) at /home/alice/git/10.7/storage/innobase/sync/srw_lock.cc:509
 
#3  0x0000563cd5b6b7c5 in ssux_lock_impl<false>::wr_lock (this=0x563cd885bf88 <dict_sys+72>) at /home/alice/git/10.7/storage/innobase/include/srw_lock.h:234
 
#4  0x0000563cd5fb2379 in srw_lock_impl<false>::psi_wr_lock (this=0x563cd885bf80 <dict_sys+64>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=4272) at /home/alice/git/10.7/storage/innobase/sync/srw_lock.cc:598
 
#5  0x0000563cd5d5e98d in srw_lock_impl<false>::wr_lock (this=0x563cd885bf80 <dict_sys+64>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=4272) at /home/alice/git/10.7/storage/innobase/include/srw_lock.h:496
 
#6  0x0000563cd6195746 in dict_sys_t::lock_wait (this=0x563cd885bf40 <dict_sys>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=4272) at /home/alice/git/10.7/storage/innobase/dict/dict0dict.cc:961
 
#7  0x0000563cd5b5ea40 in dict_sys_t::lock (this=0x563cd885bf40 <dict_sys>, file=0x563cd7638a80 "/home/alice/git/10.7/storage/innobase/dict/dict0dict.cc", line=4272) at /home/alice/git/10.7/storage/innobase/include/dict0dict.h:1568
 
#8  0x0000563cd61acaaf in dict_index_set_merge_threshold (index=0x616008d3f720, merge_threshold=50) at /home/alice/git/10.7/storage/innobase/dict/dict0dict.cc:4272
 
#9  0x0000563cd5b25b16 in innobase_parse_hint_from_comment (thd=0x62b00017a288, table=0x6180001c2120, table_share=0x7f2b18615910) at /home/alice/git/10.7/storage/innobase/handler/ha_innodb.cc:11773
 
#10 0x0000563cd5bdc270 in ha_innobase::commit_inplace_alter_table (this=0x61d000159b28, altered_table=0x7f2b186150d0, ha_alter_info=0x7f2b186149d0, commit=true) at /home/alice/git/10.7/storage/innobase/handler/handler0alter.cc:11324

Other threads are blocked, waiting for shared dict_sys.latch, because at least one exclusive lock wait is pending. In dict_sys, I see the following:

latch = {pfs_psi = 0x7f2b206d5480, lock = {writer = {
 
        lock = {<std::__atomic_base<unsigned int>> = {_M_i = 0x80000006}, <No data fields>}}, readers = {<std::__atomic_base<unsigned int>> = {_M_i = 0x80000001}, <No data fields>}}}, latch_ex = 0x0, 
  latch_readers = {m_counter = {<std::__atomic_base<unsigned int>> = {_M_i = 0x1}, <No data fields>}}

That is, there is no latch_ex holder, but there apparently is one shared latch holder (latch_readers) that is blocking all other threads. I reviewed the logic around those debug fields, and it looks correct to me. All other waits inside InnoDB are either the idle buf_flush_page_cleaner() thread or waits for shared dict_sys.latch (dict_sys_t::freeze()).

I think that we have two possibilities:

  1. There is some race condition in the 10.6 lock implementation that occurs when multiple threads are waiting for an exclusive latch at the same time.
  2. A code path forgets to release the shared lock_sys.latch, as suggested by the latch_readers=1.

I would suggest the following patch to disable the function dict_index_set_merge_threshold(). If that stops the hang from occurring, we could have a bug in the lock implementation. Else, we forget to release a shared dict_sys.latch somewhere.

diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc
 
index 34080b22095..22897210dd8 100644
 
--- a/storage/innobase/dict/dict0dict.cc
 
+++ b/storage/innobase/dict/dict0dict.cc
 
@@ -4241,6 +4241,7 @@ dict_index_set_merge_threshold(
 
 	dict_index_t*	index,
 
 	ulint		merge_threshold)
 
 {
 
+#if 0
 
 	mem_heap_t*	heap;
 
 	mtr_t		mtr;
 
 	dict_index_t*	sys_index;
 
@@ -4298,6 +4299,7 @@ dict_index_set_merge_threshold(
 
 	mem_heap_free(heap);
 
 
 	dict_sys.unlock();
 
+#endif
 
 }
 
 
 #ifdef UNIV_DEBUG

Comment by Alice Sherepa [ 2022-01-27 ]

After disabling this function, I ran the tests with --repeat=1000 and there were no hangs

Comment by Marko Mäkelä [ 2022-01-27 ]

I think that the exclusive dict_sys.latch in dict_index_set_merge_threshold() is unnecessary in any case. The dict_sys.sys_indexes cannot change after server startup. It is pointing to the hard-coded SYS_INDEXES table.
alice, because the hang went away for you with my previous patch, I think that it should also go away with this one?

diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc
 
index 34080b22095..6afa87cd528 100644
 
--- a/storage/innobase/dict/dict0dict.cc
 
+++ b/storage/innobase/dict/dict0dict.cc
 
@@ -4259,8 +4259,6 @@ dict_index_set_merge_threshold(
 
 
 	mtr.start();
 
 
-	dict_sys.lock(SRW_LOCK_CALL);
 
-
 
 	sys_index = UT_LIST_GET_FIRST(dict_sys.sys_indexes->indexes);
 
 
 	/* Find the index row in SYS_INDEXES */
 
@@ -4296,8 +4294,6 @@ dict_index_set_merge_threshold(
 
 
 	mtr_commit(&mtr);
 
 	mem_heap_free(heap);
 
-
 
-	dict_sys.unlock();
 
 }
 
 
 #ifdef UNIV_DEBUG

The actual SYS_INDEXES leaf page will be protected by a buf_page_t::lock just fine.

Still, I am worried that there could potentially be a bug in my srw_lock implementation (or theoretically, in the processor or compiler that you used), so this will need further investigation.

Comment by Rick Pizzi [ 2022-05-02 ]

reopening because it doesn't look to be fixed

Comment by Marko Mäkelä [ 2022-05-03 ]

The hang was not really fixed by removing unnecessary dict_sys.latch acquisition from dict_index_set_merge_threshold() was finally analyzed and fixed in MDEV-28289. A participant of the InnoDB internal deadlock would be fts_optimize_sync_table().

Comment by Marko Mäkelä [ 2022-05-03 ]

rpizzi, please determine the stack traces of all threads. If your experienced hang is not MDEV-28289, then you could update MDEV-27983 instead of reopening an already closed bug.

Generated at Thu Feb 08 09:49:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.