[MDEV-25450] rpl.rpl_parallel_gco_wait_kill failed in bb with timeout Created: 2021-04-19  Updated: 2023-12-11  Resolved: 2023-12-11

Status: Closed
Project: MariaDB Server
Component/s: Replication, Tests
Affects Version/s: 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 11.0, 11.1, 11.2
Fix Version/s: 10.5.24, 10.6.17, 10.11.7, 11.0.5, 11.1.4, 11.2.3

Type: Bug Priority: Critical
Reporter: Alice Sherepa Assignee: Brandon Nesterenko
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-10653 SHOW SLAVE STATUS Can Deadlock an Err... Closed

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/28134/steps/mtr_ps/logs/stdio

rpl.rpl_parallel_gco_wait_kill 'innodb,stmt' w2 [ fail ]  timeout after 900 seconds
        Test ended at 2021-04-16 08:28:11
 
Test case timeout after 900 seconds
 
== /mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/2/log/rpl_parallel_gco_wait_kill.log == 
65	65
66	66
67	67
68	68
69	69
70	70
SET debug_sync='RESET';
connection server_2;
SET debug_sync='now SIGNAL d0_cont';
SET debug_sync='now WAIT_FOR t1_waiting';
SET debug_sync='now SIGNAL d3_cont';
SET debug_sync='now WAIT_FOR t2_waiting';
SET debug_sync='now SIGNAL d1_cont';
SET debug_sync='now WAIT_FOR t3_waiting';
SET debug_sync='now SIGNAL d2_cont';
SET debug_sync='now WAIT_FOR t4_waiting';
KILL THD_ID;
SET debug_sync='now WAIT_FOR t3_killed';
SET debug_sync='now SIGNAL t1_cont';
include/wait_for_slave_sql_error.inc [errno=1317,1927,1964]
 
 == /mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/2/tmp/analyze-timeout-mysqld.1.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/tmp/2/mysqld.' (111)
 
 == /mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/2/tmp/analyze-timeout-mysqld.2.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/tmp/2/mysqld.' (111)
 
 
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/2/log/rpl.rpl_parallel_gco_wait_kill-innodb,stmt/mysqld.2/data/core
Core generated by '/mnt/buildbot/build/mariadb-10.5.10/sql/mariadbd'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 32529]
[New LWP 32555]
[New LWP 32569]
[New LWP 32551]
[New LWP 32560]
[New LWP 32552]
[New LWP 32592]
[New LWP 32554]
[New LWP 732]
[New LWP 32579]
[New LWP 733]
[New LWP 32564]
[New LWP 734]
[New LWP 735]
[New LWP 730]
[New LWP 729]
[New LWP 32676]
[New LWP 32675]
[New LWP 32575]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/mnt/buildbot/build/mariadb-10.5.10/sql/mariadbd --defaults-group-suffix=.2 --d'.
Program terminated with signal SIGABRT, Aborted.
#0  0xb7738c31 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xb6ddaa40 (LWP 32529))]
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731ece1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812a8b2c in my_write_core (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/stacktrace.c:424
#3  0x809c1173 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/signal_handler.cc:343
#4  <signal handler called>
#5  0xb7738c31 in __kernel_vsyscall ()
#6  0xb7049f8f in poll () from /lib/i386-linux-gnu/libc.so.6
#7  0x8052cff7 in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:6204
#8  0x8052c4f5 in mysqld_main (argc=176, argv=0x842882ac) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:5653
#9  0x805209eb in main (argc=26, argv=0xbff49214) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/main.cc:25
 
Thread 19 (Thread 0xaf95fb40 (LWP 32575)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731da8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ad785 in safe_cond_wait (cond=0x81ee1660 <COND_manager>, mp=0x81ee15e0 <LOCK_manager>, file=0x813abcc0 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x80660b14 in inline_mysql_cond_wait (that=0x81ee1660 <COND_manager>, mutex=0x81ee15e0 <LOCK_manager>, src_file=0x813ac2f4 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_manager.cc", src_line=103) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1222
#4  0x80660fc7 in handle_manager (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_manager.cc:103
#5  0x80cfdd4a in pfs_spawn_thread (arg=0x846c37a0) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#6  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#7  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 18 (Thread 0xa97ccb40 (LWP 32675)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731de56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ada30 in safe_cond_timedwait (cond=0x81edb100 <thread_cache>, mp=0x81edb168 <thread_cache+104>, abstime=0xa97cc17c, file=0x813faf88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807fc2ca in inline_mysql_cond_timedwait (that=0x81edb100 <thread_cache>, mutex=0x81edb168 <thread_cache+104>, abstime=0xa97cc17c, src_file=0x813fb61c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x80800acd in Thread_cache::park (this=0x81edb100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x80800156 in do_handle_one_connection (connect=0x845c9450, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807ffdde in handle_one_connection (arg=0x8470a378) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cfdd4a in pfs_spawn_thread (arg=0x845cb868) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 17 (Thread 0xa9782b40 (LWP 32676)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731de56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ada30 in safe_cond_timedwait (cond=0x81edb100 <thread_cache>, mp=0x81edb168 <thread_cache+104>, abstime=0xa978217c, file=0x813faf88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807fc2ca in inline_mysql_cond_timedwait (that=0x81edb100 <thread_cache>, mutex=0x81edb168 <thread_cache+104>, abstime=0xa978217c, src_file=0x813fb61c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x80800acd in Thread_cache::park (this=0x81edb100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x80800156 in do_handle_one_connection (connect=0x845c9450, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807ffdde in handle_one_connection (arg=0x8470a378) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cfdd4a in pfs_spawn_thread (arg=0x845cb868) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 16 (Thread 0xaa0fcb40 (LWP 729)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb7320d32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0
#2  0xb731a9e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0
#3  0x812acfbe in safe_mutex_lock (mp=0x84720dd4, my_flags=0, file=0x8137a2e4 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc", line=6234) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:290
#4  0x80554538 in inline_mysql_mutex_lock (that=0x84720dd4, src_file=0x8137a2e4 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc", src_line=6234) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:750
#5  0x80567eb4 in queue_event (mi=0x8471f5d0, buf=0xb19553a1 "", event_len=40) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:6234
#6  0x80563acf in handle_slave_io (arg=0x8471f5d0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:4882
#7  0x80cfdd4a in pfs_spawn_thread (arg=0xabb371c0) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 15 (Thread 0xa9365b40 (LWP 730)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731da8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ad785 in safe_cond_wait (cond=0x84721a74, mp=0x847214c4, file=0x815bd48c "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x80aed86c in inline_mysql_cond_wait (that=0x84721a74, mutex=0x847214c4, src_file=0x815be62c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc", src_line=8716) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1222
#4  0x80b0780a in MYSQL_BIN_LOG::wait_for_update_relay_log (this=0x847214bc, thd=0xb1b019f0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc:8716
#5  0x8056bc7f in next_event (rgi=0xb1b1dc28, event_size=0xa9364b20) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:7660
#6  0x805617b0 in exec_relay_log_event (thd=0xb1b019f0, rli=0x84721040, serial_rgi=0xb1b1dc28) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:4158
#7  0x80565a7d in handle_slave_sql (arg=0x8471f5d0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:5508
#8  0x80cfdd4a in pfs_spawn_thread (arg=0xabb371c0) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#9  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#10 0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 14 (Thread 0xa9521b40 (LWP 735)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731da8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ad785 in safe_cond_wait (cond=0xb1b218f8, mp=0xb1b22f18, file=0x8142dc40 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x808865f0 in inline_mysql_cond_wait (that=0xb1b218f8, mutex=0xb1b22f18, src_file=0x8142e588 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc", src_line=382) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1222
#4  0x80887dd5 in do_gco_wait (rgi=0xb1b221c0, gco=0xb1b218f8, did_enter_cond=0xa952119f, old_stage=0xa95211f4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:382
#5  0x8088a124 in handle_rpl_parallel_thread (arg=0xb1b48804) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:1239
#6  0x80cfdd4a in pfs_spawn_thread (arg=0xb1b47e90) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#7  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#8  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 13 (Thread 0xa956bb40 (LWP 734)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb7320d32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0
#2  0xb731a9e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0
#3  0x812acfbe in safe_mutex_lock (mp=0x84721044, my_flags=0, file=0x815d0e0c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_reporting.cc", line=43) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:290
#4  0x80b430f2 in inline_mysql_mutex_lock (that=0x84721044, src_file=0x815d0e0c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_reporting.cc", src_line=43) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:750
#5  0x80b4327f in Slave_reporting_capability::report (this=0x84721040, level=ERROR_LEVEL, err_code=1927, extra_info=0xb1b1824d "Gtid 0-1-6", msg=0x8137c4b1 "%s") at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_reporting.cc:43
#6  0x80564518 in slave_output_error_info (rgi=0xb1b17c18, thd=0xb0690110) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:5116
#7  0x80887da6 in do_gco_wait (rgi=0xb1b17c18, gco=0xb1b218f8, did_enter_cond=0xa956b19f, old_stage=0xa956b1f4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:371
#8  0x8088a124 in handle_rpl_parallel_thread (arg=0xb1b486a0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:1239
#9  0x80cfdd4a in pfs_spawn_thread (arg=0xb1b47880) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#10 0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#11 0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 12 (Thread 0xaf9f3b40 (LWP 32564)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731da8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ad785 in safe_cond_wait (cond=0x81f06ac4 <mysql_bin_log+3332>, mp=0x81f06a60 <mysql_bin_log+3232>, file=0x815bd48c "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x80aed86c in inline_mysql_cond_wait (that=0x81f06ac4 <mysql_bin_log+3332>, mutex=0x81f06a60 <mysql_bin_log+3232>, src_file=0x815be62c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc", src_line=10355) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1222
#4  0x80b0c147 in binlog_background_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc:10355
#5  0x80cfdd4a in pfs_spawn_thread (arg=0x845c82e0) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#6  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#7  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 11 (Thread 0xa95b5b40 (LWP 733)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb7320d32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0
#2  0xb731a9e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0
#3  0x812acfbe in safe_mutex_lock (mp=0xb1b22f18, my_flags=0, file=0x813f6b38 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_rli.cc", line=2460) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:290
#4  0x807e90a8 in inline_mysql_mutex_lock (that=0xb1b22f18, src_file=0x813f6b38 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_rli.cc", src_line=2460) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:750
#5  0x807f0884 in rpl_group_info::mark_start_commit (this=0xb1b1faf0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_rli.cc:2460
#6  0x8088a36f in handle_rpl_parallel_thread (arg=0xb1b4853c) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:1327
#7  0x80cfdd4a in pfs_spawn_thread (arg=0xb1b47e90) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 10 (Thread 0xaeffeb40 (LWP 32579)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731de56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ada30 in safe_cond_timedwait (cond=0x81edb100 <thread_cache>, mp=0x81edb168 <thread_cache+104>, abstime=0xaeffe17c, file=0x813faf88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807fc2ca in inline_mysql_cond_timedwait (that=0x81edb100 <thread_cache>, mutex=0x81edb168 <thread_cache+104>, abstime=0xaeffe17c, src_file=0x813fb61c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x80800acd in Thread_cache::park (this=0x81edb100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x80800156 in do_handle_one_connection (connect=0x845ebee8, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807ffdde in handle_one_connection (arg=0x8470a378) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cfdd4a in pfs_spawn_thread (arg=0x8471d050) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 9 (Thread 0xa95ffb40 (LWP 732)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731da8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ad785 in safe_cond_wait (cond=0xb1b218f8, mp=0xb1b22f18, file=0x8142dc40 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x808865f0 in inline_mysql_cond_wait (that=0xb1b218f8, mutex=0xb1b22f18, src_file=0x8142e588 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc", src_line=382) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1222
#4  0x80887dd5 in do_gco_wait (rgi=0xb1b269c8, gco=0xb1b218f8, did_enter_cond=0xa95ff19f, old_stage=0xa95ff1f4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:382
#5  0x8088a124 in handle_rpl_parallel_thread (arg=0xb1b483d8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:1239
#6  0x80cfdd4a in pfs_spawn_thread (arg=0xb1b47b60) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#7  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#8  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 8 (Thread 0xb18ffb40 (LWP 32554)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb704fbb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
#2  0x81314ed4 in my_getevents (ctx=0xb1c1f000, min_nr=1, nr=256, ev=0xb18fd1c0) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/aio_linux.cc:64
#3  0x81315039 in tpool::aio_linux::getevent_thread_routine (aio=0x843a4338) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/aio_linux.cc:107
#4  0x81316680 in std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)>::_M_invoke<0u>(std::_Index_tuple<0u>) (this=0x843343a8) at /usr/include/c++/5/functional:1531
#5  0x81316555 in std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)>::operator()() (this=0x843343a8) at /usr/include/c++/5/functional:1520
#6  0x813164dc in std::thread::_Impl<std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)> >::_M_run() (this=0x8433439c) at /usr/include/c++/5/thread:115
#7  0xb72325de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#8  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 7 (Thread 0xabc59b40 (LWP 32592)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb7320d32 in __lll_lock_wait () from /lib/i386-linux-gnu/libpthread.so.0
#2  0xb731a9e5 in pthread_mutex_lock () from /lib/i386-linux-gnu/libpthread.so.0
#3  0x812acfbe in safe_mutex_lock (mp=0xb1b22f18, my_flags=0, file=0x8142e588 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc", line=2426) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:290
#4  0x80886358 in inline_mysql_mutex_lock (that=0xb1b22f18, src_file=0x8142e588 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc", src_line=2426) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:750
#5  0x8088d388 in rpl_parallel::workers_idle (this=0x84723b80) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/rpl_parallel.cc:2426
#6  0x8055f36a in send_show_master_info_data (thd=0xabb080d8, mi=0x8471f5d0, full=false, gtid_pos=0xabc58508) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:3228
#7  0x8055d19e in show_master_info (thd=0xabb080d8, mi=0x8471f5d0, full=false) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:2875
#8  0x805590e5 in Sql_cmd_show_slave_status::execute (this=0xabb194e8, thd=0xabb080d8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/slave.cc:1530
#9  0x80676826 in mysql_execute_command (thd=0xabb080d8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_parse.cc:6055
#10 0x8067c8e8 in mysql_parse (thd=0xabb080d8, rawbuf=0xabb19470 "SHOW SLAVE STATUS", length=17, parser_state=0xabc58b3c, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_parse.cc:8099
#11 0x80667885 in dispatch_command (command=COM_QUERY, thd=0xabb080d8, packet=0xabb11391 "", packet_length=17, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_parse.cc:1892
#12 0x80665de2 in do_command (thd=0xabb080d8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_parse.cc:1371
#13 0x808000c3 in do_handle_one_connection (connect=0x845ebee8, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1410
#14 0x807ffdde in handle_one_connection (arg=0x8470a378) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#15 0x80cfdd4a in pfs_spawn_thread (arg=0x8471d050) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#16 0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#17 0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 6 (Thread 0xb2831b40 (LWP 32552)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731de56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ada30 in safe_cond_timedwait (cond=0x82709aa0 <COND_checkpoint>, mp=0x82709a20 <LOCK_checkpoint>, abstime=0xb283115c, file=0x8160f9d8 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x80c54995 in inline_mysql_cond_timedwait (that=0x82709aa0 <COND_checkpoint>, mutex=0x82709a20 <LOCK_checkpoint>, abstime=0xb283115c, src_file=0x8160fa28 "/home/buildbot/buildbot/build/mariadb-10.5.10/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x80c54efa in my_service_thread_sleep (control=0x81d2e138 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/maria/ma_servicethread.c:115
#5  0x80c4847e in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/maria/ma_checkpoint.c:707
#6  0x80cfdd4a in pfs_spawn_thread (arg=0x84451590) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#7  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#8  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 5 (Thread 0xaddfdb40 (LWP 32560)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731de56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x8130c12e in __gthread_cond_timedwait (__cond=0x84477850, __mutex=0x843997d8, __abs_timeout=0xaddfd0b0) at /usr/include/i386-linux-gnu/c++/5/bits/gthr-default.h:871
#3  0x81310fd7 in std::condition_variable::__wait_until_impl<std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> > > (this=0x84477850, __lock=..., __atime=...) at /usr/include/c++/5/condition_variable:165
#4  0x8131008c in std::condition_variable::wait_until<std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> > > (this=0x84477850, __lock=..., __atime=...) at /usr/include/c++/5/condition_variable:105
#5  0x8130f64d in std::condition_variable::wait_for<long long, std::ratio<1ll, 1000ll> > (this=0x84477850, __lock=..., __rtime=...) at /usr/include/c++/5/condition_variable:137
#6  0x8130c381 in tpool::thread_pool_generic::wait_for_tasks (this=0x84399730, lk=..., thread_data=0x84477850) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/tpool_generic.cc:446
#7  0x8130c58b in tpool::thread_pool_generic::get_task (this=0x84399730, thread_var=0x84477850, t=0xaddfd18c) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/tpool_generic.cc:499
#8  0x8130c822 in tpool::thread_pool_generic::worker_main (this=0x84399730, thread_var=0x84477850) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/tpool_generic.cc:544
#9  0x813142d5 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaee08548, __object=0x84399730) at /usr/include/c++/5/functional:600
#10 0x81314266 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaee08540) at /usr/include/c++/5/functional:1531
#11 0x813140cb in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaee08540) at /usr/include/c++/5/functional:1520
#12 0x81314052 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaee08534) at /usr/include/c++/5/thread:115
#13 0xb72325de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#14 0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#15 0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 4 (Thread 0xb3355b40 (LWP 32551)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731de56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ada30 in safe_cond_timedwait (cond=0x8277cce0 <COND_timer>, mp=0x8277cc60 <LOCK_timer>, abstime=0xb3355210, file=0x81830570 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x812ae9e2 in inline_mysql_cond_timedwait (that=0x8277cce0 <COND_timer>, mutex=0x8277cc60 <LOCK_timer>, abstime=0xb3355210, src_file=0x818305c0 "/home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_timer.c", src_line=321) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x812af700 in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_timer.c:321
#5  0x80cfdd4a in pfs_spawn_thread (arg=0x843c56b8) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#6  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#7  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 3 (Thread 0xaf9a9b40 (LWP 32569)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb6f99e8c in sigwaitinfo () from /lib/i386-linux-gnu/libc.so.6
#2  0x80520aa4 in my_sigwait (set=0xaf9a9190, sig=0xaf9a9160, code=0xaf9a9164) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/my_pthread.h:196
#3  0x805264f6 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:2957
#4  0x80cfdd4a in pfs_spawn_thread (arg=0x846520a0) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#5  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#6  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 2 (Thread 0xb05feb40 (LWP 32555)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731da8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812ad785 in safe_cond_wait (cond=0x81d3860c <buf_pool+716>, mp=0x81d38580 <buf_pool+576>, file=0x817c0684 "/home/buildbot/buildbot/build/mariadb-10.5.10/storage/innobase/buf/buf0flu.cc", line=2113) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x81174f0d in buf_flush_page_cleaner () at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/innobase/buf/buf0flu.cc:2113
#4  0xb73182b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#5  0xb705416e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 1 (Thread 0xb6ddaa40 (LWP 32529)):
#0  0xb7738c31 in __kernel_vsyscall ()
#1  0xb731ece1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812a8b2c in my_write_core (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/stacktrace.c:424
#3  0x809c1173 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/signal_handler.cc:343
#4  <signal handler called>
#5  0xb7738c31 in __kernel_vsyscall ()
#6  0xb7049f8f in poll () from /lib/i386-linux-gnu/libc.so.6
#7  0x8052cff7 in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:6204
#8  0x8052c4f5 in mysqld_main (argc=176, argv=0x842882ac) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:5653
#9  0x805209eb in main (argc=26, argv=0xbff49214) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/main.cc:25
 
 - deleting it, already saved 0
 - found 'core' (0/0)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace from coredump /mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/2/log/rpl.rpl_parallel_gco_wait_kill-innodb,stmt/mysqld.1/data/core
Core generated by '/mnt/buildbot/build/mariadb-10.5.10/sql/mariadbd'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New LWP 605]
[New LWP 621]
[New LWP 627]
[New LWP 633]
[New LWP 650]
[New LWP 626]
[New LWP 731]
[New LWP 634]
[New LWP 639]
[New LWP 668]
[New LWP 640]
[New LWP 727]
[New LWP 659]
[New LWP 637]
[New LWP 675]
[New LWP 726]
[New LWP 654]
[New LWP 620]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/mnt/buildbot/build/mariadb-10.5.10/sql/mariadbd --defaults-group-suffix=.1 --d'.
Program terminated with signal SIGABRT, Aborted.
#0  0xb774fc31 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xb6df1a40 (LWP 605))]
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7335ce1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x8127db2c in my_write_core (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/stacktrace.c:424
#3  0x80996173 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/signal_handler.cc:343
#4  <signal handler called>
#5  0xb774fc31 in __kernel_vsyscall ()
#6  0xb7060f8f in poll () from /lib/i386-linux-gnu/libc.so.6
#7  0x80501ff7 in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:6204
#8  0x805014f5 in mysqld_main (argc=167, argv=0x83c331d4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:5653
#9  0x804f59eb in main (argc=26, argv=0xbfebe0f4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/main.cc:25
 
Thread 18 (Thread 0xb336cb40 (LWP 620)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x82751ce0 <COND_timer>, mp=0x82751c60 <LOCK_timer>, abstime=0xb336c210, file=0x81805570 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x812839e2 in inline_mysql_cond_timedwait (that=0x82751ce0 <COND_timer>, mutex=0x82751c60 <LOCK_timer>, abstime=0xb336c210, src_file=0x818055c0 "/home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_timer.c", src_line=321) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x81284700 in timer_handler (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_timer.c:321
#5  0x80cd2d4a in pfs_spawn_thread (arg=0x83d706b8) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#6  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#7  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 17 (Thread 0xaef60b40 (LWP 654)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xaef6017c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xaef6017c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x8416dcf0, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x8409f5a8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x83fe7b88) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 16 (Thread 0xacc8eb40 (LWP 726)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xacc8e17c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xacc8e17c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x83f4f828, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x83f4f828) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x840b84d8) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 15 (Thread 0xaceb5b40 (LWP 675)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xaceb517c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xaceb517c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x83f4f828, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x8409f5a8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x840b6608) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 14 (Thread 0xafdb2b40 (LWP 637)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb6fb0e8c in sigwaitinfo () from /lib/i386-linux-gnu/libc.so.6
#2  0x804f5aa4 in my_sigwait (set=0xafdb2190, sig=0xafdb2160, code=0xafdb2164) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/my_pthread.h:196
#3  0x804fb4f6 in signal_hand (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:2957
#4  0x80cd2d4a in pfs_spawn_thread (arg=0x83fe3720) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#5  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#6  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 13 (Thread 0xad45cb40 (LWP 659)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xad45c17c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xad45c17c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x8416dcf0, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x8409f5a8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x83fe7b88) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 12 (Thread 0xacc44b40 (LWP 727)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xacc4417c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xacc4417c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x83f4f828, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x83f4f828) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x840b84d8) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 11 (Thread 0xaeff4b40 (LWP 640)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xaeff417c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xaeff417c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x83f4f828, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x8409f5a8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x83fe7b88) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 10 (Thread 0xaceffb40 (LWP 668)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xaceff17c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xaceff17c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x83f4f828, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x8409f5a8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x840b6608) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 9 (Thread 0xafd68b40 (LWP 639)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282785 in safe_cond_wait (cond=0x81eb6660 <COND_manager>, mp=0x81eb65e0 <LOCK_manager>, file=0x81380cc0 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x80635b14 in inline_mysql_cond_wait (that=0x81eb6660 <COND_manager>, mutex=0x81eb65e0 <LOCK_manager>, src_file=0x813812f4 "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_manager.cc", src_line=103) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1222
#4  0x80635fc7 in handle_manager (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_manager.cc:103
#5  0x80cd2d4a in pfs_spawn_thread (arg=0x8403db78) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#6  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#7  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 8 (Thread 0xafdfcb40 (LWP 634)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282785 in safe_cond_wait (cond=0x81edbac4 <mysql_bin_log+3332>, mp=0x81edba60 <mysql_bin_log+3232>, file=0x8159248c "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1222) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x80ac286c in inline_mysql_cond_wait (that=0x81edbac4 <mysql_bin_log+3332>, mutex=0x81edba60 <mysql_bin_log+3232>, src_file=0x8159362c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc", src_line=10355) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1222
#4  0x80ae1147 in binlog_background_thread (arg=0x0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc:10355
#5  0x80cd2d4a in pfs_spawn_thread (arg=0x83f92758) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#6  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#7  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 7 (Thread 0xacbfab40 (LWP 731)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81edb3ac <mysql_bin_log+1516>, mp=0x81edb27c <mysql_bin_log+1212>, abstime=0xacbf8b90, file=0x8159248c "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x80ac2928 in inline_mysql_cond_timedwait (that=0x81edb3ac <mysql_bin_log+1516>, mutex=0x81edb27c <mysql_bin_log+1212>, abstime=0xacbf8b90, src_file=0x8159362c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc", src_line=8749) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x80adca01 in MYSQL_BIN_LOG::wait_for_update_binlog_end_pos (this=0x81edadc0 <mysql_bin_log>, thd=0xb1e3c328, timeout=0xacbf8b90) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/log.cc:8749
#5  0x8067e8fd in wait_new_events (info=0xacbf91fc, linfo=0xacbf8fe0, binlog_end_pos_filename=0xacbf8c0c "./master-bin.000001", end_pos_ptr=0xacbf8bf8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_repl.cc:2512
#6  0x8067ecd6 in get_binlog_end_pos (info=0xacbf91fc, log=0xacbf8f0c, linfo=0xacbf8fe0) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_repl.cc:2632
#7  0x8067f655 in send_one_binlog_file (info=0xacbf91fc, log=0xacbf8f0c, linfo=0xacbf8fe0, start_pos=4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_repl.cc:2792
#8  0x8067fce0 in mysql_binlog_send (thd=0xb1e3c328, log_ident=0xb1e48720 "master-bin.000001", pos=4, flags=2) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_repl.cc:2937
#9  0x8063d77c in dispatch_command (command=COM_BINLOG_DUMP, thd=0xb1e3c328, packet=0xb1e40641 "", packet_length=27, is_com_multi=false, is_next_command=false) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_parse.cc:2153
#10 0x8063ade2 in do_command (thd=0xb1e3c328) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_parse.cc:1371
#11 0x807d50c3 in do_handle_one_connection (connect=0x83f4f828, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1410
#12 0x807d4dde in handle_one_connection (arg=0x83f4f828) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#13 0x80cd2d4a in pfs_spawn_thread (arg=0x840b84d8) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#14 0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#15 0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 6 (Thread 0xb19feb40 (LWP 626)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7066bb7 in syscall () from /lib/i386-linux-gnu/libc.so.6
#2  0x812e9ed4 in my_getevents (ctx=0xb3311000, min_nr=1, nr=256, ev=0xb19fc1c0) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/aio_linux.cc:64
#3  0x812ea039 in tpool::aio_linux::getevent_thread_routine (aio=0x83c2f608) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/aio_linux.cc:107
#4  0x812eb680 in std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)>::_M_invoke<0u>(std::_Index_tuple<0u>) (this=0x83cdf3a8) at /usr/include/c++/5/functional:1531
#5  0x812eb555 in std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)>::operator()() (this=0x83cdf3a8) at /usr/include/c++/5/functional:1520
#6  0x812eb4dc in std::thread::_Impl<std::_Bind_simple<void (*(tpool::aio_linux*))(tpool::aio_linux*)> >::_M_run() (this=0x83cdf39c) at /usr/include/c++/5/thread:115
#7  0xb72495de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 5 (Thread 0xaefaab40 (LWP 650)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x81eb0100 <thread_cache>, mp=0x81eb0168 <thread_cache+104>, abstime=0xaefaa17c, file=0x813cff88 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x807d12ca in inline_mysql_cond_timedwait (that=0x81eb0100 <thread_cache>, mutex=0x81eb0168 <thread_cache+104>, abstime=0xaefaa17c, src_file=0x813d061c "/home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h", src_line=177) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x807d5acd in Thread_cache::park (this=0x81eb0100 <thread_cache>) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/thread_cache.h:176
#5  0x807d5156 in do_handle_one_connection (connect=0x8416dcf0, put_in_cache=true) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1422
#6  0x807d4dde in handle_one_connection (arg=0x8409f5a8) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/sql_connect.cc:1312
#7  0x80cd2d4a in pfs_spawn_thread (arg=0x83fe7b88) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#8  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#9  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 4 (Thread 0xac2dbb40 (LWP 633)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x812e112e in __gthread_cond_timedwait (__cond=0x83e22450, __mutex=0x83d446d8, __abs_timeout=0xac2db0b0) at /usr/include/i386-linux-gnu/c++/5/bits/gthr-default.h:871
#3  0x812e5fd7 in std::condition_variable::__wait_until_impl<std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> > > (this=0x83e22450, __lock=..., __atime=...) at /usr/include/c++/5/condition_variable:165
#4  0x812e508c in std::condition_variable::wait_until<std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> > > (this=0x83e22450, __lock=..., __atime=...) at /usr/include/c++/5/condition_variable:105
#5  0x812e464d in std::condition_variable::wait_for<long long, std::ratio<1ll, 1000ll> > (this=0x83e22450, __lock=..., __rtime=...) at /usr/include/c++/5/condition_variable:137
#6  0x812e1381 in tpool::thread_pool_generic::wait_for_tasks (this=0x83d44630, lk=..., thread_data=0x83e22450) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/tpool_generic.cc:446
#7  0x812e158b in tpool::thread_pool_generic::get_task (this=0x83d44630, thread_var=0x83e22450, t=0xac2db18c) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/tpool_generic.cc:499
#8  0x812e1822 in tpool::thread_pool_generic::worker_main (this=0x83d44630, thread_var=0x83e22450) at /home/buildbot/buildbot/build/mariadb-10.5.10/tpool/tpool_generic.cc:544
#9  0x812e92d5 in std::_Mem_fn_base<void (tpool::thread_pool_generic::*)(tpool::worker_data*), true>::operator()<tpool::worker_data*, void>(tpool::thread_pool_generic*, tpool::worker_data*&&) const (this=0xaee08698, __object=0x83d44630) at /usr/include/c++/5/functional:600
#10 0x812e9266 in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::_M_invoke<0u, 1u>(std::_Index_tuple<0u, 1u>) (this=0xaee08690) at /usr/include/c++/5/functional:1531
#11 0x812e90cb in std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)>::operator()() (this=0xaee08690) at /usr/include/c++/5/functional:1520
#12 0x812e9052 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (tpool::thread_pool_generic::*)(tpool::worker_data*)> (tpool::thread_pool_generic*, tpool::worker_data*)> >::_M_run() (this=0xaee08684) at /usr/include/c++/5/thread:115
#13 0xb72495de in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#14 0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#15 0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 3 (Thread 0xb05fdb40 (LWP 627)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334a8c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282785 in safe_cond_wait (cond=0x81d0d60c <buf_pool+716>, mp=0x81d0d580 <buf_pool+576>, file=0x81795684 "/home/buildbot/buildbot/build/mariadb-10.5.10/storage/innobase/buf/buf0flu.cc", line=2113) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:492
#3  0x81149f0d in buf_flush_page_cleaner () at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/innobase/buf/buf0flu.cc:2113
#4  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#5  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 2 (Thread 0xb2726b40 (LWP 621)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7334e56 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x81282a30 in safe_cond_timedwait (cond=0x826deaa0 <COND_checkpoint>, mp=0x826dea20 <LOCK_checkpoint>, abstime=0xb272615c, file=0x815e49d8 "/home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h", line=1259) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/thr_mutex.c:546
#3  0x80c29995 in inline_mysql_cond_timedwait (that=0x826deaa0 <COND_checkpoint>, mutex=0x826dea20 <LOCK_checkpoint>, abstime=0xb272615c, src_file=0x815e4a28 "/home/buildbot/buildbot/build/mariadb-10.5.10/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/buildbot/build/mariadb-10.5.10/include/mysql/psi/mysql_thread.h:1259
#4  0x80c29efa in my_service_thread_sleep (control=0x81d03138 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/maria/ma_servicethread.c:115
#5  0x80c1d47e in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/maria/ma_checkpoint.c:707
#6  0x80cd2d4a in pfs_spawn_thread (arg=0x83dfc358) at /home/buildbot/buildbot/build/mariadb-10.5.10/storage/perfschema/pfs.cc:2201
#7  0xb732f2b5 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#8  0xb706b16e in clone () from /lib/i386-linux-gnu/libc.so.6
 
Thread 1 (Thread 0xb6df1a40 (LWP 605)):
#0  0xb774fc31 in __kernel_vsyscall ()
#1  0xb7335ce1 in pthread_kill () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x8127db2c in my_write_core (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/mysys/stacktrace.c:424
#3  0x80996173 in handle_fatal_signal (sig=6) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/signal_handler.cc:343
#4  <signal handler called>
#5  0xb774fc31 in __kernel_vsyscall ()
#6  0xb7060f8f in poll () from /lib/i386-linux-gnu/libc.so.6
#7  0x80501ff7 in handle_connections_sockets () at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:6204
#8  0x805014f5 in mysqld_main (argc=167, argv=0x83c331d4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/mysqld.cc:5653
#9  0x804f59eb in main (argc=26, argv=0xbfebe0f4) at /home/buildbot/buildbot/build/mariadb-10.5.10/sql/main.cc:25
 
 - deleting it, already saved 0
 - saving '/mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/2/log/rpl.rpl_parallel_gco_wait_kill-innodb,stmt/' to '/mnt/buildbot/build/mariadb-10.5.10/mysql-test/var/log/rpl.rpl_parallel_gco_wait_kill-innodb,stmt/'



 Comments   
Comment by Marko Mäkelä [ 2022-11-10 ]

Recent occurrence on 10.6:

10.6 fef9d6ef1db9a4648a54954c38ea4fbab2a6542c

rpl.rpl_parallel_gco_wait_kill 'innodb,stmt' w19 [ fail ]  timeout after 480 seconds
        Test ended at 2022-11-10 07:39:40
Test case timeout after 480 seconds
== /home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/var/19/log/rpl_parallel_gco_wait_kill.log == 
66	66
67	67
68	68
69	69
70	70
100	100
SET debug_sync='RESET';
connection server_2;
SET debug_sync='now SIGNAL d0_cont';
SET debug_sync='now WAIT_FOR t1_waiting';
SET debug_sync='now SIGNAL d3_cont';
SET debug_sync='now WAIT_FOR t2_waiting';
SET debug_sync='now SIGNAL d1_cont';
SET debug_sync='now WAIT_FOR t3_waiting';
SET debug_sync='now SIGNAL d2_cont';
SET debug_sync='now WAIT_FOR t4_waiting';
KILL THD_ID;
SET debug_sync='now WAIT_FOR t3_killed';
SET debug_sync='now SIGNAL t1_cont';
include/wait_for_slave_sql_error.inc [errno=1317,1927,1964]
 == /home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/var/19/tmp/analyze-timeout-mysqld.1.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/var/tmp/' (111)
 == /home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/var/19/tmp/analyze-timeout-mysqld.2.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/var/tmp/' (111)

There is also a stack trace from a core dump, showing only one thread:

#0  0x00007f753f792f68 in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000055d056f151d7 in my_write_core (sig=6) at /home/buildbot/amd64-ubuntu-2004-debug/build/mysys/stacktrace.c:424
#2  0x000055d05668f0d6 in handle_fatal_signal (sig=6) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/signal_handler.cc:355
#3  <signal handler called>
#4  0x00007f753f34e99f in poll () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x000055d0561df184 in handle_connections_sockets () at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:6182
#6  0x000055d0561dea31 in mysqld_main (argc=158, argv=0x55d058a12c60) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/mysqld.cc:5887
#7  0x000055d0561d39fd in main (argc=26, argv=0x7fff2283ad68) at /home/buildbot/amd64-ubuntu-2004-debug/build/sql/main.cc:34

Comment by Marko Mäkelä [ 2023-09-19 ]

Recent occurrence on 10.6:

10.6 8096139b3a933f5324bffb29682457510d22b10e

rpl.rpl_parallel_gco_wait_kill 'innodb,stmt' w6 [ fail ]  timeout after 900 seconds
        Test ended at 2023-09-19 08:15:23
Test case timeout after 900 seconds
== /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/6/log/rpl_parallel_gco_wait_kill.log == 
66	66
67	67
68	68
69	69
70	70
100	100
SET debug_sync='RESET';
connection server_2;
SET debug_sync='now SIGNAL d0_cont';
SET debug_sync='now WAIT_FOR t1_waiting';
SET debug_sync='now SIGNAL d3_cont';
SET debug_sync='now WAIT_FOR t2_waiting';
SET debug_sync='now SIGNAL d1_cont';
SET debug_sync='now WAIT_FOR t3_waiting';
SET debug_sync='now SIGNAL d2_cont';
SET debug_sync='now WAIT_FOR t4_waiting';
KILL THD_ID;
SET debug_sync='now WAIT_FOR t3_killed';
SET debug_sync='now SIGNAL t1_cont';
include/wait_for_slave_sql_error.inc [errno=1317,1927,1964]
 == /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/6/tmp/analyze-timeout-mysqld.2.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/t' (111)
 == /home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/6/tmp/analyze-timeout-mysqld.1.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/home/buildbot/amd64-ubuntu-2204-debug-ps/build/mysql-test/var/t' (111)

You can also find there the output of GDB thread apply all backtrace full.

Comment by Marko Mäkelä [ 2023-11-30 ]

This still keeps occurring, on the slowest mandatory builder. I think that this report deserves to be critical. Here is an example where it caused the build to consume 27 minutes before failing, instead of the normal 11 to 20 minutes.

Comment by Kristian Nielsen [ 2023-11-30 ]

bnestere, this is the same bug that you have a patch for in MDEV-10653.
So I assigned it to you, hope that's ok (it just needs to be closed when your fix is merged to 10.5, or if you prefer just assign it to me and I can do it).

Summary: This is a real bug, there's a deadlock on the mutexes err_lock and LOCK_parallel_entry as can be seen in the stack traces from the failures. There's already a patch pending for 10.4.

Comment by Brandon Nesterenko [ 2023-12-11 ]

Closing as a duplicate of MDEV-10653, as this should be fixed by the push into 10.4 of 8dad51481b1

Generated at Thu Feb 08 09:37:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.