[MDEV-13791] rpl.rpl_stm_stop_middle_group failed in buildbot, server crashes Created: 2017-09-13  Updated: 2023-04-17

Status: Stalled
Project: MariaDB Server
Component/s: Tests
Affects Version/s: 10.2, 10.3
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Alice Sherepa Assignee: Andrei Elkin
Resolution: Unresolved Votes: 0
Labels: None


 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/9567/steps/test_2/logs/stdio

rpl.rpl_stm_stop_middle_group 'innodb,stmt' w1 [ fail ]
 
        Test ended at 2017-09-13 01:03:17
 
 
 
CURRENT_TEST: rpl.rpl_stm_stop_middle_group
 
mysqltest: In included file "./include/end_include_file.inc": 
included from ./include/wait_for_slave_param.inc at line 124:
 
included from ./include/wait_for_slave_sql_to_stop.inc at line 36:
 
included from ./extra/rpl_tests/rpl_stop_middle_group.test at line 124:
 
included from /mnt/buildbot/build/mariadb-10.2.9/mysql-test/suite/rpl/t/rpl_stm_stop_middle_group.test at line 11:
 
At line 62: query 'let $_include_file_connection= `SELECT SUBSTRING('$_include_file_connection', LENGTH('$_include_file_rpl_connection_name') + 2)`' failed: 2013: Lost connection to MySQL server during query
 
 
 
The result from queries just before the failure was:
 
< snip >
 
Last_SQL_Error	true
 
Fatal error: ... Slave SQL Thread stopped with incomplete event group having non-transactional changes. If the group consists solely of row-based events, you can try to restart the slave with --slave-exec-mode=IDEMPOTENT, which ignores duplicate key, key not found, and similar errors (see documentation for details).	1
 
select count(*) as one  from tm;
 
one
 
1
 
select count(*) as zero from ti;
 
zero
 
0
 
set @@global.debug_dbug="-d";
 
stop slave;
 
truncate table tm;
 
include/start_slave.inc
 
connection master;
 
connection slave;
 
set @@global.debug_dbug="+d,stop_slave_middle_group";
 
set @@global.debug_dbug="+d,incomplete_group_in_relay_log";
 
connection master;
 
update tm as t1, ti as t2 set t1.a=t1.a * 2, t2.a=t2.a * 2;
 
connection slave;
 
include/wait_for_slave_sql_to_stop.inc


2017-09-13  1:03:16 2896620352 [Note] Deleted Master_info file '/mnt/buildbot/build/mariadb-10.2.9/mysql-test/var/1/mysqld.2/data/master.info'.
 
2017-09-13  1:03:16 2896620352 [Note] Deleted Master_info file '/mnt/buildbot/build/mariadb-10.2.9/mysql-test/var/1/mysqld.2/data/relay-log.info'.
 
2017-09-13  1:03:16 2896620352 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
 
2017-09-13  1:03:16 2896620352 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='16040', master_log_file='master-bin.000001', master_log_pos='4'.
 
2017-09-13  1:03:16 2808769344 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication started in log 'master-bin.000001' at position 4
 
2017-09-13  1:03:16 2808466240 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4
 
2017-09-13  1:03:16 2808466240 [Warning] Slave SQL: Request to stop slave SQL Thread received while applying a group that has non-transactional changes; waiting for completion of the group ... , Gtid 0-1-4, Internal MariaDB error code: 0
 
2017-09-13  1:03:16 2808466240 [Warning] Slave SQL: Request to stop slave SQL Thread received while applying a group that has non-transactional changes; waiting for completion of the group ... , Gtid 0-1-4, Internal MariaDB error code: 0
 
2017-09-13  1:03:16 2808466240 [Warning] Slave SQL: Request to stop slave SQL Thread received while applying a group that has non-transactional changes; waiting for completion of the group ... , Gtid 0-1-4, Internal MariaDB error code: 0
 
2017-09-13  1:03:16 2808466240 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 1382
 
170913  1:03:16 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.9-MariaDB-debug-log
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=4
 
max_threads=153
 
thread_count=12
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61883 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0xa73005d0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0xa7ef128c thread_stack 0x49000
 
2017-09-13  1:03:16 2808466240 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 1382, relay log './slave-relay-bin.000002' position: 1682
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld(my_print_stacktrace+0x32)[0x8cf5896]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld(handle_fatal_signal+0x3c7)[0x85a2edf]
 
[0xb76f4400]
 
/lib/i386-linux-gnu/libc.so.6(+0xbd4a5)[0xb725f4a5]
 
/lib/i386-linux-gnu/libc.so.6(fnmatch+0x69)[0xb7260af9]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x8d179fa]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld(_db_keyword_+0x79)[0x8d17ed2]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld(_db_pargs_+0x65)[0x8d172e8]
 
2017-09-13  1:03:16 2808466240 [ERROR] Slave SQL: Fatal error: ... Slave SQL Thread stopped with incomplete event group having non-transactional changes. If the group consists solely of row-based events, you can try to restart the slave with --slave-exec-mode=IDEMPOTENT, which ignores duplicate key, key not found, and similar errors (see documentation for details). Gtid 0-1-7, Internal MariaDB error code: 1593
 
2017-09-13  1:03:16 2808466240 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 1634
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld(_ZN3THD11enter_stageEPK17PSI_stage_info_v1PKcS4_j+0x28)[0x827049a]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld(_Z19close_thread_tablesP3THD+0x6c)[0x82cd755]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld(_Z9reset_thdP3THD+0x1d)[0x82fdbe9]
 
2017-09-13  1:03:16 2808769344 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 1955
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x88a2797]
 
2017-09-13  1:03:16 2808466240 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 1634, relay log './slave-relay-bin.000002' position: 1934
 
2017-09-13  1:03:16 2808769344 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:16040',replication started in log 'master-bin.000001' at position 1955
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x8a0ce92]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x899afbd]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x899b1c8]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x899b39e]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x8a59819]
 
/mnt/buildbot/build/mariadb-10.2.9/sql/mysqld[0x8a5999a]
 
2017-09-13  1:03:16 2808466240 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement accesses nontransactional table as well as transactional or temporary table, and writes to any of them Statement: update tm as t1, ti as t2 set t1.a=t1.a * 2, t2.a=t2.a * 2
 
2017-09-13  1:03:16 2808466240 [ERROR] Slave SQL: Fatal error: ... Slave SQL Thread stopped with incomplete event group having non-transactional changes. If the group consists solely of row-based events, you can try to restart the slave with --slave-exec-mode=IDEMPOTENT, which ignores duplicate key, key not found, and similar errors (see documentation for details). Gtid 0-1-8, Internal MariaDB error code: 1593
 
2017-09-13  1:03:16 2808466240 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 1955
 
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb747cd4c]
 
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb728cace]



 Comments   
Comment by Alice Sherepa [ 2020-07-15 ]

bb-10.3-release 4fef5ab504d1d6779f2411ad40dea55a

rpl.rpl_stm_stop_middle_group 'innodb,stmt' w96 [ fail ]
 
        Test ended at 2020-05-06 22:10:48
 
 
 
CURRENT_TEST: rpl.rpl_stm_stop_middle_group
 
mysqltest: In included file "/opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/suite/rpl/include/rpl_stop_middle_group.test": 
included from /opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/suite/rpl/t/rpl_stm_stop_middle_group.test at line 11:
 
At line 137: query 'SET @@GLOBAL.debug_dbug = @saved_dbug' failed: 2013: Lost connection to MySQL server during query
 
 
 
The result from queries just before the failure was:
 
< snip >
 
stop slave;
 
truncate table tm;
 
include/start_slave.inc
 
connection master;
 
connection slave;
 
set @@global.debug_dbug="d,stop_slave_middle_group,incomplete_group_in_relay_log";
 
connection master;
 
update tm as t1, ti as t2 set t1.a=t1.a * 2, t2.a=t2.a * 2;
 
connection slave;
 
include/wait_for_slave_sql_to_stop.inc
 
SELECT "Fatal error: ... Slave SQL Thread stopped with incomplete event group having non-transactional changes. If the group consists solely of row-based events, you can try to restart the slave with --slave-exec-mode=IDEMPOTENT, which ignores duplicate key, key not found, and similar errors (see documentation for details)." AS Last_SQL_Error, @check as `true`;
 
Last_SQL_Error	true
 
Fatal error: ... Slave SQL Thread stopped with incomplete event group having non-transactional changes. If the group consists solely of row-based events, you can try to restart the slave with --slave-exec-mode=IDEMPOTENT, which ignores duplicate key, key not found, and similar errors (see documentation for details).	1
 
select max(a) as two from tm;
 
two
 
2
 
select max(a) as one from ti;
 
one
 
1
 
SET @@GLOBAL.debug_dbug = @saved_dbug;
 
 
 
More results from queries before failure can be found in /opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/var/96/log/rpl_stm_stop_middle_group.log
 
 
 
 
 
Server [mysqld.2 - pid: 19428, winpid: 19428, exit: 256] failed during test run
 
Server log from this test:
 
----------SERVER LOG START-----------
 
2020-05-06 22:10:42 28 [Note] Deleted Master_info file '/opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/var/96/mysqld.2/data/master.info'.
 
2020-05-06 22:10:42 28 [Note] Deleted Master_info file '/opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/var/96/mysqld.2/data/relay-log.info'.
 
2020-05-06 22:10:42 28 [Note] Master connection name: ''  Master_info_file: 'master.info'  Relay_info_file: 'relay-log.info'
 
2020-05-06 22:10:43 28 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port='17920', master_log_file='', master_log_pos='4'. New state master_host='127.0.0.1', master_port='17920', master_log_file='master-bin.000001', master_log_pos='4'.
 
2020-05-06 22:10:43 30 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:17920' in log 'master-bin.000001' at position 4
 
2020-05-06 22:10:43 31 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 4, relay log './slave-relay-bin.000001' position: 4
 
2020-05-06 22:10:43 30 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:17920',replication started in log 'master-bin.000001' at position 4
 
2020-05-06 22:10:44 31 [Warning] Slave SQL: Request to stop slave SQL Thread received while applying a group that has non-transactional changes; waiting for completion of the group ... , Gtid 0-1-4, Internal MariaDB error code: 0
 
2020-05-06 22:10:44 31 [Warning] Slave SQL: Request to stop slave SQL Thread received while applying a group that has non-transactional changes; waiting for completion of the group ... , Gtid 0-1-4, Internal MariaDB error code: 0
 
2020-05-06 22:10:44 31 [Warning] Slave SQL: Request to stop slave SQL Thread received while applying a group that has non-transactional changes; waiting for completion of the group ... , Gtid 0-1-4, Internal MariaDB error code: 0
 
2020-05-06 22:10:44 31 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 1382
 
2020-05-06 22:10:44 34 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 1382, relay log './slave-relay-bin.000002' position: 1682
 
2020-05-06 22:10:44 34 [ERROR] Slave SQL: Fatal error: ... Slave SQL Thread stopped with incomplete event group having non-transactional changes. If the group consists solely of row-based events, you can try to restart the slave with --slave-exec-mode=IDEMPOTENT, which ignores duplicate key, key not found, and similar errors (see documentation for details). Gtid 0-1-7, Internal MariaDB error code: 1593
 
2020-05-06 22:10:44 34 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 1634
 
2020-05-06 22:10:44 30 [Note] Slave I/O thread exiting, read up to log 'master-bin.000001', position 1955
 
2020-05-06 22:10:44 35 [Note] Slave I/O thread: Start asynchronous replication to master 'root@127.0.0.1:17920' in log 'master-bin.000001' at position 1955
 
2020-05-06 22:10:44 36 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000001' at position 1634, relay log './slave-relay-bin.000002' position: 1934
 
2020-05-06 22:10:44 35 [Note] Slave I/O thread: connected to master 'root@127.0.0.1:17920',replication started in log 'master-bin.000001' at position 1955
 
2020-05-06 22:10:45 36 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement accesses nontransactional table as well as transactional or temporary table, and writes to any of them Statement: update tm as t1, ti as t2 set t1.a=t1.a * 2, t2.a=t2.a * 2
 
2020-05-06 22:10:45 36 [ERROR] Slave SQL: Fatal error: ... Slave SQL Thread stopped with incomplete event group having non-transactional changes. If the group consists solely of row-based events, you can try to restart the slave with --slave-exec-mode=IDEMPOTENT, which ignores duplicate key, key not found, and similar errors (see documentation for details). Gtid 0-1-8, Internal MariaDB error code: 1593
 
2020-05-06 22:10:45 36 [Note] Slave SQL thread exiting, replication stopped in log 'master-bin.000001' at position 1955
 
200506 22:10:45 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.3.23-MariaDB-debug-log
 
key_buffer_size=1048576
 
read_buffer_size=131072
 
max_used_connections=4
 
max_threads=153
 
thread_count=12
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63285 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7fff2c000af0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fff47ffe5f8 thread_stack 0x49000
 
/opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld(my_print_stacktrace+0x44)[0x1226f9424]
 
mysys/stacktrace.c:269(my_print_stacktrace)[0x121b8151c]
 
addr2line: '': No such file
 
[0x7fff84db04d8]
 
/opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld(_db_doprnt_+0x8c)[0x12272ca14]
 
dbug/dbug.c:1311(_db_doprnt_)[0x1215dc140]
 
/opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld(_Z19close_thread_tablesP3THD+0x6c)[0x12169b7f0]
 
/opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld(_Z9reset_thdP3THD+0x28)[0x1216e5ef8]
 
/opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld(+0xe9a8f8)[0x121eaa8f8]
 
sql/sql_class.h:2333(THD::enter_stage(PSI_stage_info_v1 const*, char const*, char const*, unsigned int))[0x122106254]
 
sql/sql_class.cc:4744(reset_thd(THD*))[0x122046090]
 
handler/ha_innodb.cc:1665(innobase_reset_background_thd(THD*))[0x122046438]
 
row/row0purge.cc:1349(row_purge_step(que_thr_t*))[0x122046758]
 
que/que0que.cc:1039(que_thr_step(que_thr_t*))[0x122179114]
 
que/que0que.cc:1101(que_run_threads_low(que_thr_t*))[0x122179474]
 
pthread_create.c:0(start_thread)[0x7fff84d68b94]
 
/lib64/libc.so.6(clone+0xe4)[0x7fff845b85f4]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): 
Connection ID (thread ID): 4
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/var/96/mysqld.2/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             769110               769110               processes 
Max open files            32230                32230                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       769110               769110               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I ...
 
----------SERVER LOG END-------------
 
 
 
 
 
 - found 'core.19430' (0/0)
 
 
 
Trying 'dbx' to get a backtrace
 
 
 
Trying 'gdb' to get a backtrace from coredump /opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/var/96/log/rpl.rpl_stm_stop_middle_group-innodb,stmt/mysqld.2/data/core.19430
 
Core generated by '/opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld'
 
Output from gdb follows. The first stack trace is from the failing thread.
 
The following stack traces are from all threads (so the failing one is
 
duplicated).
 
--------------------------
 
[New LWP 19753]
 
[New LWP 19642]
 
[New LWP 19544]
 
[New LWP 19714]
 
[New LWP 19768]
 
[New LWP 19713]
 
[New LWP 19750]
 
[New LWP 19816]
 
[New LWP 20625]
 
[New LWP 19648]
 
[New LWP 19646]
 
[New LWP 19717]
 
[New LWP 19715]
 
[New LWP 19774]
 
[New LWP 19647]
 
[New LWP 19765]
 
[New LWP 19716]
 
[New LWP 29847]
 
[New LWP 19430]
 
[New LWP 19643]
 
[New LWP 19847]
 
[New LWP 19641]
 
[New LWP 19644]
 
[New LWP 19766]
 
[New LWP 19754]
 
[New LWP 19752]
 
[New LWP 20115]
 
[New LWP 19749]
 
[New LWP 19598]
 
[New LWP 19764]
 
[New LWP 20628]
 
[New LWP 19805]
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib64/libthread_db.so.1".
 
Core was generated by `/opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld --defaults-group-suffix=.2 --'.
 
Program terminated with signal 11, Segmentation fault.
 
#0  0x00007fff84d6fe7c in pthread_kill () from /lib64/libpthread.so.0
 
#0  0x00007fff84d6fe7c in pthread_kill () from /lib64/libpthread.so.0
 
#1  0x00000001226f9578 in my_write_core (sig=11) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/stacktrace.c:481
 
#2  0x0000000121b81a18 in handle_fatal_signal (sig=11) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/signal_handler.cc:343
 
#3  <signal handler called>
 
#4  0x000000012272ca64 in _db_doprnt_ (format=0x1227d1c78 "%s at %s:%d") at /opt/buildbot-slave/bld-p9-rhel7/build/dbug/dbug.c:1314
 
#5  0x00000001215dc140 in THD::enter_stage (this=0x7fff2c000af0, stage=0x1230112b8 <stage_closing_tables>, calling_func=0x1227f39e0 "<unknown>", calling_file=0x1227f3738 "/opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_base.cc", calling_line=728) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_class.h:2333
 
#6  0x000000012169b7f0 in close_thread_tables (thd=0x7fff2c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_base.cc:728
 
#7  0x00000001216e5ef8 in reset_thd (thd=0x7fff2c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_class.cc:4744
 
#8  0x0000000121eaa8f8 in innobase_reset_background_thd (thd=0x7fff2c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/handler/ha_innodb.cc:1665
 
#9  0x0000000122106254 in row_purge_step (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/row/row0purge.cc:1349
 
#10 0x0000000122046090 in que_thr_step (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/que/que0que.cc:1039
 
#11 0x0000000122046438 in que_run_threads_low (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/que/que0que.cc:1101
 
#12 0x0000000122046758 in que_run_threads (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/que/que0que.cc:1141
 
#13 0x0000000122179114 in srv_task_execute (slot=0x12304b110 <srv_sys+1296>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2482
 
#14 0x0000000122179474 in srv_worker_thread (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2537
 
#15 0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#16 0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 32 (Thread 0x7fff7d008090 (LWP 19805)):
 
#0  0x00007fff84d750f4 in sigwait () from /lib64/libpthread.so.0
 
#1  0x00000001215c8318 in signal_hand (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld.cc:3543
 
#2  0x000000012265e65c in pfs_spawn_thread (arg=0x10014c44780) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#3  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 31 (Thread 0x7fff7c4a8090 (LWP 20628)):
 
#0  0x00007fff845a78b8 in poll () from /lib64/libc.so.6
 
#1  0x000000012278f9c0 in vio_io_wait (vio=0x10014672f70, event=VIO_IO_EVENT_READ, timeout=28800000) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:962
 
#2  0x000000012278d68c in vio_socket_io_wait (vio=0x10014672f70, event=VIO_IO_EVENT_READ) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:119
 
#3  0x000000012278d950 in vio_read (vio=0x10014672f70, buf=0x7ffefc008d70 "\001", size=4) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:196
 
#4  0x00000001215e2c40 in my_real_read (net=0x7ffefc000dd8, complen=0x7fff7c4a7420, header=1 '\001') at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:962
 
#5  0x00000001215e3958 in my_net_read_packet_reallen (net=0x7ffefc000dd8, read_from_server=1 '\001', reallen=0x7fff7c4a74a8) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:1243
 
#6  0x00000001215e38c0 in my_net_read_packet (net=0x7ffefc000dd8, read_from_server=1 '\001') at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:1227
 
#7  0x0000000121765ed0 in do_command (thd=0x7ffefc000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_parse.cc:1277
 
#8  0x000000012194339c in do_handle_one_connection (connect=0x10014d041b0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1403
 
#9  0x0000000121942f98 in handle_one_connection (arg=0x10014d041b0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1308
 
#10 0x000000012265e65c in pfs_spawn_thread (arg=0x10014673450) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#11 0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#12 0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 30 (Thread 0x7fff46fdf090 (LWP 19764)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x1001479ced0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x1001479ced0, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x1001479ced0, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x00000001222d1168 in buf_dump_thread () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/buf/buf0dump.cc:834
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 29 (Thread 0x7fff7e6ff090 (LWP 19598)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000122700e0c in safe_cond_timedwait (cond=0x123abef90 <COND_checkpoint>, mp=0x123abeee0 <LOCK_checkpoint>, abstime=0x7fff7e6fe530, file=0x122b46c38 "/opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h", line=1211) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/thr_mutex.c:546
 
#2  0x00000001224d4fdc in inline_mysql_cond_timedwait (that=0x123abef90 <COND_checkpoint>, mutex=0x123abeee0 <LOCK_checkpoint>, abstime=0x7fff7e6fe530, src_file=0x122b46ca0 "/opt/buildbot-slave/bld-p9-rhel7/build/storage/maria/ma_servicethread.c", src_line=116) at /opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h:1211
 
#3  0x00000001224d574c in my_service_thread_sleep (control=0x1230512e0 <checkpoint_control>, sleep_time=29000000000) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/maria/ma_servicethread.c:115
 
#4  0x00000001224c4790 in ma_checkpoint_background (arg=0x1e) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/maria/ma_checkpoint.c:707
 
#5  0x000000012265e65c in pfs_spawn_thread (arg=0x10014686b10) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#6  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 28 (Thread 0x7fff51fbf090 (LWP 19749)):
 
#0  0x00007fff84d742b8 in nanosleep () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5884 in os_thread_sleep (tm=1000000) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0thread.cc:231
 
#2  0x000000012217883c in srv_master_sleep () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2351
 
#3  0x0000000122178a8c in srv_master_thread (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2392
 
#4  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#5  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 27 (Thread 0x7fff7c668090 (LWP 20115)):
 
#0  0x00007fff84d72da8 in __lll_lock_wait () from /lib64/libpthread.so.0
 
#1  0x00007fff84d6b78c in pthread_mutex_lock () from /lib64/libpthread.so.0
 
#2  0x00000001227294c4 in LockMutex (cs=0x7fff7c665870) at /opt/buildbot-slave/bld-p9-rhel7/build/dbug/dbug.c:336
 
#3  0x00000001227295fc in LockIfInitSettings (cs=0x7fff7c665870) at /opt/buildbot-slave/bld-p9-rhel7/build/dbug/dbug.c:349
 
#4  0x0000000122729b14 in DbugParse (cs=0x7fff7c665870, control=0x7fff0c011808 "") at /opt/buildbot-slave/bld-p9-rhel7/build/dbug/dbug.c:500
 
#5  0x000000012272adec in _db_set_init_ (control=0x7fff0c011808 "") at /opt/buildbot-slave/bld-p9-rhel7/build/dbug/dbug.c:902
 
#6  0x00000001219844b4 in Sys_var_dbug::global_update (this=0x123223e78 <Sys_debug_dbug>, thd=0x7fff0c000af0, var=0x7fff0c0117b8) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sys_vars.ic:915
 
#7  0x000000012160590c in sys_var::update (this=0x123223e78 <Sys_debug_dbug>, thd=0x7fff0c000af0, var=0x7fff0c0117b8) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/set_var.cc:208
 
#8  0x0000000121608214 in set_var::update (this=0x7fff0c0117b8, thd=0x7fff0c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/set_var.cc:837
 
#9  0x0000000121607cf8 in sql_set_variables (thd=0x7fff0c000af0, var_list=0x7fff0c0057b8, free=true) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/set_var.cc:740
 
#10 0x0000000121772c30 in mysql_execute_command (thd=0x7fff0c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_parse.cc:4883
 
#11 0x000000012177d914 in mysql_parse (thd=0x7fff0c000af0, rawbuf=0x7fff0c0115e8 "SET @@GLOBAL.debug_dbug = @saved_dbug", length=37, parser_state=0x7fff7c666cc8, is_com_multi=false, is_next_command=false) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_parse.cc:7817
 
#12 0x0000000121767f98 in dispatch_command (command=COM_QUERY, thd=0x7fff0c000af0, packet=0x7fff0c008d71 "SET @@GLOBAL.debug_dbug = @saved_dbug", packet_length=37, is_com_multi=false, is_next_command=false) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_parse.cc:1856
 
#13 0x00000001217665c0 in do_command (thd=0x7fff0c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_parse.cc:1402
 
#14 0x000000012194339c in do_handle_one_connection (connect=0x10014d041b0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1403
 
#15 0x0000000121942f98 in handle_one_connection (arg=0x10014d041b0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1308
 
#16 0x000000012265e65c in pfs_spawn_thread (arg=0x10014d0f480) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#17 0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#18 0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 26 (Thread 0x7fff50f9f090 (LWP 19752)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x1001479ccd0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x1001479ccd0, reset_sig_count=41) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x1001479ccd0, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000122173b2c in srv_resume_thread (slot=0x12304b010 <srv_sys+1040>, sig_count=0, wait=true, timeout_usec=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:922
 
#5  0x000000012217946c in srv_worker_thread (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2535
 
#6  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 25 (Thread 0x7fff477ef090 (LWP 19754)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x1001479cc50) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x1001479cc50, reset_sig_count=52) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x1001479cc50, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000122173b2c in srv_resume_thread (slot=0x12304af10 <srv_sys+784>, sig_count=0, wait=true, timeout_usec=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:922
 
#5  0x000000012217946c in srv_worker_thread (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2535
 
#6  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 24 (Thread 0x7fff45fbf090 (LWP 19766)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x1001479cfd0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x1001479cfd0, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x1001479cfd0, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x00000001222b00e8 in buf_resize_thread () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/buf/buf0buf.cc:3182
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 23 (Thread 0x7fff76fdf090 (LWP 19644)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x100147f55c0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x100147f55c0, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x100147f55c0, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000121feb93c in os_aio_simulated_handler (global_segment=3, m1=0x7fff76fde480, m2=0x7fff76fde488, type=0x7fff76fde4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:7301
 
#5  0x0000000121fe8e98 in os_aio_handler (segment=3, m1=0x7fff76fde480, m2=0x7fff76fde488, request=0x7fff76fde4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:5737
 
#6  0x0000000122393b50 in fil_aio_wait (segment=3) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/fil/fil0fil.cc:4432
 
#7  0x000000012217e120 in io_handler_thread (arg=0x1232a5720 <n+24>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0start.cc:326
 
#8  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#9  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 22 (Thread 0x7fff7ce7f090 (LWP 19641)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x100147f5440) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x100147f5440, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x100147f5440, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000121feb93c in os_aio_simulated_handler (global_segment=0, m1=0x7fff7ce7e480, m2=0x7fff7ce7e488, type=0x7fff7ce7e4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:7301
 
#5  0x0000000121fe8e98 in os_aio_handler (segment=0, m1=0x7fff7ce7e480, m2=0x7fff7ce7e488, request=0x7fff7ce7e4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:5737
 
#6  0x0000000122393b50 in fil_aio_wait (segment=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/fil/fil0fil.cc:4432
 
#7  0x000000012217e120 in io_handler_thread (arg=0x1232a5708 <n>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0start.cc:326
 
#8  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#9  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 21 (Thread 0x7fff7cf28090 (LWP 19847)):
 
#0  0x00007fff845a78b8 in poll () from /lib64/libc.so.6
 
#1  0x000000012278f9c0 in vio_io_wait (vio=0x10014673450, event=VIO_IO_EVENT_READ, timeout=28800000) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:962
 
#2  0x000000012278d68c in vio_socket_io_wait (vio=0x10014673450, event=VIO_IO_EVENT_READ) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:119
 
#3  0x000000012278d950 in vio_read (vio=0x10014673450, buf=0x7fff18169db0 "\001", size=4) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:196
 
#4  0x00000001215e2c40 in my_real_read (net=0x7fff18000dd8, complen=0x7fff7cf27420, header=1 '\001') at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:962
 
#5  0x00000001215e3958 in my_net_read_packet_reallen (net=0x7fff18000dd8, read_from_server=1 '\001', reallen=0x7fff7cf274a8) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:1243
 
#6  0x00000001215e38c0 in my_net_read_packet (net=0x7fff18000dd8, read_from_server=1 '\001') at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:1227
 
#7  0x0000000121765ed0 in do_command (thd=0x7fff18000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_parse.cc:1277
 
#8  0x000000012194339c in do_handle_one_connection (connect=0x10014d041b0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1403
 
#9  0x0000000121942f98 in handle_one_connection (arg=0x10014d041b0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1308
 
#10 0x000000012265e65c in pfs_spawn_thread (arg=0x10014c4b6d0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#11 0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#12 0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 20 (Thread 0x7fff777ef090 (LWP 19643)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x100147f5540) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x100147f5540, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x100147f5540, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000121feb93c in os_aio_simulated_handler (global_segment=2, m1=0x7fff777ee480, m2=0x7fff777ee488, type=0x7fff777ee4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:7301
 
#5  0x0000000121fe8e98 in os_aio_handler (segment=2, m1=0x7fff777ee480, m2=0x7fff777ee488, request=0x7fff777ee4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:5737
 
#6  0x0000000122393b50 in fil_aio_wait (segment=2) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/fil/fil0fil.cc:4432
 
#7  0x000000012217e120 in io_handler_thread (arg=0x1232a5718 <n+16>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0start.cc:326
 
#8  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#9  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 19 (Thread 0x7fff84e1a850 (LWP 19430)):
 
#0  0x00007fff845a78b8 in poll () from /lib64/libc.so.6
 
#1  0x00000001215d0d58 in handle_connections_sockets () at /opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld.cc:6759
 
#2  0x00000001215cfbd4 in mysqld_main (argc=159, argv=0x100144f8340) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/mysqld.cc:6228
 
#3  0x00000001215be730 in main (argc=27, argv=0x7fffd94693b8) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/main.cc:25
 
 
 
Thread 18 (Thread 0x7fff7c588090 (LWP 29847)):
 
#0  0x00007fff845a78b8 in poll () from /lib64/libc.so.6
 
#1  0x000000012278f9c0 in vio_io_wait (vio=0x7fff100169d0, event=VIO_IO_EVENT_READ, timeout=120000) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:962
 
#2  0x000000012278d68c in vio_socket_io_wait (vio=0x7fff100169d0, event=VIO_IO_EVENT_READ) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:119
 
#3  0x000000012278d950 in vio_read (vio=0x7fff100169d0, buf=0x7fff1001da80 "+", size=16384) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:196
 
#4  0x000000012278dbec in vio_read_buff (vio=0x7fff100169d0, buf=0x7fff10021b00 "", size=4) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:244
 
#5  0x00000001215e2c40 in my_real_read (net=0x7fff10016440, complen=0x7fff7c5872d0, header=0 '\000') at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:962
 
#6  0x00000001215e3958 in my_net_read_packet_reallen (net=0x7fff10016440, read_from_server=0 '\000', reallen=0x7fff7c587530) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:1243
 
#7  0x0000000121b0cf1c in cli_safe_read_reallen (mysql=0x7fff10016440, reallen=0x7fff7c587530) at /opt/buildbot-slave/bld-p9-rhel7/build/sql-common/client.c:594
 
#8  0x000000012161c48c in read_event (mysql=0x7fff10016440, mi=0x10014d09370, suppress_warnings=0x7fff7c587506, network_read_len=0x7fff7c587530) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/slave.cc:3689
 
#9  0x000000012161fd88 in handle_slave_io (arg=0x10014d09370) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/slave.cc:4843
 
#10 0x000000012265e65c in pfs_spawn_thread (arg=0x7fff0c05ee40) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#11 0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#12 0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 17 (Thread 0x7fff52fdf090 (LWP 19716)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff46d0 in os_event::timed_wait (this=0x100149cebc0, abstime=0x7fff52fde610) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:275
 
#2  0x0000000121ff4a58 in os_event::wait_time_low (this=0x100149cebc0, time_in_usec=10000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:385
 
#3  0x0000000121ff4dc0 in os_event_wait_time_low (event=0x100149cebc0, time_in_usec=10000000, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:485
 
#4  0x0000000122372258 in dict_stats_thread () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/dict/dict0stats_bg.cc:476
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 16 (Thread 0x7fff467cf090 (LWP 19765)):
 
#0  0x00007fff84d742b8 in nanosleep () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5884 in os_thread_sleep (tm=1000000) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0thread.cc:231
 
#2  0x000000012229a548 in btr_defragment_thread () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/btr/btr0defragment.cc:708
 
#3  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#4  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 15 (Thread 0x7fff75fbf090 (LWP 19647)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x100147f56c0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x100147f56c0, reset_sig_count=4) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x100147f56c0, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000121feb93c in os_aio_simulated_handler (global_segment=5, m1=0x7fff75fbe480, m2=0x7fff75fbe488, type=0x7fff75fbe4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:7301
 
#5  0x0000000121fe8e98 in os_aio_handler (segment=5, m1=0x7fff75fbe480, m2=0x7fff75fbe488, request=0x7fff75fbe4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:5737
 
#6  0x0000000122393b50 in fil_aio_wait (segment=5) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/fil/fil0fil.cc:4432
 
#7  0x000000012217e120 in io_handler_thread (arg=0x1232a5730 <n+40>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0start.cc:326
 
#8  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#9  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 14 (Thread 0x7fff7d078090 (LWP 19774)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000122700a94 in safe_cond_wait (cond=0x123241528 <mysql_bin_log+4232>, mp=0x123241478 <mysql_bin_log+4056>, file=0x1229b0438 "/opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h", line=1174) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/thr_mutex.c:492
 
#2  0x0000000121d340a8 in inline_mysql_cond_wait (that=0x123241528 <mysql_bin_log+4232>, mutex=0x123241478 <mysql_bin_log+4056>, src_file=0x1229b0ca0 "/opt/buildbot-slave/bld-p9-rhel7/build/sql/log.cc", src_line=9945) at /opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h:1174
 
#3  0x0000000121d591f0 in binlog_background_thread (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/log.cc:9945
 
#4  0x000000012265e65c in pfs_spawn_thread (arg=0x10014c10770) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 13 (Thread 0x7fff537ef090 (LWP 19715)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff46d0 in os_event::timed_wait (this=0x1001479ce50, abstime=0x7fff537ee5d0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:275
 
#2  0x0000000121ff4a58 in os_event::wait_time_low (this=0x1001479ce50, time_in_usec=5000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:385
 
#3  0x0000000121ff4dc0 in os_event_wait_time_low (event=0x1001479ce50, time_in_usec=5000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:485
 
#4  0x0000000122176480 in srv_monitor_thread () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:1701
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 12 (Thread 0x7fff527cf090 (LWP 19717)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff46d0 in os_event::timed_wait (this=0x100148fd350, abstime=0x7fff527ce400) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:275
 
#2  0x0000000121ff4a58 in os_event::wait_time_low (this=0x100148fd350, time_in_usec=5000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:385
 
#3  0x0000000121ff4dc0 in os_event_wait_time_low (event=0x100148fd350, time_in_usec=5000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:485
 
#4  0x00000001222137f0 in ib_wqueue_timedwait (wq=0x100148fd2c0, wait_in_usecs=5000000) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/ut/ut0wqueue.cc:156
 
#5  0x00000001223f8a18 in fts_optimize_thread (arg=0x100148fd2c0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/fts/fts0opt.cc:2834
 
#6  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 11 (Thread 0x7fff767cf090 (LWP 19646)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x100147f5640) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x100147f5640, reset_sig_count=7) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x100147f5640, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000121feb93c in os_aio_simulated_handler (global_segment=4, m1=0x7fff767ce480, m2=0x7fff767ce488, type=0x7fff767ce4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:7301
 
#5  0x0000000121fe8e98 in os_aio_handler (segment=4, m1=0x7fff767ce480, m2=0x7fff767ce488, request=0x7fff767ce4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:5737
 
#6  0x0000000122393b50 in fil_aio_wait (segment=4) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/fil/fil0fil.cc:4432
 
#7  0x000000012217e120 in io_handler_thread (arg=0x1232a5728 <n+32>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0start.cc:326
 
#8  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#9  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 10 (Thread 0x7fff757af090 (LWP 19648)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff46d0 in os_event::timed_wait (this=0x1001479cf50, abstime=0x7fff757ae390) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:275
 
#2  0x0000000121ff4a58 in os_event::wait_time_low (this=0x1001479cf50, time_in_usec=942000, reset_sig_count=2) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:385
 
#3  0x0000000121ff4dc0 in os_event_wait_time_low (event=0x1001479cf50, time_in_usec=942000, reset_sig_count=2) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:485
 
#4  0x00000001222e1114 in pc_sleep_if_needed (next_loop_time=18780102006, sig_count=2, cur_time=18780101064) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/buf/buf0flu.cc:2689
 
#5  0x00000001222e2a6c in buf_flush_page_cleaner_coordinator () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/buf/buf0flu.cc:3148
 
#6  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 9 (Thread 0x7fff7c518090 (LWP 20625)):
 
#0  0x00007fff845a78b8 in poll () from /lib64/libc.so.6
 
#1  0x000000012278f9c0 in vio_io_wait (vio=0x10014d0f480, event=VIO_IO_EVENT_READ, timeout=28800000) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:962
 
#2  0x000000012278d68c in vio_socket_io_wait (vio=0x10014d0f480, event=VIO_IO_EVENT_READ) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:119
 
#3  0x000000012278d950 in vio_read (vio=0x10014d0f480, buf=0x7fff08008d70 "\001", size=4) at /opt/buildbot-slave/bld-p9-rhel7/build/vio/viosocket.c:196
 
#4  0x00000001215e2c40 in my_real_read (net=0x7fff08000dd8, complen=0x7fff7c517420, header=1 '\001') at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:962
 
#5  0x00000001215e3958 in my_net_read_packet_reallen (net=0x7fff08000dd8, read_from_server=1 '\001', reallen=0x7fff7c5174a8) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:1243
 
#6  0x00000001215e38c0 in my_net_read_packet (net=0x7fff08000dd8, read_from_server=1 '\001') at /opt/buildbot-slave/bld-p9-rhel7/build/sql/net_serv.cc:1227
 
#7  0x0000000121765ed0 in do_command (thd=0x7fff08000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_parse.cc:1277
 
#8  0x000000012194339c in do_handle_one_connection (connect=0x10014ceab50) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1403
 
#9  0x0000000121942f98 in handle_one_connection (arg=0x10014ceab50) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_connect.cc:1308
 
#10 0x000000012265e65c in pfs_spawn_thread (arg=0x10014672f70) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#11 0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#12 0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 8 (Thread 0x7fff7cf98090 (LWP 19816)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000122700a94 in safe_cond_wait (cond=0x12320eb18 <COND_slave_background>, mp=0x123211d78 <LOCK_slave_background>, file=0x1227e1c88 "/opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h", line=1174) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/thr_mutex.c:492
 
#2  0x000000012160da48 in inline_mysql_cond_wait (that=0x12320eb18 <COND_slave_background>, mutex=0x123211d78 <LOCK_slave_background>, src_file=0x1227e2888 "/opt/buildbot-slave/bld-p9-rhel7/build/sql/slave.cc", src_line=516) at /opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h:1174
 
#3  0x000000012160fc78 in handle_slave_background (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/slave.cc:516
 
#4  0x000000012265e65c in pfs_spawn_thread (arg=0x10014cef170) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 7 (Thread 0x7fff517af090 (LWP 19750)):
 
#0  0x00007fff84594560 in sched_yield () from /lib64/libc.so.6
 
#1  0x0000000121ff57ec in os_thread_yield () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0thread.cc:212
 
#2  0x00000001221bef14 in trx_purge_wait_for_workers_to_complete () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/trx/trx0purge.cc:1548
 
#3  0x00000001221bf244 in trx_purge (n_purge_threads=4, truncate=false, slot=0x12304ae10 <srv_sys+528>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/trx/trx0purge.cc:1611
 
#4  0x000000012217991c in srv_do_purge (n_total_purged=0x7fff517ae670, slot=0x12304ae10 <srv_sys+528>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2641
 
#5  0x000000012217a06c in srv_purge_coordinator_thread (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2773
 
#6  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#7  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 6 (Thread 0x7fff74f9f090 (LWP 19713)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff46d0 in os_event::timed_wait (this=0x1001499cad0, abstime=0x7fff74f9e5f0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:275
 
#2  0x0000000121ff4a58 in os_event::wait_time_low (this=0x1001499cad0, time_in_usec=1000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:385
 
#3  0x0000000121ff4dc0 in os_event_wait_time_low (event=0x1001499cad0, time_in_usec=1000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:485
 
#4  0x0000000121fa4ccc in lock_wait_timeout_thread () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/lock/lock0wait.cc:522
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 5 (Thread 0x7fff457af090 (LWP 19768)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000122700a94 in safe_cond_wait (cond=0x7fff457ae550, mp=0x7fff457ae588, file=0x1229dd7c8 "/opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h", line=1185) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/thr_mutex.c:492
 
#2  0x0000000121ea05ec in inline_mysql_cond_wait (that=0x7fff457ae550, mutex=0x7fff457ae588, src_file=0x1229df5c8 "/opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/handler/ha_innodb.cc", src_line=305) at /opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h:1185
 
#3  0x0000000121ea9770 in thd_destructor_proxy () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/handler/ha_innodb.cc:305
 
#4  0x000000012265e65c in pfs_spawn_thread (arg=0x10014c05e90) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 4 (Thread 0x7fff53fff090 (LWP 19714)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff46d0 in os_event::timed_wait (this=0x1001479cdd0, abstime=0x7fff53ffe420) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:275
 
#2  0x0000000121ff4a58 in os_event::wait_time_low (this=0x1001479cdd0, time_in_usec=1000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:385
 
#3  0x0000000121ff4dc0 in os_event_wait_time_low (event=0x1001479cdd0, time_in_usec=1000000, reset_sig_count=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:485
 
#4  0x0000000122176b94 in srv_error_monitor_thread () at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:1865
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 3 (Thread 0x7fff7faaf090 (LWP 19544)):
 
#0  0x00007fff84d6ebf4 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000122700e0c in safe_cond_timedwait (cond=0x123ad0788 <COND_timer>, mp=0x123ad06d8 <LOCK_timer>, abstime=0x7fff7faae630, file=0x122b8a920 "/opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h", line=1211) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/thr_mutex.c:546
 
#2  0x00000001227026ec in inline_mysql_cond_timedwait (that=0x123ad0788 <COND_timer>, mutex=0x123ad06d8 <LOCK_timer>, abstime=0x7fff7faae630, src_file=0x122b8a978 "/opt/buildbot-slave/bld-p9-rhel7/build/mysys/thr_timer.c", src_line=292) at /opt/buildbot-slave/bld-p9-rhel7/build/include/mysql/psi/mysql_thread.h:1211
 
#3  0x00000001227037b0 in timer_handler (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/thr_timer.c:292
 
#4  0x000000012265e65c in pfs_spawn_thread (arg=0x100146709c0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/perfschema/pfs.cc:1869
 
#5  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#6  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 2 (Thread 0x7fff77fff090 (LWP 19642)):
 
#0  0x00007fff84d6e7fc in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
 
#1  0x0000000121ff5214 in os_event::wait (this=0x100147f54c0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:158
 
#2  0x0000000121ff48a0 in os_event::wait_low (this=0x100147f54c0, reset_sig_count=3) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:325
 
#3  0x0000000121ff4e18 in os_event_wait_low (event=0x100147f54c0, reset_sig_count=0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0event.cc:502
 
#4  0x0000000121feb93c in os_aio_simulated_handler (global_segment=1, m1=0x7fff77ffe480, m2=0x7fff77ffe488, type=0x7fff77ffe4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:7301
 
#5  0x0000000121fe8e98 in os_aio_handler (segment=1, m1=0x7fff77ffe480, m2=0x7fff77ffe488, request=0x7fff77ffe4b8) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/os/os0file.cc:5737
 
#6  0x0000000122393b50 in fil_aio_wait (segment=1) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/fil/fil0fil.cc:4432
 
#7  0x000000012217e120 in io_handler_thread (arg=0x1232a5710 <n+8>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0start.cc:326
 
#8  0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#9  0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Thread 1 (Thread 0x7fff47fff090 (LWP 19753)):
 
#0  0x00007fff84d6fe7c in pthread_kill () from /lib64/libpthread.so.0
 
#1  0x00000001226f9578 in my_write_core (sig=11) at /opt/buildbot-slave/bld-p9-rhel7/build/mysys/stacktrace.c:481
 
#2  0x0000000121b81a18 in handle_fatal_signal (sig=11) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/signal_handler.cc:343
 
#3  <signal handler called>
 
#4  0x000000012272ca64 in _db_doprnt_ (format=0x1227d1c78 "%s at %s:%d") at /opt/buildbot-slave/bld-p9-rhel7/build/dbug/dbug.c:1314
 
#5  0x00000001215dc140 in THD::enter_stage (this=0x7fff2c000af0, stage=0x1230112b8 <stage_closing_tables>, calling_func=0x1227f39e0 "<unknown>", calling_file=0x1227f3738 "/opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_base.cc", calling_line=728) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_class.h:2333
 
#6  0x000000012169b7f0 in close_thread_tables (thd=0x7fff2c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_base.cc:728
 
#7  0x00000001216e5ef8 in reset_thd (thd=0x7fff2c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/sql/sql_class.cc:4744
 
#8  0x0000000121eaa8f8 in innobase_reset_background_thd (thd=0x7fff2c000af0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/handler/ha_innodb.cc:1665
 
#9  0x0000000122106254 in row_purge_step (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/row/row0purge.cc:1349
 
#10 0x0000000122046090 in que_thr_step (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/que/que0que.cc:1039
 
#11 0x0000000122046438 in que_run_threads_low (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/que/que0que.cc:1101
 
#12 0x0000000122046758 in que_run_threads (thr=0x10014be53b0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/que/que0que.cc:1141
 
#13 0x0000000122179114 in srv_task_execute (slot=0x12304b110 <srv_sys+1296>) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2482
 
#14 0x0000000122179474 in srv_worker_thread (arg=0x0) at /opt/buildbot-slave/bld-p9-rhel7/build/storage/innobase/srv/srv0srv.cc:2537
 
#15 0x00007fff84d68b94 in start_thread () from /lib64/libpthread.so.0
 
#16 0x00007fff845b85f4 in clone () from /lib64/libc.so.6
 
 
 
Compressed file /opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/var/96/log/rpl.rpl_stm_stop_middle_group-innodb,stmt/mysqld.2/data/core.19430
 
 - skipping '/opt/buildbot-slave/bld-p9-rhel7/build/mysql-test/var/96/log/rpl.rpl_stm_stop_middle_group-innodb,stmt/'

Comment by Marko Mäkelä [ 2022-11-08 ]

alice, your finding looks like MDEV-23371. The original crash could be due to a bug in DBUG_PRINT. With MDEV-29613, configuring the debug server server with cmake -DWITH_DBUG_TRACE=OFF should get rid of this type of crashes.

I think that something is broken in DBUG_PRINT and possibly also in DBUG_ENTER and DBUG_RETURN. Replication tests are notorious for enabling the DBUG spam mode (MDEV-19455, MDEV-21361).

Generated at Thu Feb 08 08:08:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.