sys_vars.innodb_fatal_semaphore_wait_threshold fails sporadically on p8-xenial-bintar-debug in buildbot

sys_vars.innodb_fatal_semaphore_wait_threshold 'innodb_plugin' w1 [ fail ]

        Test ended at 2016-08-02 23:56:32

CURRENT_TEST: sys_vars.innodb_fatal_semaphore_wait_threshold

mysqltest: At line 117: query 'drop table if exists t1' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:

< snip >

create table t1 (id integer, x integer) engine = InnoDB;

insert into t1 values(0, 0);

set DEBUG_DBUG='+d,fatal-semaphore-timeout';

set autocommit=0;

# Sending query on con1,

# the session will hold lock table mutex and sleep

SELECT * from t1 where id = 0 FOR UPDATE;

# Switch to connection con2

set autocommit=0;

# Sending query on con2,

# the session will be blocked on the lock table mutex and

# thus be put into sync arry

SELECT * from t1 where id = 0 FOR UPDATE;

# Switched to the default connection

# Waitting for mysqld to crash

# Mysqld crash was detected

# Waitting for reconnect after mysqld restarts

# Reconnected after mysqld was successfully restarted

# Cleaning up before exit

drop table if exists t1;

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace

Core generated by '/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 11594]

[New LWP 11578]

[New LWP 11562]

[New LWP 11579]

[New LWP 11581]

[New LWP 11570]

[New LWP 11595]

[New LWP 11571]

[New LWP 11597]

[New LWP 11576]

[New LWP 11599]

[New LWP 11577]

[New LWP 11580]

[New LWP 11611]

[New LWP 11593]

[New LWP 11596]

[New LWP 11613]

[New LWP 11598]

[New LWP 11600]

[New LWP 11601]

[New LWP 11602]

[New LWP 11603]

[New LWP 11612]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Core was generated by `/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/mysqld --default'.

Program terminated with signal SIGABRT, Aborted.

#0  0x00003fff88bf2e4c in pthread_kill () from /lib/powerpc64le-linux-gnu/libpthread.so.0

[Current thread is 1 (Thread 0x3fff79bfefb0 (LWP 11594))]

#0  0x00003fff88bf2e4c in pthread_kill () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x000000005fa7b348 in my_write_core (sig=6) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/mysys/stacktrace.c:477

#2  0x000000005f20780c in handle_fatal_signal (sig=6) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/signal_handler.cc:294

#3  <signal handler called>

#4  0x00003fff8864f27c in raise () from /lib/powerpc64le-linux-gnu/libc.so.6

#5  0x00003fff886518f4 in abort () from /lib/powerpc64le-linux-gnu/libc.so.6

#6  0x00003fff8125c078 in srv_error_monitor_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0srv.cc:1958

#7  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#8  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 23 (Thread 0x3fff80d073b0 (LWP 11612)):

#0  0x00003fff88729638 in select () from /lib/powerpc64le-linux-gnu/libc.so.6

#1  0x00003fff8114b184 in os_thread_sleep (tm=3600000000) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0thread.cc:261

#2  0x00003fff81107600 in lock_table (flags=0, table=0x3fff76bd03f8, mode=LOCK_IX, thr=0x3fff72441a08) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/lock/lock0lock.cc:5027

#3  0x00003fff81227ffc in row_search_for_mysql (buf=0x3fff74170188 "\377", mode=1, prebuilt=0x3fff72441078, match_mode=0, direction=0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/row/row0sel.cc:4108

#4  0x00003fff8108e7c4 in ha_innodb::index_read (this=0x3fff76be1088, buf=0x3fff74170188 "\377", key_ptr=0x0, key_len=0, find_flag=HA_READ_AFTER_KEY) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:9207

#5  0x00003fff8108f98c in ha_innodb::index_first (this=0x3fff76be1088, buf=0x3fff74170188 "\377") at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:9615

#6  0x00003fff8108fce0 in ha_innodb::rnd_next (this=0x3fff76be1088, buf=0x3fff74170188 "\377") at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:9712

#7  0x000000005f20fd3c in handler::ha_rnd_next (this=0x3fff76be1088, buf=0x3fff74170188 "\377") at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/handler.cc:2577

#8  0x000000005f3d36c0 in rr_sequential (info=0x3fff74018848) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/records.cc:470

#9  0x000000005efbbd08 in join_init_read_record (tab=0x3fff74018788) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:19207

#10 0x000000005efb9244 in sub_select (join=0x3fff74016c60, join_tab=0x3fff74018788, end_of_records=false) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:18292

#11 0x000000005efb8864 in do_select (join=0x3fff74016c60, fields=0x3fff82a962a0, table=0x0, procedure=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:17950

#12 0x000000005ef8f898 in JOIN::exec_inner (this=0x3fff74016c60) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:3214

#13 0x000000005ef8d01c in JOIN::exec (this=0x3fff74016c60) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:2504

#14 0x000000005ef901f8 in mysql_select (thd=0x3fff82a92070, rref_pointer_array=0x3fff82a96400, tables=0x3fff74016288, wild_num=1, fields=..., conds=0x3fff74016a18, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2148272640, result=0x3fff74016c40, unit=0x3fff82a95a88, select_lex=0x3fff82a96188) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:3437

#15 0x000000005ef86330 in handle_select (thd=0x3fff82a92070, lex=0x3fff82a959c0, result=0x3fff74016c40, setup_tables_done_option=0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:384

#16 0x000000005ef48f3c in execute_sqlcom_select (thd=0x3fff82a92070, all_tables=0x3fff74016288) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:5894

#17 0x000000005ef3ec88 in mysql_execute_command (thd=0x3fff82a92070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:2960

#18 0x000000005ef4cd30 in mysql_parse (thd=0x3fff82a92070, rawbuf=0x3fff74016088 "SELECT * from t1 where id = 0 FOR UPDATE", length=40, parser_state=0x3fff80d05bd0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:7314

#19 0x000000005ef3a6fc in dispatch_command (command=COM_QUERY, thd=0x3fff82a92070, packet=0x3fff87713071 "", packet_length=40) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:1486

#20 0x000000005ef393c8 in do_command (thd=0x3fff82a92070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:1107

#21 0x000000005f0af4a4 in do_handle_one_connection (thd_arg=0x3fff82a92070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_connect.cc:1350

#22 0x000000005f0af100 in handle_one_connection (arg=0x3fff82a92070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_connect.cc:1262

#23 0x000000005f887340 in pfs_spawn_thread (arg=0x3fff7fb2c1f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/perfschema/pfs.cc:1860

#24 0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#25 0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 22 (Thread 0x3fff80dc73b0 (LWP 11603)):

#0  0x00003fff88bf8514 in sigwait () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x000000005ee3539c in signal_hand (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/mysqld.cc:3474

#2  0x000000005f887340 in pfs_spawn_thread (arg=0x3fff7fb29770) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/perfschema/pfs.cc:1860

#3  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#4  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 21 (Thread 0x3fff74bfefb0 (LWP 11602)):

#0  0x00003fff88729638 in select () from /lib/powerpc64le-linux-gnu/libc.so.6

#1  0x00003fff8114b184 in os_thread_sleep (tm=1000000) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0thread.cc:261

#2  0x00003fff80f6277c in btr_defragment_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/btr/btr0defragment.cc:758

#3  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#4  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 20 (Thread 0x3fff753fefb0 (LWP 11601)):

#0  0x00003fff88bf17e0 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114932c in os_cond_wait_timed (cond=0x3fff84bfa9c0, fast_mutex=0x3fff84bfa980, abstime=0x3fff753fe3e0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:177

#2  0x00003fff81149f84 in os_event_wait_time_low (event=0x3fff84bfa980, time_in_usec=5000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:705

#3  0x00003fff812b9da0 in ib_wqueue_timedwait (wq=0x3fff87ef0df8, wait_in_usecs=5000000) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/ut/ut0wqueue.cc:154

#4  0x00003fff81065294 in fts_optimize_thread (arg=0x3fff87ef0df8) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/fts/fts0opt.cc:3064

#5  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#6  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 19 (Thread 0x3fff75bfefb0 (LWP 11600)):

#0  0x00003fff88bf17e0 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114932c in os_cond_wait_timed (cond=0x3fff84bf4e40, fast_mutex=0x3fff84bf4e00, abstime=0x3fff75bfe4f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:177

#2  0x00003fff81149f84 in os_event_wait_time_low (event=0x3fff84bf4e00, time_in_usec=10000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:705

#3  0x00003fff80ffd458 in dict_stats_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/dict/dict0stats_bg.cc:545

#4  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#5  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 18 (Thread 0x3fff7abfefb0 (LWP 11598)):

#0  0x00003fff88bf17e0 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114932c in os_cond_wait_timed (cond=0x3fff84bf42c0, fast_mutex=0x3fff84bf4280, abstime=0x3fff7abfe450) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:177

#2  0x00003fff81149f84 in os_event_wait_time_low (event=0x3fff84bf4280, time_in_usec=1000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:705

#3  0x00003fff80f91f60 in page_cleaner_sleep_if_needed (next_loop_time=1470196569887) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/buf/buf0flu.cc:2274

#4  0x00003fff80f9209c in buf_flush_page_cleaner_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/buf/buf0flu.cc:2310

#5  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#6  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 17 (Thread 0x3fff80ca73b0 (LWP 11613)):

#0  0x00003fff88bf127c in pthread_cond_wait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114948c in os_cond_wait (cond=0x3fff84bf4640, fast_mutex=0x3fff84bf4600) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:214

#2  0x00003fff81149d64 in os_event_wait_low (event=0x3fff84bf4600, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:592

#3  0x00003fff81269860 in sync_array_wait_event (arr=0x3fff87f30640, index=0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/sync/sync0arr.cc:465

#4  0x00003fff81271520 in mutex_spin_wait (mutex=0x3fff81f20078, file_name=0x3fff813179b8 "/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/lock/lock0lock.cc", line=5025) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/sync/sync0sync.cc:604

#5  0x00003fff810f38bc in mutex_enter_func (mutex=0x3fff81f20078, file_name=0x3fff813179b8 "/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/lock/lock0lock.cc", line=5025) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/include/sync0sync.ic:225

#6  0x00003fff810f397c in pfs_mutex_enter_func (mutex=0x3fff81f20078, file_name=0x3fff813179b8 "/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/lock/lock0lock.cc", line=5025) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/include/sync0sync.ic:251

#7  0x00003fff811075bc in lock_table (flags=0, table=0x3fff76bd03f8, mode=LOCK_IX, thr=0x3fff72f81708) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/lock/lock0lock.cc:5025

#8  0x00003fff81227ffc in row_search_for_mysql (buf=0x3fff72d81a88 "\377", mode=1, prebuilt=0x3fff72f81078, match_mode=0, direction=0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/row/row0sel.cc:4108

#9  0x00003fff8108e7c4 in ha_innodb::index_read (this=0x3fff72d62088, buf=0x3fff72d81a88 "\377", key_ptr=0x0, key_len=0, find_flag=HA_READ_AFTER_KEY) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:9207

#10 0x00003fff8108f98c in ha_innodb::index_first (this=0x3fff72d62088, buf=0x3fff72d81a88 "\377") at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:9615

#11 0x00003fff8108fce0 in ha_innodb::rnd_next (this=0x3fff72d62088, buf=0x3fff72d81a88 "\377") at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/handler/ha_innodb.cc:9712

#12 0x000000005f20fd3c in handler::ha_rnd_next (this=0x3fff72d62088, buf=0x3fff72d81a88 "\377") at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/handler.cc:2577

#13 0x000000005f3d36c0 in rr_sequential (info=0x3fff72c18848) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/records.cc:470

#14 0x000000005efbbd08 in join_init_read_record (tab=0x3fff72c18788) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:19207

#15 0x000000005efb9244 in sub_select (join=0x3fff72c16c60, join_tab=0x3fff72c18788, end_of_records=false) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:18292

#16 0x000000005efb8864 in do_select (join=0x3fff72c16c60, fields=0x3fff82a9c2a0, table=0x0, procedure=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:17950

#17 0x000000005ef8f898 in JOIN::exec_inner (this=0x3fff72c16c60) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:3214

#18 0x000000005ef8d01c in JOIN::exec (this=0x3fff72c16c60) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:2504

#19 0x000000005ef901f8 in mysql_select (thd=0x3fff82a98070, rref_pointer_array=0x3fff82a9c400, tables=0x3fff72c16288, wild_num=1, fields=..., conds=0x3fff72c16a18, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2148272640, result=0x3fff72c16c40, unit=0x3fff82a9ba88, select_lex=0x3fff82a9c188) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:3437

#20 0x000000005ef86330 in handle_select (thd=0x3fff82a98070, lex=0x3fff82a9b9c0, result=0x3fff72c16c40, setup_tables_done_option=0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_select.cc:384

#21 0x000000005ef48f3c in execute_sqlcom_select (thd=0x3fff82a98070, all_tables=0x3fff72c16288) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:5894

#22 0x000000005ef3ec88 in mysql_execute_command (thd=0x3fff82a98070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:2960

#23 0x000000005ef4cd30 in mysql_parse (thd=0x3fff82a98070, rawbuf=0x3fff72c16088 "SELECT * from t1 where id = 0 FOR UPDATE", length=40, parser_state=0x3fff80ca5bd0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:7314

#24 0x000000005ef3a6fc in dispatch_command (command=COM_QUERY, thd=0x3fff82a98070, packet=0x3fff87718071 "", packet_length=40) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:1486

#25 0x000000005ef393c8 in do_command (thd=0x3fff82a98070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:1107

#26 0x000000005f0af4a4 in do_handle_one_connection (thd_arg=0x3fff82a98070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_connect.cc:1350

#27 0x000000005f0af100 in handle_one_connection (arg=0x3fff82a98070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_connect.cc:1262

#28 0x000000005f887340 in pfs_spawn_thread (arg=0x3fff7fb2c6f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/perfschema/pfs.cc:1860

#29 0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#30 0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 16 (Thread 0x3fff78bfefb0 (LWP 11596)):

#0  0x00003fff88729638 in select () from /lib/powerpc64le-linux-gnu/libc.so.6

#1  0x00003fff8114b184 in os_thread_sleep (tm=1000000) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0thread.cc:261

#2  0x00003fff8125d9e0 in srv_master_sleep () at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0srv.cc:2514

#3  0x00003fff8125dbac in srv_master_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0srv.cc:2558

#4  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#5  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 15 (Thread 0x3fff7a3fefb0 (LWP 11593)):

#0  0x00003fff88bf17e0 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114932c in os_cond_wait_timed (cond=0x3fff84bf4740, fast_mutex=0x3fff84bf4700, abstime=0x3fff7a3fe4d0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:177

#2  0x00003fff81149f84 in os_event_wait_time_low (event=0x3fff84bf4700, time_in_usec=1000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:705

#3  0x00003fff81113c68 in lock_wait_timeout_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/lock/lock0wait.cc:536

#4  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#5  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 14 (Thread 0x3fff80d673b0 (LWP 11611)):

#0  0x00003fff88726818 in poll () from /lib/powerpc64le-linux-gnu/libc.so.6

#1  0x000000005faf5d04 in vio_io_wait (vio=0x3fff7fb2ba70, event=VIO_IO_EVENT_READ, timeout=28800000) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/vio/viosocket.c:972

#2  0x000000005faf3f54 in vio_socket_io_wait (vio=0x3fff7fb2ba70, event=VIO_IO_EVENT_READ) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/vio/viosocket.c:108

#3  0x000000005faf4190 in vio_read (vio=0x3fff7fb2ba70, buf=0x3fff8770e070 "\001", size=4) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/vio/viosocket.c:184

#4  0x000000005ee491ec in my_real_read (net=0x3fff82a8c330, complen=0x3fff80d66320, header=1 '\001') at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/net_serv.cc:873

#5  0x000000005ee49c30 in my_net_read_packet (net=0x3fff82a8c330, read_from_server=1 '\001') at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/net_serv.cc:1099

#6  0x000000005ef38e58 in do_command (thd=0x3fff82a8c070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_parse.cc:984

#7  0x000000005f0af4a4 in do_handle_one_connection (thd_arg=0x3fff82a8c070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_connect.cc:1350

#8  0x000000005f0af100 in handle_one_connection (arg=0x3fff82a8c070) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/sql_connect.cc:1262

#9  0x000000005f887340 in pfs_spawn_thread (arg=0x3fff7fb2bcf0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/perfschema/pfs.cc:1860

#10 0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#11 0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 13 (Thread 0x3fff7c38efb0 (LWP 11580)):

#0  0x00003fff890f0738 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1

#1  0x00003fff890f07a0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1

#2  0x00003fff81146124 in os_aio_linux_collect (array=0x3fff87f701f0, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5326

#3  0x00003fff811465c4 in os_aio_linux_handle (global_seg=4, message1=0x3fff7c38e4e0, message2=0x3fff7c38e4e8, type=0x3fff7c38e4f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5478

#4  0x00003fff810187d4 in fil_aio_wait (segment=4) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/fil/fil0fil.cc:6045

#5  0x00003fff81262e08 in io_handler_thread (arg=0x3fff813de460 <n+32>) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0start.cc:515

#6  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#7  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 12 (Thread 0x3fff7db8efb0 (LWP 11577)):

#0  0x00003fff890f0738 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1

#1  0x00003fff890f07a0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1

#2  0x00003fff81146124 in os_aio_linux_collect (array=0x3fff87f700b0, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5326

#3  0x00003fff811465c4 in os_aio_linux_handle (global_seg=1, message1=0x3fff7db8e4e0, message2=0x3fff7db8e4e8, type=0x3fff7db8e4f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5478

#4  0x00003fff810187d4 in fil_aio_wait (segment=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/fil/fil0fil.cc:6045

#5  0x00003fff81262e08 in io_handler_thread (arg=0x3fff813de448 <n+8>) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0start.cc:515

#6  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#7  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 11 (Thread 0x3fff763fefb0 (LWP 11599)):

#0  0x00003fff88bf127c in pthread_cond_wait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114948c in os_cond_wait (cond=0x3fff87e60e40, fast_mutex=0x3fff87e60e00) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:214

#2  0x00003fff81149d64 in os_event_wait_low (event=0x3fff87e60e00, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:592

#3  0x00003fff80f883a0 in buf_dump_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/buf/buf0dump.cc:703

#4  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#5  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 10 (Thread 0x3fff7e38efb0 (LWP 11576)):

#0  0x00003fff890f0738 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1

#1  0x00003fff890f07a0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1

#2  0x00003fff81146124 in os_aio_linux_collect (array=0x3fff87f70100, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5326

#3  0x00003fff811465c4 in os_aio_linux_handle (global_seg=0, message1=0x3fff7e38e4e0, message2=0x3fff7e38e4e8, type=0x3fff7e38e4f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5478

#4  0x00003fff810187d4 in fil_aio_wait (segment=0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/fil/fil0fil.cc:6045

#5  0x00003fff81262e08 in io_handler_thread (arg=0x3fff813de440 <n>) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0start.cc:515

#6  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#7  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 9 (Thread 0x3fff783fefb0 (LWP 11597)):

#0  0x00003fff88bf127c in pthread_cond_wait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114948c in os_cond_wait (cond=0x3fff87e60cc0, fast_mutex=0x3fff87e60c80) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:214

#2  0x00003fff81149d64 in os_event_wait_low (event=0x3fff87e60c80, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:592

#3  0x00003fff8125e794 in srv_purge_coordinator_suspend (slot=0x3fff87f00a80, rseg_history_len=6) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0srv.cc:2848

#4  0x00003fff8125ee00 in srv_purge_coordinator_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0srv.cc:2967

#5  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#6  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 8 (Thread 0x3fff7f7fefb0 (LWP 11571)):

#0  0x00003fff88bf17e0 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x000000005fa8265c in safe_cond_timedwait (cond=0x609c6650 <COND_checkpoint>, mp=0x609c65a0 <LOCK_checkpoint>, abstime=0x3fff7f7fe3d0, file=0x5fcfd418 "/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1191) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/mysys/thr_mutex.c:547

#2  0x000000005f912008 in inline_mysql_cond_timedwait (that=0x609c6650 <COND_checkpoint>, mutex=0x609c65a0 <LOCK_checkpoint>, abstime=0x3fff7f7fe3d0, src_file=0x5fcfd478 "/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/maria/ma_servicethread.c", src_line=116) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/include/mysql/psi/mysql_thread.h:1191

#3  0x000000005f912698 in my_service_thread_sleep (control=0x5ff7ce00 <checkpoint_control>, sleep_time=29000000000) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/maria/ma_servicethread.c:115

#4  0x000000005f9032f8 in ma_checkpoint_background (arg=0x1e) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/maria/ma_checkpoint.c:708

#5  0x000000005f887340 in pfs_spawn_thread (arg=0x3fff87e544f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/perfschema/pfs.cc:1860

#6  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#7  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 7 (Thread 0x3fff793fefb0 (LWP 11595)):

#0  0x00003fff88bf17e0 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x00003fff8114932c in os_cond_wait_timed (cond=0x3fff87e60dc0, fast_mutex=0x3fff87e60d80, abstime=0x3fff793fe460) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:177

#2  0x00003fff81149f84 in os_event_wait_time_low (event=0x3fff87e60d80, time_in_usec=5000000, reset_sig_count=1) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0sync.cc:705

#3  0x00003fff8125b898 in srv_monitor_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0srv.cc:1727

#4  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#5  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 6 (Thread 0x3fff81bfefb0 (LWP 11570)):

#0  0x00003fff88bf17e0 in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x000000005fa8265c in safe_cond_timedwait (cond=0x609cfa30 <COND_timer>, mp=0x609cf980 <LOCK_timer>, abstime=0x3fff81bfe4d0, file=0x5fd20748 "/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1191) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/mysys/thr_mutex.c:547

#2  0x000000005fa83cb0 in inline_mysql_cond_timedwait (that=0x609cfa30 <COND_timer>, mutex=0x609cf980 <LOCK_timer>, abstime=0x3fff81bfe4d0, src_file=0x5fd207a8 "/var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/include/mysql/psi/mysql_thread.h:1191

#3  0x000000005fa84bd4 in timer_handler (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/mysys/thr_timer.c:292

#4  0x000000005f887340 in pfs_spawn_thread (arg=0x3fff87e51f70) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/perfschema/pfs.cc:1860

#5  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#6  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 5 (Thread 0x3fff7bb8efb0 (LWP 11581)):

#0  0x00003fff890f0738 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1

#1  0x00003fff890f07a0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1

#2  0x00003fff81146124 in os_aio_linux_collect (array=0x3fff87f701f0, segment=1, seg_size=256) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5326

#3  0x00003fff811465c4 in os_aio_linux_handle (global_seg=5, message1=0x3fff7bb8e4e0, message2=0x3fff7bb8e4e8, type=0x3fff7bb8e4f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5478

#4  0x00003fff810187d4 in fil_aio_wait (segment=5) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/fil/fil0fil.cc:6045

#5  0x00003fff81262e08 in io_handler_thread (arg=0x3fff813de468 <n+40>) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0start.cc:515

#6  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#7  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 4 (Thread 0x3fff7cb8efb0 (LWP 11579)):

#0  0x00003fff890f0738 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1

#1  0x00003fff890f07a0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1

#2  0x00003fff81146124 in os_aio_linux_collect (array=0x3fff87f70150, segment=1, seg_size=256) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5326

#3  0x00003fff811465c4 in os_aio_linux_handle (global_seg=3, message1=0x3fff7cb8e4e0, message2=0x3fff7cb8e4e8, type=0x3fff7cb8e4f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5478

#4  0x00003fff810187d4 in fil_aio_wait (segment=3) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/fil/fil0fil.cc:6045

#5  0x00003fff81262e08 in io_handler_thread (arg=0x3fff813de458 <n+24>) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0start.cc:515

#6  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#7  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 3 (Thread 0x3fff891ec810 (LWP 11562)):

#0  0x00003fff88726818 in poll () from /lib/powerpc64le-linux-gnu/libc.so.6

#1  0x000000005ee3c884 in handle_connections_sockets () at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/mysqld.cc:6523

#2  0x000000005ee3b540 in mysqld_main (argc=144, argv=0x3fff87fe10c0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/mysqld.cc:5948

#3  0x000000005ee2c958 in main (argc=21, argv=0x3fffebafcad8) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/main.cc:25

Thread 2 (Thread 0x3fff7d38efb0 (LWP 11578)):

#0  0x00003fff890f0738 in ?? () from /lib/powerpc64le-linux-gnu/libaio.so.1

#1  0x00003fff890f07a0 in io_getevents () from /lib/powerpc64le-linux-gnu/libaio.so.1

#2  0x00003fff81146124 in os_aio_linux_collect (array=0x3fff87f70150, segment=0, seg_size=256) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5326

#3  0x00003fff811465c4 in os_aio_linux_handle (global_seg=2, message1=0x3fff7d38e4e0, message2=0x3fff7d38e4e8, type=0x3fff7d38e4f0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/os/os0file.cc:5478

#4  0x00003fff810187d4 in fil_aio_wait (segment=2) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/fil/fil0fil.cc:6045

#5  0x00003fff81262e08 in io_handler_thread (arg=0x3fff813de450 <n+16>) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0start.cc:515

#6  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#7  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Thread 1 (Thread 0x3fff79bfefb0 (LWP 11594)):

#0  0x00003fff88bf2e4c in pthread_kill () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#1  0x000000005fa7b348 in my_write_core (sig=6) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/mysys/stacktrace.c:477

#2  0x000000005f20780c in handle_fatal_signal (sig=6) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/sql/signal_handler.cc:294

#3  <signal handler called>

#4  0x00003fff8864f27c in raise () from /lib/powerpc64le-linux-gnu/libc.so.6

#5  0x00003fff886518f4 in abort () from /lib/powerpc64le-linux-gnu/libc.so.6

#6  0x00003fff8125c078 in srv_error_monitor_thread (arg=0x0) at /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/storage/innobase/srv/srv0srv.cc:1958

#7  0x00003fff88be84a0 in start_thread () from /lib/powerpc64le-linux-gnu/libpthread.so.0

#8  0x00003fff88737e74 in clone () from /lib/powerpc64le-linux-gnu/libc.so.6

Compressed file /var/lib/buildbot/maria-slave/p8-xenial-bintar-debug/build/mysql-test/var/log/sys_vars.innodb_fatal_semaphore_wait_threshold-innodb_plugin/mysqld.1/data/core

worker[1] > Restart  - not started

***Warnings generated in error logs during shutdown after running tests: sys_vars.innodb_fatal_semaphore_wait_threshold

InnoDB: Error: semaphore wait has lasted > 1 seconds

Attempting backtrace. You can use the following information to find out

InnoDB: Error: semaphore wait has lasted > 1 seconds

Attempting backtrace. You can use the following information to find out