galera mtr test galera_pc_weight crashes

Logging: /home/qa-user1/rr/server/mysql-test/mysql-test-run.pl  galera_3nodes.galera_pc_weight --testcase-timeout=120

vardir: /home/qa-user1/rr/build-mariadb/mysql-test/var

Checking leftover processes...

 - found old pid 17817 in 'mysqld.1.pid', killing it...

   process did not exist!

 - found old pid 17819 in 'mysqld.2.pid', killing it...

   process did not exist!

 - found old pid 17821 in 'mysqld.3.pid', killing it...

   process did not exist!

Removing old var directory...

Creating var directory '/home/qa-user1/rr/build-mariadb/mysql-test/var'...

Checking supported features...

MariaDB Version 10.3.7-MariaDB

 - SSL connections supported

Collecting tests...

Installing system database...

==============================================================================

TEST                                      RESULT   TIME (ms) or COMMENT

--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019

worker[1] Test still running: galera_3nodes.galera_pc_weight

worker[1] Test still running: galera_3nodes.galera_pc_weight

worker[1] Test still running: galera_3nodes.galera_pc_weight

worker[1] Test still running: galera_3nodes.galera_pc_weight

worker[1] Test still running: galera_3nodes.galera_pc_weight

worker[1] Trying to dump core for [mysqltest - pid: 18987, winpid: 18987]

worker[1] Trying to dump core for [mysqld.1 - pid: 18581, winpid: 18581]

worker[1] Trying to dump core for [mysqld.2 - pid: 18583, winpid: 18583]

worker[1] Trying to dump core for [mysqld.3 - pid: 18585, winpid: 18585]

galera_3nodes.galera_pc_weight 'innodb'  [ fail ]  timeout after 7200 seconds

        Test ended at 2018-04-26 20:52:13

Test case timeout after 7200 seconds

== /home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_pc_weight.log ==

SET SESSION wsrep_sync_wait=0;

SHOW STATUS LIKE 'wsrep_cluster_size';

Variable_name   Value

wsrep_cluster_size      2

SHOW STATUS LIKE 'wsrep_cluster_status';

Variable_name   Value

wsrep_cluster_status    non-Primary

SHOW STATUS LIKE 'wsrep_connected';

Variable_name   Value

wsrep_connected ON

SHOW STATUS LIKE 'wsrep_ready';

Variable_name   Value

wsrep_ready     OFF

SHOW STATUS LIKE 'wsrep_local_state';

Variable_name   Value

wsrep_local_state       0

SHOW STATUS LIKE 'wsrep_local_state_comment';

Variable_name   Value

wsrep_local_state_comment       Initialized

connection node_1;

 == /home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/analyze-timeout-mysqld.3.err ==

mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/mysqld.3.sock' (111)

 == /home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/analyze-timeout-mysqld.2.err ==

mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/mysqld.2.sock' (111)

 == /home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/analyze-timeout-mysqld.1.err ==

mysqltest: Could not open connection 'default': 2013 Lost connection to MySQL server at 'handshake: reading inital communication packet', system error: 110

 - saving '/home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/' to '/home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/'

 - found 'core.18584' (0/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/mysqld.2/data/core.18584

Core generated by '/home/qa-user1/rr/build-mariadb/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 18584]

[New LWP 18590]

[New LWP 18952]

[New LWP 18588]

[New LWP 18954]

[New LWP 18963]

[New LWP 18628]

[New LWP 18629]

[New LWP 18964]

[New LWP 18630]

[New LWP 18965]

[New LWP 18953]

[New LWP 18966]

[New LWP 18955]

[New LWP 18967]

[New LWP 18956]

[New LWP 18968]

[New LWP 18958]

[New LWP 18971]

[New LWP 18972]

[New LWP 18962]

[New LWP 18977]

[New LWP 18969]

[New LWP 18985]

[New LWP 18970]

[New LWP 18973]

[New LWP 18974]

[New LWP 18975]

[New LWP 18976]

[New LWP 18957]

[New LWP 18959]

[New LWP 18605]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `/home/qa-user1/rr/build-mariadb/sql/mysqld --defaults-group-suffix=.2 --default'.

Program terminated with signal 6, Aborted.

#0  0x00007fd3e2bbe9b1 in pthread_kill () from /lib64/libpthread.so.0

#0  0x00007fd3e2bbe9b1 in pthread_kill () from /lib64/libpthread.so.0

#1  0x000055b635cd75ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305

#2  <signal handler called>

#3  0x00007fd3e1196a3d in poll () from /lib64/libc.so.6

#4  0x000055b635a7a57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffc2f47fbf0) at /usr/include/bits/poll2.h:41

#5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707

#6  0x000055b635a7d759 in mysqld_main (argc=156, argv=0x55b638cd9b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176

#7  0x00007fd3e10cac05 in __libc_start_main () from /lib64/libc.so.6

#8  0x000055b635a71398 in _start ()

Thread 32 (Thread 0x7fd3d88c2700 (LWP 18605)):

#0  0x00007fd3e11a1923 in epoll_wait () from /lib64/libc.so.6

#1  0x00007fd3dad5bbd8 in asio::detail::epoll_reactor::run (this=0x55b638e95f80, block=<optimized out>, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391

#2  0x00007fd3dad5d9d0 in do_run_one (ec=..., this_thread=..., lock=..., this=0x55b638e95e20) at asio/asio/detail/impl/task_io_service.ipp:355

#3  asio::detail::task_io_service::run (this=0x55b638e95e20, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148

#4  0x00007fd3dad74f21 in run (this=0x55b638e95ae0) at asio/asio/impl/io_service.ipp:58

#5  gcomm::AsioProtonet::event_loop (this=0x55b638e95a40, period=...) at gcomm/src/asio_protonet.cpp:121

#6  0x00007fd3dad8d237 in GCommConn::run (this=0x55b638e94af0) at gcs/src/gcs_gcomm.cpp:498

#7  0x00007fd3dad92929 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:209

#8  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#9  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7fd3c1ffb700 (LWP 18959)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b638fcc750, abstime=abstime@entry=0x7fd3c1ffab80) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b638fcc750, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x000055b635fd2847 in pc_sleep_if_needed (cur_time=1524765006890, sig_count=2, next_loop_time=1524765007890) at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:2688

#5  buf_flush_page_cleaner_coordinator () at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:3154

#6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7fd3c2ffd700 (LWP 18957)):

#0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c2ffcba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c2ffcba0, m1=m1@entry=0x7fd3c2ffcc40, m2=m2@entry=0x7fd3c2ffcc48, request=request@entry=0x7fd3c2ffcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c2ffcc50, m2=0x7fd3c2ffcc48, m1=0x7fd3c2ffcc40, global_segment=4) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7fd3c2ffcc40, m2=m2@entry=0x7fd3c2ffcc48, request=request@entry=0x7fd3c2ffcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=4) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7fd3c9e61700 (LWP 18976)):

#0  0x00007fd3e2bc1371 in sigwait () from /lib64/libpthread.so.0

#1  0x000055b635a74053 in signal_hand (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/mysqld.cc:3550

#2  0x000055b636161134 in pfs_spawn_thread (arg=0x55b6391d2948) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#3  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7fd385ffb700 (LWP 18975)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635e434c3 in inline_mysql_cond_wait (src_file=0x55b63630b670 "/home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc", src_line=315, mutex=0x7fd385ffad50, that=0x7fd385ffad80) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1178

#2  thd_destructor_proxy () at /home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc:315

#3  0x000055b636161134 in pfs_spawn_thread (arg=0x55b6391494f8) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fd3867fc700 (LWP 18974)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb7791 in wait (this=0x55b638fcc7d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x55b638fcc7d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x000055b635fc5f2a in buf_resize_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0buf.cc:3124

#5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fd386ffd700 (LWP 18973)):

#0  0x00007fd3e2bc0e4d in nanosleep () from /lib64/libpthread.so.0

#1  0x000055b635eb7d1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230

#2  0x000055b635fb77ba in btr_defragment_thread () at /home/qa-user1/rr/server/storage/innobase/btr/btr0defragment.cc:768

#3  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fd3a0ff9700 (LWP 18970)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb7791 in wait (this=0x55b638fcc550) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x55b638fcc550, reset_sig_count=4, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ada8 <srv_sys+360>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x000055b635f3e834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531

#6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fd3c9dcd700 (LWP 18985)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635a7570b in inline_mysql_cond_timedwait (src_file=0x55b6361fe048 "/home/qa-user1/rr/server/sql/mysqld.cc", src_line=3001, abstime=0x7fd3c9dccd20, mutex=0x55b636b1c8a0 <LOCK_thread_cache>, that=0x55b636b1f060 <COND_thread_cache>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204

#2  cache_thread (thd=0x7fd394000a88) at /home/qa-user1/rr/server/sql/mysqld.cc:3000

#3  one_thread_per_connection_end (thd=0x7fd394000a88, put_in_cache=<optimized out>) at /home/qa-user1/rr/server/sql/mysqld.cc:3087

#4  0x000055b635bebaff in do_handle_one_connection (connect=connect@entry=0x55b6391dd638) at /home/qa-user1/rr/server/sql/sql_connect.cc:1421

#5  0x000055b635bebc94 in handle_one_connection (arg=arg@entry=0x55b6391dd638) at /home/qa-user1/rr/server/sql/sql_connect.cc:1308

#6  0x000055b636161134 in pfs_spawn_thread (arg=0x55b63924e3f8) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fd3a17fa700 (LWP 18969)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb7791 in wait (this=0x55b638fcc450) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x55b638fcc450, reset_sig_count=6, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ad38 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x000055b635f3e834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531

#6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fd3c9e17700 (LWP 18977)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635a8fca4 in inline_mysql_cond_wait (src_file=0x55b6362083b8 "/home/qa-user1/rr/server/sql/slave.cc", src_line=506, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167

#2  handle_slave_background (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/slave.cc:506

#3  0x000055b636161134 in pfs_spawn_thread (arg=0x55b639228778) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fd3c0ff9700 (LWP 18962)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b63916d610, abstime=abstime@entry=0x7fd3c0ff8c70) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x000055b635eb7891 in os_event::wait_time_low (this=this@entry=0x55b63916d610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x000055b635eb7b45 in os_event_wait_time_low (event=event@entry=0x55b63916d610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x000055b635e922e5 in lock_wait_timeout_thread () at /home/qa-user1/rr/server/storage/innobase/lock/lock0wait.cc:547

#5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fd3877fe700 (LWP 18972)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb7791 in wait (this=0x55b638fcc6d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x55b638fcc6d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x000055b635fccb7e in buf_dump_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0dump.cc:837

#5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fd387fff700 (LWP 18971)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb7791 in wait (this=0x55b638fcc4d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x55b638fcc4d0, reset_sig_count=4, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ad70 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x000055b635f3e834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531

#6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fd3c27fc700 (LWP 18958)):

#0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c27fbba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c27fbba0, m1=m1@entry=0x7fd3c27fbc40, m2=m2@entry=0x7fd3c27fbc48, request=request@entry=0x7fd3c27fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c27fbc50, m2=0x7fd3c27fbc48, m1=0x7fd3c27fbc40, global_segment=5) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7fd3c27fbc40, m2=m2@entry=0x7fd3c27fbc48, request=request@entry=0x7fd3c27fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=5) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fd3a1ffb700 (LWP 18968)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb7791 in wait (this=0x55b638fcc3d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x55b638fcc3d0, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ad00 <srv_sys+192>, sig_count=sig_count@entry=1, wait=<optimized out>, timeout_usec=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x000055b635f3ef56 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2691

#6  srv_purge_coordinator_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2778

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fd3c37fe700 (LWP 18956)):

#0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c37fdba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c37fdba0, m1=m1@entry=0x7fd3c37fdc40, m2=m2@entry=0x7fd3c37fdc48, request=request@entry=0x7fd3c37fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c37fdc50, m2=0x7fd3c37fdc48, m1=0x7fd3c37fdc40, global_segment=3) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7fd3c37fdc40, m2=m2@entry=0x7fd3c37fdc48, request=request@entry=0x7fd3c37fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=3) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fd3a27fc700 (LWP 18967)):

#0  0x00007fd3e2bc0e4d in nanosleep () from /lib64/libpthread.so.0

#1  0x000055b635eb7d1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230

#2  0x000055b635f3dfcc in srv_master_sleep () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2364

#3  srv_master_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2409

#4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fd3c8c5f700 (LWP 18955)):

#0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c8c5eba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c8c5eba0, m1=m1@entry=0x7fd3c8c5ec40, m2=m2@entry=0x7fd3c8c5ec48, request=request@entry=0x7fd3c8c5ec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c8c5ec50, m2=0x7fd3c8c5ec48, m1=0x7fd3c8c5ec40, global_segment=2) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7fd3c8c5ec40, m2=m2@entry=0x7fd3c8c5ec48, request=request@entry=0x7fd3c8c5ec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=2) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fd3a2ffd700 (LWP 18966)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b6391a1c70, abstime=abstime@entry=0x7fd3a2ffcb60) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b6391a1c70, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x000055b635f84000 in ib_wqueue_timedwait (wq=wq@entry=0x55b63912e400, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/qa-user1/rr/server/storage/innobase/ut/ut0wqueue.cc:163

#5  0x000055b63605707d in fts_optimize_thread (arg=0x55b63912e400) at /home/qa-user1/rr/server/storage/innobase/fts/fts0opt.cc:3035

#6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fd3c9c61700 (LWP 18953)):

#0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c9c60ba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c9c60ba0, m1=m1@entry=0x7fd3c9c60c40, m2=m2@entry=0x7fd3c9c60c48, request=request@entry=0x7fd3c9c60c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c9c60c50, m2=0x7fd3c9c60c48, m1=0x7fd3c9c60c40, global_segment=0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7fd3c9c60c40, m2=m2@entry=0x7fd3c9c60c48, request=request@entry=0x7fd3c9c60c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=0) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fd3a37fe700 (LWP 18965)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b6391a0e30, abstime=abstime@entry=0x7fd3a37fdc90) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b6391a0e30, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x000055b6360130b5 in dict_stats_thread () at /home/qa-user1/rr/server/storage/innobase/dict/dict0stats_bg.cc:451

#5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fd3d8077700 (LWP 18630)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fd3dac6beae in fifo_lock_get (q=0x55b638e7b750) at galerautils/src/gu_fifo.c:261

#2  gu_fifo_get_head (q=0x55b638e7b750, err=0x7fd3d8076920) at galerautils/src/gu_fifo.c:329

#3  0x00007fd3dad85605 in gcs_recv (conn=0x55b638e55140, action=0x7fd3d8076980) at gcs/src/gcs.cpp:1865

#4  0x00007fd3dadbf644 in galera::GcsActionSource::process (this=0x55b638e4dc78, recv_ctx=0x7fd3b80009a8, exit_loop=@0x7fd3d8076a00: false) at galera/src/gcs_action_source.cpp:175

#5  0x00007fd3dade169b in galera::ReplicatorSMM::async_recv (this=0x55b638e4d5f0, recv_ctx=0x7fd3b80009a8) at galera/src/replicator_smm.cpp:361

#6  0x00007fd3dadefda8 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at galera/src/wsrep_provider.cpp:244

#7  0x000055b635c5a83d in wsrep_replication_process (thd=0x7fd3b80009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:362

#8  0x000055b635c4f5ca in start_wsrep_THD (arg=0x55b635c5a800 <wsrep_replication_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980

#9  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#10 0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fd3c17fa700 (LWP 18964)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b638fcc650, abstime=abstime@entry=0x7fd3c17f9d00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b638fcc650, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x000055b635f3d820 in srv_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1711

#5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fd3d80c1700 (LWP 18629)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635c5ad15 in inline_mysql_cond_wait (src_file=0x55b6362359e8 "/home/qa-user1/rr/server/sql/wsrep_thd.cc", src_line=470, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167

#2  wsrep_rollback_process (thd=0x7fd3b40009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:470

#3  0x000055b635c4f5ca in start_wsrep_THD (arg=0x55b635c5ac40 <wsrep_rollback_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980

#4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fd3c3fff700 (LWP 18628)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fd3dad8add2 in wait (this=0x7fd3c3ffe100, cond=...) at galerautils/src/gu_lock.hpp:59

#2  front (timeout=..., this=0x55b638e94cc0) at gcs/src/gcs_gcomm.cpp:124

#3  gcomm_recv (backend=<optimized out>, msg=0x55b638e55390, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673

#4  0x00007fd3dad7fdd2 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x55b638e55390, backend=0x55b638e554a8) at gcs/src/gcs_core.cpp:468

#5  gcs_core_recv (conn=0x55b638e55330, recv_act=recv_act@entry=0x7fd3c3ffedb0, timeout=<optimized out>) at gcs/src/gcs_core.cpp:1074

#6  0x00007fd3dad86c73 in gcs_recv_thread (arg=0x55b638e55140) at gcs/src/gcs.cpp:1269

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fd3a3fff700 (LWP 18963)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b638fcc5d0, abstime=abstime@entry=0x7fd3a3ffec00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b638fcc5d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x000055b635f3c7a7 in srv_error_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1875

#5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fd3c9460700 (LWP 18954)):

#0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c945fba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c945fba0, m1=m1@entry=0x7fd3c945fc40, m2=m2@entry=0x7fd3c945fc48, request=request@entry=0x7fd3c945fc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c945fc50, m2=0x7fd3c945fc48, m1=0x7fd3c945fc40, global_segment=1) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7fd3c945fc40, m2=m2@entry=0x7fd3c945fc48, request=request@entry=0x7fd3c945fc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=1) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fd3db857700 (LWP 18588)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b6361a0c9e in inline_mysql_cond_timedwait (that=0x55b6373abde0 <COND_timer>, mutex=0x55b6373abe20 <LOCK_timer>, src_file=0x55b63635e650 "/home/qa-user1/rr/server/mysys/thr_timer.c", src_line=292, abstime=0x7fd3db856d70) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204

#2  timer_handler (arg=arg@entry=0x0) at /home/qa-user1/rr/server/mysys/thr_timer.c:292

#3  0x000055b636161134 in pfs_spawn_thread (arg=0x55b638e16a98) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fd3cbffe700 (LWP 18952)):

#0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x000055b6360afb65 in inline_mysql_cond_timedwait (src_file=0x55b63634a2c8 "/home/qa-user1/rr/server/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7fd3cbffdd00, mutex=0x55b63739c380 <LOCK_checkpoint>, that=0x55b63739c340 <COND_checkpoint>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204

#2  my_service_thread_sleep (control=control@entry=0x55b636a9d940 <checkpoint_control>, sleep_time=<optimized out>) at /home/qa-user1/rr/server/storage/maria/ma_servicethread.c:115

#3  0x000055b6360a9402 in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/qa-user1/rr/server/storage/maria/ma_checkpoint.c:707

#4  0x000055b636161134 in pfs_spawn_thread (arg=0x55b638ec7818) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fd3da8c6700 (LWP 18590)):

#0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fd3dadb7b8d in wait (this=0x7fd3da8c5c60, cond=...) at galerautils/src/gu_lock.hpp:59

#2  galera::ServiceThd::thd_func (arg=0x55b638e4db40) at galera/src/galera_service_thd.cpp:30

#3  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fd3e11a134d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fd3e2fd6840 (LWP 18584)):

#0  0x00007fd3e2bbe9b1 in pthread_kill () from /lib64/libpthread.so.0

#1  0x000055b635cd75ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305

#2  <signal handler called>

#3  0x00007fd3e1196a3d in poll () from /lib64/libc.so.6

#4  0x000055b635a7a57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffc2f47fbf0) at /usr/include/bits/poll2.h:41

#5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707

#6  0x000055b635a7d759 in mysqld_main (argc=156, argv=0x55b638cd9b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176

#7  0x00007fd3e10cac05 in __libc_start_main () from /lib64/libc.so.6

#8  0x000055b635a71398 in _start ()

 - found 'core.18586' (1/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace from coredump /home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/mysqld.3/data/core.18586

Core generated by '/home/qa-user1/rr/build-mariadb/sql/mysqld'

Output from gdb follows. The first stack trace is from the failing thread.

The following stack traces are from all threads (so the failing one is

duplicated).

--------------------------

[New LWP 18586]

[New LWP 18604]

[New LWP 18591]

[New LWP 18627]

[New LWP 18606]

[New LWP 18879]

[New LWP 18625]

[New LWP 18626]

[New LWP 18883]

[New LWP 18884]

[New LWP 18886]

[New LWP 18892]

[New LWP 18893]

[New LWP 18894]

[New LWP 18895]

[New LWP 18897]

[New LWP 18898]

[New LWP 18899]

[New LWP 18900]

[New LWP 18901]

[New LWP 18905]

[New LWP 18906]

[New LWP 18986]

[New LWP 18880]

[New LWP 18881]

[New LWP 18882]

[New LWP 18896]

[New LWP 18902]

[New LWP 18903]

[New LWP 18904]

[New LWP 18907]

[New LWP 18885]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `/home/qa-user1/rr/build-mariadb/sql/mysqld --defaults-group-suffix=.3 --default'.

Program terminated with signal 6, Aborted.

#0  0x00007fb5cec2f9b1 in pthread_kill () from /lib64/libpthread.so.0

#0  0x00007fb5cec2f9b1 in pthread_kill () from /lib64/libpthread.so.0

#1  0x00005619f75ab5ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305

#2  <signal handler called>

#3  0x00007fb5cd207a3d in poll () from /lib64/libc.so.6

#4  0x00005619f734e57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffe0026f720) at /usr/include/bits/poll2.h:41

#5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707

#6  0x00005619f7351759 in mysqld_main (argc=156, argv=0x5619fa929b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176

#7  0x00007fb5cd13bc05 in __libc_start_main () from /lib64/libc.so.6

#8  0x00005619f7345398 in _start ()

Thread 32 (Thread 0x7fb59e7fc700 (LWP 18885)):

#0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59e7fbba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59e7fbba0, m1=m1@entry=0x7fb59e7fbc40, m2=m2@entry=0x7fb59e7fbc48, request=request@entry=0x7fb59e7fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59e7fbc50, m2=0x7fb59e7fbc48, m1=0x7fb59e7fbc40, global_segment=5) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7fb59e7fbc40, m2=m2@entry=0x7fb59e7fbc48, request=request@entry=0x7fb59e7fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=5) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7fb5b569e700 (LWP 18907)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f7363ca4 in inline_mysql_cond_wait (src_file=0x5619f7adc3b8 "/home/qa-user1/rr/server/sql/slave.cc", src_line=506, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167

#2  handle_slave_background (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/slave.cc:506

#3  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fae94598) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7fb58a7fc700 (LWP 18904)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b791 in wait (this=0x5619fac1c7d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x5619fac1c7d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x00005619f7899f2a in buf_resize_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0buf.cc:3124

#5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7fb58affd700 (LWP 18903)):

#0  0x00007fb5cec31e4d in nanosleep () from /lib64/libpthread.so.0

#1  0x00005619f778bd1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230

#2  0x00005619f788b7ba in btr_defragment_thread () at /home/qa-user1/rr/server/storage/innobase/btr/btr0defragment.cc:768

#3  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7fb59d7fa700 (LWP 18902)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b791 in wait (this=0x5619fac1c6d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x5619fac1c6d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x00005619f78a0b7e in buf_dump_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0dump.cc:837

#5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fb5927fc700 (LWP 18896)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fad8bdb0, abstime=abstime@entry=0x7fb5927fbb60) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fad8bdb0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x00005619f7858000 in ib_wqueue_timedwait (wq=wq@entry=0x5619fad7e400, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/qa-user1/rr/server/storage/innobase/ut/ut0wqueue.cc:163

#5  0x00005619f792b07d in fts_optimize_thread (arg=0x5619fad7e400) at /home/qa-user1/rr/server/storage/innobase/fts/fts0opt.cc:3035

#6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fb59ffff700 (LWP 18882)):

#0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59fffeba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59fffeba0, m1=m1@entry=0x7fb59fffec40, m2=m2@entry=0x7fb59fffec48, request=request@entry=0x7fb59fffec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59fffec50, m2=0x7fb59fffec48, m1=0x7fb59fffec40, global_segment=2) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7fb59fffec40, m2=m2@entry=0x7fb59fffec48, request=request@entry=0x7fb59fffec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=2) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fb5b4ce7700 (LWP 18881)):

#0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb5b4ce6ba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb5b4ce6ba0, m1=m1@entry=0x7fb5b4ce6c40, m2=m2@entry=0x7fb5b4ce6c48, request=request@entry=0x7fb5b4ce6c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb5b4ce6c50, m2=0x7fb5b4ce6c48, m1=0x7fb5b4ce6c40, global_segment=1) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7fb5b4ce6c40, m2=m2@entry=0x7fb5b4ce6c48, request=request@entry=0x7fb5b4ce6c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=1) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fb5b54e8700 (LWP 18880)):

#0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb5b54e7ba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb5b54e7ba0, m1=m1@entry=0x7fb5b54e7c40, m2=m2@entry=0x7fb5b54e7c48, request=request@entry=0x7fb5b54e7c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb5b54e7c50, m2=0x7fb5b54e7c48, m1=0x7fb5b54e7c40, global_segment=0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7fb5b54e7c40, m2=m2@entry=0x7fb5b54e7c48, request=request@entry=0x7fb5b54e7c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=0) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fb5b5654700 (LWP 18986)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f734970b in inline_mysql_cond_timedwait (src_file=0x5619f7ad2048 "/home/qa-user1/rr/server/sql/mysqld.cc", src_line=3001, abstime=0x7fb5b5653d20, mutex=0x5619f83f08a0 <LOCK_thread_cache>, that=0x5619f83f3060 <COND_thread_cache>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204

#2  cache_thread (thd=0x7fb584000a88) at /home/qa-user1/rr/server/sql/mysqld.cc:3000

#3  one_thread_per_connection_end (thd=0x7fb584000a88, put_in_cache=<optimized out>) at /home/qa-user1/rr/server/sql/mysqld.cc:3087

#4  0x00005619f74bfaff in do_handle_one_connection (connect=connect@entry=0x5619fae62138) at /home/qa-user1/rr/server/sql/sql_connect.cc:1421

#5  0x00005619f74bfc94 in handle_one_connection (arg=arg@entry=0x5619fae62138) at /home/qa-user1/rr/server/sql/sql_connect.cc:1308

#6  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fae9e498) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fb5b56e8700 (LWP 18906)):

#0  0x00007fb5cec32371 in sigwait () from /lib64/libpthread.so.0

#1  0x00005619f7348053 in signal_hand (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/mysqld.cc:3550

#2  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fabd0b48) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#3  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fb589ffb700 (LWP 18905)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f77174c3 in inline_mysql_cond_wait (src_file=0x5619f7bdf670 "/home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc", src_line=315, mutex=0x7fb589ffad50, that=0x7fb589ffad80) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1178

#2  thd_destructor_proxy () at /home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc:315

#3  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fad98518) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fb58b7fe700 (LWP 18901)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b791 in wait (this=0x5619fac1c4d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x5619fac1c4d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836ed70 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x00005619f7812834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531

#6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fb58bfff700 (LWP 18900)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b791 in wait (this=0x5619fac1c450) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x5619fac1c450, reset_sig_count=10, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836ed38 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x00005619f7812834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531

#6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fb590ff9700 (LWP 18899)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b791 in wait (this=0x5619fac1c550) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x5619fac1c550, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836eda8 <srv_sys+360>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x00005619f7812834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531

#6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fb5917fa700 (LWP 18898)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b791 in wait (this=0x5619fac1c3d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x5619fac1c3d0, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333

#3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527

#4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836ed00 <srv_sys+192>, sig_count=sig_count@entry=1, wait=<optimized out>, timeout_usec=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912

#5  0x00005619f7812f56 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2691

#6  srv_purge_coordinator_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2778

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fb591ffb700 (LWP 18897)):

#0  0x00007fb5cec31e4d in nanosleep () from /lib64/libpthread.so.0

#1  0x00005619f778bd1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230

#2  0x00005619f7811fcc in srv_master_sleep () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2364

#3  srv_master_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2409

#4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fb592ffd700 (LWP 18895)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fadf0e30, abstime=abstime@entry=0x7fb592ffcc90) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fadf0e30, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x00005619f78e70b5 in dict_stats_thread () at /home/qa-user1/rr/server/storage/innobase/dict/dict0stats_bg.cc:451

#5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fb5937fe700 (LWP 18894)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fac1c650, abstime=abstime@entry=0x7fb5937fdd00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fac1c650, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x00005619f7811820 in srv_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1711

#5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fb593fff700 (LWP 18893)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fac1c5d0, abstime=abstime@entry=0x7fb593ffec00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fac1c5d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x00005619f78107a7 in srv_error_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1875

#5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fb59cff9700 (LWP 18892)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fadbd610, abstime=abstime@entry=0x7fb59cff8c70) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x00005619f778b891 in os_event::wait_time_low (this=this@entry=0x5619fadbd610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x00005619f778bb45 in os_event_wait_time_low (event=event@entry=0x5619fadbd610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x00005619f77662e5 in lock_wait_timeout_thread () at /home/qa-user1/rr/server/storage/innobase/lock/lock0wait.cc:547

#5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fb59dffb700 (LWP 18886)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fac1c750, abstime=abstime@entry=0x7fb59dffab80) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283

#2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fac1c750, time_in_usec=time_in_usec@entry=956000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405

#3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=956000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510

#4  0x00005619f78a6847 in pc_sleep_if_needed (cur_time=1524765010939, sig_count=2, next_loop_time=1524765011895) at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:2688

#5  buf_flush_page_cleaner_coordinator () at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:3154

#6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fb59effd700 (LWP 18884)):

#0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59effcba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59effcba0, m1=m1@entry=0x7fb59effcc40, m2=m2@entry=0x7fb59effcc48, request=request@entry=0x7fb59effcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59effcc50, m2=0x7fb59effcc48, m1=0x7fb59effcc40, global_segment=4) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7fb59effcc40, m2=m2@entry=0x7fb59effcc48, request=request@entry=0x7fb59effcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=4) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fb59f7fe700 (LWP 18883)):

#0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59f7fdba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922

#2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59f7fdba0, m1=m1@entry=0x7fb59f7fdc40, m2=m2@entry=0x7fb59f7fdc48, request=request@entry=0x7fb59f7fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067

#3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59f7fdc50, m2=0x7fb59f7fdc48, m1=0x7fb59f7fdc40, global_segment=3) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121

#4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7fb59f7fdc40, m2=m2@entry=0x7fb59f7fdc48, request=request@entry=0x7fb59f7fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649

#5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=3) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653

#6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fb5c4132700 (LWP 18626)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f752ed15 in inline_mysql_cond_wait (src_file=0x5619f7b099e8 "/home/qa-user1/rr/server/sql/wsrep_thd.cc", src_line=470, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167

#2  wsrep_rollback_process (thd=0x7fb5ac0009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:470

#3  0x00005619f75235ca in start_wsrep_THD (arg=0x5619f752ec40 <wsrep_rollback_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980

#4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fb5b7ffe700 (LWP 18625)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fb5c6dfbdd2 in wait (this=0x7fb5b7ffd100, cond=...) at galerautils/src/gu_lock.hpp:59

#2  front (timeout=..., this=0x5619faae4cd0) at gcs/src/gcs_gcomm.cpp:124

#3  gcomm_recv (backend=<optimized out>, msg=0x5619faaa53a0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673

#4  0x00007fb5c6df0dd2 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x5619faaa53a0, backend=0x5619faaa54b8) at gcs/src/gcs_core.cpp:468

#5  gcs_core_recv (conn=0x5619faaa5340, recv_act=recv_act@entry=0x7fb5b7ffddb0, timeout=<optimized out>) at gcs/src/gcs_core.cpp:1074

#6  0x00007fb5c6df7c73 in gcs_recv_thread (arg=0x5619faaa5150) at gcs/src/gcs.cpp:1269

#7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fb5b77fd700 (LWP 18879)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f7983b65 in inline_mysql_cond_timedwait (src_file=0x5619f7c1e2c8 "/home/qa-user1/rr/server/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7fb5b77fcd00, mutex=0x5619f8c70380 <LOCK_checkpoint>, that=0x5619f8c70340 <COND_checkpoint>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204

#2  my_service_thread_sleep (control=control@entry=0x5619f8371940 <checkpoint_control>, sleep_time=<optimized out>) at /home/qa-user1/rr/server/storage/maria/ma_servicethread.c:115

#3  0x00005619f797d402 in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/qa-user1/rr/server/storage/maria/ma_checkpoint.c:707

#4  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fab17808) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fb5c4933700 (LWP 18606)):

#0  0x00007fb5cd212923 in epoll_wait () from /lib64/libc.so.6

#1  0x00007fb5c6dccbd8 in asio::detail::epoll_reactor::run (this=0x5619faae5f90, block=<optimized out>, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391

#2  0x00007fb5c6dce9d0 in do_run_one (ec=..., this_thread=..., lock=..., this=0x5619faae5e30) at asio/asio/detail/impl/task_io_service.ipp:355

#3  asio::detail::task_io_service::run (this=0x5619faae5e30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148

#4  0x00007fb5c6de5f21 in run (this=0x5619faae5af0) at asio/asio/impl/io_service.ipp:58

#5  gcomm::AsioProtonet::event_loop (this=0x5619faae5a50, period=...) at gcomm/src/asio_protonet.cpp:121

#6  0x00007fb5c6dfe237 in GCommConn::run (this=0x5619faae4b00) at gcs/src/gcs_gcomm.cpp:498

#7  0x00007fb5c6e03929 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:209

#8  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#9  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fb5c40e8700 (LWP 18627)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fb5c6cdceae in fifo_lock_get (q=0x5619faacb760) at galerautils/src/gu_fifo.c:261

#2  gu_fifo_get_head (q=0x5619faacb760, err=0x7fb5c40e7920) at galerautils/src/gu_fifo.c:329

#3  0x00007fb5c6df6605 in gcs_recv (conn=0x5619faaa5150, action=0x7fb5c40e7980) at gcs/src/gcs.cpp:1865

#4  0x00007fb5c6e30644 in galera::GcsActionSource::process (this=0x5619faa9dc88, recv_ctx=0x7fb5a00009a8, exit_loop=@0x7fb5c40e7a00: false) at galera/src/gcs_action_source.cpp:175

#5  0x00007fb5c6e5269b in galera::ReplicatorSMM::async_recv (this=0x5619faa9d600, recv_ctx=0x7fb5a00009a8) at galera/src/replicator_smm.cpp:361

#6  0x00007fb5c6e60da8 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at galera/src/wsrep_provider.cpp:244

#7  0x00005619f752e83d in wsrep_replication_process (thd=0x7fb5a00009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:362

#8  0x00005619f75235ca in start_wsrep_THD (arg=0x5619f752e800 <wsrep_replication_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980

#9  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#10 0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fb5c78c8700 (LWP 18591)):

#0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00005619f7a74c9e in inline_mysql_cond_timedwait (that=0x5619f8c7fde0 <COND_timer>, mutex=0x5619f8c7fe20 <LOCK_timer>, src_file=0x5619f7c32650 "/home/qa-user1/rr/server/mysys/thr_timer.c", src_line=292, abstime=0x7fb5c78c7d70) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204

#2  timer_handler (arg=arg@entry=0x0) at /home/qa-user1/rr/server/mysys/thr_timer.c:292

#3  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619faa66a98) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862

#4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fb5c6937700 (LWP 18604)):

#0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00007fb5c6e28b8d in wait (this=0x7fb5c6936c60, cond=...) at galerautils/src/gu_lock.hpp:59

#2  galera::ServiceThd::thd_func (arg=0x5619faa9db50) at galera/src/galera_service_thd.cpp:30

#3  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fb5cd21234d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fb5cf047840 (LWP 18586)):

#0  0x00007fb5cec2f9b1 in pthread_kill () from /lib64/libpthread.so.0

#1  0x00005619f75ab5ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305

#2  <signal handler called>

#3  0x00007fb5cd207a3d in poll () from /lib64/libc.so.6

#4  0x00005619f734e57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffe0026f720) at /usr/include/bits/poll2.h:41

#5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707

#6  0x00005619f7351759 in mysqld_main (argc=156, argv=0x5619fa929b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176

#7  0x00007fb5cd13bc05 in __libc_start_main () from /lib64/libc.so.6

#8  0x00005619f7345398 in _start ()

--------------------------------------------------------------------------

The servers were restarted 0 times

Spent 0.000 of 7380 seconds executing testcases

Failure: Failed 1/1 tests, 0.00% were successful.