Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16047

galera mtr test galera_pc_weight crashes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 10.3.6
    • Fix Version/s: 10.3.20
    • Component/s: Galera, Tests
    • Labels:
      None
    • Environment:
      CentOS 7.4

      Description

      galera mtr test galera_pc_weight from gelera galera_3nodes test suite crashes
      on mariadb-10.3.6-76-g461de7e
      problem appears with the default test timeout of 900 sec
      but it's also recreated with test timeout of 120 minutes

      Logging: /home/qa-user1/rr/server/mysql-test/mysql-test-run.pl  galera_3nodes.galera_pc_weight --testcase-timeout=120
      vardir: /home/qa-user1/rr/build-mariadb/mysql-test/var
      Checking leftover processes...
       - found old pid 17817 in 'mysqld.1.pid', killing it...
         process did not exist!
       - found old pid 17819 in 'mysqld.2.pid', killing it...
         process did not exist!
       - found old pid 17821 in 'mysqld.3.pid', killing it...
         process did not exist!
      Removing old var directory...
      Creating var directory '/home/qa-user1/rr/build-mariadb/mysql-test/var'...
      Checking supported features...
      MariaDB Version 10.3.7-MariaDB
       - SSL connections supported
      Collecting tests...
      Installing system database...
       
      ==============================================================================
       
      TEST                                      RESULT   TIME (ms) or COMMENT
      --------------------------------------------------------------------------
       
      worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
      worker[1] Test still running: galera_3nodes.galera_pc_weight
      worker[1] Test still running: galera_3nodes.galera_pc_weight
      worker[1] Test still running: galera_3nodes.galera_pc_weight
      worker[1] Test still running: galera_3nodes.galera_pc_weight
      worker[1] Test still running: galera_3nodes.galera_pc_weight
      worker[1] Trying to dump core for [mysqltest - pid: 18987, winpid: 18987]
      worker[1] Trying to dump core for [mysqld.1 - pid: 18581, winpid: 18581]
      worker[1] Trying to dump core for [mysqld.2 - pid: 18583, winpid: 18583]
      worker[1] Trying to dump core for [mysqld.3 - pid: 18585, winpid: 18585]
      galera_3nodes.galera_pc_weight 'innodb'  [ fail ]  timeout after 7200 seconds
              Test ended at 2018-04-26 20:52:13
       
      Test case timeout after 7200 seconds
       
      == /home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_pc_weight.log ==
      SET SESSION wsrep_sync_wait=0;
      SHOW STATUS LIKE 'wsrep_cluster_size';
      Variable_name   Value
      wsrep_cluster_size      2
      SHOW STATUS LIKE 'wsrep_cluster_status';
      Variable_name   Value
      wsrep_cluster_status    non-Primary
      SHOW STATUS LIKE 'wsrep_connected';
      Variable_name   Value
      wsrep_connected ON
      SHOW STATUS LIKE 'wsrep_ready';
      Variable_name   Value
      wsrep_ready     OFF
      SHOW STATUS LIKE 'wsrep_local_state';
      Variable_name   Value
      wsrep_local_state       0
      SHOW STATUS LIKE 'wsrep_local_state_comment';
      Variable_name   Value
      wsrep_local_state_comment       Initialized
      connection node_1;
       
       == /home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/analyze-timeout-mysqld.3.err ==
      mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/mysqld.3.sock' (111)
       
       == /home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/analyze-timeout-mysqld.2.err ==
      mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/mysqld.2.sock' (111)
       
       == /home/qa-user1/rr/build-mariadb/mysql-test/var/tmp/analyze-timeout-mysqld.1.err ==
      mysqltest: Could not open connection 'default': 2013 Lost connection to MySQL server at 'handshake: reading inital communication packet', system error: 110
       
       
       - saving '/home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/' to '/home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/'
       - found 'core.18584' (0/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/mysqld.2/data/core.18584
      Core generated by '/home/qa-user1/rr/build-mariadb/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 18584]
      [New LWP 18590]
      [New LWP 18952]
      [New LWP 18588]
      [New LWP 18954]
      [New LWP 18963]
      [New LWP 18628]
      [New LWP 18629]
      [New LWP 18964]
      [New LWP 18630]
      [New LWP 18965]
      [New LWP 18953]
      [New LWP 18966]
      [New LWP 18955]
      [New LWP 18967]
      [New LWP 18956]
      [New LWP 18968]
      [New LWP 18958]
      [New LWP 18971]
      [New LWP 18972]
      [New LWP 18962]
      [New LWP 18977]
      [New LWP 18969]
      [New LWP 18985]
      [New LWP 18970]
      [New LWP 18973]
      [New LWP 18974]
      [New LWP 18975]
      [New LWP 18976]
      [New LWP 18957]
      [New LWP 18959]
      [New LWP 18605]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Core was generated by `/home/qa-user1/rr/build-mariadb/sql/mysqld --defaults-group-suffix=.2 --default'.
      Program terminated with signal 6, Aborted.
      #0  0x00007fd3e2bbe9b1 in pthread_kill () from /lib64/libpthread.so.0
      #0  0x00007fd3e2bbe9b1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x000055b635cd75ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305
      #2  <signal handler called>
      #3  0x00007fd3e1196a3d in poll () from /lib64/libc.so.6
      #4  0x000055b635a7a57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffc2f47fbf0) at /usr/include/bits/poll2.h:41
      #5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707
      #6  0x000055b635a7d759 in mysqld_main (argc=156, argv=0x55b638cd9b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176
      #7  0x00007fd3e10cac05 in __libc_start_main () from /lib64/libc.so.6
      #8  0x000055b635a71398 in _start ()
       
      Thread 32 (Thread 0x7fd3d88c2700 (LWP 18605)):
      #0  0x00007fd3e11a1923 in epoll_wait () from /lib64/libc.so.6
      #1  0x00007fd3dad5bbd8 in asio::detail::epoll_reactor::run (this=0x55b638e95f80, block=<optimized out>, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391
      #2  0x00007fd3dad5d9d0 in do_run_one (ec=..., this_thread=..., lock=..., this=0x55b638e95e20) at asio/asio/detail/impl/task_io_service.ipp:355
      #3  asio::detail::task_io_service::run (this=0x55b638e95e20, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148
      #4  0x00007fd3dad74f21 in run (this=0x55b638e95ae0) at asio/asio/impl/io_service.ipp:58
      #5  gcomm::AsioProtonet::event_loop (this=0x55b638e95a40, period=...) at gcomm/src/asio_protonet.cpp:121
      #6  0x00007fd3dad8d237 in GCommConn::run (this=0x55b638e94af0) at gcs/src/gcs_gcomm.cpp:498
      #7  0x00007fd3dad92929 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:209
      #8  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 31 (Thread 0x7fd3c1ffb700 (LWP 18959)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b638fcc750, abstime=abstime@entry=0x7fd3c1ffab80) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b638fcc750, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x000055b635fd2847 in pc_sleep_if_needed (cur_time=1524765006890, sig_count=2, next_loop_time=1524765007890) at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:2688
      #5  buf_flush_page_cleaner_coordinator () at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:3154
      #6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 30 (Thread 0x7fd3c2ffd700 (LWP 18957)):
      #0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c2ffcba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c2ffcba0, m1=m1@entry=0x7fd3c2ffcc40, m2=m2@entry=0x7fd3c2ffcc48, request=request@entry=0x7fd3c2ffcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c2ffcc50, m2=0x7fd3c2ffcc48, m1=0x7fd3c2ffcc40, global_segment=4) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7fd3c2ffcc40, m2=m2@entry=0x7fd3c2ffcc48, request=request@entry=0x7fd3c2ffcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=4) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 29 (Thread 0x7fd3c9e61700 (LWP 18976)):
      #0  0x00007fd3e2bc1371 in sigwait () from /lib64/libpthread.so.0
      #1  0x000055b635a74053 in signal_hand (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/mysqld.cc:3550
      #2  0x000055b636161134 in pfs_spawn_thread (arg=0x55b6391d2948) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #3  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 28 (Thread 0x7fd385ffb700 (LWP 18975)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635e434c3 in inline_mysql_cond_wait (src_file=0x55b63630b670 "/home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc", src_line=315, mutex=0x7fd385ffad50, that=0x7fd385ffad80) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1178
      #2  thd_destructor_proxy () at /home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc:315
      #3  0x000055b636161134 in pfs_spawn_thread (arg=0x55b6391494f8) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 27 (Thread 0x7fd3867fc700 (LWP 18974)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb7791 in wait (this=0x55b638fcc7d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x55b638fcc7d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x000055b635fc5f2a in buf_resize_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0buf.cc:3124
      #5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 26 (Thread 0x7fd386ffd700 (LWP 18973)):
      #0  0x00007fd3e2bc0e4d in nanosleep () from /lib64/libpthread.so.0
      #1  0x000055b635eb7d1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230
      #2  0x000055b635fb77ba in btr_defragment_thread () at /home/qa-user1/rr/server/storage/innobase/btr/btr0defragment.cc:768
      #3  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 25 (Thread 0x7fd3a0ff9700 (LWP 18970)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb7791 in wait (this=0x55b638fcc550) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x55b638fcc550, reset_sig_count=4, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ada8 <srv_sys+360>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x000055b635f3e834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531
      #6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 24 (Thread 0x7fd3c9dcd700 (LWP 18985)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635a7570b in inline_mysql_cond_timedwait (src_file=0x55b6361fe048 "/home/qa-user1/rr/server/sql/mysqld.cc", src_line=3001, abstime=0x7fd3c9dccd20, mutex=0x55b636b1c8a0 <LOCK_thread_cache>, that=0x55b636b1f060 <COND_thread_cache>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204
      #2  cache_thread (thd=0x7fd394000a88) at /home/qa-user1/rr/server/sql/mysqld.cc:3000
      #3  one_thread_per_connection_end (thd=0x7fd394000a88, put_in_cache=<optimized out>) at /home/qa-user1/rr/server/sql/mysqld.cc:3087
      #4  0x000055b635bebaff in do_handle_one_connection (connect=connect@entry=0x55b6391dd638) at /home/qa-user1/rr/server/sql/sql_connect.cc:1421
      #5  0x000055b635bebc94 in handle_one_connection (arg=arg@entry=0x55b6391dd638) at /home/qa-user1/rr/server/sql/sql_connect.cc:1308
      #6  0x000055b636161134 in pfs_spawn_thread (arg=0x55b63924e3f8) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 23 (Thread 0x7fd3a17fa700 (LWP 18969)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb7791 in wait (this=0x55b638fcc450) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x55b638fcc450, reset_sig_count=6, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ad38 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x000055b635f3e834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531
      #6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 22 (Thread 0x7fd3c9e17700 (LWP 18977)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635a8fca4 in inline_mysql_cond_wait (src_file=0x55b6362083b8 "/home/qa-user1/rr/server/sql/slave.cc", src_line=506, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167
      #2  handle_slave_background (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/slave.cc:506
      #3  0x000055b636161134 in pfs_spawn_thread (arg=0x55b639228778) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 21 (Thread 0x7fd3c0ff9700 (LWP 18962)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b63916d610, abstime=abstime@entry=0x7fd3c0ff8c70) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x000055b635eb7891 in os_event::wait_time_low (this=this@entry=0x55b63916d610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x000055b635eb7b45 in os_event_wait_time_low (event=event@entry=0x55b63916d610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x000055b635e922e5 in lock_wait_timeout_thread () at /home/qa-user1/rr/server/storage/innobase/lock/lock0wait.cc:547
      #5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 20 (Thread 0x7fd3877fe700 (LWP 18972)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb7791 in wait (this=0x55b638fcc6d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x55b638fcc6d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x000055b635fccb7e in buf_dump_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0dump.cc:837
      #5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 19 (Thread 0x7fd387fff700 (LWP 18971)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb7791 in wait (this=0x55b638fcc4d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x55b638fcc4d0, reset_sig_count=4, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ad70 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x000055b635f3e834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531
      #6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 18 (Thread 0x7fd3c27fc700 (LWP 18958)):
      #0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c27fbba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c27fbba0, m1=m1@entry=0x7fd3c27fbc40, m2=m2@entry=0x7fd3c27fbc48, request=request@entry=0x7fd3c27fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c27fbc50, m2=0x7fd3c27fbc48, m1=0x7fd3c27fbc40, global_segment=5) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7fd3c27fbc40, m2=m2@entry=0x7fd3c27fbc48, request=request@entry=0x7fd3c27fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=5) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 17 (Thread 0x7fd3a1ffb700 (LWP 18968)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb7791 in wait (this=0x55b638fcc3d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x55b638fcc3d0, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x000055b635eb7b55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x000055b635f39553 in srv_resume_thread (slot=slot@entry=0x55b636a9ad00 <srv_sys+192>, sig_count=sig_count@entry=1, wait=<optimized out>, timeout_usec=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x000055b635f3ef56 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2691
      #6  srv_purge_coordinator_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2778
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 16 (Thread 0x7fd3c37fe700 (LWP 18956)):
      #0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c37fdba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c37fdba0, m1=m1@entry=0x7fd3c37fdc40, m2=m2@entry=0x7fd3c37fdc48, request=request@entry=0x7fd3c37fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c37fdc50, m2=0x7fd3c37fdc48, m1=0x7fd3c37fdc40, global_segment=3) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7fd3c37fdc40, m2=m2@entry=0x7fd3c37fdc48, request=request@entry=0x7fd3c37fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=3) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 15 (Thread 0x7fd3a27fc700 (LWP 18967)):
      #0  0x00007fd3e2bc0e4d in nanosleep () from /lib64/libpthread.so.0
      #1  0x000055b635eb7d1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230
      #2  0x000055b635f3dfcc in srv_master_sleep () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2364
      #3  srv_master_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2409
      #4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 14 (Thread 0x7fd3c8c5f700 (LWP 18955)):
      #0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c8c5eba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c8c5eba0, m1=m1@entry=0x7fd3c8c5ec40, m2=m2@entry=0x7fd3c8c5ec48, request=request@entry=0x7fd3c8c5ec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c8c5ec50, m2=0x7fd3c8c5ec48, m1=0x7fd3c8c5ec40, global_segment=2) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7fd3c8c5ec40, m2=m2@entry=0x7fd3c8c5ec48, request=request@entry=0x7fd3c8c5ec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=2) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 13 (Thread 0x7fd3a2ffd700 (LWP 18966)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b6391a1c70, abstime=abstime@entry=0x7fd3a2ffcb60) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b6391a1c70, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x000055b635f84000 in ib_wqueue_timedwait (wq=wq@entry=0x55b63912e400, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/qa-user1/rr/server/storage/innobase/ut/ut0wqueue.cc:163
      #5  0x000055b63605707d in fts_optimize_thread (arg=0x55b63912e400) at /home/qa-user1/rr/server/storage/innobase/fts/fts0opt.cc:3035
      #6  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 12 (Thread 0x7fd3c9c61700 (LWP 18953)):
      #0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c9c60ba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c9c60ba0, m1=m1@entry=0x7fd3c9c60c40, m2=m2@entry=0x7fd3c9c60c48, request=request@entry=0x7fd3c9c60c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c9c60c50, m2=0x7fd3c9c60c48, m1=0x7fd3c9c60c40, global_segment=0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7fd3c9c60c40, m2=m2@entry=0x7fd3c9c60c48, request=request@entry=0x7fd3c9c60c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=0) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 11 (Thread 0x7fd3a37fe700 (LWP 18965)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b6391a0e30, abstime=abstime@entry=0x7fd3a37fdc90) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b6391a0e30, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x000055b6360130b5 in dict_stats_thread () at /home/qa-user1/rr/server/storage/innobase/dict/dict0stats_bg.cc:451
      #5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 10 (Thread 0x7fd3d8077700 (LWP 18630)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fd3dac6beae in fifo_lock_get (q=0x55b638e7b750) at galerautils/src/gu_fifo.c:261
      #2  gu_fifo_get_head (q=0x55b638e7b750, err=0x7fd3d8076920) at galerautils/src/gu_fifo.c:329
      #3  0x00007fd3dad85605 in gcs_recv (conn=0x55b638e55140, action=0x7fd3d8076980) at gcs/src/gcs.cpp:1865
      #4  0x00007fd3dadbf644 in galera::GcsActionSource::process (this=0x55b638e4dc78, recv_ctx=0x7fd3b80009a8, exit_loop=@0x7fd3d8076a00: false) at galera/src/gcs_action_source.cpp:175
      #5  0x00007fd3dade169b in galera::ReplicatorSMM::async_recv (this=0x55b638e4d5f0, recv_ctx=0x7fd3b80009a8) at galera/src/replicator_smm.cpp:361
      #6  0x00007fd3dadefda8 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at galera/src/wsrep_provider.cpp:244
      #7  0x000055b635c5a83d in wsrep_replication_process (thd=0x7fd3b80009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:362
      #8  0x000055b635c4f5ca in start_wsrep_THD (arg=0x55b635c5a800 <wsrep_replication_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980
      #9  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #10 0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 9 (Thread 0x7fd3c17fa700 (LWP 18964)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b638fcc650, abstime=abstime@entry=0x7fd3c17f9d00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b638fcc650, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x000055b635f3d820 in srv_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1711
      #5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 8 (Thread 0x7fd3d80c1700 (LWP 18629)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635c5ad15 in inline_mysql_cond_wait (src_file=0x55b6362359e8 "/home/qa-user1/rr/server/sql/wsrep_thd.cc", src_line=470, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167
      #2  wsrep_rollback_process (thd=0x7fd3b40009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:470
      #3  0x000055b635c4f5ca in start_wsrep_THD (arg=0x55b635c5ac40 <wsrep_rollback_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980
      #4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 7 (Thread 0x7fd3c3fff700 (LWP 18628)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fd3dad8add2 in wait (this=0x7fd3c3ffe100, cond=...) at galerautils/src/gu_lock.hpp:59
      #2  front (timeout=..., this=0x55b638e94cc0) at gcs/src/gcs_gcomm.cpp:124
      #3  gcomm_recv (backend=<optimized out>, msg=0x55b638e55390, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673
      #4  0x00007fd3dad7fdd2 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x55b638e55390, backend=0x55b638e554a8) at gcs/src/gcs_core.cpp:468
      #5  gcs_core_recv (conn=0x55b638e55330, recv_act=recv_act@entry=0x7fd3c3ffedb0, timeout=<optimized out>) at gcs/src/gcs_core.cpp:1074
      #6  0x00007fd3dad86c73 in gcs_recv_thread (arg=0x55b638e55140) at gcs/src/gcs.cpp:1269
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 6 (Thread 0x7fd3a3fff700 (LWP 18963)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b635eb766e in os_event::timed_wait (this=this@entry=0x55b638fcc5d0, abstime=abstime@entry=0x7fd3a3ffec00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x000055b635eb7891 in os_event::wait_time_low (this=0x55b638fcc5d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x000055b635eb7b45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x000055b635f3c7a7 in srv_error_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1875
      #5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 5 (Thread 0x7fd3c9460700 (LWP 18954)):
      #0  0x00007fd3e29b0644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x000055b635eb2c94 in LinuxAIOHandler::collect (this=this@entry=0x7fd3c945fba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x000055b635eb30b4 in LinuxAIOHandler::poll (this=this@entry=0x7fd3c945fba0, m1=m1@entry=0x7fd3c945fc40, m2=m2@entry=0x7fd3c945fc48, request=request@entry=0x7fd3c945fc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x000055b635eb6cb7 in os_aio_linux_handler (request=0x7fd3c945fc50, m2=0x7fd3c945fc48, m1=0x7fd3c945fc40, global_segment=1) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7fd3c945fc40, m2=m2@entry=0x7fd3c945fc48, request=request@entry=0x7fd3c945fc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x000055b63601a663 in fil_aio_wait (segment=segment@entry=1) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x000055b635f3f5e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 4 (Thread 0x7fd3db857700 (LWP 18588)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b6361a0c9e in inline_mysql_cond_timedwait (that=0x55b6373abde0 <COND_timer>, mutex=0x55b6373abe20 <LOCK_timer>, src_file=0x55b63635e650 "/home/qa-user1/rr/server/mysys/thr_timer.c", src_line=292, abstime=0x7fd3db856d70) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204
      #2  timer_handler (arg=arg@entry=0x0) at /home/qa-user1/rr/server/mysys/thr_timer.c:292
      #3  0x000055b636161134 in pfs_spawn_thread (arg=0x55b638e16a98) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #4  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 3 (Thread 0x7fd3cbffe700 (LWP 18952)):
      #0  0x00007fd3e2bbdcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x000055b6360afb65 in inline_mysql_cond_timedwait (src_file=0x55b63634a2c8 "/home/qa-user1/rr/server/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7fd3cbffdd00, mutex=0x55b63739c380 <LOCK_checkpoint>, that=0x55b63739c340 <COND_checkpoint>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204
      #2  my_service_thread_sleep (control=control@entry=0x55b636a9d940 <checkpoint_control>, sleep_time=<optimized out>) at /home/qa-user1/rr/server/storage/maria/ma_servicethread.c:115
      #3  0x000055b6360a9402 in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/qa-user1/rr/server/storage/maria/ma_checkpoint.c:707
      #4  0x000055b636161134 in pfs_spawn_thread (arg=0x55b638ec7818) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #5  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 2 (Thread 0x7fd3da8c6700 (LWP 18590)):
      #0  0x00007fd3e2bbd945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fd3dadb7b8d in wait (this=0x7fd3da8c5c60, cond=...) at galerautils/src/gu_lock.hpp:59
      #2  galera::ServiceThd::thd_func (arg=0x55b638e4db40) at galera/src/galera_service_thd.cpp:30
      #3  0x00007fd3e2bb9e25 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fd3e11a134d in clone () from /lib64/libc.so.6
       
      Thread 1 (Thread 0x7fd3e2fd6840 (LWP 18584)):
      #0  0x00007fd3e2bbe9b1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x000055b635cd75ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305
      #2  <signal handler called>
      #3  0x00007fd3e1196a3d in poll () from /lib64/libc.so.6
      #4  0x000055b635a7a57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffc2f47fbf0) at /usr/include/bits/poll2.h:41
      #5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707
      #6  0x000055b635a7d759 in mysqld_main (argc=156, argv=0x55b638cd9b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176
      #7  0x00007fd3e10cac05 in __libc_start_main () from /lib64/libc.so.6
      #8  0x000055b635a71398 in _start ()
       
       - found 'core.18586' (1/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace from coredump /home/qa-user1/rr/build-mariadb/mysql-test/var/log/galera_3nodes.galera_pc_weight-innodb/mysqld.3/data/core.18586
      Core generated by '/home/qa-user1/rr/build-mariadb/sql/mysqld'
      Output from gdb follows. The first stack trace is from the failing thread.
      The following stack traces are from all threads (so the failing one is
      duplicated).
      --------------------------
      [New LWP 18586]
      [New LWP 18604]
      [New LWP 18591]
      [New LWP 18627]
      [New LWP 18606]
      [New LWP 18879]
      [New LWP 18625]
      [New LWP 18626]
      [New LWP 18883]
      [New LWP 18884]
      [New LWP 18886]
      [New LWP 18892]
      [New LWP 18893]
      [New LWP 18894]
      [New LWP 18895]
      [New LWP 18897]
      [New LWP 18898]
      [New LWP 18899]
      [New LWP 18900]
      [New LWP 18901]
      [New LWP 18905]
      [New LWP 18906]
      [New LWP 18986]
      [New LWP 18880]
      [New LWP 18881]
      [New LWP 18882]
      [New LWP 18896]
      [New LWP 18902]
      [New LWP 18903]
      [New LWP 18904]
      [New LWP 18907]
      [New LWP 18885]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib64/libthread_db.so.1".
      Core was generated by `/home/qa-user1/rr/build-mariadb/sql/mysqld --defaults-group-suffix=.3 --default'.
      Program terminated with signal 6, Aborted.
      #0  0x00007fb5cec2f9b1 in pthread_kill () from /lib64/libpthread.so.0
      #0  0x00007fb5cec2f9b1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x00005619f75ab5ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305
      #2  <signal handler called>
      #3  0x00007fb5cd207a3d in poll () from /lib64/libc.so.6
      #4  0x00005619f734e57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffe0026f720) at /usr/include/bits/poll2.h:41
      #5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707
      #6  0x00005619f7351759 in mysqld_main (argc=156, argv=0x5619fa929b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176
      #7  0x00007fb5cd13bc05 in __libc_start_main () from /lib64/libc.so.6
      #8  0x00005619f7345398 in _start ()
       
      Thread 32 (Thread 0x7fb59e7fc700 (LWP 18885)):
      #0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59e7fbba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59e7fbba0, m1=m1@entry=0x7fb59e7fbc40, m2=m2@entry=0x7fb59e7fbc48, request=request@entry=0x7fb59e7fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59e7fbc50, m2=0x7fb59e7fbc48, m1=0x7fb59e7fbc40, global_segment=5) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7fb59e7fbc40, m2=m2@entry=0x7fb59e7fbc48, request=request@entry=0x7fb59e7fbc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=5) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 31 (Thread 0x7fb5b569e700 (LWP 18907)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f7363ca4 in inline_mysql_cond_wait (src_file=0x5619f7adc3b8 "/home/qa-user1/rr/server/sql/slave.cc", src_line=506, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167
      #2  handle_slave_background (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/slave.cc:506
      #3  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fae94598) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 30 (Thread 0x7fb58a7fc700 (LWP 18904)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b791 in wait (this=0x5619fac1c7d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x5619fac1c7d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x00005619f7899f2a in buf_resize_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0buf.cc:3124
      #5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 29 (Thread 0x7fb58affd700 (LWP 18903)):
      #0  0x00007fb5cec31e4d in nanosleep () from /lib64/libpthread.so.0
      #1  0x00005619f778bd1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230
      #2  0x00005619f788b7ba in btr_defragment_thread () at /home/qa-user1/rr/server/storage/innobase/btr/btr0defragment.cc:768
      #3  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 28 (Thread 0x7fb59d7fa700 (LWP 18902)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b791 in wait (this=0x5619fac1c6d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x5619fac1c6d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x00005619f78a0b7e in buf_dump_thread () at /home/qa-user1/rr/server/storage/innobase/buf/buf0dump.cc:837
      #5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 27 (Thread 0x7fb5927fc700 (LWP 18896)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fad8bdb0, abstime=abstime@entry=0x7fb5927fbb60) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fad8bdb0, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x00005619f7858000 in ib_wqueue_timedwait (wq=wq@entry=0x5619fad7e400, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/qa-user1/rr/server/storage/innobase/ut/ut0wqueue.cc:163
      #5  0x00005619f792b07d in fts_optimize_thread (arg=0x5619fad7e400) at /home/qa-user1/rr/server/storage/innobase/fts/fts0opt.cc:3035
      #6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 26 (Thread 0x7fb59ffff700 (LWP 18882)):
      #0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59fffeba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59fffeba0, m1=m1@entry=0x7fb59fffec40, m2=m2@entry=0x7fb59fffec48, request=request@entry=0x7fb59fffec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59fffec50, m2=0x7fb59fffec48, m1=0x7fb59fffec40, global_segment=2) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7fb59fffec40, m2=m2@entry=0x7fb59fffec48, request=request@entry=0x7fb59fffec50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=2) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 25 (Thread 0x7fb5b4ce7700 (LWP 18881)):
      #0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb5b4ce6ba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb5b4ce6ba0, m1=m1@entry=0x7fb5b4ce6c40, m2=m2@entry=0x7fb5b4ce6c48, request=request@entry=0x7fb5b4ce6c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb5b4ce6c50, m2=0x7fb5b4ce6c48, m1=0x7fb5b4ce6c40, global_segment=1) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7fb5b4ce6c40, m2=m2@entry=0x7fb5b4ce6c48, request=request@entry=0x7fb5b4ce6c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=1) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 24 (Thread 0x7fb5b54e8700 (LWP 18880)):
      #0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb5b54e7ba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb5b54e7ba0, m1=m1@entry=0x7fb5b54e7c40, m2=m2@entry=0x7fb5b54e7c48, request=request@entry=0x7fb5b54e7c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb5b54e7c50, m2=0x7fb5b54e7c48, m1=0x7fb5b54e7c40, global_segment=0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7fb5b54e7c40, m2=m2@entry=0x7fb5b54e7c48, request=request@entry=0x7fb5b54e7c50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=0) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 23 (Thread 0x7fb5b5654700 (LWP 18986)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f734970b in inline_mysql_cond_timedwait (src_file=0x5619f7ad2048 "/home/qa-user1/rr/server/sql/mysqld.cc", src_line=3001, abstime=0x7fb5b5653d20, mutex=0x5619f83f08a0 <LOCK_thread_cache>, that=0x5619f83f3060 <COND_thread_cache>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204
      #2  cache_thread (thd=0x7fb584000a88) at /home/qa-user1/rr/server/sql/mysqld.cc:3000
      #3  one_thread_per_connection_end (thd=0x7fb584000a88, put_in_cache=<optimized out>) at /home/qa-user1/rr/server/sql/mysqld.cc:3087
      #4  0x00005619f74bfaff in do_handle_one_connection (connect=connect@entry=0x5619fae62138) at /home/qa-user1/rr/server/sql/sql_connect.cc:1421
      #5  0x00005619f74bfc94 in handle_one_connection (arg=arg@entry=0x5619fae62138) at /home/qa-user1/rr/server/sql/sql_connect.cc:1308
      #6  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fae9e498) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 22 (Thread 0x7fb5b56e8700 (LWP 18906)):
      #0  0x00007fb5cec32371 in sigwait () from /lib64/libpthread.so.0
      #1  0x00005619f7348053 in signal_hand (arg=arg@entry=0x0) at /home/qa-user1/rr/server/sql/mysqld.cc:3550
      #2  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fabd0b48) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #3  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 21 (Thread 0x7fb589ffb700 (LWP 18905)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f77174c3 in inline_mysql_cond_wait (src_file=0x5619f7bdf670 "/home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc", src_line=315, mutex=0x7fb589ffad50, that=0x7fb589ffad80) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1178
      #2  thd_destructor_proxy () at /home/qa-user1/rr/server/storage/innobase/handler/ha_innodb.cc:315
      #3  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fad98518) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 20 (Thread 0x7fb58b7fe700 (LWP 18901)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b791 in wait (this=0x5619fac1c4d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x5619fac1c4d0, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836ed70 <srv_sys+304>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x00005619f7812834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531
      #6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 19 (Thread 0x7fb58bfff700 (LWP 18900)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b791 in wait (this=0x5619fac1c450) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x5619fac1c450, reset_sig_count=10, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836ed38 <srv_sys+248>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x00005619f7812834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531
      #6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 18 (Thread 0x7fb590ff9700 (LWP 18899)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b791 in wait (this=0x5619fac1c550) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x5619fac1c550, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836eda8 <srv_sys+360>, sig_count=sig_count@entry=0, wait=wait@entry=true, timeout_usec=timeout_usec@entry=0) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x00005619f7812834 in srv_worker_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2531
      #6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 17 (Thread 0x7fb5917fa700 (LWP 18898)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b791 in wait (this=0x5619fac1c3d0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:163
      #2  os_event::wait_low (this=0x5619fac1c3d0, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:333
      #3  0x00005619f778bb55 in os_event_wait_low (event=<optimized out>, reset_sig_count=reset_sig_count@entry=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:527
      #4  0x00005619f780d553 in srv_resume_thread (slot=slot@entry=0x5619f836ed00 <srv_sys+192>, sig_count=sig_count@entry=1, wait=<optimized out>, timeout_usec=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:912
      #5  0x00005619f7812f56 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2691
      #6  srv_purge_coordinator_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2778
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 16 (Thread 0x7fb591ffb700 (LWP 18897)):
      #0  0x00007fb5cec31e4d in nanosleep () from /lib64/libpthread.so.0
      #1  0x00005619f778bd1c in os_thread_sleep (tm=tm@entry=1000000) at /home/qa-user1/rr/server/storage/innobase/os/os0thread.cc:230
      #2  0x00005619f7811fcc in srv_master_sleep () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2364
      #3  srv_master_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:2409
      #4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 15 (Thread 0x7fb592ffd700 (LWP 18895)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fadf0e30, abstime=abstime@entry=0x7fb592ffcc90) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fadf0e30, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=1, reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=reset_sig_count@entry=0) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x00005619f78e70b5 in dict_stats_thread () at /home/qa-user1/rr/server/storage/innobase/dict/dict0stats_bg.cc:451
      #5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 14 (Thread 0x7fb5937fe700 (LWP 18894)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fac1c650, abstime=abstime@entry=0x7fb5937fdd00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fac1c650, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x00005619f7811820 in srv_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1711
      #5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 13 (Thread 0x7fb593fff700 (LWP 18893)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fac1c5d0, abstime=abstime@entry=0x7fb593ffec00) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fac1c5d0, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x00005619f78107a7 in srv_error_monitor_thread () at /home/qa-user1/rr/server/storage/innobase/srv/srv0srv.cc:1875
      #5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 12 (Thread 0x7fb59cff9700 (LWP 18892)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fadbd610, abstime=abstime@entry=0x7fb59cff8c70) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x00005619f778b891 in os_event::wait_time_low (this=this@entry=0x5619fadbd610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=1) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x00005619f778bb45 in os_event_wait_time_low (event=event@entry=0x5619fadbd610, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x00005619f77662e5 in lock_wait_timeout_thread () at /home/qa-user1/rr/server/storage/innobase/lock/lock0wait.cc:547
      #5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 11 (Thread 0x7fb59dffb700 (LWP 18886)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f778b66e in os_event::timed_wait (this=this@entry=0x5619fac1c750, abstime=abstime@entry=0x7fb59dffab80) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:283
      #2  0x00005619f778b891 in os_event::wait_time_low (this=0x5619fac1c750, time_in_usec=time_in_usec@entry=956000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:405
      #3  0x00005619f778bb45 in os_event_wait_time_low (event=<optimized out>, time_in_usec=time_in_usec@entry=956000, reset_sig_count=reset_sig_count@entry=2) at /home/qa-user1/rr/server/storage/innobase/os/os0event.cc:510
      #4  0x00005619f78a6847 in pc_sleep_if_needed (cur_time=1524765010939, sig_count=2, next_loop_time=1524765011895) at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:2688
      #5  buf_flush_page_cleaner_coordinator () at /home/qa-user1/rr/server/storage/innobase/buf/buf0flu.cc:3154
      #6  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 10 (Thread 0x7fb59effd700 (LWP 18884)):
      #0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59effcba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59effcba0, m1=m1@entry=0x7fb59effcc40, m2=m2@entry=0x7fb59effcc48, request=request@entry=0x7fb59effcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59effcc50, m2=0x7fb59effcc48, m1=0x7fb59effcc40, global_segment=4) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7fb59effcc40, m2=m2@entry=0x7fb59effcc48, request=request@entry=0x7fb59effcc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=4) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 9 (Thread 0x7fb59f7fe700 (LWP 18883)):
      #0  0x00007fb5cea21644 in __io_getevents_0_4 () from /lib64/libaio.so.1
      #1  0x00005619f7786c94 in LinuxAIOHandler::collect (this=this@entry=0x7fb59f7fdba0) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:1922
      #2  0x00005619f77870b4 in LinuxAIOHandler::poll (this=this@entry=0x7fb59f7fdba0, m1=m1@entry=0x7fb59f7fdc40, m2=m2@entry=0x7fb59f7fdc48, request=request@entry=0x7fb59f7fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2067
      #3  0x00005619f778acb7 in os_aio_linux_handler (request=0x7fb59f7fdc50, m2=0x7fb59f7fdc48, m1=0x7fb59f7fdc40, global_segment=3) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:2121
      #4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7fb59f7fdc40, m2=m2@entry=0x7fb59f7fdc48, request=request@entry=0x7fb59f7fdc50) at /home/qa-user1/rr/server/storage/innobase/os/os0file.cc:5649
      #5  0x00005619f78ee663 in fil_aio_wait (segment=segment@entry=3) at /home/qa-user1/rr/server/storage/innobase/fil/fil0fil.cc:4653
      #6  0x00005619f78135e8 in io_handler_thread (arg=<optimized out>) at /home/qa-user1/rr/server/storage/innobase/srv/srv0start.cc:339
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 8 (Thread 0x7fb5c4132700 (LWP 18626)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f752ed15 in inline_mysql_cond_wait (src_file=0x5619f7b099e8 "/home/qa-user1/rr/server/sql/wsrep_thd.cc", src_line=470, mutex=<optimized out>, that=<optimized out>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1167
      #2  wsrep_rollback_process (thd=0x7fb5ac0009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:470
      #3  0x00005619f75235ca in start_wsrep_THD (arg=0x5619f752ec40 <wsrep_rollback_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980
      #4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 7 (Thread 0x7fb5b7ffe700 (LWP 18625)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fb5c6dfbdd2 in wait (this=0x7fb5b7ffd100, cond=...) at galerautils/src/gu_lock.hpp:59
      #2  front (timeout=..., this=0x5619faae4cd0) at gcs/src/gcs_gcomm.cpp:124
      #3  gcomm_recv (backend=<optimized out>, msg=0x5619faaa53a0, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673
      #4  0x00007fb5c6df0dd2 in core_msg_recv (timeout=9223372035999999999, recv_msg=0x5619faaa53a0, backend=0x5619faaa54b8) at gcs/src/gcs_core.cpp:468
      #5  gcs_core_recv (conn=0x5619faaa5340, recv_act=recv_act@entry=0x7fb5b7ffddb0, timeout=<optimized out>) at gcs/src/gcs_core.cpp:1074
      #6  0x00007fb5c6df7c73 in gcs_recv_thread (arg=0x5619faaa5150) at gcs/src/gcs.cpp:1269
      #7  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #8  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 6 (Thread 0x7fb5b77fd700 (LWP 18879)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f7983b65 in inline_mysql_cond_timedwait (src_file=0x5619f7c1e2c8 "/home/qa-user1/rr/server/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7fb5b77fcd00, mutex=0x5619f8c70380 <LOCK_checkpoint>, that=0x5619f8c70340 <COND_checkpoint>) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204
      #2  my_service_thread_sleep (control=control@entry=0x5619f8371940 <checkpoint_control>, sleep_time=<optimized out>) at /home/qa-user1/rr/server/storage/maria/ma_servicethread.c:115
      #3  0x00005619f797d402 in ma_checkpoint_background (arg=arg@entry=0x1e) at /home/qa-user1/rr/server/storage/maria/ma_checkpoint.c:707
      #4  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619fab17808) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #5  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #6  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 5 (Thread 0x7fb5c4933700 (LWP 18606)):
      #0  0x00007fb5cd212923 in epoll_wait () from /lib64/libc.so.6
      #1  0x00007fb5c6dccbd8 in asio::detail::epoll_reactor::run (this=0x5619faae5f90, block=<optimized out>, ops=...) at asio/asio/detail/impl/epoll_reactor.ipp:391
      #2  0x00007fb5c6dce9d0 in do_run_one (ec=..., this_thread=..., lock=..., this=0x5619faae5e30) at asio/asio/detail/impl/task_io_service.ipp:355
      #3  asio::detail::task_io_service::run (this=0x5619faae5e30, ec=...) at asio/asio/detail/impl/task_io_service.ipp:148
      #4  0x00007fb5c6de5f21 in run (this=0x5619faae5af0) at asio/asio/impl/io_service.ipp:58
      #5  gcomm::AsioProtonet::event_loop (this=0x5619faae5a50, period=...) at gcomm/src/asio_protonet.cpp:121
      #6  0x00007fb5c6dfe237 in GCommConn::run (this=0x5619faae4b00) at gcs/src/gcs_gcomm.cpp:498
      #7  0x00007fb5c6e03929 in GCommConn::run_fn (arg=<optimized out>) at gcs/src/gcs_gcomm.cpp:209
      #8  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 4 (Thread 0x7fb5c40e8700 (LWP 18627)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fb5c6cdceae in fifo_lock_get (q=0x5619faacb760) at galerautils/src/gu_fifo.c:261
      #2  gu_fifo_get_head (q=0x5619faacb760, err=0x7fb5c40e7920) at galerautils/src/gu_fifo.c:329
      #3  0x00007fb5c6df6605 in gcs_recv (conn=0x5619faaa5150, action=0x7fb5c40e7980) at gcs/src/gcs.cpp:1865
      #4  0x00007fb5c6e30644 in galera::GcsActionSource::process (this=0x5619faa9dc88, recv_ctx=0x7fb5a00009a8, exit_loop=@0x7fb5c40e7a00: false) at galera/src/gcs_action_source.cpp:175
      #5  0x00007fb5c6e5269b in galera::ReplicatorSMM::async_recv (this=0x5619faa9d600, recv_ctx=0x7fb5a00009a8) at galera/src/replicator_smm.cpp:361
      #6  0x00007fb5c6e60da8 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at galera/src/wsrep_provider.cpp:244
      #7  0x00005619f752e83d in wsrep_replication_process (thd=0x7fb5a00009a8) at /home/qa-user1/rr/server/sql/wsrep_thd.cc:362
      #8  0x00005619f75235ca in start_wsrep_THD (arg=0x5619f752e800 <wsrep_replication_process(THD*)>) at /home/qa-user1/rr/server/sql/wsrep_mysqld.cc:1980
      #9  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #10 0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 3 (Thread 0x7fb5c78c8700 (LWP 18591)):
      #0  0x00007fb5cec2ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005619f7a74c9e in inline_mysql_cond_timedwait (that=0x5619f8c7fde0 <COND_timer>, mutex=0x5619f8c7fe20 <LOCK_timer>, src_file=0x5619f7c32650 "/home/qa-user1/rr/server/mysys/thr_timer.c", src_line=292, abstime=0x7fb5c78c7d70) at /home/qa-user1/rr/server/include/mysql/psi/mysql_thread.h:1204
      #2  timer_handler (arg=arg@entry=0x0) at /home/qa-user1/rr/server/mysys/thr_timer.c:292
      #3  0x00005619f7a35134 in pfs_spawn_thread (arg=0x5619faa66a98) at /home/qa-user1/rr/server/storage/perfschema/pfs.cc:1862
      #4  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 2 (Thread 0x7fb5c6937700 (LWP 18604)):
      #0  0x00007fb5cec2e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00007fb5c6e28b8d in wait (this=0x7fb5c6936c60, cond=...) at galerautils/src/gu_lock.hpp:59
      #2  galera::ServiceThd::thd_func (arg=0x5619faa9db50) at galera/src/galera_service_thd.cpp:30
      #3  0x00007fb5cec2ae25 in start_thread () from /lib64/libpthread.so.0
      #4  0x00007fb5cd21234d in clone () from /lib64/libc.so.6
       
      Thread 1 (Thread 0x7fb5cf047840 (LWP 18586)):
      #0  0x00007fb5cec2f9b1 in pthread_kill () from /lib64/libpthread.so.0
      #1  0x00005619f75ab5ec in handle_fatal_signal (sig=6) at /home/qa-user1/rr/server/sql/signal_handler.cc:305
      #2  <signal handler called>
      #3  0x00007fb5cd207a3d in poll () from /lib64/libc.so.6
      #4  0x00005619f734e57b in poll (__timeout=-1, __nfds=2, __fds=0x7ffe0026f720) at /usr/include/bits/poll2.h:41
      #5  handle_connections_sockets () at /home/qa-user1/rr/server/sql/mysqld.cc:6707
      #6  0x00005619f7351759 in mysqld_main (argc=156, argv=0x5619fa929b68) at /home/qa-user1/rr/server/sql/mysqld.cc:6176
      #7  0x00007fb5cd13bc05 in __libc_start_main () from /lib64/libc.so.6
      #8  0x00005619f7345398 in _start ()
       
      --------------------------------------------------------------------------
      The servers were restarted 0 times
      Spent 0.000 of 7380 seconds executing testcases
       
      Failure: Failed 1/1 tests, 0.00% were successful.
      
      

        Attachments

          Activity

            People

            Assignee:
            stepan.patryshev Stepan Patryshev (Inactive)
            Reporter:
            winstone Zdravelina Sokolovska (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.