Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6860

Parallel async replication hangs on a Galera node

Details

    Description

      In a setup where a master replicates to a Galera node acting as a slave with slave-parallel-threads > 1, the slave threads hang which brings replication to a halt.

      How to reproduce:

      • Run the attached script on master.

      On slave:

      MariaDB [test]> show processlist;
      +----+-------------+-----------------+------+---------+------+--------------------------------------------------------------------------------+------------------+----------+
      | Id | User        | Host            | db   | Command | Time | State                                                                          | Info             | Progress |
      +----+-------------+-----------------+------+---------+------+--------------------------------------------------------------------------------+------------------+----------+
      |  1 | system user |                 | NULL | Sleep   |  840 | NULL                                                                           | NULL             |    0.000 |
      |  2 | system user |                 | NULL | Sleep   |  840 | wsrep aborter idle                                                             | NULL             |    0.000 |
      |  5 | system user |                 | NULL | Sleep   |  838 | NULL                                                                           | NULL             |    0.000 |
      |  6 | system user |                 | NULL | Sleep   |  838 | NULL                                                                           | NULL             |    0.000 |
      |  7 | system user |                 | NULL | Sleep   |  838 | NULL                                                                           | NULL             |    0.000 |
      |  8 | system user |                 | NULL | Connect |  263 | Waiting for prior transaction to start commit before starting next transaction | NULL             |    0.000 |
      |  9 | system user |                 | NULL | Connect |  263 | closing tables                                                                 | NULL             |    0.000 |
      | 10 | system user |                 | NULL | Connect |  263 | closing tables                                                                 | NULL             |    0.000 |
      | 11 | system user |                 | NULL | Connect |  263 | closing tables                                                                 | NULL             |    0.000 |
      | 12 | root        | localhost:52597 | test | Query   |    0 | init                                                                           | show processlist |    0.000 |
      | 13 | system user |                 | NULL | Connect |  336 | Waiting for master to send event                                               | NULL             |    0.000 |
      | 14 | system user |                 | NULL | Connect |  336 | Slave has read all relay log; waiting for the slave I/O thread to update it    | NULL             |    0.000 |
      +----+-------------+-----------------+------+---------+------+--------------------------------------------------------------------------------+------------------+----------+
      12 rows in set (0.00 sec)
       
      MariaDB [test]> show slave status\G
      *************************** 1. row ***************************
                     Slave_IO_State: Waiting for master to send event
                        Master_Host: 127.0.0.1
                        Master_User: root
                        Master_Port: 15999
                      Connect_Retry: 60
                    Master_Log_File: nirbhay-VirtualBox-bin.000001
                Read_Master_Log_Pos: 146163
                     Relay_Log_File: nirbhay-VirtualBox-relay-bin.000002
                      Relay_Log_Pos: 1463
              Relay_Master_Log_File: nirbhay-VirtualBox-bin.000001
                   Slave_IO_Running: Yes
                  Slave_SQL_Running: Yes
                    Replicate_Do_DB: 
                Replicate_Ignore_DB: 
                 Replicate_Do_Table: 
             Replicate_Ignore_Table: 
            Replicate_Wild_Do_Table: 
        Replicate_Wild_Ignore_Table: 
                         Last_Errno: 0
                         Last_Error: 
                       Skip_Counter: 0
                Exec_Master_Log_Pos: 1163
                    Relay_Log_Space: 146773
                    Until_Condition: None
                     Until_Log_File: 
                      Until_Log_Pos: 0
                 Master_SSL_Allowed: No
                 Master_SSL_CA_File: 
                 Master_SSL_CA_Path: 
                    Master_SSL_Cert: 
                  Master_SSL_Cipher: 
                     Master_SSL_Key: 
              Seconds_Behind_Master: 268
      Master_SSL_Verify_Server_Cert: No
                      Last_IO_Errno: 0
                      Last_IO_Error: 
                     Last_SQL_Errno: 0
                     Last_SQL_Error: 
        Replicate_Ignore_Server_Ids: 
                   Master_Server_Id: 1
                     Master_SSL_Crl: 
                 Master_SSL_Crlpath: 
                         Using_Gtid: Slave_Pos
                        Gtid_IO_Pos: 0-1-1006
      1 row in set (0.00 sec)
      

      The node also hangs when shutdown.

      141009 11:38:17 [Note] ./bin/mysqld: Normal shutdown
       
      141009 11:38:17 [Note] WSREP: Stop replication
      141009 11:38:17 [Note] WSREP: Provider disconnect
      141009 11:38:17 [Note] WSREP: Closing send monitor...
      141009 11:38:17 [Note] WSREP: Closed send monitor.
      141009 11:38:17 [Note] WSREP: gcomm: terminating thread
      141009 11:38:17 [Note] WSREP: gcomm: joining thread
      141009 11:38:17 [Note] WSREP: gcomm: closing backend
      141009 11:38:17 [Note] WSREP: view((empty))
      141009 11:38:17 [Note] WSREP: Received self-leave message.
      141009 11:38:17 [Note] WSREP: gcomm: closed
      141009 11:38:17 [Note] WSREP: Flow-control interval: [0, 0]
      141009 11:38:17 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      141009 11:38:17 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 10)
      141009 11:38:17 [Note] WSREP: RECV thread exiting 0: Success
      141009 11:38:17 [Note] WSREP: recv_thread() joined.
      141009 11:38:17 [Note] WSREP: Closing replication queue.
      141009 11:38:17 [Note] WSREP: Closing slave action queue.
      141009 11:38:19 [Note] WSREP: waiting for client connections to close: 5
      

      Why this is happening

      So the actual issue , Galera sends event at the time of prepare phase
      and And creates deadlock.

      For example lets us consider the replication A -> B <==> C (A,B
      parallel replication optimistic, B,C Galera cluster nodes)
      Lets assume 2 inserts(T1 gtid x-x-1 and T2 x-x-2) from master A arrive
      to slave B.
      2nd insert prepares faster then 1st insert, So it has already sent the
      writeset to node C. Now it is the queue waiting for its turn to commit
      While the first insert does prepare on galera
      (wsrep_run_wsrep_commit), but it is stuck because T2 transaction still
      haven't run post_commit on galera
      so galera state is still in S_WAITING
      T2 cant run post_commit on galera because it is waiting for T1 commit
      , T1 cant commit because it is waiting in prepare stage for
      transaction T2 to clear the galera state.
      Backtrace from gdb

       
       
      Gtid_seq_no= 2
      Thread 34 (Thread 0x7fcd966d2700 (LWP 23891)):
      #0  0x00007fcda6d56415 in pthread_cond_wait@@GLIBC_2.3.2 () from
      /usr/lib/libpthread.so.0
      #1  0x00005569d607d380 in safe_cond_wait (cond=0x7fcd854078e8,
      mp=0x7fcd85407838, file=0x5569d6240360
      "/home/sachin/10.1/server/include/mysql/psi/mysql_thread.h",
      line=1154) at /home/sachin/10.1/server/mysys/thr_mutex.c:493
      #2  0x00005569d5aec4d0 in inline_mysql_cond_wait (that=0x7fcd854078e8,
      mutex=0x7fcd85407838, src_file=0x5569d6240cb8
      "/home/sachin/10.1/server/sql/log.cc", src_line=7387) at
      /home/sachin/10.1/server/include/mysql/psi/mysql_thread.h:1154
      #3  0x00005569d5afeee5 in MYSQL_BIN_LOG::queue_for_group_commit
      (this=0x5569d692d7c0 <mysql_bin_log>, orig_entry=0x7fcd966cf440) at
      /home/sachin/10.1/server/sql/log.cc:7387
      #4  0x00005569d5aff5c9 in
      MYSQL_BIN_LOG::write_transaction_to_binlog_events (this=0x5569d692d7c0
      <mysql_bin_log>, entry=0x7fcd966cf440) at
      /home/sachin/10.1/server/sql/log.cc:7607
      #5  0x00005569d5afecff in MYSQL_BIN_LOG::write_transaction_to_binlog
      (this=0x5569d692d7c0 <mysql_bin_log>, thd=0x7fcd84c068b0,
      cache_mngr=0x7fcd84c72c70, end_ev=0x7fcd966cf5e0, all=true,
      using_stmt_cache=true, using_trx_cache=true) at
      /home/sachin/10.1/server/sql/log.cc:7290
      #6  0x00005569d5af0ce6 in binlog_flush_cache (thd=0x7fcd84c068b0,
      cache_mngr=0x7fcd84c72c70, end_ev=0x7fcd966cf5e0, all=true,
      using_stmt=true, using_trx=true) at
      /home/sachin/10.1/server/sql/log.cc:1751
      #7  0x00005569d5af11bb in binlog_commit_flush_xid_caches
      (thd=0x7fcd84c068b0, cache_mngr=0x7fcd84c72c70, all=true, xid=2) at
      /home/sachin/10.1/server/sql/log.cc:1859
      #8  0x00005569d5b045c8 in MYSQL_BIN_LOG::log_and_order
      (this=0x5569d692d7c0 <mysql_bin_log>, thd=0x7fcd84c068b0, xid=2,
      all=true, need_prepare_ordered=false, need_commit_ordered=true) at
      /home/sachin/10.1/server/sql/log.cc:9575
      #9  0x00005569d5a1ec0d in ha_commit_trans (thd=0x7fcd84c068b0,
      all=true) at /home/sachin/10.1/server/sql/handler.cc:1497
      #10 0x00005569d5925e7e in trans_commit (thd=0x7fcd84c068b0) at
      /home/sachin/10.1/server/sql/transaction.cc:235
      #11 0x00005569d5b1b1fa in Xid_log_event::do_apply_event
      (this=0x7fcd8542a770, rgi=0x7fcd85407800) at
      /home/sachin/10.1/server/sql/log_event.cc:7720
      #12 0x00005569d5743fa1 in Log_event::apply_event (this=0x7fcd8542a770,
      rgi=0x7fcd85407800) at /home/sachin/10.1/server/sql/log_event.h:1343
      #13 0x00005569d573987e in apply_event_and_update_pos_apply
      (ev=0x7fcd8542a770, thd=0x7fcd84c068b0, rgi=0x7fcd85407800, reason=0)
      at /home/sachin/10.1/server/sql/slave.cc:3479
      #14 0x00005569d5739deb in apply_event_and_update_pos_for_parallel
      (ev=0x7fcd8542a770, thd=0x7fcd84c068b0, rgi=0x7fcd85407800) at
      /home/sachin/10.1/server/sql/slave.cc:3623
      #15 0x00005569d597bfbe in rpt_handle_event (qev=0x7fcd85424770,
      rpt=0x7fcd85421c88) at /home/sachin/10.1/server/sql/rpl_parallel.cc:50
      #16 0x00005569d597ed57 in handle_rpl_parallel_thread
      (arg=0x7fcd85421c88) at
      /home/sachin/10.1/server/sql/rpl_parallel.cc:1258
      #17 0x00005569d5d42aa4 in pfs_spawn_thread (arg=0x7fcd85415570) at
      /home/sachin/10.1/server/storage/perfschema/pfs.cc:1861
      #18 0x00007fcda6d5057f in start_thread () from /usr/lib/libpthread.so.0
      #19 0x00007fcda5f0f0e3 in clone () from /usr/lib/libc.so.6
       
      Gtid_seq_no= 1
      Thread 33 (Thread 0x7fcd9671d700 (LWP 23890)):
      #0  0x00007fcda6d56415 in pthread_cond_wait@@GLIBC_2.3.2 () from
      /usr/lib/libpthread.so.0
      #1  0x00007fcd9e7778ab in gu::Lock::wait (this=0x7fcd9671a0c0,
      cond=...) at galerautils/src/gu_mutex.hpp:40
      #2  galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter
      (this=this@entry=0x7fcda12d5da0, obj=...) at
      galera/src/monitor.hpp:124
      #3  0x00007fcd9e771f28 in galera::ReplicatorSMM::pre_commit
      (this=0x7fcda12d5000, trx=0x7fcd8507e000, meta=<optimized out>) at
      galera/src/replicator_smm.cpp:796
      #5  0x00005569d59864d0 in wsrep_run_wsrep_commit (thd=0x7fcd85006a70,
      all=true) at /home/sachin/10.1/server/sql/wsrep_hton.cc:492
      #6  0x00005569d5984d6a in wsrep_prepare (hton=0x7fcda583e270,
      thd=0x7fcd85006a70, all=true) at
      /home/sachin/10.1/server/sql/wsrep_hton.cc:208
      #7  0x00005569d5a1e1b0 in prepare_or_error (ht=0x7fcda583e270,
      thd=0x7fcd85006a70, all=true) at
      /home/sachin/10.1/server/sql/handler.cc:1196
      #8  0x00005569d5a1ea1c in ha_commit_trans (thd=0x7fcd85006a70,
      all=true) at /home/sachin/10.1/server/sql/handler.cc:1475
      #9  0x00005569d5925e7e in trans_commit (thd=0x7fcd85006a70) at
      /home/sachin/10.1/server/sql/transaction.cc:235
      #10 0x00005569d5b1b1fa in Xid_log_event::do_apply_event
      (this=0x7fcd8542a570, rgi=0x7fcd85407000) at
      /home/sachin/10.1/server/sql/log_event.cc:7720
      #11 0x00005569d5743fa1 in Log_event::apply_event (this=0x7fcd8542a570,
      rgi=0x7fcd85407000) at /home/sachin/10.1/server/sql/log_event.h:1343
      #12 0x00005569d573987e in apply_event_and_update_pos_apply
      (ev=0x7fcd8542a570, thd=0x7fcd85006a70, rgi=0x7fcd85407000, reason=0)
      at /home/sachin/10.1/server/sql/slave.cc:3479
      #13 0x00005569d5739deb in apply_event_and_update_pos_for_parallel
      (ev=0x7fcd8542a570, thd=0x7fcd85006a70, rgi=0x7fcd85407000) at
      /home/sachin/10.1/server/sql/slave.cc:3623
      #14 0x00005569d597bfbe in rpt_handle_event (qev=0x7fcd85423870,
      rpt=0x7fcd85421a80) at /home/sachin/10.1/server/sql/rpl_parallel.cc:50
      #15 0x00005569d597ed57 in handle_rpl_parallel_thread
      (arg=0x7fcd85421a80) at
      /home/sachin/10.1/server/sql/rpl_parallel.cc:1258
      #16 0x00005569d5d42aa4 in pfs_spawn_thread (arg=0x7fcd854152f0) at
      /home/sachin/10.1/server/storage/perfschema/pfs.cc:1861
      #17 0x00007fcda6d5057f in start_thread () from /usr/lib/libpthread.so.0
      #18 0x00007fcda5f0f0e3 in clone () from /usr/lib/libc.so.6
      

      Attachments

        1. 6860.cnf
          0.1 kB
          Sachin Setiya
        2. 6860.test
          1 kB
          Sachin Setiya
        3. load.sql
          0.4 kB
          Nirbhay Choubey
        4. slave.cnf
          0.5 kB
          Nirbhay Choubey

        Issue Links

          Activity

            This is thread back-trace of all threads

            (gdb) thread apply all bt
             
            Thread 45 (Thread 0x7f54d7cd9700 (LWP 18076)):
            #0  0x00007f54e493448d in poll () from /usr/lib/libc.so.6
            #1  0x0000000000eb5eda in vio_io_wait (vio=0x7f54b5f88f70, event=VIO_IO_EVENT_READ, timeout=28800000) at /home/xyz/mdev-6860/server/vio/viosocket.c:992
            #2  0x0000000000eb482d in vio_socket_io_wait (vio=0x7f54b5f88f70, event=VIO_IO_EVENT_READ) at /home/xyz/mdev-6860/server/vio/viosocket.c:108
            #3  0x0000000000eb49e0 in vio_read (vio=0x7f54b5f88f70, buf=0x7f54b612a870 "\001", size=4) at /home/xyz/mdev-6860/server/vio/viosocket.c:184
            #4  0x00000000005a656c in my_real_read (net=0x7f54e44cc0e8, complen=0x7f54d7cd8940, header=1 '\001') at /home/xyz/mdev-6860/server/sql/net_serv.cc:874
            #5  0x00000000005a6dbc in my_net_read_packet (net=0x7f54e44cc0e8, read_from_server=1 '\001') at /home/xyz/mdev-6860/server/sql/net_serv.cc:1137
            #6  0x0000000000655a29 in do_command (thd=0x7f54e44cbe30) at /home/xyz/mdev-6860/server/sql/sql_parse.cc:953
            #7  0x000000000077aa45 in do_handle_one_connection (thd_arg=0x7f54e44cbe30) at /home/xyz/mdev-6860/server/sql/sql_connect.cc:1395
            #8  0x000000000077a78b in handle_one_connection (arg=0x7f54e44cbe30) at /home/xyz/mdev-6860/server/sql/sql_connect.cc:1307
            #9  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #10 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 44 (Thread 0x7f54d7d22700 (LWP 18074)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000e6e45b in safe_cond_wait (cond=0x7f54c5ffcf20, mp=0x7f54c5ffc828, file=0xfa3b40 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h", line=1165) at /home/xyz/mdev-6860/server/mysys/thr_mutex.c:493
            #2  0x000000000092d5d5 in inline_mysql_cond_wait (that=0x7f54c5ffcf20, mutex=0x7f54c5ffc828, src_file=0xfa3fb8 "/home/xyz/mdev-6860/server/sql/log.cc", src_line=7917) at /home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h:1165
            #3  0x0000000000941198 in MYSQL_BIN_LOG::wait_for_update_relay_log (this=0x7f54c5ffc820, thd=0x7f54ae2b08b0) at /home/xyz/mdev-6860/server/sql/log.cc:7917
            #4  0x00000000005c9de8 in next_event (rgi=0x7f54ae2a6000, event_size=0x7f54d7d214b0) at /home/xyz/mdev-6860/server/sql/slave.cc:6740
            #5  0x00000000005c19d7 in exec_relay_log_event (thd=0x7f54ae2b08b0, rli=0x7f54c5ffc348, serial_rgi=0x7f54ae2a6000) at /home/xyz/mdev-6860/server/sql/slave.cc:3511
            #6  0x00000000005c4f8c in handle_slave_sql (arg=0x7f54c5ffa700) at /home/xyz/mdev-6860/server/sql/slave.cc:4756
            #7  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #8  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 43 (Thread 0x7f54d7d6b700 (LWP 18073)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f58e83 in gu::Lock::wait (this=0x7f54d7d69900, cond=...) at galerautils/src/gu_lock.hpp:56
            #2  0x00007f54e40f1ede in galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x7f54e452ad70, obj=...) at galera/src/monitor.hpp:123
            #3  0x00007f54e40e7bd8 in galera::ReplicatorSMM::pre_commit (this=0x7f54e452a000, trx=0x7f54b310a000, meta=0x7f54b301a820) at galera/src/replicator_smm.cpp:778
            #4  0x00007f54e4102a0f in galera_pre_commit (gh=0x7f54e444cc80, conn_id=15, trx_handle=0x7f54b301a858, flags=1, meta=0x7f54b301a820) at galera/src/wsrep_provider.cpp:518
            #5  0x00000000007e3b17 in wsrep_run_wsrep_commit (thd=0x7f54b3015630, hton=0x7f54e4571c70, all=true) at /home/xyz/mdev-6860/server/sql/wsrep_hton.cc:467
            #6  0x00000000007e250c in wsrep_prepare (hton=0x7f54e4571c70, thd=0x7f54b3015630, all=true) at /home/xyz/mdev-6860/server/sql/wsrep_hton.cc:192
            #7  0x0000000000867ec9 in ha_commit_trans (thd=0x7f54b3015630, all=true) at /home/xyz/mdev-6860/server/sql/handler.cc:1429
            #8  0x000000000078ec08 in trans_commit (thd=0x7f54b3015630) at /home/xyz/mdev-6860/server/sql/transaction.cc:239
            #9  0x0000000000958ded in Xid_log_event::do_apply_event (this=0x7f54ad6b4b70, rgi=0x7f54ad41f800) at /home/xyz/mdev-6860/server/sql/log_event.cc:7461
            #10 0x00000000005cb3fe in Log_event::apply_event (this=0x7f54ad6b4b70, rgi=0x7f54ad41f800) at /home/xyz/mdev-6860/server/sql/log_event.h:1360
            #11 0x00000000005c15e1 in apply_event_and_update_pos (ev=0x7f54ad6b4b70, thd=0x7f54b3015630, rgi=0x7f54ad41f800, rpt=0x7f54b6094ea8) at /home/xyz/mdev-6860/server/sql/slave.cc:3308
            #12 0x00000000007d898a in rpt_handle_event (qev=0x7f54ad428c70, rpt=0x7f54b6094ea8) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:52
            #13 0x00000000007db164 in handle_rpl_parallel_thread (arg=0x7f54b6094ea8) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:1201
            #14 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #15 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 42 (Thread 0x7f54d7db4700 (LWP 18072)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000e6e45b in safe_cond_wait (cond=0x7f54ad7148e8, mp=0x7f54ad714838, file=0xfa3b40 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h", line=1165) at /home/xyz/mdev-6860/server/mysys/thr_mutex.c:493
            #2  0x000000000092d5d5 in inline_mysql_cond_wait (that=0x7f54ad7148e8, mutex=0x7f54ad714838, src_file=0xfa3fb8 "/home/xyz/mdev-6860/server/sql/log.cc", src_line=7081) at /home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h:1165
            #3  0x000000000093f3e2 in MYSQL_BIN_LOG::queue_for_group_commit (this=0x17fdda0 <mysql_bin_log>, orig_entry=0x7f54d7db2ec0) at /home/xyz/mdev-6860/server/sql/log.cc:7081
            #4  0x000000000093fa6b in MYSQL_BIN_LOG::write_transaction_to_binlog_events (this=0x17fdda0 <mysql_bin_log>, entry=0x7f54d7db2ec0) at /home/xyz/mdev-6860/server/sql/log.cc:7300
            #5  0x000000000093f211 in MYSQL_BIN_LOG::write_transaction_to_binlog (this=0x17fdda0 <mysql_bin_log>, thd=0x7f54b1c15330, cache_mngr=0x7f54b1c59770, end_ev=0x7f54d7db2fe0, all=true, using_stmt_cache=true, using_trx_cache=true) at /home/xyz/mdev-6860/server/sql/log.cc:6984
            #6  0x0000000000931a3a in binlog_flush_cache (thd=0x7f54b1c15330, cache_mngr=0x7f54b1c59770, end_ev=0x7f54d7db2fe0, all=true, using_stmt=true, using_trx=true) at /home/xyz/mdev-6860/server/sql/log.cc:1817
            #7  0x0000000000931e2b in binlog_commit_flush_xid_caches (thd=0x7f54b1c15330, cache_mngr=0x7f54b1c59770, all=true, xid=825) at /home/xyz/mdev-6860/server/sql/log.cc:1923
            #8  0x0000000000944349 in MYSQL_BIN_LOG::log_and_order (this=0x17fdda0 <mysql_bin_log>, thd=0x7f54b1c15330, xid=825, all=true, need_prepare_ordered=false, need_commit_ordered=true) at /home/xyz/mdev-6860/server/sql/log.cc:9189
            #9  0x00000000008681a0 in ha_commit_trans (thd=0x7f54b1c15330, all=true) at /home/xyz/mdev-6860/server/sql/handler.cc:1484
            #10 0x000000000078ec08 in trans_commit (thd=0x7f54b1c15330) at /home/xyz/mdev-6860/server/sql/transaction.cc:239
            #11 0x0000000000958ded in Xid_log_event::do_apply_event (this=0x7f54ad659470, rgi=0x7f54ad714800) at /home/xyz/mdev-6860/server/sql/log_event.cc:7461
            #12 0x00000000005cb3fe in Log_event::apply_event (this=0x7f54ad659470, rgi=0x7f54ad714800) at /home/xyz/mdev-6860/server/sql/log_event.h:1360
            #13 0x00000000005c15e1 in apply_event_and_update_pos (ev=0x7f54ad659470, thd=0x7f54b1c15330, rgi=0x7f54ad714800, rpt=0x7f54b6094ca0) at /home/xyz/mdev-6860/server/sql/slave.cc:3308
            #14 0x00000000007d898a in rpt_handle_event (qev=0x7f54ad713b70, rpt=0x7f54b6094ca0) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:52
            #15 0x00000000007db164 in handle_rpl_parallel_thread (arg=0x7f54b6094ca0) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:1201
            #16 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #17 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 41 (Thread 0x7f54d7dfd700 (LWP 18071)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f58e83 in gu::Lock::wait (this=0x7f54d7dfb900, cond=...) at galerautils/src/gu_lock.hpp:56
            #2  0x00007f54e40f1ede in galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x7f54e452ad70, obj=...) at galera/src/monitor.hpp:123
            #3  0x00007f54e40e7bd8 in galera::ReplicatorSMM::pre_commit (this=0x7f54e452a000, trx=0x7f54b1ccd000, meta=0x7f54b4c19b20) at galera/src/replicator_smm.cpp:778
            #4  0x00007f54e4102a0f in galera_pre_commit (gh=0x7f54e444cc80, conn_id=17, trx_handle=0x7f54b4c19b58, flags=1, meta=0x7f54b4c19b20) at galera/src/wsrep_provider.cpp:518
            #5  0x00000000007e3b17 in wsrep_run_wsrep_commit (thd=0x7f54b4c14930, hton=0x7f54e4571c70, all=true) at /home/xyz/mdev-6860/server/sql/wsrep_hton.cc:467
            #6  0x00000000007e250c in wsrep_prepare (hton=0x7f54e4571c70, thd=0x7f54b4c14930, all=true) at /home/xyz/mdev-6860/server/sql/wsrep_hton.cc:192
            #7  0x0000000000867ec9 in ha_commit_trans (thd=0x7f54b4c14930, all=true) at /home/xyz/mdev-6860/server/sql/handler.cc:1429
            #8  0x000000000078ec08 in trans_commit (thd=0x7f54b4c14930) at /home/xyz/mdev-6860/server/sql/transaction.cc:239
            #9  0x0000000000958ded in Xid_log_event::do_apply_event (this=0x7f54ad63b770, rgi=0x7f54ad41e800) at /home/xyz/mdev-6860/server/sql/log_event.cc:7461
            #10 0x00000000005cb3fe in Log_event::apply_event (this=0x7f54ad63b770, rgi=0x7f54ad41e800) at /home/xyz/mdev-6860/server/sql/log_event.h:1360
            #11 0x00000000005c15e1 in apply_event_and_update_pos (ev=0x7f54ad63b770, thd=0x7f54b4c14930, rgi=0x7f54ad41e800, rpt=0x7f54b6094a98) at /home/xyz/mdev-6860/server/sql/slave.cc:3308
            #12 0x00000000007d898a in rpt_handle_event (qev=0x7f54ad426470, rpt=0x7f54b6094a98) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:52
            #13 0x00000000007db164 in handle_rpl_parallel_thread (arg=0x7f54b6094a98) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:1201
            #14 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #15 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 40 (Thread 0x7f54dae48700 (LWP 18070)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f58e83 in gu::Lock::wait (this=0x7f54dae46900, cond=...) at galerautils/src/gu_lock.hpp:56
            #2  0x00007f54e40f1ede in galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter (this=0x7f54e452ad70, obj=...) at galera/src/monitor.hpp:123
            #3  0x00007f54e40e7bd8 in galera::ReplicatorSMM::pre_commit (this=0x7f54e452a000, trx=0x7f54d7533000, meta=0x7f54d74830e0) at galera/src/replicator_smm.cpp:778
            #4  0x00007f54e4102a0f in galera_pre_commit (gh=0x7f54e444cc80, conn_id=16, trx_handle=0x7f54d7483118, flags=1, meta=0x7f54d74830e0) at galera/src/wsrep_provider.cpp:518
            #5  0x00000000007e3b17 in wsrep_run_wsrep_commit (thd=0x7f54d747def0, hton=0x7f54e4571c70, all=true) at /home/xyz/mdev-6860/server/sql/wsrep_hton.cc:467
            #6  0x00000000007e250c in wsrep_prepare (hton=0x7f54e4571c70, thd=0x7f54d747def0, all=true) at /home/xyz/mdev-6860/server/sql/wsrep_hton.cc:192
            #7  0x0000000000867ec9 in ha_commit_trans (thd=0x7f54d747def0, all=true) at /home/xyz/mdev-6860/server/sql/handler.cc:1429
            #8  0x000000000078ec08 in trans_commit (thd=0x7f54d747def0) at /home/xyz/mdev-6860/server/sql/transaction.cc:239
            #9  0x0000000000958ded in Xid_log_event::do_apply_event (this=0x7f54ad6b4270, rgi=0x7f54ad41f000) at /home/xyz/mdev-6860/server/sql/log_event.cc:7461
            #10 0x00000000005cb3fe in Log_event::apply_event (this=0x7f54ad6b4270, rgi=0x7f54ad41f000) at /home/xyz/mdev-6860/server/sql/log_event.h:1360
            #11 0x00000000005c15e1 in apply_event_and_update_pos (ev=0x7f54ad6b4270, thd=0x7f54d747def0, rgi=0x7f54ad41f000, rpt=0x7f54b6094890) at /home/xyz/mdev-6860/server/sql/slave.cc:3308
            #12 0x00000000007d898a in rpt_handle_event (qev=0x7f54ad427870, rpt=0x7f54b6094890) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:52
            #13 0x00000000007db164 in handle_rpl_parallel_thread (arg=0x7f54b6094890) at /home/xyz/mdev-6860/server/sql/rpl_parallel.cc:1201
            #14 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #15 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 39 (Thread 0x7f54dae91700 (LWP 18069)):
            #0  0x00007f54e493448d in poll () from /usr/lib/libc.so.6
            #1  0x0000000000eb5eda in vio_io_wait (vio=0x7f54b1a82570, event=VIO_IO_EVENT_READ, timeout=3600000) at /home/xyz/mdev-6860/server/vio/viosocket.c:992
            #2  0x0000000000eb482d in vio_socket_io_wait (vio=0x7f54b1a82570, event=VIO_IO_EVENT_READ) at /home/xyz/mdev-6860/server/vio/viosocket.c:108
            #3  0x0000000000eb49e0 in vio_read (vio=0x7f54b1a82570, buf=0x7f54b1ac00f0 "", size=16384) at /home/xyz/mdev-6860/server/vio/viosocket.c:184
            #4  0x0000000000eb4c00 in vio_read_buff (vio=0x7f54b1a82570, buf=0x7f54b1ac62b0 "", size=4) at /home/xyz/mdev-6860/server/vio/viosocket.c:232
            #5  0x00000000005a656c in my_real_read (net=0x7f54b1a87c70, complen=0x7f54dae90c50, header=0 '\000') at /home/xyz/mdev-6860/server/sql/net_serv.cc:874
            #6  0x00000000005a6dbc in my_net_read_packet (net=0x7f54b1a87c70, read_from_server=0 '\000') at /home/xyz/mdev-6860/server/sql/net_serv.cc:1137
            #7  0x000000000082cb88 in cli_safe_read (mysql=0x7f54b1a87c70) at /home/xyz/mdev-6860/server/sql-common/client.c:581
            #8  0x00000000005c0fe4 in read_event (mysql=0x7f54b1a87c70, mi=0x7f54c5ffa700, suppress_warnings=0x7f54dae90e1b) at /home/xyz/mdev-6860/server/sql/slave.cc:3116
            #9  0x00000000005c3394 in handle_slave_io (arg=0x7f54c5ffa700) at /home/xyz/mdev-6860/server/sql/slave.cc:4112
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 38 (Thread 0x7f54daeda700 (LWP 18067)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54daed8c30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54daed8cb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54b4e379b0, exit_loop=@0x7f54daed8d3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54b4e379b0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54b4e379b0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54b4e379b0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 37 (Thread 0x7f54daf23700 (LWP 18066)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54daf21c30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54daf21cb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54c5223cf0, exit_loop=@0x7f54daf21d3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54c5223cf0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54c5223cf0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54c5223cf0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 36 (Thread 0x7f54daf6c700 (LWP 18065)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54daf6ac30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54daf6acb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54d76396b0, exit_loop=@0x7f54daf6ad3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54d76396b0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54d76396b0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54d76396b0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 35 (Thread 0x7f54dafb5700 (LWP 18064)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54dafb3c30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54dafb3cb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54d7a1f6b0, exit_loop=@0x7f54dafb3d3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54d7a1f6b0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54d7a1f6b0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54d7a1f6b0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 34 (Thread 0x7f54daffe700 (LWP 18063)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54daffcc30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54daffccb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54d78542f0, exit_loop=@0x7f54daffcd3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54d78542f0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54d78542f0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54d78542f0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 33 (Thread 0x7f54dba85700 (LWP 18062)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54dba83c30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54dba83cb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54d8e199b0, exit_loop=@0x7f54dba83d3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54d8e199b0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54d8e199b0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54d8e199b0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 32 (Thread 0x7f54dbace700 (LWP 18061)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54dbaccc30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54dbacccb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54b5ff4b30, exit_loop=@0x7f54dbaccd3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54b5ff4b30) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54b5ff4b30) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54b5ff4b30) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 31 (Thread 0x7f54e4854700 (LWP 18060)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54e4852c30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54e4852cb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54ade133b0, exit_loop=@0x7f54e4852d3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54ade133b0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54ade133b0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54ade133b0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 30 (Thread 0x7f54e6b58700 (LWP 18059)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54e6b56c30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54e6b56cb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54adc136f0, exit_loop=@0x7f54e6b56d3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54adc136f0) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54adc136f0) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54adc136f0) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 29 (Thread 0x7f54e6ba1700 (LWP 18058)):
            #0  0x00007f54e689ac37 in do_sigwait () from /usr/lib/libpthread.so.0
            #1  0x00007f54e689acad in sigwait () from /usr/lib/libpthread.so.0
            #2  0x0000000000597b0d in signal_hand (arg=0x0) at /home/xyz/mdev-6860/server/sql/mysqld.cc:3507
            #3  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #4  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 28 (Thread 0x7f54e6bea700 (LWP 18057)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000e6e45b in safe_cond_wait (cond=0x17fed88 <mysql_bin_log+4072>, mp=0x17fecd8 <mysql_bin_log+3896>, file=0xfa3b40 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h", line=1165) at /home/xyz/mdev-6860/server/mysys/thr_mutex.c:493
            #2  0x000000000092d5d5 in inline_mysql_cond_wait (that=0x17fed88 <mysql_bin_log+4072>, mutex=0x17fecd8 <mysql_bin_log+3896>, src_file=0xfa3fb8 "/home/xyz/mdev-6860/server/sql/log.cc", src_line=9452) at /home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h:1165
            #3  0x0000000000944c43 in binlog_background_thread (arg=0x0) at /home/xyz/mdev-6860/server/sql/log.cc:9452
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 27 (Thread 0x7f54b01fc700 (LWP 18056)):
            #0  0x00007f54e68974b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000c182cd in os_cond_wait_timed (cond=0x7f54e455cf98, fast_mutex=0x7f54e455cf60, abstime=0x7f54b01fbda0) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:159
            #2  0x0000000000c188f1 in os_event_wait_time_low (event=0x7f54e455cf60, time_in_usec=5000000, reset_sig_count=1) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:591
            #3  0x0000000000d1c1fa in ib_wqueue_timedwait (wq=0x7f54c5fca1f8, wait_in_usecs=5000000) at /home/xyz/mdev-6860/server/storage/xtradb/ut/ut0wqueue.cc:154
            #4  0x0000000000e037f3 in fts_optimize_thread (arg=0x7f54c5fca1f8) at /home/xyz/mdev-6860/server/storage/xtradb/fts/fts0opt.cc:3064
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 26 (Thread 0x7f54b09fd700 (LWP 18055)):
            #0  0x00007f54e68974b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000c182cd in os_cond_wait_timed (cond=0x7f54e455ccf8, fast_mutex=0x7f54e455ccc0, abstime=0x7f54b09fce50) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:159
            #2  0x0000000000c188f1 in os_event_wait_time_low (event=0x7f54e455ccc0, time_in_usec=10000000, reset_sig_count=7) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:591
            #3  0x0000000000dc3c93 in dict_stats_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/dict/dict0stats_bg.cc:349
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 25 (Thread 0x7f54b11fe700 (LWP 18054)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000c183b9 in os_cond_wait (cond=0x7f54e455b408, fast_mutex=0x7f54e455b3d0) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:196
            #2  0x0000000000c1872f in os_event_wait_low (event=0x7f54e455b3d0, reset_sig_count=1) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:477
            #3  0x0000000000d6e652 in buf_dump_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/buf/buf0dump.cc:695
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 24 (Thread 0x7f54b19ff700 (LWP 18053)):
            #0  0x00007f54e49360b3 in select () from /usr/lib/libc.so.6
            #1  0x0000000000c18fa8 in os_thread_sleep (tm=1000000) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0thread.cc:307
            #2  0x0000000000d765a0 in page_cleaner_sleep_if_needed (next_loop_time=1482129511501) at /home/xyz/mdev-6860/server/storage/xtradb/buf/buf0flu.cc:2583
            #3  0x0000000000d76e2a in buf_flush_lru_manager_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/buf/buf0flu.cc:2858
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 23 (Thread 0x7f54b59ff700 (LWP 18052)):
            #0  0x00007f54e49360b3 in select () from /usr/lib/libc.so.6
            #1  0x0000000000c18fa8 in os_thread_sleep (tm=1000000) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0thread.cc:307
            #2  0x0000000000d765a0 in page_cleaner_sleep_if_needed (next_loop_time=1482129511498) at /home/xyz/mdev-6860/server/storage/xtradb/buf/buf0flu.cc:2583
            #3  0x0000000000d76892 in buf_flush_page_cleaner_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/buf/buf0flu.cc:2707
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 22 (Thread 0x7f54b27fe700 (LWP 18051)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000c183b9 in os_cond_wait (cond=0x7f54e455b2b8, fast_mutex=0x7f54e455b280) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:196
            #2  0x0000000000c1872f in os_event_wait_low (event=0x7f54e455b280, reset_sig_count=563) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:477
            #3  0x0000000000cd8768 in srv_purge_coordinator_suspend (slot=0x7f54e45657c8, rseg_history_len=1742) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0srv.cc:3427
            #4  0x0000000000cd8c7c in srv_purge_coordinator_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0srv.cc:3549
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 21 (Thread 0x7f54b2fff700 (LWP 18050)):
            #0  0x00007f54e49360b3 in select () from /usr/lib/libc.so.6
            #1  0x0000000000c18fa8 in os_thread_sleep (tm=1000000) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0thread.cc:307
            #2  0x0000000000cd7c84 in srv_master_sleep () at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0srv.cc:3067
            #3  0x0000000000cd7dd3 in srv_master_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0srv.cc:3117
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 20 (Thread 0x7f54b3bfd700 (LWP 18049)):
            #0  0x00007f54e68974b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000c182cd in os_cond_wait_timed (cond=0x7f54e455b398, fast_mutex=0x7f54e455b360, abstime=0x7f54b3bfce00) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:159
            #2  0x0000000000c188f1 in os_event_wait_time_low (event=0x7f54e455b360, time_in_usec=5000000, reset_sig_count=1) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:591
            #3  0x0000000000cd5f22 in srv_monitor_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0srv.cc:2030
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 19 (Thread 0x7f54b43fe700 (LWP 18048)):
            #0  0x00007f54e68974b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000c182cd in os_cond_wait_timed (cond=0x7f54e455b328, fast_mutex=0x7f54e455b2f0, abstime=0x7f54b43fddd0) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:159
            #2  0x0000000000c188f1 in os_event_wait_time_low (event=0x7f54e455b2f0, time_in_usec=1000000, reset_sig_count=1) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:591
            #3  0x0000000000cd6715 in srv_error_monitor_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0srv.cc:2306
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 18 (Thread 0x7f54b4bff700 (LWP 18047)):
            #0  0x00007f54e68974b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000c182cd in os_cond_wait_timed (cond=0x7f54e455ca58, fast_mutex=0x7f54e455ca20, abstime=0x7f54b4bfee30) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:159
            #2  0x0000000000c188f1 in os_event_wait_time_low (event=0x7f54e455ca20, time_in_usec=1000000, reset_sig_count=1) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0sync.cc:591
            #3  0x0000000000bee4f9 in lock_wait_timeout_thread (arg=0x0) at /home/xyz/mdev-6860/server/storage/xtradb/lock/lock0wait.cc:534
            #4  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #5  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 17 (Thread 0x7f54b6df6700 (LWP 18044)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45666e0, segment=3, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=9, message1=0x7f54b6df5e30, message2=0x7f54b6df5e28, type=0x7f54b6df5e20, space_id=0x7f54b6df5e18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=9) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201be08 <n+72>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 16 (Thread 0x7f54b75f7700 (LWP 18043)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45666e0, segment=2, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=8, message1=0x7f54b75f6e30, message2=0x7f54b75f6e28, type=0x7f54b75f6e20, space_id=0x7f54b75f6e18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=8) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201be00 <n+64>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 15 (Thread 0x7f54b7df8700 (LWP 18042)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45666e0, segment=1, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=7, message1=0x7f54b7df7e30, message2=0x7f54b7df7e28, type=0x7f54b7df7e20, space_id=0x7f54b7df7e18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=7) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bdf8 <n+56>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 14 (Thread 0x7f54b85f9700 (LWP 18041)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45666e0, segment=0, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=6, message1=0x7f54b85f8e30, message2=0x7f54b85f8e28, type=0x7f54b85f8e20, space_id=0x7f54b85f8e18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=6) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bdf0 <n+48>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 13 (Thread 0x7f54b8dfa700 (LWP 18040)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45665f0, segment=3, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=5, message1=0x7f54b8df9e30, message2=0x7f54b8df9e28, type=0x7f54b8df9e20, space_id=0x7f54b8df9e18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=5) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bde8 <n+40>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 12 (Thread 0x7f54b95fb700 (LWP 18039)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45665f0, segment=2, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=4, message1=0x7f54b95fae30, message2=0x7f54b95fae28, type=0x7f54b95fae20, space_id=0x7f54b95fae18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=4) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bde0 <n+32>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 11 (Thread 0x7f54b9dfc700 (LWP 18038)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45665f0, segment=1, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=3, message1=0x7f54b9dfbe30, message2=0x7f54b9dfbe28, type=0x7f54b9dfbe20, space_id=0x7f54b9dfbe18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=3) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bdd8 <n+24>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 10 (Thread 0x7f54ba5fd700 (LWP 18037)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e45665f0, segment=0, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=2, message1=0x7f54ba5fce30, message2=0x7f54ba5fce28, type=0x7f54ba5fce20, space_id=0x7f54ba5fce18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=2) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bdd0 <n+16>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 9 (Thread 0x7f54badfe700 (LWP 18036)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e4566640, segment=0, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=1, message1=0x7f54badfde30, message2=0x7f54badfde28, type=0x7f54badfde20, space_id=0x7f54badfde18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=1) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bdc8 <n+8>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 8 (Thread 0x7f54bb5ff700 (LWP 18035)):
            #0  0x00007f54e6688674 in ?? () from /usr/lib/libaio.so.1
            #1  0x0000000000c15dad in os_aio_linux_collect (array=0x7f54e4566690, segment=0, seg_size=256) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5203
            #2  0x0000000000c1611c in os_aio_linux_handle (global_seg=0, message1=0x7f54bb5fee30, message2=0x7f54bb5fee28, type=0x7f54bb5fee20, space_id=0x7f54bb5fee18) at /home/xyz/mdev-6860/server/storage/xtradb/os/os0file.cc:5349
            #3  0x0000000000dd5603 in fil_aio_wait (segment=0) at /home/xyz/mdev-6860/server/storage/xtradb/fil/fil0fil.cc:5783
            #4  0x0000000000cdb41d in io_handler_thread (arg=0x201bdc0 <n>) at /home/xyz/mdev-6860/server/storage/xtradb/srv/srv0start.cc:535
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 7 (Thread 0x7f54c5dff700 (LWP 18034)):
            #0  0x00007f54e68974b8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000e6e736 in safe_cond_timedwait (cond=0x2001d40 <COND_checkpoint>, mp=0x2001c80 <LOCK_checkpoint>, abstime=0x7f54c5dfedf0, file=0xfd8650 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h", line=1202) at /home/xyz/mdev-6860/server/mysys/thr_mutex.c:547
            #2  0x0000000000a57f37 in inline_mysql_cond_timedwait (that=0x2001d40 <COND_checkpoint>, mutex=0x2001c80 <LOCK_checkpoint>, abstime=0x7f54c5dfedf0, src_file=0xfd8690 "/home/xyz/mdev-6860/server/storage/maria/ma_servicethread.c", src_line=116) at /home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h:1202
            #3  0x0000000000a583ca in my_service_thread_sleep (control=0x161b180 <checkpoint_control>, sleep_time=30000000000) at /home/xyz/mdev-6860/server/storage/maria/ma_servicethread.c:115
            #4  0x0000000000a4d786 in ma_checkpoint_background (arg=0x1e) at /home/xyz/mdev-6860/server/storage/maria/ma_checkpoint.c:708
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 6 (Thread 0x7f54e6c33700 (LWP 18033)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f4cb48 in fifo_lock_get (q=0x7f54e45e9400) at galerautils/src/gu_fifo.c:235
            #2  0x00007f54e3f4cd6c in gu_fifo_get_head (q=0x7f54e45e9400, err=0x7f54e6c31c30) at galerautils/src/gu_fifo.c:294
            #3  0x00007f54e408ffa5 in gcs_recv (conn=0x7f54e4570200, action=0x7f54e6c31cb0) at gcs/src/gcs.cpp:1787
            #4  0x00007f54e40ee05b in galera::Gcs::recv (this=0x7f54e452a5e0, act=...) at galera/src/galera_gcs.hpp:118
            #5  0x00007f54e40c87a5 in galera::GcsActionSource::process (this=0x7f54e452a728, recv_ctx=0x7f54d7413d70, exit_loop=@0x7f54e6c31d3f: false) at galera/src/gcs_action_source.cpp:175
            #6  0x00007f54e40e57af in galera::ReplicatorSMM::async_recv (this=0x7f54e452a000, recv_ctx=0x7f54d7413d70) at galera/src/replicator_smm.cpp:355
            #7  0x00007f54e4101749 in galera_recv (gh=0x7f54e444cc80, recv_ctx=0x7f54d7413d70) at galera/src/wsrep_provider.cpp:244
            #8  0x00000000007f4d40 in wsrep_replication_process (thd=0x7f54d7413d70) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:328
            #9  0x000000000059b833 in start_wsrep_THD (arg=0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #11 0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 5 (Thread 0x7f54e6c7c700 (LWP 18032)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x0000000000e6e45b in safe_cond_wait (cond=0x17d57c0 <COND_wsrep_rollback>, mp=0x17d5700 <LOCK_wsrep_rollback>, file=0xf27a60 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h", line=1165) at /home/xyz/mdev-6860/server/mysys/thr_mutex.c:493
            #2  0x00000000007f385b in inline_mysql_cond_wait (that=0x17d57c0 <COND_wsrep_rollback>, mutex=0x17d5700 <LOCK_wsrep_rollback>, src_file=0xf27bc8 "/home/xyz/mdev-6860/server/sql/wsrep_thd.cc", src_line=422) at /home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h:1165
            #3  0x00000000007f532a in wsrep_rollback_process (thd=0x7f54d7613070) at /home/xyz/mdev-6860/server/sql/wsrep_thd.cc:422
            #4  0x000000000059b833 in start_wsrep_THD (arg=0x7f522f <wsrep_rollback_process(THD*)>) at /home/xyz/mdev-6860/server/sql/mysqld.cc:5475
            #5  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #6  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 4 (Thread 0x7f54d85fe700 (LWP 18031)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f58e83 in gu::Lock::wait (this=0x7f54d85fd880, cond=...) at galerautils/src/gu_lock.hpp:56
            #2  0x00007f54e409531f in RecvBuf::front (this=0x7f54e4594140, timeout=...) at gcs/src/gcs_gcomm.cpp:124
            #3  0x00007f54e4092482 in gcomm_recv (backend=0x7f54e4570578, msg=0x7f54e4570460, timeout=9223372035999999999) at gcs/src/gcs_gcomm.cpp:673
            #4  0x00007f54e4086cc2 in core_msg_recv (backend=0x7f54e4570578, recv_msg=0x7f54e4570460, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:468
            #5  0x00007f54e4088a70 in gcs_core_recv (conn=0x7f54e4570400, recv_act=0x7f54d85fdde0, timeout=9223372035999999999) at gcs/src/gcs_core.cpp:1067
            #6  0x00007f54e408e23b in gcs_recv_thread (arg=0x7f54e4570200) at gcs/src/gcs.cpp:1191
            #7  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #8  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 3 (Thread 0x7f54d8dff700 (LWP 18030)):
            #0  0x00007f54e493ddd3 in epoll_wait () from /usr/lib/libc.so.6
            #1  0x00007f54e40468db in asio::detail::epoll_reactor::run (this=0x7f54e44b9f00, block=true, ops=...) at /usr/include/asio/detail/impl/epoll_reactor.ipp:391
            #2  0x00007f54e4047a26 in asio::detail::task_io_service::do_run_one (this=0x7f54e448a7e0, lock=..., this_thread=..., ec=...) at /usr/include/asio/detail/impl/task_io_service.ipp:355
            #3  0x00007f54e404759e in asio::detail::task_io_service::run (this=0x7f54e448a7e0, ec=...) at /usr/include/asio/detail/impl/task_io_service.ipp:148
            #4  0x00007f54e4047cbb in asio::io_service::run (this=0x7f54e444d4b8) at /usr/include/asio/impl/io_service.ipp:58
            #5  0x00007f54e4076c6c in gcomm::AsioProtonet::event_loop (this=0x7f54e444d400, period=...) at gcomm/src/asio_protonet.cpp:121
            #6  0x00007f54e4091755 in GCommConn::run (this=0x7f54e4593f00) at gcs/src/gcs_gcomm.cpp:498
            #7  0x00007f54e4095bdc in GCommConn::run_fn (arg=0x7f54e4593f00) at gcs/src/gcs_gcomm.cpp:209
            #8  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #9  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 2 (Thread 0x7f54db7ff700 (LWP 18029)):
            #0  0x00007f54e689710f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
            #1  0x00007f54e3f58e83 in gu::Lock::wait (this=0x7f54db7fe790, cond=...) at galerautils/src/gu_lock.hpp:56
            #2  0x00007f54e40c44fb in galera::ServiceThd::thd_func (arg=0x7f54e452a5f0) at galera/src/galera_service_thd.cpp:30
            #3  0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so.0
            #4  0x00007f54e493d7df in clone () from /usr/lib/libc.so.6
             
            Thread 1 (Thread 0x7f54e6c7fc00 (LWP 18022)):
            #0  0x00007f54e493448d in poll () from /usr/lib/libc.so.6
            #1  0x000000000059e5f8 in handle_connections_sockets () at /home/xyz/mdev-6860/server/sql/mysqld.cc:6938
            #2  0x000000000059d61c in mysqld_main (argc=26, argv=0x7f54e44373a0) at /home/xyz/mdev-6860/server/sql/mysqld.cc:6364
            #3  0x0000000000591816 in main (argc=25, argv=0x7ffff684aa88) at /home/xyz/mdev-6860/server/sql/main.cc:25
            
            

            sachin.setiya.007 Sachin Setiya (Inactive) added a comment - This is thread back-trace of all threads (gdb) thread apply all bt   Thread 45 (Thread 0x7f54d7cd9700 (LWP 18076 )): # 0 0x00007f54e493448d in poll () from /usr/lib/libc.so. 6 # 1 0x0000000000eb5eda in vio_io_wait (vio= 0x7f54b5f88f70 , event=VIO_IO_EVENT_READ, timeout= 28800000 ) at /home/xyz/mdev- 6860 /server/vio/viosocket.c: 992 # 2 0x0000000000eb482d in vio_socket_io_wait (vio= 0x7f54b5f88f70 , event=VIO_IO_EVENT_READ) at /home/xyz/mdev- 6860 /server/vio/viosocket.c: 108 # 3 0x0000000000eb49e0 in vio_read (vio= 0x7f54b5f88f70 , buf= 0x7f54b612a870 "\001" , size= 4 ) at /home/xyz/mdev- 6860 /server/vio/viosocket.c: 184 # 4 0x00000000005a656c in my_real_read (net= 0x7f54e44cc0e8 , complen= 0x7f54d7cd8940 , header= 1 '\001' ) at /home/xyz/mdev- 6860 /server/sql/net_serv.cc: 874 # 5 0x00000000005a6dbc in my_net_read_packet (net= 0x7f54e44cc0e8 , read_from_server= 1 '\001' ) at /home/xyz/mdev- 6860 /server/sql/net_serv.cc: 1137 # 6 0x0000000000655a29 in do_command (thd= 0x7f54e44cbe30 ) at /home/xyz/mdev- 6860 /server/sql/sql_parse.cc: 953 # 7 0x000000000077aa45 in do_handle_one_connection (thd_arg= 0x7f54e44cbe30 ) at /home/xyz/mdev- 6860 /server/sql/sql_connect.cc: 1395 # 8 0x000000000077a78b in handle_one_connection (arg= 0x7f54e44cbe30 ) at /home/xyz/mdev- 6860 /server/sql/sql_connect.cc: 1307 # 9 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 10 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 44 (Thread 0x7f54d7d22700 (LWP 18074 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000e6e45b in safe_cond_wait (cond= 0x7f54c5ffcf20 , mp= 0x7f54c5ffc828 , file= 0xfa3b40 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h" , line= 1165 ) at /home/xyz/mdev- 6860 /server/mysys/thr_mutex.c: 493 # 2 0x000000000092d5d5 in inline_mysql_cond_wait (that= 0x7f54c5ffcf20 , mutex= 0x7f54c5ffc828 , src_file= 0xfa3fb8 "/home/xyz/mdev-6860/server/sql/log.cc" , src_line= 7917 ) at /home/xyz/mdev- 6860 /server/include/mysql/psi/mysql_thread.h: 1165 # 3 0x0000000000941198 in MYSQL_BIN_LOG::wait_for_update_relay_log ( this = 0x7f54c5ffc820 , thd= 0x7f54ae2b08b0 ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 7917 # 4 0x00000000005c9de8 in next_event (rgi= 0x7f54ae2a6000 , event_size= 0x7f54d7d214b0 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 6740 # 5 0x00000000005c19d7 in exec_relay_log_event (thd= 0x7f54ae2b08b0 , rli= 0x7f54c5ffc348 , serial_rgi= 0x7f54ae2a6000 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 3511 # 6 0x00000000005c4f8c in handle_slave_sql (arg= 0x7f54c5ffa700 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 4756 # 7 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 8 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 43 (Thread 0x7f54d7d6b700 (LWP 18073 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f58e83 in gu::Lock::wait ( this = 0x7f54d7d69900 , cond=...) at galerautils/src/gu_lock.hpp: 56 # 2 0x00007f54e40f1ede in galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter ( this = 0x7f54e452ad70 , obj=...) at galera/src/monitor.hpp: 123 # 3 0x00007f54e40e7bd8 in galera::ReplicatorSMM::pre_commit ( this = 0x7f54e452a000 , trx= 0x7f54b310a000 , meta= 0x7f54b301a820 ) at galera/src/replicator_smm.cpp: 778 # 4 0x00007f54e4102a0f in galera_pre_commit (gh= 0x7f54e444cc80 , conn_id= 15 , trx_handle= 0x7f54b301a858 , flags= 1 , meta= 0x7f54b301a820 ) at galera/src/wsrep_provider.cpp: 518 # 5 0x00000000007e3b17 in wsrep_run_wsrep_commit (thd= 0x7f54b3015630 , hton= 0x7f54e4571c70 , all= true ) at /home/xyz/mdev- 6860 /server/sql/wsrep_hton.cc: 467 # 6 0x00000000007e250c in wsrep_prepare (hton= 0x7f54e4571c70 , thd= 0x7f54b3015630 , all= true ) at /home/xyz/mdev- 6860 /server/sql/wsrep_hton.cc: 192 # 7 0x0000000000867ec9 in ha_commit_trans (thd= 0x7f54b3015630 , all= true ) at /home/xyz/mdev- 6860 /server/sql/handler.cc: 1429 # 8 0x000000000078ec08 in trans_commit (thd= 0x7f54b3015630 ) at /home/xyz/mdev- 6860 /server/sql/transaction.cc: 239 # 9 0x0000000000958ded in Xid_log_event::do_apply_event ( this = 0x7f54ad6b4b70 , rgi= 0x7f54ad41f800 ) at /home/xyz/mdev- 6860 /server/sql/log_event.cc: 7461 # 10 0x00000000005cb3fe in Log_event::apply_event ( this = 0x7f54ad6b4b70 , rgi= 0x7f54ad41f800 ) at /home/xyz/mdev- 6860 /server/sql/log_event.h: 1360 # 11 0x00000000005c15e1 in apply_event_and_update_pos (ev= 0x7f54ad6b4b70 , thd= 0x7f54b3015630 , rgi= 0x7f54ad41f800 , rpt= 0x7f54b6094ea8 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 3308 # 12 0x00000000007d898a in rpt_handle_event (qev= 0x7f54ad428c70 , rpt= 0x7f54b6094ea8 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 52 # 13 0x00000000007db164 in handle_rpl_parallel_thread (arg= 0x7f54b6094ea8 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 1201 # 14 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 15 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 42 (Thread 0x7f54d7db4700 (LWP 18072 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000e6e45b in safe_cond_wait (cond= 0x7f54ad7148e8 , mp= 0x7f54ad714838 , file= 0xfa3b40 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h" , line= 1165 ) at /home/xyz/mdev- 6860 /server/mysys/thr_mutex.c: 493 # 2 0x000000000092d5d5 in inline_mysql_cond_wait (that= 0x7f54ad7148e8 , mutex= 0x7f54ad714838 , src_file= 0xfa3fb8 "/home/xyz/mdev-6860/server/sql/log.cc" , src_line= 7081 ) at /home/xyz/mdev- 6860 /server/include/mysql/psi/mysql_thread.h: 1165 # 3 0x000000000093f3e2 in MYSQL_BIN_LOG::queue_for_group_commit ( this = 0x17fdda0 <mysql_bin_log>, orig_entry= 0x7f54d7db2ec0 ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 7081 # 4 0x000000000093fa6b in MYSQL_BIN_LOG::write_transaction_to_binlog_events ( this = 0x17fdda0 <mysql_bin_log>, entry= 0x7f54d7db2ec0 ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 7300 # 5 0x000000000093f211 in MYSQL_BIN_LOG::write_transaction_to_binlog ( this = 0x17fdda0 <mysql_bin_log>, thd= 0x7f54b1c15330 , cache_mngr= 0x7f54b1c59770 , end_ev= 0x7f54d7db2fe0 , all= true , using_stmt_cache= true , using_trx_cache= true ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 6984 # 6 0x0000000000931a3a in binlog_flush_cache (thd= 0x7f54b1c15330 , cache_mngr= 0x7f54b1c59770 , end_ev= 0x7f54d7db2fe0 , all= true , using_stmt= true , using_trx= true ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 1817 # 7 0x0000000000931e2b in binlog_commit_flush_xid_caches (thd= 0x7f54b1c15330 , cache_mngr= 0x7f54b1c59770 , all= true , xid= 825 ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 1923 # 8 0x0000000000944349 in MYSQL_BIN_LOG::log_and_order ( this = 0x17fdda0 <mysql_bin_log>, thd= 0x7f54b1c15330 , xid= 825 , all= true , need_prepare_ordered= false , need_commit_ordered= true ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 9189 # 9 0x00000000008681a0 in ha_commit_trans (thd= 0x7f54b1c15330 , all= true ) at /home/xyz/mdev- 6860 /server/sql/handler.cc: 1484 # 10 0x000000000078ec08 in trans_commit (thd= 0x7f54b1c15330 ) at /home/xyz/mdev- 6860 /server/sql/transaction.cc: 239 # 11 0x0000000000958ded in Xid_log_event::do_apply_event ( this = 0x7f54ad659470 , rgi= 0x7f54ad714800 ) at /home/xyz/mdev- 6860 /server/sql/log_event.cc: 7461 # 12 0x00000000005cb3fe in Log_event::apply_event ( this = 0x7f54ad659470 , rgi= 0x7f54ad714800 ) at /home/xyz/mdev- 6860 /server/sql/log_event.h: 1360 # 13 0x00000000005c15e1 in apply_event_and_update_pos (ev= 0x7f54ad659470 , thd= 0x7f54b1c15330 , rgi= 0x7f54ad714800 , rpt= 0x7f54b6094ca0 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 3308 # 14 0x00000000007d898a in rpt_handle_event (qev= 0x7f54ad713b70 , rpt= 0x7f54b6094ca0 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 52 # 15 0x00000000007db164 in handle_rpl_parallel_thread (arg= 0x7f54b6094ca0 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 1201 # 16 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 17 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 41 (Thread 0x7f54d7dfd700 (LWP 18071 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f58e83 in gu::Lock::wait ( this = 0x7f54d7dfb900 , cond=...) at galerautils/src/gu_lock.hpp: 56 # 2 0x00007f54e40f1ede in galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter ( this = 0x7f54e452ad70 , obj=...) at galera/src/monitor.hpp: 123 # 3 0x00007f54e40e7bd8 in galera::ReplicatorSMM::pre_commit ( this = 0x7f54e452a000 , trx= 0x7f54b1ccd000 , meta= 0x7f54b4c19b20 ) at galera/src/replicator_smm.cpp: 778 # 4 0x00007f54e4102a0f in galera_pre_commit (gh= 0x7f54e444cc80 , conn_id= 17 , trx_handle= 0x7f54b4c19b58 , flags= 1 , meta= 0x7f54b4c19b20 ) at galera/src/wsrep_provider.cpp: 518 # 5 0x00000000007e3b17 in wsrep_run_wsrep_commit (thd= 0x7f54b4c14930 , hton= 0x7f54e4571c70 , all= true ) at /home/xyz/mdev- 6860 /server/sql/wsrep_hton.cc: 467 # 6 0x00000000007e250c in wsrep_prepare (hton= 0x7f54e4571c70 , thd= 0x7f54b4c14930 , all= true ) at /home/xyz/mdev- 6860 /server/sql/wsrep_hton.cc: 192 # 7 0x0000000000867ec9 in ha_commit_trans (thd= 0x7f54b4c14930 , all= true ) at /home/xyz/mdev- 6860 /server/sql/handler.cc: 1429 # 8 0x000000000078ec08 in trans_commit (thd= 0x7f54b4c14930 ) at /home/xyz/mdev- 6860 /server/sql/transaction.cc: 239 # 9 0x0000000000958ded in Xid_log_event::do_apply_event ( this = 0x7f54ad63b770 , rgi= 0x7f54ad41e800 ) at /home/xyz/mdev- 6860 /server/sql/log_event.cc: 7461 # 10 0x00000000005cb3fe in Log_event::apply_event ( this = 0x7f54ad63b770 , rgi= 0x7f54ad41e800 ) at /home/xyz/mdev- 6860 /server/sql/log_event.h: 1360 # 11 0x00000000005c15e1 in apply_event_and_update_pos (ev= 0x7f54ad63b770 , thd= 0x7f54b4c14930 , rgi= 0x7f54ad41e800 , rpt= 0x7f54b6094a98 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 3308 # 12 0x00000000007d898a in rpt_handle_event (qev= 0x7f54ad426470 , rpt= 0x7f54b6094a98 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 52 # 13 0x00000000007db164 in handle_rpl_parallel_thread (arg= 0x7f54b6094a98 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 1201 # 14 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 15 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 40 (Thread 0x7f54dae48700 (LWP 18070 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f58e83 in gu::Lock::wait ( this = 0x7f54dae46900 , cond=...) at galerautils/src/gu_lock.hpp: 56 # 2 0x00007f54e40f1ede in galera::Monitor<galera::ReplicatorSMM::CommitOrder>::enter ( this = 0x7f54e452ad70 , obj=...) at galera/src/monitor.hpp: 123 # 3 0x00007f54e40e7bd8 in galera::ReplicatorSMM::pre_commit ( this = 0x7f54e452a000 , trx= 0x7f54d7533000 , meta= 0x7f54d74830e0 ) at galera/src/replicator_smm.cpp: 778 # 4 0x00007f54e4102a0f in galera_pre_commit (gh= 0x7f54e444cc80 , conn_id= 16 , trx_handle= 0x7f54d7483118 , flags= 1 , meta= 0x7f54d74830e0 ) at galera/src/wsrep_provider.cpp: 518 # 5 0x00000000007e3b17 in wsrep_run_wsrep_commit (thd= 0x7f54d747def0 , hton= 0x7f54e4571c70 , all= true ) at /home/xyz/mdev- 6860 /server/sql/wsrep_hton.cc: 467 # 6 0x00000000007e250c in wsrep_prepare (hton= 0x7f54e4571c70 , thd= 0x7f54d747def0 , all= true ) at /home/xyz/mdev- 6860 /server/sql/wsrep_hton.cc: 192 # 7 0x0000000000867ec9 in ha_commit_trans (thd= 0x7f54d747def0 , all= true ) at /home/xyz/mdev- 6860 /server/sql/handler.cc: 1429 # 8 0x000000000078ec08 in trans_commit (thd= 0x7f54d747def0 ) at /home/xyz/mdev- 6860 /server/sql/transaction.cc: 239 # 9 0x0000000000958ded in Xid_log_event::do_apply_event ( this = 0x7f54ad6b4270 , rgi= 0x7f54ad41f000 ) at /home/xyz/mdev- 6860 /server/sql/log_event.cc: 7461 # 10 0x00000000005cb3fe in Log_event::apply_event ( this = 0x7f54ad6b4270 , rgi= 0x7f54ad41f000 ) at /home/xyz/mdev- 6860 /server/sql/log_event.h: 1360 # 11 0x00000000005c15e1 in apply_event_and_update_pos (ev= 0x7f54ad6b4270 , thd= 0x7f54d747def0 , rgi= 0x7f54ad41f000 , rpt= 0x7f54b6094890 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 3308 # 12 0x00000000007d898a in rpt_handle_event (qev= 0x7f54ad427870 , rpt= 0x7f54b6094890 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 52 # 13 0x00000000007db164 in handle_rpl_parallel_thread (arg= 0x7f54b6094890 ) at /home/xyz/mdev- 6860 /server/sql/rpl_parallel.cc: 1201 # 14 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 15 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 39 (Thread 0x7f54dae91700 (LWP 18069 )): # 0 0x00007f54e493448d in poll () from /usr/lib/libc.so. 6 # 1 0x0000000000eb5eda in vio_io_wait (vio= 0x7f54b1a82570 , event=VIO_IO_EVENT_READ, timeout= 3600000 ) at /home/xyz/mdev- 6860 /server/vio/viosocket.c: 992 # 2 0x0000000000eb482d in vio_socket_io_wait (vio= 0x7f54b1a82570 , event=VIO_IO_EVENT_READ) at /home/xyz/mdev- 6860 /server/vio/viosocket.c: 108 # 3 0x0000000000eb49e0 in vio_read (vio= 0x7f54b1a82570 , buf= 0x7f54b1ac00f0 "" , size= 16384 ) at /home/xyz/mdev- 6860 /server/vio/viosocket.c: 184 # 4 0x0000000000eb4c00 in vio_read_buff (vio= 0x7f54b1a82570 , buf= 0x7f54b1ac62b0 "" , size= 4 ) at /home/xyz/mdev- 6860 /server/vio/viosocket.c: 232 # 5 0x00000000005a656c in my_real_read (net= 0x7f54b1a87c70 , complen= 0x7f54dae90c50 , header= 0 '\000' ) at /home/xyz/mdev- 6860 /server/sql/net_serv.cc: 874 # 6 0x00000000005a6dbc in my_net_read_packet (net= 0x7f54b1a87c70 , read_from_server= 0 '\000' ) at /home/xyz/mdev- 6860 /server/sql/net_serv.cc: 1137 # 7 0x000000000082cb88 in cli_safe_read (mysql= 0x7f54b1a87c70 ) at /home/xyz/mdev- 6860 /server/sql-common/client.c: 581 # 8 0x00000000005c0fe4 in read_event (mysql= 0x7f54b1a87c70 , mi= 0x7f54c5ffa700 , suppress_warnings= 0x7f54dae90e1b ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 3116 # 9 0x00000000005c3394 in handle_slave_io (arg= 0x7f54c5ffa700 ) at /home/xyz/mdev- 6860 /server/sql/slave.cc: 4112 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 38 (Thread 0x7f54daeda700 (LWP 18067 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54daed8c30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54daed8cb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54b4e379b0 , exit_loop= @0x7f54daed8d3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54b4e379b0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54b4e379b0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54b4e379b0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 37 (Thread 0x7f54daf23700 (LWP 18066 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54daf21c30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54daf21cb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54c5223cf0 , exit_loop= @0x7f54daf21d3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54c5223cf0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54c5223cf0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54c5223cf0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 36 (Thread 0x7f54daf6c700 (LWP 18065 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54daf6ac30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54daf6acb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54d76396b0 , exit_loop= @0x7f54daf6ad3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54d76396b0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54d76396b0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54d76396b0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 35 (Thread 0x7f54dafb5700 (LWP 18064 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54dafb3c30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54dafb3cb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54d7a1f6b0 , exit_loop= @0x7f54dafb3d3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54d7a1f6b0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54d7a1f6b0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54d7a1f6b0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 34 (Thread 0x7f54daffe700 (LWP 18063 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54daffcc30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54daffccb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54d78542f0 , exit_loop= @0x7f54daffcd3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54d78542f0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54d78542f0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54d78542f0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 33 (Thread 0x7f54dba85700 (LWP 18062 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54dba83c30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54dba83cb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54d8e199b0 , exit_loop= @0x7f54dba83d3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54d8e199b0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54d8e199b0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54d8e199b0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 32 (Thread 0x7f54dbace700 (LWP 18061 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54dbaccc30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54dbacccb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54b5ff4b30 , exit_loop= @0x7f54dbaccd3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54b5ff4b30 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54b5ff4b30 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54b5ff4b30 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 31 (Thread 0x7f54e4854700 (LWP 18060 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54e4852c30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54e4852cb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54ade133b0 , exit_loop= @0x7f54e4852d3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54ade133b0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54ade133b0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54ade133b0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 30 (Thread 0x7f54e6b58700 (LWP 18059 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54e6b56c30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54e6b56cb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54adc136f0 , exit_loop= @0x7f54e6b56d3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54adc136f0 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54adc136f0 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54adc136f0 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 29 (Thread 0x7f54e6ba1700 (LWP 18058 )): # 0 0x00007f54e689ac37 in do_sigwait () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e689acad in sigwait () from /usr/lib/libpthread.so. 0 # 2 0x0000000000597b0d in signal_hand (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 3507 # 3 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 4 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 28 (Thread 0x7f54e6bea700 (LWP 18057 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000e6e45b in safe_cond_wait (cond= 0x17fed88 <mysql_bin_log+ 4072 >, mp= 0x17fecd8 <mysql_bin_log+ 3896 >, file= 0xfa3b40 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h" , line= 1165 ) at /home/xyz/mdev- 6860 /server/mysys/thr_mutex.c: 493 # 2 0x000000000092d5d5 in inline_mysql_cond_wait (that= 0x17fed88 <mysql_bin_log+ 4072 >, mutex= 0x17fecd8 <mysql_bin_log+ 3896 >, src_file= 0xfa3fb8 "/home/xyz/mdev-6860/server/sql/log.cc" , src_line= 9452 ) at /home/xyz/mdev- 6860 /server/include/mysql/psi/mysql_thread.h: 1165 # 3 0x0000000000944c43 in binlog_background_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/sql/log.cc: 9452 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 27 (Thread 0x7f54b01fc700 (LWP 18056 )): # 0 0x00007f54e68974b8 in pthread_cond_timedwait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000c182cd in os_cond_wait_timed (cond= 0x7f54e455cf98 , fast_mutex= 0x7f54e455cf60 , abstime= 0x7f54b01fbda0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 159 # 2 0x0000000000c188f1 in os_event_wait_time_low (event= 0x7f54e455cf60 , time_in_usec= 5000000 , reset_sig_count= 1 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 591 # 3 0x0000000000d1c1fa in ib_wqueue_timedwait (wq= 0x7f54c5fca1f8 , wait_in_usecs= 5000000 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/ut/ut0wqueue.cc: 154 # 4 0x0000000000e037f3 in fts_optimize_thread (arg= 0x7f54c5fca1f8 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fts/fts0opt.cc: 3064 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 26 (Thread 0x7f54b09fd700 (LWP 18055 )): # 0 0x00007f54e68974b8 in pthread_cond_timedwait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000c182cd in os_cond_wait_timed (cond= 0x7f54e455ccf8 , fast_mutex= 0x7f54e455ccc0 , abstime= 0x7f54b09fce50 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 159 # 2 0x0000000000c188f1 in os_event_wait_time_low (event= 0x7f54e455ccc0 , time_in_usec= 10000000 , reset_sig_count= 7 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 591 # 3 0x0000000000dc3c93 in dict_stats_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/dict/dict0stats_bg.cc: 349 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 25 (Thread 0x7f54b11fe700 (LWP 18054 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000c183b9 in os_cond_wait (cond= 0x7f54e455b408 , fast_mutex= 0x7f54e455b3d0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 196 # 2 0x0000000000c1872f in os_event_wait_low (event= 0x7f54e455b3d0 , reset_sig_count= 1 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 477 # 3 0x0000000000d6e652 in buf_dump_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/buf/buf0dump.cc: 695 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 24 (Thread 0x7f54b19ff700 (LWP 18053 )): # 0 0x00007f54e49360b3 in select () from /usr/lib/libc.so. 6 # 1 0x0000000000c18fa8 in os_thread_sleep (tm= 1000000 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0thread.cc: 307 # 2 0x0000000000d765a0 in page_cleaner_sleep_if_needed (next_loop_time= 1482129511501 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/buf/buf0flu.cc: 2583 # 3 0x0000000000d76e2a in buf_flush_lru_manager_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/buf/buf0flu.cc: 2858 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 23 (Thread 0x7f54b59ff700 (LWP 18052 )): # 0 0x00007f54e49360b3 in select () from /usr/lib/libc.so. 6 # 1 0x0000000000c18fa8 in os_thread_sleep (tm= 1000000 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0thread.cc: 307 # 2 0x0000000000d765a0 in page_cleaner_sleep_if_needed (next_loop_time= 1482129511498 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/buf/buf0flu.cc: 2583 # 3 0x0000000000d76892 in buf_flush_page_cleaner_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/buf/buf0flu.cc: 2707 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 22 (Thread 0x7f54b27fe700 (LWP 18051 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000c183b9 in os_cond_wait (cond= 0x7f54e455b2b8 , fast_mutex= 0x7f54e455b280 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 196 # 2 0x0000000000c1872f in os_event_wait_low (event= 0x7f54e455b280 , reset_sig_count= 563 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 477 # 3 0x0000000000cd8768 in srv_purge_coordinator_suspend (slot= 0x7f54e45657c8 , rseg_history_len= 1742 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0srv.cc: 3427 # 4 0x0000000000cd8c7c in srv_purge_coordinator_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0srv.cc: 3549 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 21 (Thread 0x7f54b2fff700 (LWP 18050 )): # 0 0x00007f54e49360b3 in select () from /usr/lib/libc.so. 6 # 1 0x0000000000c18fa8 in os_thread_sleep (tm= 1000000 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0thread.cc: 307 # 2 0x0000000000cd7c84 in srv_master_sleep () at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0srv.cc: 3067 # 3 0x0000000000cd7dd3 in srv_master_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0srv.cc: 3117 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 20 (Thread 0x7f54b3bfd700 (LWP 18049 )): # 0 0x00007f54e68974b8 in pthread_cond_timedwait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000c182cd in os_cond_wait_timed (cond= 0x7f54e455b398 , fast_mutex= 0x7f54e455b360 , abstime= 0x7f54b3bfce00 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 159 # 2 0x0000000000c188f1 in os_event_wait_time_low (event= 0x7f54e455b360 , time_in_usec= 5000000 , reset_sig_count= 1 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 591 # 3 0x0000000000cd5f22 in srv_monitor_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0srv.cc: 2030 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 19 (Thread 0x7f54b43fe700 (LWP 18048 )): # 0 0x00007f54e68974b8 in pthread_cond_timedwait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000c182cd in os_cond_wait_timed (cond= 0x7f54e455b328 , fast_mutex= 0x7f54e455b2f0 , abstime= 0x7f54b43fddd0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 159 # 2 0x0000000000c188f1 in os_event_wait_time_low (event= 0x7f54e455b2f0 , time_in_usec= 1000000 , reset_sig_count= 1 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 591 # 3 0x0000000000cd6715 in srv_error_monitor_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0srv.cc: 2306 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 18 (Thread 0x7f54b4bff700 (LWP 18047 )): # 0 0x00007f54e68974b8 in pthread_cond_timedwait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000c182cd in os_cond_wait_timed (cond= 0x7f54e455ca58 , fast_mutex= 0x7f54e455ca20 , abstime= 0x7f54b4bfee30 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 159 # 2 0x0000000000c188f1 in os_event_wait_time_low (event= 0x7f54e455ca20 , time_in_usec= 1000000 , reset_sig_count= 1 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0sync.cc: 591 # 3 0x0000000000bee4f9 in lock_wait_timeout_thread (arg= 0x0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/lock/lock0wait.cc: 534 # 4 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 5 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 17 (Thread 0x7f54b6df6700 (LWP 18044 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45666e0 , segment= 3 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 9 , message1= 0x7f54b6df5e30 , message2= 0x7f54b6df5e28 , type= 0x7f54b6df5e20 , space_id= 0x7f54b6df5e18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 9 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201be08 <n+ 72 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 16 (Thread 0x7f54b75f7700 (LWP 18043 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45666e0 , segment= 2 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 8 , message1= 0x7f54b75f6e30 , message2= 0x7f54b75f6e28 , type= 0x7f54b75f6e20 , space_id= 0x7f54b75f6e18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 8 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201be00 <n+ 64 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 15 (Thread 0x7f54b7df8700 (LWP 18042 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45666e0 , segment= 1 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 7 , message1= 0x7f54b7df7e30 , message2= 0x7f54b7df7e28 , type= 0x7f54b7df7e20 , space_id= 0x7f54b7df7e18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 7 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bdf8 <n+ 56 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 14 (Thread 0x7f54b85f9700 (LWP 18041 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45666e0 , segment= 0 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 6 , message1= 0x7f54b85f8e30 , message2= 0x7f54b85f8e28 , type= 0x7f54b85f8e20 , space_id= 0x7f54b85f8e18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 6 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bdf0 <n+ 48 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 13 (Thread 0x7f54b8dfa700 (LWP 18040 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45665f0 , segment= 3 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 5 , message1= 0x7f54b8df9e30 , message2= 0x7f54b8df9e28 , type= 0x7f54b8df9e20 , space_id= 0x7f54b8df9e18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 5 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bde8 <n+ 40 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 12 (Thread 0x7f54b95fb700 (LWP 18039 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45665f0 , segment= 2 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 4 , message1= 0x7f54b95fae30 , message2= 0x7f54b95fae28 , type= 0x7f54b95fae20 , space_id= 0x7f54b95fae18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 4 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bde0 <n+ 32 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 11 (Thread 0x7f54b9dfc700 (LWP 18038 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45665f0 , segment= 1 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 3 , message1= 0x7f54b9dfbe30 , message2= 0x7f54b9dfbe28 , type= 0x7f54b9dfbe20 , space_id= 0x7f54b9dfbe18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 3 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bdd8 <n+ 24 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 10 (Thread 0x7f54ba5fd700 (LWP 18037 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e45665f0 , segment= 0 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 2 , message1= 0x7f54ba5fce30 , message2= 0x7f54ba5fce28 , type= 0x7f54ba5fce20 , space_id= 0x7f54ba5fce18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 2 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bdd0 <n+ 16 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 9 (Thread 0x7f54badfe700 (LWP 18036 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e4566640 , segment= 0 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 1 , message1= 0x7f54badfde30 , message2= 0x7f54badfde28 , type= 0x7f54badfde20 , space_id= 0x7f54badfde18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 1 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bdc8 <n+ 8 >) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 8 (Thread 0x7f54bb5ff700 (LWP 18035 )): # 0 0x00007f54e6688674 in ?? () from /usr/lib/libaio.so. 1 # 1 0x0000000000c15dad in os_aio_linux_collect (array= 0x7f54e4566690 , segment= 0 , seg_size= 256 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5203 # 2 0x0000000000c1611c in os_aio_linux_handle (global_seg= 0 , message1= 0x7f54bb5fee30 , message2= 0x7f54bb5fee28 , type= 0x7f54bb5fee20 , space_id= 0x7f54bb5fee18 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/os/os0file.cc: 5349 # 3 0x0000000000dd5603 in fil_aio_wait (segment= 0 ) at /home/xyz/mdev- 6860 /server/storage/xtradb/fil/fil0fil.cc: 5783 # 4 0x0000000000cdb41d in io_handler_thread (arg= 0x201bdc0 <n>) at /home/xyz/mdev- 6860 /server/storage/xtradb/srv/srv0start.cc: 535 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 7 (Thread 0x7f54c5dff700 (LWP 18034 )): # 0 0x00007f54e68974b8 in pthread_cond_timedwait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000e6e736 in safe_cond_timedwait (cond= 0x2001d40 <COND_checkpoint>, mp= 0x2001c80 <LOCK_checkpoint>, abstime= 0x7f54c5dfedf0 , file= 0xfd8650 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h" , line= 1202 ) at /home/xyz/mdev- 6860 /server/mysys/thr_mutex.c: 547 # 2 0x0000000000a57f37 in inline_mysql_cond_timedwait (that= 0x2001d40 <COND_checkpoint>, mutex= 0x2001c80 <LOCK_checkpoint>, abstime= 0x7f54c5dfedf0 , src_file= 0xfd8690 "/home/xyz/mdev-6860/server/storage/maria/ma_servicethread.c" , src_line= 116 ) at /home/xyz/mdev- 6860 /server/include/mysql/psi/mysql_thread.h: 1202 # 3 0x0000000000a583ca in my_service_thread_sleep (control= 0x161b180 <checkpoint_control>, sleep_time= 30000000000 ) at /home/xyz/mdev- 6860 /server/storage/maria/ma_servicethread.c: 115 # 4 0x0000000000a4d786 in ma_checkpoint_background (arg= 0x1e ) at /home/xyz/mdev- 6860 /server/storage/maria/ma_checkpoint.c: 708 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 6 (Thread 0x7f54e6c33700 (LWP 18033 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f4cb48 in fifo_lock_get (q= 0x7f54e45e9400 ) at galerautils/src/gu_fifo.c: 235 # 2 0x00007f54e3f4cd6c in gu_fifo_get_head (q= 0x7f54e45e9400 , err= 0x7f54e6c31c30 ) at galerautils/src/gu_fifo.c: 294 # 3 0x00007f54e408ffa5 in gcs_recv (conn= 0x7f54e4570200 , action= 0x7f54e6c31cb0 ) at gcs/src/gcs.cpp: 1787 # 4 0x00007f54e40ee05b in galera::Gcs::recv ( this = 0x7f54e452a5e0 , act=...) at galera/src/galera_gcs.hpp: 118 # 5 0x00007f54e40c87a5 in galera::GcsActionSource::process ( this = 0x7f54e452a728 , recv_ctx= 0x7f54d7413d70 , exit_loop= @0x7f54e6c31d3f : false ) at galera/src/gcs_action_source.cpp: 175 # 6 0x00007f54e40e57af in galera::ReplicatorSMM::async_recv ( this = 0x7f54e452a000 , recv_ctx= 0x7f54d7413d70 ) at galera/src/replicator_smm.cpp: 355 # 7 0x00007f54e4101749 in galera_recv (gh= 0x7f54e444cc80 , recv_ctx= 0x7f54d7413d70 ) at galera/src/wsrep_provider.cpp: 244 # 8 0x00000000007f4d40 in wsrep_replication_process (thd= 0x7f54d7413d70 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 328 # 9 0x000000000059b833 in start_wsrep_THD (arg= 0x7f4c8e <wsrep_replication_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 10 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 11 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 5 (Thread 0x7f54e6c7c700 (LWP 18032 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x0000000000e6e45b in safe_cond_wait (cond= 0x17d57c0 <COND_wsrep_rollback>, mp= 0x17d5700 <LOCK_wsrep_rollback>, file= 0xf27a60 "/home/xyz/mdev-6860/server/include/mysql/psi/mysql_thread.h" , line= 1165 ) at /home/xyz/mdev- 6860 /server/mysys/thr_mutex.c: 493 # 2 0x00000000007f385b in inline_mysql_cond_wait (that= 0x17d57c0 <COND_wsrep_rollback>, mutex= 0x17d5700 <LOCK_wsrep_rollback>, src_file= 0xf27bc8 "/home/xyz/mdev-6860/server/sql/wsrep_thd.cc" , src_line= 422 ) at /home/xyz/mdev- 6860 /server/include/mysql/psi/mysql_thread.h: 1165 # 3 0x00000000007f532a in wsrep_rollback_process (thd= 0x7f54d7613070 ) at /home/xyz/mdev- 6860 /server/sql/wsrep_thd.cc: 422 # 4 0x000000000059b833 in start_wsrep_THD (arg= 0x7f522f <wsrep_rollback_process(THD*)>) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 5475 # 5 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 6 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 4 (Thread 0x7f54d85fe700 (LWP 18031 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f58e83 in gu::Lock::wait ( this = 0x7f54d85fd880 , cond=...) at galerautils/src/gu_lock.hpp: 56 # 2 0x00007f54e409531f in RecvBuf::front ( this = 0x7f54e4594140 , timeout=...) at gcs/src/gcs_gcomm.cpp: 124 # 3 0x00007f54e4092482 in gcomm_recv (backend= 0x7f54e4570578 , msg= 0x7f54e4570460 , timeout= 9223372035999999999 ) at gcs/src/gcs_gcomm.cpp: 673 # 4 0x00007f54e4086cc2 in core_msg_recv (backend= 0x7f54e4570578 , recv_msg= 0x7f54e4570460 , timeout= 9223372035999999999 ) at gcs/src/gcs_core.cpp: 468 # 5 0x00007f54e4088a70 in gcs_core_recv (conn= 0x7f54e4570400 , recv_act= 0x7f54d85fdde0 , timeout= 9223372035999999999 ) at gcs/src/gcs_core.cpp: 1067 # 6 0x00007f54e408e23b in gcs_recv_thread (arg= 0x7f54e4570200 ) at gcs/src/gcs.cpp: 1191 # 7 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 8 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 3 (Thread 0x7f54d8dff700 (LWP 18030 )): # 0 0x00007f54e493ddd3 in epoll_wait () from /usr/lib/libc.so. 6 # 1 0x00007f54e40468db in asio::detail::epoll_reactor::run ( this = 0x7f54e44b9f00 , block= true , ops=...) at /usr/include/asio/detail/impl/epoll_reactor.ipp: 391 # 2 0x00007f54e4047a26 in asio::detail::task_io_service::do_run_one ( this = 0x7f54e448a7e0 , lock=..., this_thread=..., ec=...) at /usr/include/asio/detail/impl/task_io_service.ipp: 355 # 3 0x00007f54e404759e in asio::detail::task_io_service::run ( this = 0x7f54e448a7e0 , ec=...) at /usr/include/asio/detail/impl/task_io_service.ipp: 148 # 4 0x00007f54e4047cbb in asio::io_service::run ( this = 0x7f54e444d4b8 ) at /usr/include/asio/impl/io_service.ipp: 58 # 5 0x00007f54e4076c6c in gcomm::AsioProtonet::event_loop ( this = 0x7f54e444d400 , period=...) at gcomm/src/asio_protonet.cpp: 121 # 6 0x00007f54e4091755 in GCommConn::run ( this = 0x7f54e4593f00 ) at gcs/src/gcs_gcomm.cpp: 498 # 7 0x00007f54e4095bdc in GCommConn::run_fn (arg= 0x7f54e4593f00 ) at gcs/src/gcs_gcomm.cpp: 209 # 8 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 9 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 2 (Thread 0x7f54db7ff700 (LWP 18029 )): # 0 0x00007f54e689710f in pthread_cond_wait@ @GLIBC_2 . 3.2 () from /usr/lib/libpthread.so. 0 # 1 0x00007f54e3f58e83 in gu::Lock::wait ( this = 0x7f54db7fe790 , cond=...) at galerautils/src/gu_lock.hpp: 56 # 2 0x00007f54e40c44fb in galera::ServiceThd::thd_func (arg= 0x7f54e452a5f0 ) at galera/src/galera_service_thd.cpp: 30 # 3 0x00007f54e6891454 in start_thread () from /usr/lib/libpthread.so. 0 # 4 0x00007f54e493d7df in clone () from /usr/lib/libc.so. 6   Thread 1 (Thread 0x7f54e6c7fc00 (LWP 18022 )): # 0 0x00007f54e493448d in poll () from /usr/lib/libc.so. 6 # 1 0x000000000059e5f8 in handle_connections_sockets () at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 6938 # 2 0x000000000059d61c in mysqld_main (argc= 26 , argv= 0x7f54e44373a0 ) at /home/xyz/mdev- 6860 /server/sql/mysqld.cc: 6364 # 3 0x0000000000591816 in main (argc= 25 , argv= 0x7ffff684aa88 ) at /home/xyz/mdev- 6860 /server/sql/main.cc: 25

            Any plans with this please?

            Unfortunately, the slave catchup is really slow without the parallelism.

            hydrapolic Tomáš Mózes added a comment - Any plans with this please? Unfortunately, the slave catchup is really slow without the parallelism.

            Got bitten by this today. We really need a fix, or at least an error when we try to enable parallel async replication in a Galera setup.

            rpizzi Rick Pizzi (Inactive) added a comment - Got bitten by this today. We really need a fix, or at least an error when we try to enable parallel async replication in a Galera setup.

            I dont think that it is race condition , may be some simple error, I am uploading the test case with less inserts , It is failing deterministic-ally for me
            6860.cnf 6860.test

            sachin.setiya.007 Sachin Setiya (Inactive) added a comment - I dont think that it is race condition , may be some simple error, I am uploading the test case with less inserts , It is failing deterministic-ally for me 6860.cnf 6860.test

            Actually it is not deterministic after restart it was not failing , SO increased the no of inserts

            sachin.setiya.007 Sachin Setiya (Inactive) added a comment - Actually it is not deterministic after restart it was not failing , SO increased the no of inserts
            Alex/AT Alex/AT added a comment -

            BUMP

            Encountering the same issue with latest 10.3

            Alex/AT Alex/AT added a comment - BUMP Encountering the same issue with latest 10.3

            Fix is not feasible to apply earlier GA-releases.

            jplindst Jan Lindström (Inactive) added a comment - Fix is not feasible to apply earlier GA-releases.
            mihaQ MikaH added a comment -

            Undortunately this is broken on MariaDB 10.5-series. At least with 10.5.6. If increasing slave_paraller_treads from 1 to 3, few mins later the node is dropped from the cluster

            mihaQ MikaH added a comment - Undortunately this is broken on MariaDB 10.5-series. At least with 10.5.6. If increasing slave_paraller_treads from 1 to 3, few mins later the node is dropped from the cluster

            People

              jplindst Jan Lindström (Inactive)
              nirbhay_c Nirbhay Choubey (Inactive)
              Votes:
              10 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.