Apr 25 22:06:09 galera-node-2 mysqld_safe: WSREP: Recovered position 32c39115-8d32-11e5-9122-b6a2b3e41530:724808778 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] /usr/sbin/mysqld (mysqld 10.0.34-MariaDB-1~trusty-wsrep) starting as process 29529 ... Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Read nil XID from storage engines, skipping position init Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: wsrep_load(): Galera 25.3.22(r3764) by Codership Oy loaded successfully. Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: CRC-32C: using hardware acceleration. Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Found saved state: 32c39115-8d32-11e5-9122-b6a2b3e41530:-1, safe_to_bootstrap: 0 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.55.55.86; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; Apr 25 22:06:09 galera-node-2 mysqld: pc.checksum = false; pc Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: GCache history reset: 32c39115-8d32-11e5-9122-b6a2b3e41530:0 -> 32c39115-8d32-11e5-9122-b6a2b3e41530:724808778 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Assign initial position for certification: 724808778, protocol version: -1 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: wsrep_sst_grab() Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Start replication Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Setting initial position to 32c39115-8d32-11e5-9122-b6a2b3e41530:724808778 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: protonet asio version 0 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Using CRC-32C for message checksums. Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: backend: asio Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: gcomm thread scheduling priority set to other:0 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: restore pc from disk failed Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: GMCast version 0 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: EVS version 0 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: gcomm: connecting to group 'GALERACLUSTER', peer '10.55.55.85:,10.55.55.86:,10.55.55.87:' Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') connection established to 185196d1 tcp://10.55.55.86:4567 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Warning] WSREP: (185196d1, 'tcp://0.0.0.0:4567') address 'tcp://10.55.55.86:4567' points to own listening address, blacklisting Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') connection established to acfd2d2f tcp://10.55.55.87:4567 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') connection established to d607ef0f tcp://10.55.55.85:4567 Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: declaring acfd2d2f at tcp://10.55.55.87:4567 stable Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: declaring d607ef0f at tcp://10.55.55.85:4567 stable Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: Node acfd2d2f state prim Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: view(view_id(PRIM,185196d1,12) memb { Apr 25 22:06:09 galera-node-2 mysqld: #011185196d1,0 Apr 25 22:06:09 galera-node-2 mysqld: #011acfd2d2f,0 Apr 25 22:06:09 galera-node-2 mysqld: #011d607ef0f,0 Apr 25 22:06:09 galera-node-2 mysqld: } joined { Apr 25 22:06:09 galera-node-2 mysqld: } left { Apr 25 22:06:09 galera-node-2 mysqld: } partitioned { Apr 25 22:06:09 galera-node-2 mysqld: }) Apr 25 22:06:09 galera-node-2 mysqld: 180425 22:06:09 [Note] WSREP: save pc into disk Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: gcomm: connected Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Opened channel 'GALERACLUSTER' Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Waiting for SST to complete. Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 18ea8d6b-48c4-11e8-8a41-0fef3fa593ed Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: STATE EXCHANGE: sent state msg: 18ea8d6b-48c4-11e8-8a41-0fef3fa593ed Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: STATE EXCHANGE: got state msg: 18ea8d6b-48c4-11e8-8a41-0fef3fa593ed from 0 (mysql02) Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: STATE EXCHANGE: got state msg: 18ea8d6b-48c4-11e8-8a41-0fef3fa593ed from 1 (mysql03) Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: STATE EXCHANGE: got state msg: 18ea8d6b-48c4-11e8-8a41-0fef3fa593ed from 2 (mysql01) Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Quorum results: Apr 25 22:06:10 galera-node-2 mysqld: #011version = 4, Apr 25 22:06:10 galera-node-2 mysqld: #011component = PRIMARY, Apr 25 22:06:10 galera-node-2 mysqld: #011conf_id = 10, Apr 25 22:06:10 galera-node-2 mysqld: #011members = 2/3 (joined/total), Apr 25 22:06:10 galera-node-2 mysqld: #011act_id = 724815064, Apr 25 22:06:10 galera-node-2 mysqld: #011last_appl. = -1, Apr 25 22:06:10 galera-node-2 mysqld: #011protocols = 0/7/3 (gcs/repl/appl), Apr 25 22:06:10 galera-node-2 mysqld: #011group UUID = 32c39115-8d32-11e5-9122-b6a2b3e41530 Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Flow-control interval: [28, 28] Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Trying to continue unpaused monitor Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 724815064) Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: State transfer required: Apr 25 22:06:10 galera-node-2 mysqld: #011Group state: 32c39115-8d32-11e5-9122-b6a2b3e41530:724815064 Apr 25 22:06:10 galera-node-2 mysqld: #011Local state: 32c39115-8d32-11e5-9122-b6a2b3e41530:724808778 Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: New cluster view: global state: 32c39115-8d32-11e5-9122-b6a2b3e41530:724815064, view# 11: Primary, number of nodes: 3, my index: 0, protocol version 3 Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Warning] WSREP: Gap in state sequence. Need state transfer. Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '10.55.55.86' --datadir '/var/lib/mysql/' --parent '29529' --binlog '/var/log/mysql/mariadb-bin' ' Apr 25 22:06:10 galera-node-2 mysqld: /usr//bin/wsrep_sst_xtrabackup: line 397: setup_ports: command not found Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [ERROR] WSREP: Failed to read 'ready ' from: wsrep_sst_xtrabackup --role 'joiner' --address '10.55.55.86' --datadir '/var/lib/mysql/' --parent '29529' --binlog '/var/log/mysql/mariadb-bin' Apr 25 22:06:10 galera-node-2 mysqld: #011Read: '(null)' Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '10.55.55.86' --datadir '/var/lib/mysql/' --parent '29529' --binlog '/var/log/mysql/mariadb-bin' : 2 (No such file or directory) Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [ERROR] WSREP: Failed to prepare for 'xtrabackup' SST. Unrecoverable. Apr 25 22:06:10 galera-node-2 mysqld: 180425 22:06:10 [ERROR] Aborting Apr 25 22:06:10 galera-node-2 mysqld: Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Closing send monitor... Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Closed send monitor. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: gcomm: terminating thread Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: gcomm: joining thread Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: gcomm: closing backend Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: view(view_id(NON_PRIM,185196d1,12) memb { Apr 25 22:06:12 galera-node-2 mysqld: #011185196d1,0 Apr 25 22:06:12 galera-node-2 mysqld: } joined { Apr 25 22:06:12 galera-node-2 mysqld: } left { Apr 25 22:06:12 galera-node-2 mysqld: } partitioned { Apr 25 22:06:12 galera-node-2 mysqld: #011acfd2d2f,0 Apr 25 22:06:12 galera-node-2 mysqld: #011d607ef0f,0 Apr 25 22:06:12 galera-node-2 mysqld: }) Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') connection to peer 185196d1 with addr tcp://10.55.55.86:4567 timed out, no messages seen in PT3S Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: (185196d1, 'tcp://0.0.0.0:4567') turning message relay requesting off Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: view((empty)) Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: gcomm: closed Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Flow-control interval: [16, 16] Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Trying to continue unpaused monitor Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Received NON-PRIMARY. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 724815071) Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Received self-leave message. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Flow-control interval: [0, 0] Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Trying to continue unpaused monitor Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Received SELF-LEAVE. Closing connection. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 724815071) Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: RECV thread exiting 0: Success Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: recv_thread() joined. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Closing replication queue. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Closing slave action queue. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: Service disconnected. Apr 25 22:06:12 galera-node-2 mysqld: 180425 22:06:12 [Note] WSREP: rollbacker thread exiting Apr 25 22:06:13 galera-node-2 mysqld: 180425 22:06:13 [Note] WSREP: Some threads may fail to exit. Apr 25 22:06:13 galera-node-2 mysqld: 180425 22:06:13 [Note] /usr/sbin/mysqld: Shutdown complete Apr 25 22:06:13 galera-node-2 mysqld: Apr 25 22:06:18 galera-node-2 mysqld: Error in my_thread_global_end(): 1 threads didn't exit Apr 25 22:06:18 galera-node-2 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended