Node3 , join attempts 2018-03-30 20:03:29 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2018-03-30 20:03:29 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 2018-03-30 20:03:29 0 [Note] WSREP: wsrep_load(): Galera 25.3.23(r3789) by Codership Oy loaded successfully. 2018-03-30 20:03:29 0 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm. 2018-03-30 20:03:29 0 [Note] WSREP: Found saved state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, safe_to_bootstrap: 0 2018-03-30 20:03:29 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.104.196; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs .delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_wi ndow = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache. mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_li mit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.s egment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.bootstrap = YES; pc 2018-03-30 20:03:29 0 [Note] WSREP: GCache history reset: dff6e041-1005-11e8-85c9-965f304f37bc:0 -> dff6e041-1005-11e8-85c9-965f304f37bc:131618 2018-03-30 20:03:29 0 [Note] WSREP: Assign initial position for certification: 131618, protocol version: -1 2018-03-30 20:03:29 0 [Note] WSREP: wsrep_sst_grab() 2018-03-30 20:03:29 0 [Note] WSREP: Start replication 2018-03-30 20:03:29 0 [Note] WSREP: Setting initial position to dff6e041-1005-11e8-85c9-965f304f37bc:131618 2018-03-30 20:03:29 0 [Note] WSREP: protonet asio version 0 2018-03-30 20:03:29 0 [Note] WSREP: Using CRC-32C for message checksums. 2018-03-30 20:03:29 0 [Note] WSREP: backend: asio 2018-03-30 20:03:29 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2018-03-30 20:03:29 0 [Note] WSREP: restore pc from disk successfully 2018-03-30 20:03:29 0 [Note] WSREP: GMCast version 0 2018-03-30 20:03:29 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2018-03-30 20:03:29 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2018-03-30 20:03:29 0 [Note] WSREP: EVS version 0 2018-03-30 20:03:29 0 [Note] WSREP: gcomm: connecting to group 'cluster1', peer '192.168.104.193:,192.168.104.195:,192.168.104.196:' 2018-03-30 20:03:29 0 [ERROR] WSREP: bind: Address already in use 2018-03-30 20:03:29 0 [ERROR] WSREP: failed to open gcomm backend connection: 98: error while trying to listen 'tcp://0.0.0.0:4567?socket.non_blocking=1', asio error 'bind: Address already in us e': 98 (Address already in use) 2018-03-30 20:03:29 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -98 (Address already in use) 2018-03-30 20:03:29 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel 'cluster1' at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196': -98 (Address already in use) 2018-03-30 20:03:29 0 [ERROR] WSREP: gcs connect failed: Address already in use 2018-03-30 20:03:29 0 [ERROR] WSREP: wsrep::connect(gcomm://192.168.104.193,192.168.104.195,192.168.104.196) failed: 7 2018-03-30 20:03:29 0 [ERROR] Aborting 2018-03-30 20:09:13 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2018-03-30 20:09:13 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 2018-03-30 20:09:13 0 [Note] WSREP: wsrep_load(): Galera 25.3.23(r3789) by Codership Oy loaded successfully. 2018-03-30 20:09:13 0 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm. 2018-03-30 20:09:13 0 [Note] WSREP: Found saved state: dff6e041-1005-11e8-85c9-965f304f37bc:-1, safe_to_bootstrap: 0 2018-03-30 20:09:13 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.104.196; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs .delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_wi ndow = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache. mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_li mit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.s egment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.bootstrap = YES; pc 2018-03-30 20:09:13 0 [Note] WSREP: GCache history reset: dff6e041-1005-11e8-85c9-965f304f37bc:0 -> dff6e041-1005-11e8-85c9-965f304f37bc:131618 2018-03-30 20:09:13 0 [Note] WSREP: Assign initial position for certification: 131618, protocol version: -1 2018-03-30 20:09:13 0 [Note] WSREP: wsrep_sst_grab() 2018-03-30 20:09:13 0 [Note] WSREP: Start replication 2018-03-30 20:09:13 0 [Note] WSREP: Setting initial position to dff6e041-1005-11e8-85c9-965f304f37bc:131618 2018-03-30 20:09:13 0 [Note] WSREP: protonet asio version 0 2018-03-30 20:09:13 0 [Note] WSREP: Using CRC-32C for message checksums. 2018-03-30 20:09:13 0 [Note] WSREP: backend: asio 2018-03-30 20:09:13 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2018-03-30 20:09:13 0 [Note] WSREP: restore pc from disk successfully 2018-03-30 20:09:13 0 [Note] WSREP: GMCast version 0 2018-03-30 20:09:13 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2018-03-30 20:09:13 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2018-03-30 20:09:13 0 [Note] WSREP: EVS version 0 2018-03-30 20:09:13 0 [Note] WSREP: gcomm: connecting to group 'cluster1', peer '192.168.104.193:,192.168.104.195:,192.168.104.196:' 2018-03-30 20:09:13 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') connection established to de19f79a tcp://192.168.104.196:4567 2018-03-30 20:09:13 0 [Warning] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') address 'tcp://192.168.104.196:4567' points to own listening address, blacklisting 2018-03-30 20:09:13 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') connection established to 7059a99a tcp://192.168.104.193:4567 2018-03-30 20:09:13 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2018-03-30 20:09:13 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') connection established to e3153dfe tcp://192.168.104.195:4567 2018-03-30 20:09:13 0 [Note] WSREP: declaring 7059a99a at tcp://192.168.104.193:4567 stable 2018-03-30 20:09:13 0 [Note] WSREP: declaring e3153dfe at tcp://192.168.104.195:4567 stable 2018-03-30 20:09:13 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2018-03-30 20:09:13 0 [Note] WSREP: view(view_id(NON_PRIM,7059a99a,107) memb { 2018-03-30 20:09:14 0 [Note] WSREP: gcomm: connected 2018-03-30 20:09:14 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2018-03-30 20:09:14 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2018-03-30 20:09:14 0 [Note] WSREP: Opened channel 'cluster1' 2018-03-30 20:09:14 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 3 2018-03-30 20:09:14 0 [Note] WSREP: Flow-control interval: [28, 28] 2018-03-30 20:09:14 0 [Note] WSREP: Trying to continue unpaused monitor 2018-03-30 20:09:14 0 [Note] WSREP: Received NON-PRIMARY. 2018-03-30 20:09:14 0 [Note] WSREP: Waiting for SST to complete. 2018-03-30 20:09:14 1 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version -1 2018-03-30 20:09:14 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2018-03-30 20:09:17 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') connection to peer de19f79a with addr tcp://192.168.104.196:4567 timed out, no messages seen in PT3S 2018-03-30 20:09:17 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') turning message relay requesting off 2018-03-30 20:17:36 0 [Note] WSREP: Read nil XID from storage engines, skipping position init 2018-03-30 20:17:36 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 2018-03-30 20:17:36 0 [Note] WSREP: wsrep_load(): Galera 25.3.23(r3789) by Codership Oy loaded successfully. 2018-03-30 20:17:36 0 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm. 2018-03-30 20:17:36 0 [Note] WSREP: Found saved state: dff6e041-1005-11e8-85c9-965f304f37bc:131618, safe_to_bootstrap: 1 2018-03-30 20:17:36 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.104.196; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs .delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_wi ndow = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache. mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_li mit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.s egment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.bootstrap = YES; pc 2018-03-30 20:17:36 0 [Note] WSREP: GCache history reset: dff6e041-1005-11e8-85c9-965f304f37bc:0 -> dff6e041-1005-11e8-85c9-965f304f37bc:131618 2018-03-30 20:17:36 0 [Note] WSREP: Assign initial position for certification: 131618, protocol version: -1 2018-03-30 20:17:36 0 [Note] WSREP: wsrep_sst_grab() 2018-03-30 20:17:36 0 [Note] WSREP: Start replication 2018-03-30 20:17:36 0 [Note] WSREP: Setting initial position to dff6e041-1005-11e8-85c9-965f304f37bc:131618 2018-03-30 20:17:36 0 [Note] WSREP: protonet asio version 0 2018-03-30 20:17:36 0 [Note] WSREP: Using CRC-32C for message checksums. 2018-03-30 20:17:36 0 [Note] WSREP: backend: asio 2018-03-30 20:17:36 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2018-03-30 20:17:36 0 [Note] WSREP: restore pc from disk successfully 2018-03-30 20:17:36 0 [Note] WSREP: GMCast version 0 2018-03-30 20:17:36 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2018-03-30 20:17:36 0 [Note] WSREP: (de19f79a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2018-03-30 20:17:36 0 [Note] WSREP: EVS version 0 2018-03-30 20:17:36 0 [Note] WSREP: gcomm: connecting to group 'cluster1', peer '192.168.104.193:,192.168.104.195:,192.168.104.196:' 2018-03-30 20:17:36 0 [ERROR] WSREP: bind: Address already in use 2018-03-30 20:17:36 0 [ERROR] WSREP: failed to open gcomm backend connection: 98: error while trying to listen 'tcp://0.0.0.0:4567?socket.non_blocking=1', asio error 'bind: Address already in us e': 98 (Address already in use) 2018-03-30 20:17:36 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -98 (Address already in use) 2018-03-30 20:17:36 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel 'cluster1' at 'gcomm://192.168.104.193,192.168.104.195,192.168.104.196': -98 (Address already in use) 2018-03-30 20:17:36 0 [ERROR] WSREP: gcs connect failed: Address already in use 2018-03-30 20:17:36 0 [ERROR] WSREP: wsrep::connect(gcomm://192.168.104.193,192.168.104.195,192.168.104.196) failed: 7 2018-03-30 20:17:36 0 [ERROR] Aborting Node 2 2018-03-30 20:08:51 0 [Note] WSREP: (e3153dfe, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.104.196:4567 2018-03-30 20:08:52 0 [Note] WSREP: (e3153dfe, 'tcp://0.0.0.0:4567') reconnecting to de19f79a (tcp://192.168.104.196:4567), attempt 0 2018-03-30 20:08:56 0 [Note] WSREP: evs::proto(e3153dfe, GATHER, view_id(REG,7059a99a,105)) suspecting node: de19f79a 2018-03-30 20:08:56 0 [Note] WSREP: evs::proto(e3153dfe, GATHER, view_id(REG,7059a99a,105)) suspected node without join message, declaring inactive 2018-03-30 20:08:56 0 [Note] WSREP: declaring 7059a99a at tcp://192.168.104.193:4567 stable 2018-03-30 20:08:56 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2018-03-30 20:08:56 0 [Note] WSREP: view(view_id(NON_PRIM,7059a99a,106) memb { 2018-03-30 20:08:56 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 2 2018-03-30 20:08:56 0 [Note] WSREP: Flow-control interval: [23, 23] 2018-03-30 20:08:56 0 [Note] WSREP: Trying to continue unpaused monitor 2018-03-30 20:08:56 0 [Note] WSREP: Received NON-PRIMARY. 2018-03-30 20:08:56 1 [Note] WSREP: New cluster view: global state: :-1, view# -1: non-Primary, number of nodes: 2, my index: 1, protocol version -1 2018-03-30 20:08:56 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2018-03-30 20:09:13 0 [Note] WSREP: (e3153dfe, 'tcp://0.0.0.0:4567') connection established to de19f79a tcp://192.168.104.196:4567 2018-03-30 20:09:13 0 [Note] WSREP: declaring 7059a99a at tcp://192.168.104.193:4567 stable 2018-03-30 20:09:13 0 [Note] WSREP: declaring de19f79a at tcp://192.168.104.196:4567 stable 2018-03-30 20:09:13 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2018-03-30 20:09:13 0 [Note] WSREP: view(view_id(NON_PRIM,7059a99a,107) memb { 2018-03-30 20:09:13 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 3 2018-03-30 20:09:13 0 [Note] WSREP: Flow-control interval: [28, 28] 2018-03-30 20:09:13 0 [Note] WSREP: Trying to continue unpaused monitor 2018-03-30 20:09:13 0 [Note] WSREP: Received NON-PRIMARY. 2018-03-30 20:09:13 1 [Note] WSREP: New cluster view: global state: :-1, view# -1: non-Primary, number of nodes: 3, my index: 2, protocol version -1 2018-03-30 20:09:13 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2018-03-30 20:09:16 0 [Note] WSREP: (e3153dfe, 'tcp://0.0.0.0:4567') turning message relay requesting off Node 1 2018-03-30 20:08:51 0 [Note] WSREP: (7059a99a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.104.196:4567 2018-03-30 20:08:52 0 [Note] WSREP: (7059a99a, 'tcp://0.0.0.0:4567') reconnecting to de19f79a (tcp://192.168.104.196:4567), attempt 0 2018-03-30 20:08:55 0 [Note] WSREP: evs::proto(7059a99a, OPERATIONAL, view_id(REG,7059a99a,105)) suspecting node: de19f79a 2018-03-30 20:08:55 0 [Note] WSREP: evs::proto(7059a99a, OPERATIONAL, view_id(REG,7059a99a,105)) suspected node without join message, declaring inactive 2018-03-30 20:08:56 0 [Note] WSREP: declaring e3153dfe at tcp://192.168.104.195:4567 stable 2018-03-30 20:08:56 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2018-03-30 20:08:56 0 [Note] WSREP: view(view_id(NON_PRIM,7059a99a,106) memb { 2018-03-30 20:08:56 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 2 2018-03-30 20:08:56 0 [Note] WSREP: Flow-control interval: [23, 23] 2018-03-30 20:08:56 0 [Note] WSREP: Trying to continue unpaused monitor 2018-03-30 20:08:56 0 [Note] WSREP: Received NON-PRIMARY. 2018-03-30 20:08:56 2 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131588, view# -1: non-Primary, number of nodes: 2, my index: 0, protocol version -1 2018-03-30 20:08:56 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2018-03-30 20:09:13 0 [Note] WSREP: (7059a99a, 'tcp://0.0.0.0:4567') connection established to de19f79a tcp://192.168.104.196:4567 2018-03-30 20:09:13 0 [Note] WSREP: declaring de19f79a at tcp://192.168.104.196:4567 stable 2018-03-30 20:09:13 0 [Note] WSREP: declaring e3153dfe at tcp://192.168.104.195:4567 stable 2018-03-30 20:09:13 0 [Warning] WSREP: no nodes coming from prim view, prim not possible 2018-03-30 20:09:13 0 [Note] WSREP: view(view_id(NON_PRIM,7059a99a,107) memb { 2018-03-30 20:09:13 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 3 2018-03-30 20:09:13 0 [Note] WSREP: Flow-control interval: [28, 28] 2018-03-30 20:09:13 0 [Note] WSREP: Trying to continue unpaused monitor 2018-03-30 20:09:13 0 [Note] WSREP: Received NON-PRIMARY. 2018-03-30 20:09:13 2 [Note] WSREP: New cluster view: global state: dff6e041-1005-11e8-85c9-965f304f37bc:131588, view# -1: non-Primary, number of nodes: 3, my index: 0, protocol version -1 2018-03-30 20:09:13 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2018-03-30 20:09:16 0 [Note] WSREP: (7059a99a, 'tcp://0.0.0.0:4567') turning message relay requesting off