Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13687

MariaDB 10.2.8 galera SST fails

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Not a Bug
    • 10.2
    • N/A
    • Galera, Galera SST
    • None
    • CentOS 7.x x86_64 w/ MariaDB 10.2.8

    Description

      Two brand new servers with CentOS 7 latest version, MariaDB 10.2.8.

      MBA2 - 192.168.1.249
      MBA1 - 192.168.1.250

      1) startup MariaDB service on MBA2, create sst user, grant root access
      2) stop MariaDB service on MBA2
      3) run galera_new_cluster
      4) startup MariaDB service again, now I can seee

      MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_clu%';
      		 
      +--------------------------+--------------------------------------+
      		 
      | Variable_name            | Value                                |
      		 
      +--------------------------+--------------------------------------+
      		 
      | wsrep_cluster_conf_id    | 1                                    |
      		 
      | wsrep_cluster_size       | 1                                    |
      		 
      | wsrep_cluster_state_uuid | 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8 |
      		 
      | wsrep_cluster_status     | Primary                              |
      		 
      +--------------------------+--------------------------------------+

      5) startup MariaDB service on MBA1 fails with following error messsages no matter which sst_method I choose.

      • mariabackup

      .
      		 
      .
      		 
      .
      		 
      2017-08-31 16:57:59 140565635680384 [Warning] WSREP: access file(/data/mysql/clst/gvwstate.dat) failed(No such file or directory)
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: restore pc from disk failed
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: GMCast version 0
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: EVS version 0
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: gcomm: connecting to group 'Galera', peer '192.168.1.249:'
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') connection established to 9ba3ac27 tcp://192.168.1.249:4567
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: declaring 9ba3ac27 at tcp://192.168.1.249:4567 stable
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: Node 9ba3ac27 state prim
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: view(view_id(PRIM,7d22ec57,8) memb {
      		 
              7d22ec57,0
      		 
              9ba3ac27,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
      })
      		 
      2017-08-31 16:57:59 140565635680384 [Note] WSREP: save pc into disk
      		 
      2017-08-31 16:58:00 140565635680384 [Note] WSREP: gcomm: connected
      		 
      2017-08-31 16:58:00 140565635680384 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      		 
      2017-08-31 16:58:00 140565635680384 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      		 
      2017-08-31 16:58:00 140565635680384 [Note] WSREP: Opened channel 'Galera'
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      		 
      2017-08-31 16:58:00 140565635680384 [Note] WSREP: Waiting for SST to complete.
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 7d6f76a4-8e2a-11e7-8ba4-f25dcf018c49
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: STATE EXCHANGE: sent state msg: 7d6f76a4-8e2a-11e7-8ba4-f25dcf018c49
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: STATE EXCHANGE: got state msg: 7d6f76a4-8e2a-11e7-8ba4-f25dcf018c49 from 0 (MBA1)
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: STATE EXCHANGE: got state msg: 7d6f76a4-8e2a-11e7-8ba4-f25dcf018c49 from 1 (MBA2)
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: Quorum results:
      		 
              version    = 4,
      		 
              component  = PRIMARY,
      		 
              conf_id    = 7,
      		 
              members    = 1/2 (joined/total),
      		 
              act_id     = 0,
      		 
              last_appl. = -1,
      		 
              protocols  = 0/7/3 (gcs/repl/appl),
      		 
              group UUID = 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: Flow-control interval: [23, 23]
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: State transfer required: 
              Group state: 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8:0
      		 
              Local state: 00000000-0000-0000-0000-000000000000:-1
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: New cluster view: global state: 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8:0, view# 8: Primary, number of nodes: 2, my index: 0, protocol version 3
      		 
      2017-08-31 16:58:00 140565427668736 [Warning] WSREP: Gap in state sequence. Need state transfer.
      		 
      2017-08-31 16:58:00 140563181451008 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '192.168.1.250' --datadir '/data/mysql/data/'   --parent '19302' --binlog '/data/mysql/tran/mysql-bin' '
      		 
      WSREP_SST: [INFO] Streaming with xbstream (20170831 16:58:00.349)
      		 
      WSREP_SST: [INFO] Using socat as streamer (20170831 16:58:00.350)
      		 
      WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20170831 16:58:00.375)
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: Prepared SST request: mariabackup|192.168.1.250:4444/xtrabackup_sst//1
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: REPL Protocols: 7 (3, 2)
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
      		 
      2017-08-31 16:58:00 140565452982016 [Note] WSREP: Service thread queue flushed.
      		 
      2017-08-31 16:58:00 140565427668736 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (9ba441c2-8e29-11e7-b7e0-2f564ca18fa8): 1 (Operation not permitted)
      		 
               at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: Member 0.0 (MBA1) requested state transfer from '*any*'. Selected 1.0 (MBA2)(SYNCED) as donor.
      		 
      2017-08-31 16:58:00 140563189843712 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: Requesting state transfer: success, donor: 1
      		 
      2017-08-31 16:58:00 140565427668736 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(9ba441c2-8e29-11e7-b7e0-2f564ca18fa8:0)
      		 
      2017-08-31 16:58:01 140563189843712 [Warning] WSREP: 1.0 (MBA2): State transfer to 0.0 (MBA1) failed: -2 (No such file or directory)
      		 
      2017-08-31 16:58:01 140563189843712 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort.
      		 
      2017-08-31 16:58:01 140563189843712 [Note] WSREP: gcomm: terminating thread
      		 
      2017-08-31 16:58:01 140563189843712 [Note] WSREP: gcomm: joining thread
      		 
      2017-08-31 16:58:01 140563189843712 [Note] WSREP: gcomm: closing backend
      		 
      2017-08-31 16:58:03 140563189843712 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') turning message relay requesting off
      		 
      2017-08-31 16:58:04 140563189843712 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') connection to peer 9ba3ac27 with addr tcp://192.168.1.249:4567 timed out, no messages seen in PT3S
      		 
      2017-08-31 16:58:04 140563189843712 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.1.249:4567 
      2017-08-31 16:58:05 140563189843712 [Note] WSREP: (7d22ec57, 'tcp://0.0.0.0:4567') reconnecting to 9ba3ac27 (tcp://192.168.1.249:4567), attempt 0
      		 
      2017-08-31 16:58:06 140563189843712 [Note] WSREP: evs::proto(7d22ec57, LEAVING, view_id(REG,7d22ec57,8)) suspecting node: 9ba3ac27
      		 
      2017-08-31 16:58:06 140563189843712 [Note] WSREP: evs::proto(7d22ec57, LEAVING, view_id(REG,7d22ec57,8)) suspected node without join message, declaring inactive
      		 
      2017-08-31 16:58:06 140563189843712 [Note] WSREP: view(view_id(NON_PRIM,7d22ec57,8) memb {
      		 
              7d22ec57,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
              9ba3ac27,0
      		 
      })
      		 
      2017-08-31 16:58:06 140563189843712 [Note] WSREP: view((empty))
      		 
      2017-08-31 16:58:06 140563189843712 [Note] WSREP: gcomm: closed
      		 
      2017-08-31 16:58:06 140563189843712 [Note] WSREP: /usr/sbin/mysqld: Terminated.

      • mysqldump

      .
      		 
      .
      		 
      .
      		 
      2017-08-31 17:09:00 140583527037056 [Warning] WSREP: access file(/data/mysql/clst/gvwstate.dat) failed(No such file or directory)
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: restore pc from disk failed
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: GMCast version 0
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: EVS version 0
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: gcomm: connecting to group 'Galera', peer '192.168.1.249:'
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') connection established to 9ba3ac27 tcp://192.168.1.249:4567
      		 
      2017-08-31 17:09:00 140583527037056 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: declaring 9ba3ac27 at tcp://192.168.1.249:4567 stable
      		 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: Node 9ba3ac27 state prim
      		 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: view(view_id(PRIM,07410f0a,32) memb {
      		 
              07410f0a,0
      		 
              9ba3ac27,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
      })
      		 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: save pc into disk
      		 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: gcomm: connected
      		 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      		 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      		 
      2017-08-31 17:09:01 140583527037056 [Note] WSREP: Opened channel 'Galera'
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 078d99d8-8e2c-11e7-92cc-97ad3b2774c3
      		 
      2017-08-31 17:09:01 140583527037056 [Note] Reading of all Master_info entries succeded
      		 
      2017-08-31 17:09:01 140583527037056 [Note] Added new Master_info '' to hash table
      		 
      2017-08-31 17:09:01 140583527037056 [Note] /usr/sbin/mysqld: ready for connections.
      		 
      Version: '10.2.8-MariaDB-log'  socket: '/data/mysql/vars/mysql.sock'  port: 3306  MariaDB Server
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: STATE EXCHANGE: sent state msg: 078d99d8-8e2c-11e7-92cc-97ad3b2774c3
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: STATE EXCHANGE: got state msg: 078d99d8-8e2c-11e7-92cc-97ad3b2774c3 from 0 (MBA1)
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: STATE EXCHANGE: got state msg: 078d99d8-8e2c-11e7-92cc-97ad3b2774c3 from 1 (MBA2)
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: Quorum results:
      		 
              version    = 4,
      		 
              component  = PRIMARY,
      		 
              conf_id    = 31,
      		 
              members    = 1/2 (joined/total),
      		 
              act_id     = 0,
      		 
              last_appl. = -1,
      		 
              protocols  = 0/7/3 (gcs/repl/appl),
      		 
              group UUID = 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: Flow-control interval: [23, 23]
      		 
      2017-08-31 17:09:01 140577005881088 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
      		 
      2017-08-31 17:09:01 140583073695488 [Note] WSREP: State transfer required: 
              Group state: 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8:0
      		 
              Local state: 00000000-0000-0000-0000-000000000000:-1
      		 
      2017-08-31 17:09:01 140583073695488 [Note] WSREP: New cluster view: global state: 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8:0, view# 32: Primary, number of nodes: 2, my index: 0, protocol version 3
      		 
      2017-08-31 17:09:01 140583073695488 [Warning] WSREP: Gap in state sequence. Need state transfer.
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: Prepared SST request: mysqldump|192.168.1.250:3306
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: REPL Protocols: 7 (3, 2)
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
      		 
      2017-08-31 17:09:03 140577022666496 [Note] WSREP: Service thread queue flushed.
      		 
      2017-08-31 17:09:03 140583073695488 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (9ba441c2-8e29-11e7-b7e0-2f564ca18fa8): 1 (Operation not permitted)
      		 
               at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      		 
      2017-08-31 17:09:03 140577005881088 [Warning] WSREP: Member 0.0 (MBA1) requested state transfer from '192.168.1.249', but it is impossible to select State Transfer donor: No route to host
      		 
      2017-08-31 17:09:03 140583073695488 [ERROR] WSREP: Requesting state transfer failed: -113(No route to host)
      		 
      2017-08-31 17:09:03 140583073695488 [ERROR] WSREP: State transfer request failed unrecoverably: 113 (No route to host). Most likely it is due to inability to communicate with the cluster primary component. Restart required.
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: Closing send monitor...
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: Closed send monitor.
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: gcomm: terminating thread
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: gcomm: joining thread
      		 
      2017-08-31 17:09:03 140583073695488 [Note] WSREP: gcomm: closing backend
      		 
      2017-08-31 17:09:04 140583073695488 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') turning message relay requesting off
      		 
      2017-08-31 17:09:06 140583073695488 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') connection to peer 9ba3ac27 with addr tcp://192.168.1.249:4567 timed out, no messages seen in PT3S
      		 
      2017-08-31 17:09:06 140583073695488 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.1.249:4567 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: (07410f0a, 'tcp://0.0.0.0:4567') reconnecting to 9ba3ac27 (tcp://192.168.1.249:4567), attempt 0
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: evs::proto(07410f0a, LEAVING, view_id(REG,07410f0a,32)) suspecting node: 9ba3ac27
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: evs::proto(07410f0a, LEAVING, view_id(REG,07410f0a,32)) suspected node without join message, declaring inactive
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: view(view_id(NON_PRIM,07410f0a,32) memb {
      		 
              07410f0a,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
              9ba3ac27,0
      		 
      })
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: view((empty))
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: gcomm: closed
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: Flow-control interval: [16, 16]
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: Received NON-PRIMARY.
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 0)
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: Received self-leave message.
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: Flow-control interval: [0, 0]
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 0)
      		 
      2017-08-31 17:09:08 140577005881088 [Note] WSREP: RECV thread exiting 0: Success
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: recv_thread() joined.
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: Closing replication queue.
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: Closing slave action queue.
      		 
      2017-08-31 17:09:08 140583073695488 [Note] WSREP: /usr/sbin/mysqld: Terminated.

      • rsync

      .
      		 
      .
      		 
      .
      		 
      2017-08-31 17:11:15 139844106639488 [Warning] WSREP: access file(/data/mysql/clst/gvwstate.dat) failed(No such file or directory)
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: restore pc from disk failed
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: GMCast version 0
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: EVS version 0
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: gcomm: connecting to group 'Galera', peer '192.168.1.249:'
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') connection established to 9ba3ac27 tcp://192.168.1.249:4567
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: declaring 9ba3ac27 at tcp://192.168.1.249:4567 stable
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: Node 9ba3ac27 state prim
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: view(view_id(PRIM,5753d2a1,36) memb {
      		 
              5753d2a1,0
      		 
              9ba3ac27,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
      })
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: save pc into disk
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: gcomm: connected
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: Opened channel 'Galera'
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      		 
      2017-08-31 17:11:15 139844106639488 [Note] WSREP: Waiting for SST to complete.
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 57a057d2-8e2c-11e7-907c-bea4d0033439
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: STATE EXCHANGE: sent state msg: 57a057d2-8e2c-11e7-907c-bea4d0033439
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: STATE EXCHANGE: got state msg: 57a057d2-8e2c-11e7-907c-bea4d0033439 from 0 (MBA1)
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: STATE EXCHANGE: got state msg: 57a057d2-8e2c-11e7-907c-bea4d0033439 from 1 (MBA2)
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: Quorum results:
      		 
              version    = 4,
      		 
              component  = PRIMARY,
      		 
              conf_id    = 35,
      		 
              members    = 1/2 (joined/total),
      		 
              act_id     = 0,
      		 
              last_appl. = -1,
      		 
              protocols  = 0/7/3 (gcs/repl/appl),
      		 
              group UUID = 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: Flow-control interval: [23, 23]
      		 
      2017-08-31 17:11:15 139843881977600 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: State transfer required: 
              Group state: 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8:0
      		 
              Local state: 00000000-0000-0000-0000-000000000000:-1
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: New cluster view: global state: 9ba441c2-8e29-11e7-b7e0-2f564ca18fa8:0, view# 36: Primary, number of nodes: 2, my index: 0, protocol version 3
      		 
      2017-08-31 17:11:15 139843873449728 [Warning] WSREP: Gap in state sequence. Need state transfer.
      		 
      2017-08-31 17:11:15 139841652123392 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.1.250' --datadir '/data/mysql/data/'   --parent '2418' --binlog '/data/mysql/tran/mysql-bin' '
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: Prepared SST request: rsync|192.168.1.250:4444/rsync_sst
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: REPL Protocols: 7 (3, 2)
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
      		 
      2017-08-31 17:11:15 139843923941120 [Note] WSREP: Service thread queue flushed.
      		 
      2017-08-31 17:11:15 139843873449728 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (9ba441c2-8e29-11e7-b7e0-2f564ca18fa8): 1 (Operation not permitted)
      		 
               at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      		 
      2017-08-31 17:11:15 139843881977600 [Warning] WSREP: Member 0.0 (MBA1) requested state transfer from '192.168.1.249', but it is impossible to select State Transfer donor: No route to host
      		 
      2017-08-31 17:11:15 139843873449728 [ERROR] WSREP: Requesting state transfer failed: -113(No route to host)
      		 
      2017-08-31 17:11:15 139843873449728 [ERROR] WSREP: State transfer request failed unrecoverably: 113 (No route to host). Most likely it is due to inability to communicate with the cluster primary component. Restart required.
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: Closing send monitor...
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: Closed send monitor.
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: gcomm: terminating thread
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: gcomm: joining thread
      		 
      2017-08-31 17:11:15 139843873449728 [Note] WSREP: gcomm: closing backend
      		 
      2017-08-31 17:11:18 139843873449728 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') turning message relay requesting off
      		 
      2017-08-31 17:11:19 139843873449728 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') connection to peer 9ba3ac27 with addr tcp://192.168.1.249:4567 timed out, no messages seen in PT3S
      		 
      2017-08-31 17:11:19 139843873449728 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.1.249:4567 
      2017-08-31 17:11:20 139843873449728 [Note] WSREP: (5753d2a1, 'tcp://0.0.0.0:4567') reconnecting to 9ba3ac27 (tcp://192.168.1.249:4567), attempt 0
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: evs::proto(5753d2a1, LEAVING, view_id(REG,5753d2a1,36)) suspecting node: 9ba3ac27
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: evs::proto(5753d2a1, LEAVING, view_id(REG,5753d2a1,36)) suspected node without join message, declaring inactive
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: view(view_id(NON_PRIM,5753d2a1,36) memb {
      		 
              5753d2a1,0
      		 
      } joined {
      		 
      } left {
      		 
      } partitioned {
      		 
              9ba3ac27,0
      		 
      })
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: view((empty))
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: gcomm: closed
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: Flow-control interval: [16, 16]
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: Received NON-PRIMARY.
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 0)
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: Received self-leave message.
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: Flow-control interval: [0, 0]
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 0)
      		 
      2017-08-31 17:11:21 139843881977600 [Note] WSREP: RECV thread exiting 0: Success
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: recv_thread() joined.
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: Closing replication queue.
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: Closing slave action queue.
      		 
      2017-08-31 17:11:21 139843873449728 [Note] WSREP: /usr/sbin/mysqld: Terminated.
      		 
      WSREP_SST: [ERROR] Parent mysqld process (PID:2418) terminated unexpectedly. (20170831 17:11:21.934)
      		 
      WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2466 (20170831 17:11:21.936)
      		 
      WSREP_SST: [INFO] Joiner cleanup done. (20170831 17:11:22.439)

      galera.cnf

      [galera]
      		 
      wsrep_on                        = 1
      		 
      wsrep_cluster_address           = "gcomm://192.168.1.249"
      		 
      wsrep_cluster_name              = Galera
      		 
      wsrep_gtid_domain_id            = 192
      		 
      wsrep_node_address              = "192.168.1.250"
      		 
      wsrep_node_name                 = MBA1
      		 
      wsrep_sst_donor                 = 192.168.1.249
      		 
      wsrep_sst_method                = rsync
      		 
      wsrep_sst_auth                  = sst:changeme
      		 
      wsrep_data_home_dir             = /data/mysql/clst
      		 
      wsrep_auto_increment_control    = 1
      		 
      wsrep_provider                  = /usr/lib64/galera/libgalera_smm.so
      		 
      wsrep_slave_threads             = 4
      		 
      wsrep_provider_options          = "gcache.size=2048M"
      		 
      wsrep_debug                     = 0
      		 
      wsrep_gtid_mode                 = 1
      		 
      wsrep_max_ws_rows               = 0
      		 
      wsrep_max_ws_size               = 2147483647
      		 
      wsrep_recover                   = 0
      		 
      wsrep_replicate_myisam          = 0
      		 
      wsrep_sst_donor_rejects_queries = 0
      		 
      wsrep_sync_wait                 = 0

      And I don't any network connection problems.

      [root@MBA1 clst]# telnet 192.168.1.249 3306
      		 
      Trying 192.168.1.249...
      		 
      Connected to 192.168.1.249.
      		 
      Escape character is '^]'.
      		 
      \
      		 
      5.5.5-10.2.8-MariaDB-log0]bWk6h-��-��CJ(+=1|?e`3{mysql_native_password^]
      		 
      telnet> quit
      		 
      Connection closed.
      		 
      [root@MBA1 clst]# telnet 192.168.1.249 4567
      		 
      Trying 192.168.1.249...
      		 
      Connected to 192.168.1.249.
      		 
      Escape character is '^]'.
      		 
      $�Ӕ     )��j��,纑J�Ǿ�^]
      		 
      telnet> quit
      		 
      Connection closed.
      		 
      [root@MBA1 clst]# ping 192.168.1.249  
      PING 192.168.1.249 (192.168.1.249) 56(84) bytes of data.
      		 
      64 bytes from 192.168.1.249: icmp_seq=1 ttl=64 time=0.248 ms
      		 
      64 bytes from 192.168.1.249: icmp_seq=2 ttl=64 time=0.278 ms
      		 
      64 bytes from 192.168.1.249: icmp_seq=3 ttl=64 time=0.285 ms
      		 
      ^C
      		 
      --- 192.168.1.249 ping statistics ---
      		 
      3 packets transmitted, 3 received, 0% packet loss, time 2000ms
      		 
      rtt min/avg/max/mdev = 0.248/0.270/0.285/0.020 ms

      Attachments

        Activity

          People

            anikitin Andrii Nikitin (Inactive)
            mxu Michael Xu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.