Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9599

When using systemd and an SST fails, mysqld on the joiner automatically restarts

    XMLWordPrintable

Details

    Description

      When using systemd in MariaDB 10.1 and an SST fails, mysqld on the joiner automatically restarts.

      Is this intentional?

      Below is an example with wsrep_sst_rsync attempting to do an SST 4 times after automatically restarting.

      Feb 19 12:35:08 ip-172-31-19-192 systemd: Starting MariaDB database server...
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] /usr/sbin/mysqld (mysqld 10.1.11-MariaDB) starting as process 2383 ...
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Read nil XID from storage engines, skipping position init
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: wsrep_load(): Galera 25.3.12(r3516) by Codership Oy <info@codership.com> loaded successfully.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: CRC-32C: using hardware acceleration.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.19.192; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564825634560 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: wsrep_sst_grab()
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Start replication
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: protonet asio version 0
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Using CRC-32C for message checksums.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: backend: asio
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: restore pc from disk failed
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: GMCast version 0
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: (1f255a15, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: (1f255a15, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: EVS version 0
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '172.31.22.174:,172.31.19.192:'
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Warning] WSREP: (1f255a15, 'tcp://0.0.0.0:4567') address 'tcp://172.31.19.192:4567' points to own listening address, blacklisting
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: (1f255a15, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: declaring 474dfc6f at tcp://172.31.22.174:4567 stable
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Node 474dfc6f state prim
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: view(view_id(PRIM,1f255a15,4) memb {
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 1f255a15,0
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: save pc into disk
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: gcomm: connected
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Opened channel 'my_wsrep_cluster'
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 1f721fbf-d72f-11e5-8415-d2f3d6bbba50
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565056141440 [Note] WSREP: Waiting for SST to complete.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: STATE EXCHANGE: sent state msg: 1f721fbf-d72f-11e5-8415-d2f3d6bbba50
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: STATE EXCHANGE: got state msg: 1f721fbf-d72f-11e5-8415-d2f3d6bbba50 from 0 ()
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: STATE EXCHANGE: got state msg: 1f721fbf-d72f-11e5-8415-d2f3d6bbba50 from 1 ()
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: Quorum results:
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: version    = 3,
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: component  = PRIMARY,
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: conf_id    = 3,
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: members    = 1/2 (joined/total),
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: act_id     = 1,
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: last_appl. = -1,
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: protocols  = 0/7/3 (gcs/repl/appl),
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: group UUID = 474f3f92-d723-11e5-8da3-d3e87bd5db9a
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: Flow-control interval: [23, 23]
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1)
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Note] WSREP: State transfer required:
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: Group state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: Local state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Note] WSREP: New cluster view: global state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 3
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Warning] WSREP: Gap in state sequence. Need state transfer.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564758517504 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.19.192' --datadir '/var/lib/mysql/'   --parent '2383'  '' '
      Feb 19 12:35:09 ip-172-31-19-192 rsyncd[2421]: rsyncd version 3.0.9 starting, listening on port 4444
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Note] WSREP: Prepared SST request: rsync|172.31.19.192:4444/rsync_sst
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Note] WSREP: REPL Protocols: 7 (3, 2)
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564825634560 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564825634560 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (474f3f92-d723-11e5-8da3-d3e87bd5db9a): 1 (Operation not permitted)
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor.
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140564766910208 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
      Feb 19 12:35:09 ip-172-31-19-192 mysqld: 2016-02-19 12:35:09 140565055826688 [Note] WSREP: Requesting state transfer: success, donor: 1
      Feb 19 12:35:10 ip-172-31-19-192 mysqld: 2016-02-19 12:35:10 140564766910208 [Warning] WSREP: 1.0 (): State transfer to 0.0 () failed: -78 (Remote address changed)
      Feb 19 12:35:10 ip-172-31-19-192 mysqld: 2016-02-19 12:35:10 140564766910208 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():731: Will never receive state. Need to abort.
      Feb 19 12:35:10 ip-172-31-19-192 mysqld: 2016-02-19 12:35:10 140564766910208 [Note] WSREP: gcomm: terminating thread
      Feb 19 12:35:10 ip-172-31-19-192 mysqld: 2016-02-19 12:35:10 140564766910208 [Note] WSREP: gcomm: joining thread
      Feb 19 12:35:10 ip-172-31-19-192 mysqld: 2016-02-19 12:35:10 140564766910208 [Note] WSREP: gcomm: closing backend
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: 2016-02-19 12:35:11 140564766910208 [Note] WSREP: view(view_id(NON_PRIM,1f255a15,4) memb {
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: 1f255a15,0
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: 2016-02-19 12:35:11 140564766910208 [Note] WSREP: view((empty))
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: 2016-02-19 12:35:11 140564766910208 [Note] WSREP: gcomm: closed
      Feb 19 12:35:11 ip-172-31-19-192 mysqld: 2016-02-19 12:35:11 140564766910208 [Note] WSREP: /usr/sbin/mysqld: Terminated.
      Feb 19 12:35:11 ip-172-31-19-192 systemd: mariadb.service: main process exited, code=killed, status=6/ABRT
      Feb 19 12:35:11 ip-172-31-19-192 systemd: Failed to start MariaDB database server.
      Feb 19 12:35:11 ip-172-31-19-192 systemd: Unit mariadb.service entered failed state.
      Feb 19 12:35:12 ip-172-31-19-192 mysqld: WSREP_SST: [ERROR] Parent mysqld process (PID:2383) terminated unexpectedly. (20160219 12:35:12.696)
      Feb 19 12:35:12 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2421 (20160219 12:35:12.698)
      Feb 19 12:35:13 ip-172-31-19-192 rsyncd[2421]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(551) [Receiver=3.0.9]
      Feb 19 12:35:13 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20160219 12:35:13.202)
      Feb 19 12:35:16 ip-172-31-19-192 systemd: mariadb.service holdoff time over, scheduling restart.
      Feb 19 12:35:16 ip-172-31-19-192 systemd: Stopping MariaDB database server...
      Feb 19 12:35:16 ip-172-31-19-192 systemd: Starting MariaDB database server...
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] /usr/sbin/mysqld (mysqld 10.1.11-MariaDB) starting as process 2463 ...
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Read nil XID from storage engines, skipping position init
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: wsrep_load(): Galera 25.3.12(r3516) by Codership Oy <info@codership.com> loaded successfully.
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: CRC-32C: using hardware acceleration.
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.19.192; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140684967278336 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: wsrep_sst_grab()
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Start replication
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: protonet asio version 0
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Using CRC-32C for message checksums.
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: backend: asio
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: restore pc from disk failed
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: GMCast version 0
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: (23e701f3, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: (23e701f3, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: EVS version 0
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '172.31.22.174:,172.31.19.192:'
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Warning] WSREP: (23e701f3, 'tcp://0.0.0.0:4567') address 'tcp://172.31.19.192:4567' points to own listening address, blacklisting
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: (23e701f3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: declaring 474dfc6f at tcp://172.31.22.174:4567 stable
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: Node 474dfc6f state prim
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: view(view_id(PRIM,23e701f3,6) memb {
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 23e701f3,0
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:17 ip-172-31-19-192 mysqld: 2016-02-19 12:35:17 140685195495552 [Note] WSREP: save pc into disk
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195495552 [Note] WSREP: gcomm: connected
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195495552 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195495552 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195495552 [Note] WSREP: Opened channel 'my_wsrep_cluster'
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 248016a6-d72f-11e5-afc0-d3e786fff5e6
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195495552 [Note] WSREP: Waiting for SST to complete.
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: STATE EXCHANGE: sent state msg: 248016a6-d72f-11e5-afc0-d3e786fff5e6
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: STATE EXCHANGE: got state msg: 248016a6-d72f-11e5-afc0-d3e786fff5e6 from 0 ()
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: STATE EXCHANGE: got state msg: 248016a6-d72f-11e5-afc0-d3e786fff5e6 from 1 ()
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: Quorum results:
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: version    = 3,
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: component  = PRIMARY,
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: conf_id    = 5,
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: members    = 1/2 (joined/total),
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: act_id     = 1,
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: last_appl. = -1,
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: protocols  = 0/7/3 (gcs/repl/appl),
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: group UUID = 474f3f92-d723-11e5-8da3-d3e87bd5db9a
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: Flow-control interval: [23, 23]
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1)
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Note] WSREP: State transfer required:
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: Group state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: Local state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Note] WSREP: New cluster view: global state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 3
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Warning] WSREP: Gap in state sequence. Need state transfer.
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684900161280 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.19.192' --datadir '/var/lib/mysql/'   --parent '2463'  '' '
      Feb 19 12:35:18 ip-172-31-19-192 rsyncd[2501]: rsyncd version 3.0.9 starting, listening on port 4444
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Note] WSREP: Prepared SST request: rsync|172.31.19.192:4444/rsync_sst
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Note] WSREP: REPL Protocols: 7 (3, 2)
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684967278336 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684967278336 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (474f3f92-d723-11e5-8da3-d3e87bd5db9a): 1 (Operation not permitted)
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor.
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140684908553984 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
      Feb 19 12:35:18 ip-172-31-19-192 mysqld: 2016-02-19 12:35:18 140685195180800 [Note] WSREP: Requesting state transfer: success, donor: 1
      Feb 19 12:35:19 ip-172-31-19-192 mysqld: 2016-02-19 12:35:19 140684908553984 [Warning] WSREP: 1.0 (): State transfer to 0.0 () failed: -78 (Remote address changed)
      Feb 19 12:35:19 ip-172-31-19-192 mysqld: 2016-02-19 12:35:19 140684908553984 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():731: Will never receive state. Need to abort.
      Feb 19 12:35:19 ip-172-31-19-192 mysqld: 2016-02-19 12:35:19 140684908553984 [Note] WSREP: gcomm: terminating thread
      Feb 19 12:35:19 ip-172-31-19-192 mysqld: 2016-02-19 12:35:19 140684908553984 [Note] WSREP: gcomm: joining thread
      Feb 19 12:35:19 ip-172-31-19-192 mysqld: 2016-02-19 12:35:19 140684908553984 [Note] WSREP: gcomm: closing backend
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: 2016-02-19 12:35:20 140684908553984 [Note] WSREP: view(view_id(NON_PRIM,23e701f3,6) memb {
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: 23e701f3,0
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: 2016-02-19 12:35:20 140684908553984 [Note] WSREP: (23e701f3, 'tcp://0.0.0.0:4567') turning message relay requesting off
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: 2016-02-19 12:35:20 140684908553984 [Note] WSREP: view((empty))
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: 2016-02-19 12:35:20 140684908553984 [Note] WSREP: gcomm: closed
      Feb 19 12:35:20 ip-172-31-19-192 mysqld: 2016-02-19 12:35:20 140684908553984 [Note] WSREP: /usr/sbin/mysqld: Terminated.
      Feb 19 12:35:20 ip-172-31-19-192 systemd: mariadb.service: main process exited, code=killed, status=6/ABRT
      Feb 19 12:35:20 ip-172-31-19-192 systemd: Failed to start MariaDB database server.
      Feb 19 12:35:20 ip-172-31-19-192 systemd: Unit mariadb.service entered failed state.
      Feb 19 12:35:21 ip-172-31-19-192 mysqld: WSREP_SST: [ERROR] Parent mysqld process (PID:2463) terminated unexpectedly. (20160219 12:35:21.184)
      Feb 19 12:35:21 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2501 (20160219 12:35:21.186)
      Feb 19 12:35:21 ip-172-31-19-192 rsyncd[2501]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(551) [Receiver=3.0.9]
      Feb 19 12:35:21 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20160219 12:35:21.691)
      Feb 19 12:35:25 ip-172-31-19-192 systemd: mariadb.service holdoff time over, scheduling restart.
      Feb 19 12:35:25 ip-172-31-19-192 systemd: Stopping MariaDB database server...
      Feb 19 12:35:25 ip-172-31-19-192 systemd: Starting MariaDB database server...
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] /usr/sbin/mysqld (mysqld 10.1.11-MariaDB) starting as process 2543 ...
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Read nil XID from storage engines, skipping position init
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: wsrep_load(): Galera 25.3.12(r3516) by Codership Oy <info@codership.com> loaded successfully.
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: CRC-32C: using hardware acceleration.
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.19.192; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140673818818304 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: wsrep_sst_grab()
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Start replication
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: protonet asio version 0
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Using CRC-32C for message checksums.
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: backend: asio
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: restore pc from disk failed
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: GMCast version 0
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: (28e8b828, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: (28e8b828, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: EVS version 0
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '172.31.22.174:,172.31.19.192:'
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Warning] WSREP: (28e8b828, 'tcp://0.0.0.0:4567') address 'tcp://172.31.19.192:4567' points to own listening address, blacklisting
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: (28e8b828, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: declaring 474dfc6f at tcp://172.31.22.174:4567 stable
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: Node 474dfc6f state prim
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: view(view_id(PRIM,28e8b828,8) memb {
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 28e8b828,0
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:25 ip-172-31-19-192 mysqld: 2016-02-19 12:35:25 140674047051904 [Note] WSREP: save pc into disk
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674047051904 [Note] WSREP: gcomm: connected
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674047051904 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674047051904 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674047051904 [Note] WSREP: Opened channel 'my_wsrep_cluster'
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 2981cc91-d72f-11e5-8035-6a7f61753a42
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674047051904 [Note] WSREP: Waiting for SST to complete.
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: STATE EXCHANGE: sent state msg: 2981cc91-d72f-11e5-8035-6a7f61753a42
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: STATE EXCHANGE: got state msg: 2981cc91-d72f-11e5-8035-6a7f61753a42 from 0 ()
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: STATE EXCHANGE: got state msg: 2981cc91-d72f-11e5-8035-6a7f61753a42 from 1 ()
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: Quorum results:
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: version    = 3,
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: component  = PRIMARY,
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: conf_id    = 7,
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: members    = 1/2 (joined/total),
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: act_id     = 1,
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: last_appl. = -1,
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: protocols  = 0/7/3 (gcs/repl/appl),
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: group UUID = 474f3f92-d723-11e5-8da3-d3e87bd5db9a
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: Flow-control interval: [23, 23]
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1)
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Note] WSREP: State transfer required:
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: Group state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: Local state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Note] WSREP: New cluster view: global state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1, view# 8: Primary, number of nodes: 2, my index: 0, protocol version 3
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Warning] WSREP: Gap in state sequence. Need state transfer.
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673751701248 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.19.192' --datadir '/var/lib/mysql/'   --parent '2543'  '' '
      Feb 19 12:35:26 ip-172-31-19-192 rsyncd[2581]: rsyncd version 3.0.9 starting, listening on port 4444
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Note] WSREP: Prepared SST request: rsync|172.31.19.192:4444/rsync_sst
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Note] WSREP: REPL Protocols: 7 (3, 2)
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673818818304 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673818818304 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (474f3f92-d723-11e5-8da3-d3e87bd5db9a): 1 (Operation not permitted)
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor.
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140673760093952 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
      Feb 19 12:35:26 ip-172-31-19-192 mysqld: 2016-02-19 12:35:26 140674046737152 [Note] WSREP: Requesting state transfer: success, donor: 1
      Feb 19 12:35:28 ip-172-31-19-192 mysqld: 2016-02-19 12:35:28 140673760093952 [Warning] WSREP: 1.0 (): State transfer to 0.0 () failed: -78 (Remote address changed)
      Feb 19 12:35:28 ip-172-31-19-192 mysqld: 2016-02-19 12:35:28 140673760093952 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():731: Will never receive state. Need to abort.
      Feb 19 12:35:28 ip-172-31-19-192 mysqld: 2016-02-19 12:35:28 140673760093952 [Note] WSREP: gcomm: terminating thread
      Feb 19 12:35:28 ip-172-31-19-192 mysqld: 2016-02-19 12:35:28 140673760093952 [Note] WSREP: gcomm: joining thread
      Feb 19 12:35:28 ip-172-31-19-192 mysqld: 2016-02-19 12:35:28 140673760093952 [Note] WSREP: gcomm: closing backend
      Feb 19 12:35:28 ip-172-31-19-192 mysqld: 2016-02-19 12:35:28 140673760093952 [Note] WSREP: (28e8b828, 'tcp://0.0.0.0:4567') turning message relay requesting off
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: 2016-02-19 12:35:29 140673760093952 [Note] WSREP: view(view_id(NON_PRIM,28e8b828,8) memb {
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: 28e8b828,0
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: 2016-02-19 12:35:29 140673760093952 [Note] WSREP: view((empty))
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: 2016-02-19 12:35:29 140673760093952 [Note] WSREP: gcomm: closed
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: 2016-02-19 12:35:29 140673760093952 [Note] WSREP: /usr/sbin/mysqld: Terminated.
      Feb 19 12:35:29 ip-172-31-19-192 systemd: mariadb.service: main process exited, code=killed, status=6/ABRT
      Feb 19 12:35:29 ip-172-31-19-192 systemd: Failed to start MariaDB database server.
      Feb 19 12:35:29 ip-172-31-19-192 systemd: Unit mariadb.service entered failed state.
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: WSREP_SST: [ERROR] Parent mysqld process (PID:2543) terminated unexpectedly. (20160219 12:35:29.597)
      Feb 19 12:35:29 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2581 (20160219 12:35:29.599)
      Feb 19 12:35:30 ip-172-31-19-192 rsyncd[2581]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(551) [Receiver=3.0.9]
      Feb 19 12:35:30 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20160219 12:35:30.104)
      Feb 19 12:35:34 ip-172-31-19-192 systemd: mariadb.service holdoff time over, scheduling restart.
      Feb 19 12:35:34 ip-172-31-19-192 systemd: Stopping MariaDB database server...
      Feb 19 12:35:34 ip-172-31-19-192 systemd: Starting MariaDB database server...
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] /usr/sbin/mysqld (mysqld 10.1.11-MariaDB) starting as process 2625 ...
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Read nil XID from storage engines, skipping position init
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: wsrep_load(): Galera 25.3.12(r3516) by Codership Oy <info@codership.com> loaded successfully.
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: CRC-32C: using hardware acceleration.
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.31.19.192; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139911814444800 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: wsrep_sst_grab()
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Start replication
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: protonet asio version 0
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Using CRC-32C for message checksums.
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: backend: asio
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: restore pc from disk failed
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: GMCast version 0
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: (2e322b1a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: (2e322b1a, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: EVS version 0
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '172.31.22.174:,172.31.19.192:'
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Warning] WSREP: (2e322b1a, 'tcp://0.0.0.0:4567') address 'tcp://172.31.19.192:4567' points to own listening address, blacklisting
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: (2e322b1a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: declaring 474dfc6f at tcp://172.31.22.174:4567 stable
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: Node 474dfc6f state prim
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: view(view_id(PRIM,2e322b1a,10) memb {
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2e322b1a,0
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:34 ip-172-31-19-192 mysqld: 2016-02-19 12:35:34 139912045295744 [Note] WSREP: save pc into disk
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912045295744 [Note] WSREP: gcomm: connected
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912045295744 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912045295744 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912045295744 [Note] WSREP: Opened channel 'my_wsrep_cluster'
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 2ecb3fac-d72f-11e5-983f-a263893355be
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912045295744 [Note] WSREP: Waiting for SST to complete.
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: STATE EXCHANGE: sent state msg: 2ecb3fac-d72f-11e5-983f-a263893355be
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: STATE EXCHANGE: got state msg: 2ecb3fac-d72f-11e5-983f-a263893355be from 0 ()
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: STATE EXCHANGE: got state msg: 2ecb3fac-d72f-11e5-983f-a263893355be from 1 ()
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: Quorum results:
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: version    = 3,
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: component  = PRIMARY,
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: conf_id    = 9,
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: members    = 1/2 (joined/total),
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: act_id     = 1,
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: last_appl. = -1,
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: protocols  = 0/7/3 (gcs/repl/appl),
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: group UUID = 474f3f92-d723-11e5-8da3-d3e87bd5db9a
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: Flow-control interval: [23, 23]
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1)
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Note] WSREP: State transfer required:
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: Group state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: Local state: 00000000-0000-0000-0000-000000000000:-1
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Note] WSREP: New cluster view: global state: 474f3f92-d723-11e5-8da3-d3e87bd5db9a:1, view# 10: Primary, number of nodes: 2, my index: 0, protocol version 3
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Warning] WSREP: Gap in state sequence. Need state transfer.
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911747327744 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.31.19.192' --datadir '/var/lib/mysql/'   --parent '2625'  '' '
      Feb 19 12:35:35 ip-172-31-19-192 rsyncd[2663]: rsyncd version 3.0.9 starting, listening on port 4444
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Note] WSREP: Prepared SST request: rsync|172.31.19.192:4444/rsync_sst
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Note] WSREP: REPL Protocols: 7 (3, 2)
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911814444800 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911814444800 [Note] WSREP: Service thread queue flushed.
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (474f3f92-d723-11e5-8da3-d3e87bd5db9a): 1 (Operation not permitted)
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor.
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139911755720448 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
      Feb 19 12:35:35 ip-172-31-19-192 mysqld: 2016-02-19 12:35:35 139912044980992 [Note] WSREP: Requesting state transfer: success, donor: 1
      Feb 19 12:35:36 ip-172-31-19-192 mysqld: 2016-02-19 12:35:36 139911755720448 [Warning] WSREP: 1.0 (): State transfer to 0.0 () failed: -78 (Remote address changed)
      Feb 19 12:35:36 ip-172-31-19-192 mysqld: 2016-02-19 12:35:36 139911755720448 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():731: Will never receive state. Need to abort.
      Feb 19 12:35:36 ip-172-31-19-192 mysqld: 2016-02-19 12:35:36 139911755720448 [Note] WSREP: gcomm: terminating thread
      Feb 19 12:35:36 ip-172-31-19-192 mysqld: 2016-02-19 12:35:36 139911755720448 [Note] WSREP: gcomm: joining thread
      Feb 19 12:35:36 ip-172-31-19-192 mysqld: 2016-02-19 12:35:36 139911755720448 [Note] WSREP: gcomm: closing backend
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: 2016-02-19 12:35:37 139911755720448 [Note] WSREP: view(view_id(NON_PRIM,2e322b1a,10) memb {
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: 2e322b1a,0
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: } joined {
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: } left {
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: } partitioned {
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: 474dfc6f,0
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: })
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: 2016-02-19 12:35:37 139911755720448 [Note] WSREP: (2e322b1a, 'tcp://0.0.0.0:4567') turning message relay requesting off
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: 2016-02-19 12:35:37 139911755720448 [Note] WSREP: view((empty))
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: 2016-02-19 12:35:37 139911755720448 [Note] WSREP: gcomm: closed
      Feb 19 12:35:37 ip-172-31-19-192 mysqld: 2016-02-19 12:35:37 139911755720448 [Note] WSREP: /usr/sbin/mysqld: Terminated.
      Feb 19 12:35:37 ip-172-31-19-192 systemd: mariadb.service: main process exited, code=killed, status=6/ABRT
      Feb 19 12:35:37 ip-172-31-19-192 systemd: Failed to start MariaDB database server.
      Feb 19 12:35:37 ip-172-31-19-192 systemd: Unit mariadb.service entered failed state.
      Feb 19 12:35:38 ip-172-31-19-192 mysqld: WSREP_SST: [ERROR] Parent mysqld process (PID:2625) terminated unexpectedly. (20160219 12:35:38.462)
      Feb 19 12:35:38 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2663 (20160219 12:35:38.464)
      Feb 19 12:35:38 ip-172-31-19-192 systemd: Stopping MariaDB database server...
      Feb 19 12:35:38 ip-172-31-19-192 systemd: Stopped MariaDB database server.
      Feb 19 12:35:38 ip-172-31-19-192 rsyncd[2663]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(551) [Receiver=3.0.9]
      Feb 19 12:35:38 ip-172-31-19-192 mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20160219 12:35:38.969)

      Attachments

        Activity

          People

            jplindst Jan Lindström (Inactive)
            GeoffMontee Geoff Montee (Inactive)
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.