-- Logs begin at Mon 2018-03-26 22:43:28 CEST, end at Fri 2018-04-06 20:02:38 CEST. -- Mar 29 14:02:31 db2.fraggelberget.nu sshd[13297]: Connection closed by 172.16.66.90 port 42200 [preauth] Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762556528384 [Note] WSREP: (15e1b953, 'tcp://0.0.0.0:4567') connection established to 2c0ead3f tcp://172.16.66.43:4567 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762556528384 [Note] WSREP: (15e1b953, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762556528384 [Note] WSREP: declaring 2c0ead3f at tcp://172.16.66.43:4567 stable Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762556528384 [Note] WSREP: declaring 3a7f6ad5 at tcp://172.16.66.41:4567 stable Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762556528384 [Note] WSREP: Node 15e1b953 state prim Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762556528384 [Note] WSREP: view(view_id(PRIM,15e1b953,211) memb { Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 15e1b953,0 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2c0ead3f,0 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 3a7f6ad5,0 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: } joined { Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: } left { Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: } partitioned { Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: }) Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762556528384 [Note] WSREP: save pc into disk Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 2c5ddf12-3349-11e8-86a7-0eef57676d91 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: STATE EXCHANGE: sent state msg: 2c5ddf12-3349-11e8-86a7-0eef57676d91 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: STATE EXCHANGE: got state msg: 2c5ddf12-3349-11e8-86a7-0eef57676d91 from 0 (db2) Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: STATE EXCHANGE: got state msg: 2c5ddf12-3349-11e8-86a7-0eef57676d91 from 2 (db1) Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: STATE EXCHANGE: got state msg: 2c5ddf12-3349-11e8-86a7-0eef57676d91 from 1 (db3) Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: Quorum results: Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: version = 4, Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: component = PRIMARY, Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: conf_id = 210, Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: members = 2/3 (joined/total), Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: act_id = 28320690, Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: last_appl. = 28320620, Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: protocols = 0/8/3 (gcs/repl/appl), Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: group UUID = da51edca-0520-11e5-9c41-1b0454cc1522 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: Flow-control interval: [28, 28] Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762545960704 [Note] WSREP: Trying to continue unpaused monitor Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762537568000 [Note] WSREP: New cluster view: global state: da51edca-0520-11e5-9c41-1b0454cc1522:28320690, view# 211: Primary, number of nodes: 3, my index: 0, protocol version 3 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762537568000 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762537568000 [Note] WSREP: REPL Protocols: 8 (3, 2) Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762537568000 [Note] WSREP: Assign initial position for certification: 28320690, protocol version: 3 Mar 29 14:03:20 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:20 139762590099200 [Note] WSREP: Service thread queue flushed. Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139762545960704 [Note] WSREP: Member 1.0 (db3) requested state transfer from '*any*'. Selected 0.0 (db2)(SYNCED) as donor. Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139762545960704 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 28320699) Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139762537568000 [Note] WSREP: IST request: da51edca-0520-11e5-9c41-1b0454cc1522:28182031-28320690|tcp://172.16.66.43:4568 Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139762537568000 [Note] WSREP: IST first seqno 28182032 not found from cache, falling back to SST Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139762537568000 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139761321821952 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '172.16.66.43:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' '' --gtid 'da51edca-0520-11e5-9c41-1b0454cc1522:28320699' --g Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139762537568000 [Note] WSREP: sst_donor_thread signaled with 0 Mar 29 14:03:21 db2.fraggelberget.nu maxscale[2240]: Server changed state: db1[172.16.66.41:3307]: new_master. [Slave, Synced, Running] -> [Master, Synced, Running] Mar 29 14:03:21 db2.fraggelberget.nu maxscale[2240]: Server changed state: db2[172.16.66.42:3307]: lost_master. [Master, Synced, Running] -> [Running] Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139761321821952 [Note] WSREP: Flushing tables for SST... Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139761321821952 [Note] WSREP: Provider paused at da51edca-0520-11e5-9c41-1b0454cc1522:28320703 (532455) Mar 29 14:03:21 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:21 139761321821952 [Note] WSREP: Tables flushed. Mar 29 14:03:23 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:03:23 139762556528384 [Note] WSREP: (15e1b953, 'tcp://0.0.0.0:4567') turning message relay requesting off Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Connection reset by peer (104) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync: connection unexpectedly closed (271 bytes received so far) [sender] Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync: failed to connect to 172.16.66.43 (172.16.66.43): Connection refused (111) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync error: error in socket IO (code 10) at clientserver.c(122) [sender=3.0.9] Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync: failed to connect to 172.16.66.43 (172.16.66.43): Connection refused (111) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync error: error in socket IO (code 10) at clientserver.c(122) [sender=3.0.9] Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync: failed to connect to 172.16.66.43 (172.16.66.43): Connection refused (111) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync error: error in socket IO (code 10) at clientserver.c(122) [sender=3.0.9] Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync: failed to connect to 172.16.66.43 (172.16.66.43): Connection refused (111) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync error: error in socket IO (code 10) at clientserver.c(122) [sender=3.0.9] Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync: failed to connect to 172.16.66.43 (172.16.66.43): Connection refused (111) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: rsync error: error in socket IO (code 10) at clientserver.c(122) [sender=3.0.9] Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: WSREP_SST: [ERROR] find/rsync returned code 123: (20180329 14:04:50.378) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:50 139761321821952 [ERROR] WSREP: Failed to read from: wsrep_sst_rsync --role 'donor' --address '172.16.66.43:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' '' --gtid 'da51edca-0520-11e5-9c41-1b0454cc1522:2 Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:50 139761321821952 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'donor' --address '172.16.66.43:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' '' --gtid 'da51edca-0520-11e5-9c41-1b045 Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:50 139761321821952 [Note] WSREP: resuming provider at 532455 Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:50 139761321821952 [Note] WSREP: Provider resumed. Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:50 139761321821952 [ERROR] WSREP: Command did not run: wsrep_sst_rsync --role 'donor' --address '172.16.66.43:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' '' --gtid 'da51edca-0520-11e5-9c41-1b0454cc1522:2 Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:50 139762545960704 [Warning] WSREP: 0.0 (db2): State transfer to 1.0 (db3) failed: -255 (Unknown error 255) Mar 29 14:04:50 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:50 139762545960704 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 28321957) Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762556528384 [Note] WSREP: declaring 3a7f6ad5 at tcp://172.16.66.41:4567 stable Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762556528384 [Note] WSREP: forgetting 2c0ead3f (tcp://172.16.66.43:4567) Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762556528384 [Note] WSREP: Node 15e1b953 state prim Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762556528384 [Note] WSREP: view(view_id(PRIM,15e1b953,212) memb { Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 15e1b953,0 Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 3a7f6ad5,0 Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: } joined { Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: } left { Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: } partitioned { Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2c0ead3f,0 Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: }) Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762556528384 [Note] WSREP: save pc into disk Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762556528384 [Note] WSREP: forgetting 2c0ead3f (tcp://172.16.66.43:4567) Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 628ef79f-3349-11e8-80ae-bbe33983b8d8 Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: STATE EXCHANGE: sent state msg: 628ef79f-3349-11e8-80ae-bbe33983b8d8 Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: STATE EXCHANGE: got state msg: 628ef79f-3349-11e8-80ae-bbe33983b8d8 from 0 (db2) Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: STATE EXCHANGE: got state msg: 628ef79f-3349-11e8-80ae-bbe33983b8d8 from 1 (db1) Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: Quorum results: Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: version = 4, Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: component = PRIMARY, Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: conf_id = 211, Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: members = 2/2 (joined/total), Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: act_id = 28321957, Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: last_appl. = 28320620, Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: protocols = 0/8/3 (gcs/repl/appl), Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: group UUID = da51edca-0520-11e5-9c41-1b0454cc1522 Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: Flow-control interval: [23, 23] Mar 29 14:04:51 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:51 139762545960704 [Note] WSREP: Trying to continue unpaused monitor Mar 29 14:04:56 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:04:56 139762556528384 [Note] WSREP: cleaning up 2c0ead3f (tcp://172.16.66.43:4567) Mar 29 14:05:30 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:05:30 139762537568000 [Note] WSREP: New cluster view: global state: da51edca-0520-11e5-9c41-1b0454cc1522:28321957, view# 212: Primary, number of nodes: 2, my index: 0, protocol version 3 Mar 29 14:05:30 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:05:30 139762537568000 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. Mar 29 14:05:30 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:05:30 139762537568000 [Note] WSREP: REPL Protocols: 8 (3, 2) Mar 29 14:05:30 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:05:30 139762537568000 [Note] WSREP: Assign initial position for certification: 28321957, protocol version: 3 Mar 29 14:05:30 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:05:30 139762590099200 [Note] WSREP: Service thread queue flushed. Mar 29 14:06:03 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:06:03 139762545960704 [Note] WSREP: Member 0.0 (db2) synced with group. Mar 29 14:06:03 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:06:03 139762545960704 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 28322986) Mar 29 14:06:04 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:06:04 139762537568000 [Note] WSREP: Synchronized with group, ready for connections Mar 29 14:06:04 db2.fraggelberget.nu mysqld[1453]: 2018-03-29 14:06:04 139762537568000 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. Mar 29 14:06:05 db2.fraggelberget.nu maxscale[2240]: Server changed state: db1[172.16.66.41:3307]: new_slave. [Master, Synced, Running] -> [Slave, Synced, Running]