Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Not a Bug
-
10.4.15
-
None
-
Ubuntu 18.04
Description
The goal was to join a node from dev to make a test to upgrade in 10.5 (get prod as master, to got activity).
sst_auth was different (forgot to update)
prod don't got right to go to 4444 of the node 4
node 1~3 : 10.4.15
node 4 : 10.4.14
the node 4 try to join the cluster
node3 as donor => failed to process IST, ... => New SELF-LEAVE
node1 as donor => failed to process IST, ... => New SELF-LEAVE
node2 as donor => alone => non primary
on global status all this node was in status : "initialized"
So if we know the name of the cluster and one node to join we can switchof the cluster.
===== Node 1 log =====
2020-10-13 13:05:13 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:14 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') connection established to f7b801a1-a84f tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:14 0 [Note] WSREP: declaring 68c69353-9bc1 at tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring f7b801a1-a84f at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: Node 68c69353-9bc1 state prim |
2020-10-13 13:05:14 0 [Note] WSREP: view(view_id(PRIM,68c69353-9bc1,156) memb { |
68c69353-9bc1,0 |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:05:14 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:14 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 4 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 0 (db1-c2m4p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 1 (db1-c2m2p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 2 (db1-c2m3p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 3 (db1-c2m2d) |
2020-10-13 13:05:14 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 65, |
members = 3/4 (joined/total), |
act_id = 3286865882, |
last_appl. = 3286865813, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:05:14 0 [Note] WSREP: Flow-control interval: [32, 32] |
2020-10-13 13:05:14 2 [Note] WSREP: ####### processing CC 3286865883, local, ordered |
2020-10-13 13:05:14 2 [Note] WSREP: ####### My UUID: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf |
2020-10-13 13:05:14 2 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:05:14 2 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:05:14 2 [Note] WSREP: ####### Adjusting cert position: 3286865882 -> 3286865883 |
2020-10-13 13:05:14 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:05:14 2 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286865883 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 1 |
members(4): |
0: 68c69353-09a8-11eb-9bc1-3376b5cfc3ea, db1-c2m4p |
1: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf, db1-c2m2p |
2: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
3: f7b801a1-0d43-11eb-a84f-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:05:14 2 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865814 |
2020-10-13 13:05:14 2 [Note] WSREP: Min available from gcache for CC from group: 3285368863 |
2020-10-13 13:05:16 0 [Note] WSREP: Member 3.0 (db1-c2m2d) requested state transfer from '*any*'. Selected 0.0 (db1-c2m4p)(SYNCED) as donor. |
2020-10-13 13:05:16 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:05:16 0 [Note] WSREP: declaring f7b801a1-a84f at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:05:16 0 [Note] WSREP: forgetting 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:17 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:05:19 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:05:19 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,157) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
} joined {
|
} left {
|
} partitioned {
|
68c69353-9bc1,0 |
})
|
2020-10-13 13:05:19 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:19 0 [Note] WSREP: forgetting 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:19 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 |
2020-10-13 13:05:19 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 1 (db1-c2m3p) |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 0 (db1-c2m2p) |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 2 (db1-c2m2d) |
2020-10-13 13:05:19 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 66, |
members = 2/3 (joined/total), |
act_id = 3286866015, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:05:19 0 [Note] WSREP: Flow-control interval: [28, 28] |
2020-10-13 13:05:19 2 [Note] WSREP: ####### processing CC 3286866016, local, ordered |
2020-10-13 13:05:19 2 [Note] WSREP: ####### My UUID: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf |
2020-10-13 13:05:19 2 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:05:19 2 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:05:19 2 [Note] WSREP: ####### Adjusting cert position: 3286866015 -> 3286866016 |
2020-10-13 13:05:19 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:05:19 2 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866016 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(3): |
0: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf, db1-c2m2p |
1: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
2: f7b801a1-0d43-11eb-a84f-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:05:19 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:19 2 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865943 |
2020-10-13 13:05:19 2 [Note] WSREP: Min available from gcache for CC from group: 3285369050 |
2020-10-13 13:05:20 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') reconnecting to f7b801a1-a84f (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 0 |
2020-10-13 13:05:21 0 [Note] WSREP: cleaning up 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:40 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') connection established to f7b801a1-a850 tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:40 0 [Note] WSREP: remote endpoint tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 changed identity f7b801a1-0d43-11eb-a84f-878d2f0020e7 -> f7b801a1-0d43-11eb-a850-878d2f0020e7 |
2020-10-13 13:05:44 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:05:49 0 [Note] WSREP: evs::proto(b51bd9d6-8d21, OPERATIONAL, view_id(REG,b51bd9d6-8d21,157)) suspecting node: f7b801a1-a84f |
2020-10-13 13:05:49 0 [Note] WSREP: evs::proto(b51bd9d6-8d21, OPERATIONAL, view_id(REG,b51bd9d6-8d21,157)) suspected node without join message, declaring inactive |
2020-10-13 13:05:50 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:05:50 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:05:50 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,158) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a84f,0 |
})
|
2020-10-13 13:05:50 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:50 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 |
2020-10-13 13:05:50 0 [Note] WSREP: forgetting f7b801a1-a850 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567) |
2020-10-13 13:05:50 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0dda9600-0d44-11eb-aaaf-3b6a45970173 |
2020-10-13 13:05:51 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') connection established to f7b801a1-a850 tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:51 0 [Warning] WSREP: discarding established (time wait) f7b801a1-a850 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567) |
2020-10-13 13:05:53 0 [Note] WSREP: cleaning up f7b801a1-a850 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567) |
2020-10-13 13:05:53 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') connection established to f7b801a1-a850 tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:53 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: |
2020-10-13 13:05:56 0 [Note] WSREP: (b51bd9d6-8d21, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:05:58 0 [Warning] WSREP: Failed to report last committed 77d86ba4-c581-11e8-b013-173e23a1017d:3286866014, -110 (Connection timed out) |
2020-10-13 13:06:21 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:06:21 0 [Note] WSREP: declaring f7b801a1-a850 at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:06:21 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:06:21 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,159) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a850,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:06:21 0 [Note] WSREP: save pc into disk |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0dda9600-0d44-11eb-aaaf-3b6a45970173 |
2020-10-13 13:06:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 |
2020-10-13 13:06:21 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 2058e43e-0d44-11eb-8787-1f99ecd795c6 |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 0 (db1-c2m2p) |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 2 (db1-c2m2d) |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 1 (db1-c2m3p) |
2020-10-13 13:06:21 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 67, |
members = 2/3 (joined/total), |
act_id = 3286866099, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:06:21 0 [Note] WSREP: Flow-control interval: [28, 28] |
2020-10-13 13:06:21 2 [Note] WSREP: ####### processing CC 3286866100, local, ordered |
2020-10-13 13:06:21 2 [Note] WSREP: ####### My UUID: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf |
2020-10-13 13:06:21 2 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:06:21 2 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:06:21 2 [Note] WSREP: ####### Adjusting cert position: 3286866099 -> 3286866100 |
2020-10-13 13:06:21 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:06:21 2 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866100 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(3): |
0: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf, db1-c2m2p |
1: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
2: f7b801a1-0d43-11eb-a850-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:06:22 2 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865943 |
2020-10-13 13:06:22 2 [Note] WSREP: Min available from gcache for CC from group: 3285369224 |
2020-10-13 13:06:29 0 [Note] WSREP: Member 2.0 (db1-c2m2d) requested state transfer from '*any*'. Selected 0.0 (db1-c2m2p)(SYNCED) as donor. |
2020-10-13 13:06:29 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 3286866166) |
2020-10-13 13:06:29 2 [Note] WSREP: Detected STR version: 1, req_len: 155, req: STRv1 |
2020-10-13 13:06:29 2 [Note] WSREP: Cert index preload: 3286865943 -> 3286866100 |
2020-10-13 13:06:29 2 [ERROR] WSREP: Failed to process action STATE_REQUEST, g: 3286866166, l: 20899575, ptr: 0x7f222d992380, size: 155: IST sender, failed to connect 'tcp://db1-c2m2d.os.acsc2012:4568': connect: Connection refused: 111 (Connection refused) |
at galera/src/ist.cpp:Sender():735 |
2020-10-13 13:06:29 2 [Note] WSREP: Closing send monitor... |
2020-10-13 13:06:29 2 [Note] WSREP: Closed send monitor. |
2020-10-13 13:06:29 2 [Note] WSREP: gcomm: terminating thread |
2020-10-13 13:06:29 2 [Note] WSREP: gcomm: joining thread |
2020-10-13 13:06:29 2 [Note] WSREP: gcomm: closing backend |
2020-10-13 13:06:29 2 [Note] WSREP: view(view_id(NON_PRIM,b51bd9d6-8d21,159) memb { |
b51bd9d6-8d21,0 |
} joined {
|
} left {
|
} partitioned {
|
c7ca3602-9def,0 |
f7b801a1-a850,0 |
})
|
2020-10-13 13:06:29 2 [Note] WSREP: PC protocol downgrade 1 -> 0 |
2020-10-13 13:06:29 2 [Note] WSREP: view((empty)) |
2020-10-13 13:06:29 2 [Note] WSREP: gcomm: closed |
2020-10-13 13:06:29 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2020-10-13 13:06:29 0 [Note] WSREP: Flow-control interval: [16, 16] |
2020-10-13 13:06:29 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:06:29 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 3286866279) |
2020-10-13 13:06:29 0 [Note] WSREP: New SELF-LEAVE. |
2020-10-13 13:06:29 0 [Note] WSREP: Flow-control interval: [0, 0] |
2020-10-13 13:06:29 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. |
2020-10-13 13:06:29 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 3286866279) |
2020-10-13 13:06:29 0 [Note] WSREP: RECV thread exiting 0: Success |
2020-10-13 13:06:29 2 [Note] WSREP: recv_thread() joined. |
2020-10-13 13:06:29 2 [Note] WSREP: Closing replication queue. |
2020-10-13 13:06:29 2 [Note] WSREP: Closing slave action queue. |
2020-10-13 13:06:29 2 [Note] WSREP: ================================================ |
View:
|
id: 00000000-0000-0000-0000-000000000000:-1 |
status: non-primary
|
protocol_version: -1 |
capabilities:
|
final: yes |
own_index: -1 |
members(0): |
=================================================
|
2020-10-13 13:06:29 2 [Note] WSREP: Non-primary view |
2020-10-13 13:06:29 2 [Note] WSREP: Server status change synced -> disconnected |
2020-10-13 13:06:30 2 [Note] WSREP: Applier thread exiting ret: 6 thd: 2 |
2020-10-13 13:06:30 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) |
2020-10-13 13:08:50 14167028 [Warning] Aborted connection 14167028 to db: 'sfr_pims' user: 'tolkien' host: 'sva6.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167024 [Warning] Aborted connection 14167024 to db: 'sfr_pims' user: 'tolkien' host: 'sva9.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167021 [Warning] Aborted connection 14167021 to db: 'sfr_pims' user: 'tolkien' host: 'sva8.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167020 [Warning] Aborted connection 14167020 to db: 'sfr_pims' user: 'tolkien' host: 'sva8.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167030 [Warning] Aborted connection 14167030 to db: 'sfr_pims' user: 'tolkien' host: 'sva8.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167038 [Warning] Aborted connection 14167038 to db: 'sfr_pims' user: 'tolkien' host: 'sva8.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167037 [Warning] Aborted connection 14167037 to db: 'sfr_pims' user: 'tolkien' host: 'sva7.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167040 [Warning] Aborted connection 14167040 to db: 'sfr_pims' user: 'tolkien' host: 'sva7.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167034 [Warning] Aborted connection 14167034 to db: 'sfr_pims' user: 'tolkien' host: 'sva9.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167022 [Warning] Aborted connection 14167022 to db: 'sfr_pims' user: 'tolkien' host: 'sva9.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167023 [Warning] Aborted connection 14167023 to db: 'sfr_pims' user: 'tolkien' host: 'sva6.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167035 [Warning] Aborted connection 14167035 to db: 'sfr_pims' user: 'tolkien' host: 'sva6.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167031 [Warning] Aborted connection 14167031 to db: 'sfr_pims' user: 'tolkien' host: 'sva6.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167041 [Warning] Aborted connection 14167041 to db: 'sfr_pims' user: 'tolkien' host: 'sva7.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:50 14167044 [Warning] Aborted connection 14167044 to db: 'sfr_pims' user: 'tolkien' host: 'sva7.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:08:56 14167083 [Warning] Aborted connection 14167083 to db: 'sfr_pims' user: 'tolkien' host: 'sva7.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:09:01 14166441 [Warning] Aborted connection 14166441 to db: 'sfr_pims' user: 'tolkien' host: 'sva7.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:09:22 14167138 [Warning] Aborted connection 14167138 to db: 'sfr_pims' user: 'tolkien' host: 'sva7.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:09:22 14167135 [Warning] Aborted connection 14167135 to db: 'sfr_pims' user: 'tolkien' host: 'sva8.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:09:22 14167140 [Warning] Aborted connection 14167140 to db: 'sfr_pims' user: 'tolkien' host: 'sva9.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:09:22 14167139 [Warning] Aborted connection 14167139 to db: 'sfr_pims' user: 'tolkien' host: 'sva6.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:12:45 14117071 [Warning] Aborted connection 14117071 to db: 'sfr_pims' user: 'tolkien' host: 'pims-celery-c0m1p.os.pfpims.sfrpims.fr' (Got timeout reading communication packets) |
2020-10-13 13:19:47 14167305 [Warning] WSREP: Unknown parameter 'pc.bootstrap' |
2020-10-13 13:19:47 14167305 [ERROR] WSREP: Set options returned 7 |
2020-10-13 13:20:03 14167305 [Warning] WSREP: Unknown parameter 'pc.bootstrap' |
2020-10-13 13:20:03 14167305 [ERROR] WSREP: Set options returned 7 |
2020-10-13 13:21:11 14167305 [Warning] WSREP: Unknown parameter 'pc. bootstrap' |
2020-10-13 13:21:11 14167305 [ERROR] WSREP: Set options returned 7 |
2020-10-13 13:21:28 14168200 [Warning] WSREP: Unknown parameter 'pc.bootstrap' |
2020-10-13 13:21:28 14168200 [ERROR] WSREP: Set options returned 7 |
2020-10-13 13:23:04 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown |
2020-10-13 13:23:04 0 [Note] WSREP: Shutdown replication |
|
===== Node 2 log =====
2020-10-13 13:05:13 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') connection established to f7b801a1-a84f tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:13 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: |
2020-10-13 13:05:14 0 [Note] WSREP: declaring 68c69353-9bc1 at tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring f7b801a1-a84f at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: Node 68c69353-9bc1 state prim |
2020-10-13 13:05:14 0 [Note] WSREP: view(view_id(PRIM,68c69353-9bc1,156) memb { |
68c69353-9bc1,0 |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:05:14 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:14 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 4 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 0 (db1-c2m4p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 1 (db1-c2m2p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 2 (db1-c2m3p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 3 (db1-c2m2d) |
2020-10-13 13:05:14 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 65, |
members = 3/4 (joined/total), |
act_id = 3286865882, |
last_appl. = 3286865813, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:05:14 0 [Note] WSREP: Flow-control interval: [32, 32] |
2020-10-13 13:05:14 1 [Note] WSREP: ####### processing CC 3286865883, local, ordered |
2020-10-13 13:05:14 1 [Note] WSREP: ####### My UUID: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44 |
2020-10-13 13:05:14 1 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:05:14 1 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:05:14 1 [Note] WSREP: ####### Adjusting cert position: 3286865882 -> 3286865883 |
2020-10-13 13:05:14 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:05:14 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286865883 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 2 |
members(4): |
0: 68c69353-09a8-11eb-9bc1-3376b5cfc3ea, db1-c2m4p |
1: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf, db1-c2m2p |
2: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
3: f7b801a1-0d43-11eb-a84f-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:05:14 1 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865814 |
2020-10-13 13:05:14 1 [Note] WSREP: Min available from gcache for CC from group: 3285368819 |
2020-10-13 13:05:16 0 [Note] WSREP: Member 3.0 (db1-c2m2d) requested state transfer from '*any*'. Selected 0.0 (db1-c2m4p)(SYNCED) as donor. |
2020-10-13 13:05:16 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:05:16 0 [Note] WSREP: declaring f7b801a1-a84f at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:05:16 0 [Note] WSREP: forgetting 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:16 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:05:19 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:05:19 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,157) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
} joined {
|
} left {
|
} partitioned {
|
68c69353-9bc1,0 |
})
|
2020-10-13 13:05:19 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:19 0 [Note] WSREP: forgetting 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:19 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3 |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 1 (db1-c2m3p) |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 0 (db1-c2m2p) |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 2 (db1-c2m2d) |
2020-10-13 13:05:19 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 66, |
members = 2/3 (joined/total), |
act_id = 3286866015, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:05:19 0 [Note] WSREP: Flow-control interval: [28, 28] |
2020-10-13 13:05:19 1 [Note] WSREP: ####### processing CC 3286866016, local, ordered |
2020-10-13 13:05:19 1 [Note] WSREP: ####### My UUID: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44 |
2020-10-13 13:05:19 1 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:05:19 1 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:05:19 1 [Note] WSREP: ####### Adjusting cert position: 3286866015 -> 3286866016 |
2020-10-13 13:05:19 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:05:19 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866016 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 1 |
members(3): |
0: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf, db1-c2m2p |
1: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
2: f7b801a1-0d43-11eb-a84f-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:05:19 1 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865943 |
2020-10-13 13:05:19 1 [Note] WSREP: Min available from gcache for CC from group: 3285369041 |
2020-10-13 13:05:19 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:20 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a84f (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 0 |
2020-10-13 13:05:21 0 [Note] WSREP: cleaning up 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:40 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') connection established to f7b801a1-a850 tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:40 0 [Note] WSREP: remote endpoint tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 changed identity f7b801a1-0d43-11eb-a84f-878d2f0020e7 -> f7b801a1-0d43-11eb-a850-878d2f0020e7 |
2020-10-13 13:05:43 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:05:49 0 [Note] WSREP: evs::proto(c7ca3602-9def, OPERATIONAL, view_id(REG,b51bd9d6-8d21,157)) suspecting node: f7b801a1-a84f |
2020-10-13 13:05:49 0 [Note] WSREP: evs::proto(c7ca3602-9def, OPERATIONAL, view_id(REG,b51bd9d6-8d21,157)) suspected node without join message, declaring inactive |
2020-10-13 13:05:50 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:05:50 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:05:50 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,158) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a84f,0 |
})
|
2020-10-13 13:05:50 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:50 0 [Note] WSREP: forgetting f7b801a1-a850 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567) |
2020-10-13 13:05:51 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') connection established to f7b801a1-a850 tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:51 0 [Warning] WSREP: discarding established (time wait) f7b801a1-a850 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567) |
2020-10-13 13:05:53 0 [Note] WSREP: cleaning up f7b801a1-a850 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567) |
2020-10-13 13:05:53 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:53 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') connection established to f7b801a1-a850 tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:56 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:06:21 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:06:21 0 [Note] WSREP: declaring f7b801a1-a850 at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:06:21 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:06:21 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,159) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a850,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:06:21 0 [Note] WSREP: save pc into disk |
2020-10-13 13:06:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 0dda9600-0d44-11eb-aaaf-3b6a45970173 |
2020-10-13 13:06:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3 |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 0 (db1-c2m2p) |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 2 (db1-c2m2d) |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 1 (db1-c2m3p) |
2020-10-13 13:06:21 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 67, |
members = 2/3 (joined/total), |
act_id = 3286866099, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:06:21 0 [Note] WSREP: Flow-control interval: [28, 28] |
2020-10-13 13:06:21 1 [Note] WSREP: ####### processing CC 3286866100, local, ordered |
2020-10-13 13:06:21 1 [Note] WSREP: ####### My UUID: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44 |
2020-10-13 13:06:21 1 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:06:21 1 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:06:21 1 [Note] WSREP: ####### Adjusting cert position: 3286866099 -> 3286866100 |
2020-10-13 13:06:29 0 [Warning] WSREP: Failed to report last committed 77d86ba4-c581-11e8-b013-173e23a1017d:3286866016, -110 (Connection timed out) |
2020-10-13 13:06:29 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:06:29 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866100 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 1 |
members(3): |
0: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf, db1-c2m2p |
1: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
2: f7b801a1-0d43-11eb-a850-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:06:29 1 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865943 |
2020-10-13 13:06:29 1 [Note] WSREP: Min available from gcache for CC from group: 3285369215 |
2020-10-13 13:06:29 0 [Note] WSREP: Member 2.0 (db1-c2m2d) requested state transfer from '*any*'. Selected 0.0 (db1-c2m2p)(SYNCED) as donor. |
2020-10-13 13:06:29 0 [Note] WSREP: declaring f7b801a1-a850 at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:06:29 0 [Note] WSREP: forgetting b51bd9d6-8d21 (tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567) |
2020-10-13 13:06:29 0 [Note] WSREP: Node c7ca3602-9def state prim |
2020-10-13 13:06:29 0 [Note] WSREP: view(view_id(PRIM,c7ca3602-9def,160) memb { |
c7ca3602-9def,0 |
f7b801a1-a850,0 |
} joined {
|
} left {
|
} partitioned {
|
b51bd9d6-8d21,0 |
})
|
2020-10-13 13:06:29 0 [Note] WSREP: save pc into disk |
2020-10-13 13:06:29 0 [Note] WSREP: forgetting b51bd9d6-8d21 (tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567) |
2020-10-13 13:06:29 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 |
2020-10-13 13:06:29 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 253a3cb3-0d44-11eb-bf81-b6c941f0a1d5 |
2020-10-13 13:06:29 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 253a3cb3-0d44-11eb-bf81-b6c941f0a1d5 |
2020-10-13 13:06:29 0 [Note] WSREP: STATE EXCHANGE: got state msg: 253a3cb3-0d44-11eb-bf81-b6c941f0a1d5 from 0 (db1-c2m3p) |
2020-10-13 13:06:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 253a3cb3-0d44-11eb-bf81-b6c941f0a1d5 from 1 (db1-c2m2d) |
2020-10-13 13:06:30 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 68, |
members = 1/2 (joined/total), |
act_id = 3286866279, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:06:30 0 [Note] WSREP: Flow-control interval: [23, 23] |
2020-10-13 13:06:30 1 [Note] WSREP: ####### processing CC 3286866280, local, ordered |
2020-10-13 13:06:30 1 [Note] WSREP: ####### My UUID: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44 |
2020-10-13 13:06:30 1 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:06:30 1 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:06:30 1 [Note] WSREP: ####### Adjusting cert position: 3286866279 -> 3286866280 |
2020-10-13 13:06:30 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:06:30 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866280 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(2): |
0: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
1: f7b801a1-0d43-11eb-a850-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:06:30 1 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865943 |
2020-10-13 13:06:30 1 [Note] WSREP: Min available from gcache for CC from group: 3285369349 |
2020-10-13 13:06:31 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:06:32 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a850 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 0 |
2020-10-13 13:06:35 0 [Note] WSREP: cleaning up b51bd9d6-8d21 (tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567) |
2020-10-13 13:06:46 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') connection established to f7b801a1-a851 tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:06:46 0 [Note] WSREP: remote endpoint tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 changed identity f7b801a1-0d43-11eb-a850-878d2f0020e7 -> f7b801a1-0d43-11eb-a851-878d2f0020e7 |
2020-10-13 13:06:50 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:07:00 0 [Note] WSREP: evs::proto(c7ca3602-9def, OPERATIONAL, view_id(REG,c7ca3602-9def,160)) suspecting node: f7b801a1-a850 |
2020-10-13 13:07:00 0 [Note] WSREP: evs::proto(c7ca3602-9def, OPERATIONAL, view_id(REG,c7ca3602-9def,160)) suspected node without join message, declaring inactive |
2020-10-13 13:07:01 0 [Note] WSREP: view(view_id(NON_PRIM,c7ca3602-9def,160) memb { |
c7ca3602-9def,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a850,0 |
})
|
2020-10-13 13:07:01 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2020-10-13 13:07:01 0 [Note] WSREP: view(view_id(NON_PRIM,c7ca3602-9def,161) memb { |
c7ca3602-9def,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a850,0 |
})
|
2020-10-13 13:07:01 0 [Note] WSREP: Flow-control interval: [16, 16] |
2020-10-13 13:07:01 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:07:01 0 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 3286866280) |
2020-10-13 13:07:01 0 [Warning] WSREP: FLOW message from member 0 in non-primary configuration. Ignored. |
2020-10-13 13:07:01 0 [Warning] WSREP: Last Applied Action message in non-primary configuration from member 0 |
2020-10-13 13:07:01 0 [Warning] WSREP: FLOW message from member 0 in non-primary configuration. Ignored. |
2020-10-13 13:07:01 0 [Warning] WSREP: FLOW message from member 0 in non-primary configuration. Ignored. |
2020-10-13 13:07:01 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866280 |
status: non-primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(1): |
0: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
=================================================
|
2020-10-13 13:07:01 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2020-10-13 13:07:01 1 [Note] WSREP: Non-primary view |
2020-10-13 13:07:01 0 [Note] WSREP: Flow-control interval: [16, 16] |
2020-10-13 13:07:01 1 [Note] WSREP: Server status change synced -> connected |
2020-10-13 13:07:01 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:07:01 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866280 |
status: non-primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(1): |
0: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
=================================================
|
2020-10-13 13:07:01 1 [Note] WSREP: Non-primary view |
2020-10-13 13:07:01 1 [Note] WSREP: Server status change connected -> connected |
2020-10-13 13:07:32 0 [Note] WSREP: declaring f7b801a1-a851 at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:07:32 0 [Note] WSREP: view(view_id(NON_PRIM,c7ca3602-9def,162) memb { |
c7ca3602-9def,0 |
f7b801a1-a851,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a850,0 |
})
|
2020-10-13 13:07:32 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 2 |
2020-10-13 13:07:32 0 [Note] WSREP: Flow-control interval: [23, 23] |
2020-10-13 13:07:32 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:07:32 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866280 |
status: non-primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(2): |
0: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
1: f7b801a1-0d43-11eb-a851-878d2f0020e7, unspecified |
=================================================
|
2020-10-13 13:07:32 1 [Note] WSREP: Non-primary view |
2020-10-13 13:07:32 1 [Note] WSREP: Server status change connected -> connected |
2020-10-13 13:16:13 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:16:14 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 0 |
2020-10-13 13:16:41 0 [Note] WSREP: evs::proto(c7ca3602-9def, OPERATIONAL, view_id(REG,c7ca3602-9def,162)) suspecting node: f7b801a1-a851 |
2020-10-13 13:16:41 0 [Note] WSREP: evs::proto(c7ca3602-9def, OPERATIONAL, view_id(REG,c7ca3602-9def,162)) suspected node without join message, declaring inactive |
2020-10-13 13:16:42 0 [Note] WSREP: view(view_id(NON_PRIM,c7ca3602-9def,162) memb { |
c7ca3602-9def,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a850,0 |
f7b801a1-a851,0 |
})
|
2020-10-13 13:16:42 0 [Note] WSREP: view(view_id(NON_PRIM,c7ca3602-9def,163) memb { |
c7ca3602-9def,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a850,0 |
f7b801a1-a851,0 |
})
|
2020-10-13 13:16:42 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2020-10-13 13:16:42 0 [Note] WSREP: Flow-control interval: [16, 16] |
2020-10-13 13:16:42 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:16:42 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2020-10-13 13:16:42 0 [Note] WSREP: Flow-control interval: [16, 16] |
2020-10-13 13:16:42 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:16:42 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866280 |
status: non-primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(1): |
0: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
=================================================
|
2020-10-13 13:16:42 1 [Note] WSREP: Non-primary view |
2020-10-13 13:16:42 1 [Note] WSREP: Server status change connected -> connected |
2020-10-13 13:16:42 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866280 |
status: non-primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(1): |
0: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
=================================================
|
2020-10-13 13:16:42 1 [Note] WSREP: Non-primary view |
2020-10-13 13:16:42 1 [Note] WSREP: Server status change connected -> connected |
2020-10-13 13:16:59 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 30 |
2020-10-13 13:17:44 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 60 |
2020-10-13 13:18:29 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 90 |
2020-10-13 13:19:14 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 120 |
2020-10-13 13:19:59 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 150 |
2020-10-13 13:20:44 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 180 |
2020-10-13 13:21:29 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 210 |
2020-10-13 13:22:14 0 [Note] WSREP: (c7ca3602-9def, 'tcp://[::]:4567') reconnecting to f7b801a1-a851 (tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567), attempt 240 |
2020-10-13 13:22:24 0 [Note] /usr/sbin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown |
2020-10-13 13:22:24 0 [Note] WSREP: Shutdown replication |
2020-10-13 13:22:24 0 [Note] WSREP: Server status change connected -> disconnecting |
2020-10-13 13:22:24 0 [Note] WSREP: Closing send monitor... |
2020-10-13 13:22:24 0 [Note] WSREP: Closed send monitor. |
2020-10-13 13:22:24 0 [Note] WSREP: gcomm: terminating thread |
2020-10-13 13:22:24 0 [Note] WSREP: gcomm: joining thread |
2020-10-13 13:22:24 0 [Note] WSREP: gcomm: closing backend |
2020-10-13 13:22:24 0 [Note] WSREP: PC protocol downgrade 1 -> 0 |
2020-10-13 13:22:24 0 [Note] WSREP: view((empty)) |
2020-10-13 13:22:24 0 [Note] WSREP: gcomm: closed |
2020-10-13 13:22:24 0 [Note] WSREP: New SELF-LEAVE. |
2020-10-13 13:22:24 0 [Note] WSREP: Flow-control interval: [0, 0] |
2020-10-13 13:22:24 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. |
2020-10-13 13:22:24 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 3286866280) |
2020-10-13 13:22:24 0 [Note] WSREP: RECV thread exiting 0: Success |
2020-10-13 13:22:24 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866280 |
status: non-primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: yes |
own_index: -1 |
members(0): |
=================================================
|
2020-10-13 13:22:24 1 [Note] WSREP: Non-primary view |
2020-10-13 13:22:24 1 [Note] WSREP: Server status change disconnecting -> disconnected |
2020-10-13 13:22:24 0 [Note] WSREP: recv_thread() joined. |
2020-10-13 13:22:24 0 [Note] WSREP: Closing replication queue. |
2020-10-13 13:22:24 0 [Note] WSREP: Closing slave action queue. |
2020-10-13 13:22:24 1 [Note] WSREP: Applier thread exiting ret: 0 thd: 1 |
2020-10-13 13:22:24 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 20762 |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 20679 |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 20676 |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 19373 |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 19372 |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 15229 |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 12005 |
2020-10-13 13:22:24 0 [Note] WSREP: killing local connection: 32 |
2020-10-13 13:22:25 2 [Note] WSREP: rollbacker thread exiting 2 |
2020-10-13 13:22:25 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) |
2020-10-13 13:22:25 0 [Note] Event Scheduler: Purging the queue. 0 events |
2020-10-13 13:22:25 0 [Note] InnoDB: FTS optimize thread exiting. |
2020-10-13 13:22:25 4 [Note] InnoDB: to purge 5 transactions |
2020-10-13 13:22:25 0 [Note] WSREP: dtor state: CLOSED |
2020-10-13 13:22:25 0 [Note] WSREP: MemPool(TrxHandleSlave): hit ratio: 0.998545, misses: 4185, in use: 335, in pool: 1715 |
2020-10-13 13:22:25 0 [Note] WSREP: mon: entered 2876225 oooe fraction 0 oool fraction 0 |
2020-10-13 13:22:25 0 [Note] WSREP: mon: entered 2876225 oooe fraction 0 oool fraction 0 |
2020-10-13 13:22:25 0 [Note] WSREP: mon: entered 2917930 oooe fraction 0 oool fraction 3.42709e-07 |
2020-10-13 13:22:25 0 [Note] WSREP: cert index usage at exit 318 |
2020-10-13 13:22:25 0 [Note] WSREP: cert trx map usage at exit 335 |
2020-10-13 13:22:25 0 [Note] WSREP: deps set usage at exit 0 |
2020-10-13 13:22:25 0 [Note] WSREP: avg deps dist 33.5026 |
2020-10-13 13:22:25 0 [Note] WSREP: avg cert interval 51455.4 |
2020-10-13 13:22:25 0 [Note] WSREP: cert index size 318 |
2020-10-13 13:22:25 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:22:25 0 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0 |
2020-10-13 13:22:25 0 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 |
2020-10-13 13:22:25 0 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 3286866280) |
2020-10-13 13:22:25 0 [Note] WSREP: Flushing memory map to disk... |
2020-10-13 13:22:25 0 [Note] InnoDB: Starting shutdown... |
2020-10-13 13:22:25 0 [Note] InnoDB: Dumping buffer pool(s) to /opt/db1/var/lib/mysql/ib_buffer_pool |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 0, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 1, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 2, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 3, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 4, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 5, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 6, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Instance 7, restricted to 8023 pages due to innodb_buf_pool_dump_pct=25 |
2020-10-13 13:22:25 0 [Note] InnoDB: Buffer pool(s) dump completed at 201013 13:22:25 |
2020-10-13 13:22:27 0 [Note] InnoDB: Shutdown completed; log sequence number 13045936597626; transaction id 24834143897 |
2020-10-13 13:22:27 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" |
2020-10-13 13:22:27 0 [Note] /usr/sbin/mysqld: Shutdown complete |
|
===== Node 3 log =====
2020-10-13 13:05:13 0 [Note] WSREP: (68c69353-9bc1, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:14 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring f7b801a1-a84f at tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: (68c69353-9bc1, 'tcp://[::]:4567') connection established to f7b801a1-a84f tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4567 |
2020-10-13 13:05:14 0 [Note] WSREP: Node 68c69353-9bc1 state prim |
2020-10-13 13:05:14 0 [Note] WSREP: view(view_id(PRIM,68c69353-9bc1,156) memb { |
68c69353-9bc1,0 |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:05:14 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:14 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 4 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 0 (db1-c2m4p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 1 (db1-c2m2p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 2 (db1-c2m3p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 3 (db1-c2m2d) |
2020-10-13 13:05:14 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 65, |
members = 3/4 (joined/total), |
act_id = 3286865882, |
last_appl. = 3286865813, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:05:14 0 [Note] WSREP: Flow-control interval: [32, 32] |
2020-10-13 13:05:14 1 [Note] WSREP: ####### processing CC 3286865883, local, ordered |
2020-10-13 13:05:14 1 [Note] WSREP: ####### My UUID: 68c69353-09a8-11eb-9bc1-3376b5cfc3ea |
2020-10-13 13:05:14 1 [Note] WSREP: Skipping cert index reset |
2020-10-13 13:05:14 1 [Note] WSREP: REPL Protocols: 10 (5) |
2020-10-13 13:05:14 1 [Note] WSREP: ####### Adjusting cert position: 3286865882 -> 3286865883 |
2020-10-13 13:05:14 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:05:14 1 [Note] WSREP: ================================================ |
View:
|
id: 77d86ba4-c581-11e8-b013-173e23a1017d:3286865883 |
status: primary
|
protocol_version: 4 |
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no |
own_index: 0 |
members(4): |
0: 68c69353-09a8-11eb-9bc1-3376b5cfc3ea, db1-c2m4p |
1: b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf, db1-c2m2p |
2: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, db1-c2m3p |
3: f7b801a1-0d43-11eb-a84f-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:05:14 1 [Note] WSREP: Lowest cert index boundary for CC from group: 3286865814 |
2020-10-13 13:05:14 1 [Note] WSREP: Min available from gcache for CC from group: 3285368852 |
2020-10-13 13:05:16 0 [Note] WSREP: Member 3.0 (db1-c2m2d) requested state transfer from '*any*'. Selected 0.0 (db1-c2m4p)(SYNCED) as donor. |
2020-10-13 13:05:16 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 3286866009) |
2020-10-13 13:05:16 1 [Note] WSREP: Detected STR version: 1, req_len: 163, req: STRv1 |
2020-10-13 13:05:16 1 [Note] WSREP: IST request: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340-3286865883|tcp://db1-c2m2d.os.acsc2012:4568 |
2020-10-13 13:05:16 1 [Note] WSREP: IST first seqno 2893434341 not found from cache, falling back to SST |
2020-10-13 13:05:16 1 [Note] WSREP: Cert index preload: 3286865814 -> 3286865883 |
2020-10-13 13:05:16 1 [ERROR] WSREP: Failed to process action STATE_REQUEST, g: 3286866009, l: 20886837, ptr: 0x7f041096dac8, size: 163: IST sender, failed to connect 'tcp://db1-c2m2d.os.acsc2012:4568': connect: Connection refused: 111 (Connection refused) |
at galera/src/ist.cpp:Sender():735 |
2020-10-13 13:05:16 1 [Note] WSREP: Closing send monitor... |
2020-10-13 13:05:16 1 [Note] WSREP: Closed send monitor. |
2020-10-13 13:05:16 1 [Note] WSREP: gcomm: terminating thread |
2020-10-13 13:05:16 1 [Note] WSREP: gcomm: joining thread |
2020-10-13 13:05:16 1 [Note] WSREP: gcomm: closing backend |
2020-10-13 13:05:16 1 [Note] WSREP: view(view_id(NON_PRIM,68c69353-9bc1,156) memb { |
68c69353-9bc1,0 |
} joined {
|
} left {
|
} partitioned {
|
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
})
|
2020-10-13 13:05:16 1 [Note] WSREP: PC protocol downgrade 1 -> 0 |
2020-10-13 13:05:16 1 [Note] WSREP: view((empty)) |
2020-10-13 13:05:16 1 [Note] WSREP: gcomm: closed |
2020-10-13 13:05:16 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 |
2020-10-13 13:05:16 0 [Note] WSREP: Flow-control interval: [16, 16] |
2020-10-13 13:05:16 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:05:16 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 3286866011) |
2020-10-13 13:05:16 0 [Note] WSREP: New SELF-LEAVE. |
2020-10-13 13:05:16 0 [Note] WSREP: Flow-control interval: [0, 0] |
2020-10-13 13:05:16 0 [Note] WSREP: Received SELF-LEAVE. Closing connection. |
2020-10-13 13:05:16 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 3286866011) |
2020-10-13 13:05:16 0 [Note] WSREP: RECV thread exiting 0: Success |
2020-10-13 13:05:16 1 [Note] WSREP: recv_thread() joined. |
2020-10-13 13:05:16 1 [Note] WSREP: Closing replication queue. |
2020-10-13 13:05:16 1 [Note] WSREP: Closing slave action queue. |
2020-10-13 13:05:16 1 [Note] WSREP: ================================================ |
View:
|
id: 00000000-0000-0000-0000-000000000000:-1 |
status: non-primary
|
protocol_version: -1 |
capabilities:
|
final: yes |
own_index: -1 |
members(0): |
=================================================
|
2020-10-13 13:05:16 1 [Note] WSREP: Non-primary view |
2020-10-13 13:05:16 1 [Note] WSREP: Server status change synced -> disconnected |
2020-10-13 13:05:16 1 [Note] WSREP: Applier thread exiting ret: 6 thd: 1 |
2020-10-13 13:05:16 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) |
2020-10-13 13:22:29 0 [Note] /usr/sbin/mysqld (initiated by: root[root] @ localhost []): Normal shutdown |
2020-10-13 13:22:29 0 [Note] WSREP: Shutdown replication |
2020-10-13 13:22:29 0 [Note] WSREP: killing local connection: 43850 |
2020-10-13 13:22:29 0 [Note] WSREP: killing local connection: 43848 |
2020-10-13 13:22:29 0 [Note] WSREP: killing local connection: 42948 |
2020-10-13 13:22:29 0 [Note] WSREP: killing local connection: 38814 |
2020-10-13 13:22:30 2 [Note] WSREP: rollbacker thread exiting 2 |
2020-10-13 13:22:30 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) |
2020-10-13 13:22:30 0 [Note] Event Scheduler: Purging the queue. 0 events |
2020-10-13 13:22:30 0 [Note] InnoDB: FTS optimize thread exiting. |
2020-10-13 13:22:30 4 [Note] InnoDB: to purge 40 transactions |
2020-10-13 13:22:30 0 [Note] WSREP: dtor state: CLOSED |
2020-10-13 13:22:30 0 [Note] WSREP: MemPool(TrxHandleSlave): hit ratio: 0.999594, misses: 8359, in use: 67, in pool: 1983 |
2020-10-13 13:22:30 0 [Note] WSREP: mon: entered 20573805 oooe fraction 0 oool fraction 0 |
2020-10-13 13:22:30 0 [Note] WSREP: mon: entered 20573805 oooe fraction 0 oool fraction 0 |
2020-10-13 13:22:30 0 [Note] WSREP: mon: entered 20886836 oooe fraction 0 oool fraction 4.7877e-08 |
2020-10-13 13:22:30 0 [Note] WSREP: cert index usage at exit 90 |
2020-10-13 13:22:30 0 [Note] WSREP: cert trx map usage at exit 67 |
2020-10-13 13:22:30 0 [Note] WSREP: deps set usage at exit 0 |
2020-10-13 13:22:30 0 [Note] WSREP: avg deps dist 32.3305 |
2020-10-13 13:22:30 0 [Note] WSREP: avg cert interval 1.3031e+06 |
2020-10-13 13:22:30 0 [Note] WSREP: cert index size 90 |
2020-10-13 13:22:30 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:22:30 0 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0 |
2020-10-13 13:22:30 0 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 |
2020-10-13 13:22:30 0 [Warning] WSREP: Waiting for 4 items to be fetched. |
2020-10-13 13:23:48 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866009 |
2020-10-13 13:23:48 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so' |
2020-10-13 13:23:48 0 [Note] WSREP: wsrep_load(): Galera 26.4.5(rb3764ab6) by Codership Oy <info@codership.com> loaded successfully. |
2020-10-13 13:23:48 0 [Note] WSREP: CRC-32C: using hardware acceleration. |
2020-10-13 13:23:48 0 [Note] WSREP: Found saved state: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866009, safe_to_bootstrap: 0 |
2020-10-13 13:23:48 0 [Note] WSREP: GCache DEBUG: opened preamble: |
Version: 2 |
UUID: 77d86ba4-c581-11e8-b013-173e23a1017d
|
Seqno: -1 - -1 |
Offset: -1 |
Synced: 0 |
2020-10-13 13:23:48 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 77d86ba4-c581-11e8-b013-173e23a1017d, offset: -1 |
2020-10-13 13:23:48 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete. |
2020-10-13 13:23:50 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete. |
2020-10-13 13:23:50 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 3285369036-3286866009 |
2020-10-13 13:23:50 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/2147481200 bytes) complete. |
2020-10-13 13:23:50 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (2147481200/2147481200 bytes) complete. |
2020-10-13 13:23:50 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 0/1496974 locked buffers |
2020-10-13 13:23:50 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 2448/2147483648 |
2020-10-13 13:23:50 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = db1-c2m4p.os; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.keepalive_period = PT3S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /opt/db1/tmp/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://[: |
2020-10-13 13:23:50 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:23:50 0 [Note] WSREP: ####### Assign initial position for certification: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866009, protocol version: -1 |
2020-10-13 13:23:50 0 [Note] WSREP: Start replication |
2020-10-13 13:23:50 0 [Note] WSREP: Connecting with bootstrap option: 0 |
2020-10-13 13:23:50 0 [Note] WSREP: Setting GCS initial position to 77d86ba4-c581-11e8-b013-173e23a1017d:3286866009 |
2020-10-13 13:23:50 0 [Note] WSREP: protonet asio version 0 |
2020-10-13 13:23:50 0 [Note] WSREP: Using CRC-32C for message checksums. |
2020-10-13 13:23:50 0 [Note] WSREP: backend: asio |
2020-10-13 13:23:50 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 |
2020-10-13 13:23:50 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) |
2020-10-13 13:23:50 0 [Note] WSREP: restore pc from disk failed |
2020-10-13 13:23:50 0 [Note] WSREP: GMCast version 0 |
2020-10-13 13:23:50 0 [Note] WSREP: (916e5530-991e, 'tcp://[::]:4567') listening at tcp://[::]:4567 |
2020-10-13 13:23:50 0 [Note] WSREP: (916e5530-991e, 'tcp://[::]:4567') multicast: , ttl: 1 |
2020-10-13 13:23:50 0 [Note] WSREP: EVS version 1 |
2020-10-13 13:23:50 0 [Note] WSREP: gcomm: connecting to group 'db1-c2mXp', peer 'db1-c2m4p.os.abv1:,db1-c2m3p.os.abv1:,db1-c2m2p.os.abv1:' |
2020-10-13 13:23:50 0 [Note] WSREP: (916e5530-991e, 'tcp://[::]:4567') Found matching local endpoint for a connection, blacklisting address tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567 |
2020-10-13 13:23:53 0 [Note] WSREP: EVS version upgrade 0 -> 1 |
2020-10-13 13:23:53 0 [Note] WSREP: PC protocol upgrade 0 -> 1 |
2020-10-13 13:23:53 0 [Warning] WSREP: no nodes coming from prim view, prim not possible |
2020-10-13 13:23:53 0 [Note] WSREP: view(view_id(NON_PRIM,916e5530-991e,1) memb { |
916e5530-991e,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:23:53 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50271S), skipping check |
2020-10-13 13:24:23 0 [Note] WSREP: PC protocol downgrade 1 -> 0 |
2020-10-13 13:24:23 0 [Note] WSREP: view((empty)) |
2020-10-13 13:24:23 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) |
at gcomm/src/pc.cpp:connect():160 |
2020-10-13 13:24:23 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out) |
2020-10-13 13:24:23 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1632: Failed to open channel 'db1-c2mXp' at 'gcomm://db1-c2m4p.os.abv1,db1-c2m3p.os.abv1,db1-c2m2p.os.abv1': -110 (Connection timed out) |
2020-10-13 13:24:23 0 [ERROR] WSREP: gcs connect failed: Connection timed out |
2020-10-13 13:24:23 0 [ERROR] WSREP: wsrep::connect(gcomm://db1-c2m4p.os.abv1,db1-c2m3p.os.abv1,db1-c2m2p.os.abv1) failed: 7 |
2020-10-13 13:24:23 0 [ERROR] Aborting |
|
===== Node 4 log =====
1602587107 2020-10-13 13:05:07 /bin/bash [381/940/940] begin... |
2020-10-13 13:05:12 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340 |
2020-10-13 13:05:12 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so' |
2020-10-13 13:05:12 0 [Note] WSREP: wsrep_load(): Galera 26.4.5(rb3764ab6) by Codership Oy <info@codership.com> loaded successfully. |
2020-10-13 13:05:12 0 [Note] WSREP: CRC-32C: using hardware acceleration. |
2020-10-13 13:05:12 0 [Note] WSREP: Found saved state: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340, safe_to_bootstrap: 0 |
2020-10-13 13:05:12 0 [Note] WSREP: GCache DEBUG: opened preamble: |
Version: 2 |
UUID: 77d86ba4-c581-11e8-b013-173e23a1017d
|
Seqno: 2892554740 - 2893434340 |
Offset: 1280 |
Synced: 1 |
2020-10-13 13:05:12 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 77d86ba4-c581-11e8-b013-173e23a1017d, offset: 1280 |
2020-10-13 13:05:12 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete. |
2020-10-13 13:05:13 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete. |
2020-10-13 13:05:13 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 2892554740-2893434340 |
2020-10-13 13:05:13 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/2063254672 bytes) complete. |
2020-10-13 13:05:13 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (2063254672/2063254672 bytes) complete. |
2020-10-13 13:05:13 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 255/879856 locked buffers |
2020-10-13 13:05:13 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 84281592/2147483648 |
2020-10-13 13:05:13 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = db1-c2m2d.os.acsc2012; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.keepalive_period = PT3S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /opt/db1/tmp/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = |
2020-10-13 13:05:13 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:05:13 0 [Note] WSREP: ####### Assign initial position for certification: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340, protocol version: -1 |
2020-10-13 13:05:13 0 [Note] WSREP: Start replication |
2020-10-13 13:05:13 0 [Note] WSREP: Connecting with bootstrap option: 0 |
2020-10-13 13:05:13 0 [Note] WSREP: Setting GCS initial position to 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340 |
2020-10-13 13:05:13 0 [Note] WSREP: protonet asio version 0 |
2020-10-13 13:05:13 0 [Note] WSREP: Using CRC-32C for message checksums. |
2020-10-13 13:05:13 0 [Note] WSREP: backend: asio |
2020-10-13 13:05:13 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 |
2020-10-13 13:05:13 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) |
2020-10-13 13:05:13 0 [Note] WSREP: restore pc from disk failed |
2020-10-13 13:05:13 0 [Note] WSREP: GMCast version 0 |
2020-10-13 13:05:13 0 [Note] WSREP: (f7b801a1-a84f, 'tcp://[::]:4567') listening at tcp://[::]:4567 |
2020-10-13 13:05:13 0 [Note] WSREP: (f7b801a1-a84f, 'tcp://[::]:4567') multicast: , ttl: 1 |
2020-10-13 13:05:13 0 [Note] WSREP: EVS version 1 |
2020-10-13 13:05:13 0 [Note] WSREP: gcomm: connecting to group 'db1-c2mXp', peer 'db1-c2m3p.os:' |
2020-10-13 13:05:13 0 [Note] WSREP: (f7b801a1-a84f, 'tcp://[::]:4567') connection established to c7ca3602-9def tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 |
2020-10-13 13:05:13 0 [Note] WSREP: (f7b801a1-a84f, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567 tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 |
2020-10-13 13:05:14 0 [Note] WSREP: (f7b801a1-a84f, 'tcp://[::]:4567') connection established to b51bd9d6-8d21 tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 |
2020-10-13 13:05:14 0 [Note] WSREP: EVS version upgrade 0 -> 1 |
2020-10-13 13:05:14 0 [Note] WSREP: declaring 68c69353-9bc1 at tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:05:14 0 [Note] WSREP: PC protocol upgrade 0 -> 1 |
2020-10-13 13:05:14 0 [Note] WSREP: (f7b801a1-a84f, 'tcp://[::]:4567') connection established to 68c69353-9bc1 tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567 |
2020-10-13 13:05:14 0 [Note] WSREP: Node 68c69353-9bc1 state prim |
2020-10-13 13:05:14 0 [Note] WSREP: view(view_id(PRIM,68c69353-9bc1,156) memb { |
68c69353-9bc1,0 |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:05:14 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:14 0 [Note] WSREP: gcomm: connected |
2020-10-13 13:05:14 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 |
2020-10-13 13:05:14 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) |
2020-10-13 13:05:14 0 [Note] WSREP: Opened channel 'db1-c2mXp' |
2020-10-13 13:05:14 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 4 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: sent state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 0 (db1-c2m4p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 1 (db1-c2m2p) |
2020-10-13 13:05:14 1 [Note] WSREP: Starting rollbacker thread 1 |
2020-10-13 13:05:14 2 [Note] WSREP: Starting applier thread 2 |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 2 (db1-c2m3p) |
2020-10-13 13:05:14 0 [Note] WSREP: STATE EXCHANGE: got state msg: f80d5f91-0d43-11eb-8f13-1b5f9b2a5ec1 from 3 (db1-c2m2d) |
2020-10-13 13:05:14 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 65, |
members = 3/4 (joined/total), |
act_id = 3286865882, |
last_appl. = 3286865813, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:05:14 0 [Note] WSREP: Flow-control interval: [32, 32] |
2020-10-13 13:05:14 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 3286865883) |
2020-10-13 13:05:14 2 [Note] WSREP: ####### processing CC 3286865883, local, ordered |
2020-10-13 13:05:14 2 [Note] WSREP: Process first view: 77d86ba4-c581-11e8-b013-173e23a1017d my uuid: f7b801a1-0d43-11eb-a84f-878d2f0020e7 |
2020-10-13 13:05:14 2 [Note] WSREP: Server db1-c2m2d connected to cluster at position 77d86ba4-c581-11e8-b013-173e23a1017d:3286865883 with ID f7b801a1-0d43-11eb-a84f-878d2f0020e7 |
2020-10-13 13:05:14 2 [Note] WSREP: Server status change disconnected -> connected |
2020-10-13 13:05:15 2 [Note] WSREP: ####### My UUID: f7b801a1-0d43-11eb-a84f-878d2f0020e7 |
2020-10-13 13:05:15 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes |
2020-10-13 13:05:15 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:05:15 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1 |
2020-10-13 13:05:15 2 [Note] WSREP: State transfer required: |
Group state: 77d86ba4-c581-11e8-b013-173e23a1017d:3286865883 |
Local state: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340 |
2020-10-13 13:05:15 2 [Note] WSREP: Server status change connected -> joiner |
2020-10-13 13:05:15 0 [Note] WSREP: Joiner monitor thread started to monitor |
2020-10-13 13:05:15 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'db1-c2m2d.os.acsc2012' --datadir '/var/lib/mysql/' --parent '31744' --binlog 'mysql-bin' --binlog-index 'mysql-bin.index' --mysqld-args --wsrep_start_position=77d86ba4-c581-11e8-b013-173e23a1017d:2893434340' |
WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20201013 13:05:15.742) |
2020-10-13 13:05:16 2 [Note] WSREP: Prepared SST request: mariabackup|db1-c2m2d.os.acsc2012:4444/xtrabackup_sst//1 |
2020-10-13 13:05:16 2 [Note] WSREP: ####### IST uuid:77d86ba4-c581-11e8-b013-173e23a1017d f: 2893434341, l: 3286865883, STRv: 3 |
2020-10-13 13:05:16 2 [Note] WSREP: IST receiver addr using tcp://db1-c2m2d.os.acsc2012:4568 |
2020-10-13 13:05:16 2 [Note] WSREP: Prepared IST receiver for 2893434341-3286865883, listening at: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4568 |
2020-10-13 13:05:16 0 [Note] WSREP: Member 3.0 (db1-c2m2d) requested state transfer from '*any*'. Selected 0.0 (db1-c2m4p)(SYNCED) as donor. |
2020-10-13 13:05:16 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3286866009) |
2020-10-13 13:05:16 2 [Note] WSREP: Requesting state transfer: success, donor: 0 |
2020-10-13 13:05:16 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:05:16 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:05:16 0 [Note] WSREP: forgetting 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:17 0 [Note] WSREP: (f7b801a1-a84f, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:05:19 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:05:19 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,157) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a84f,0 |
} joined {
|
} left {
|
} partitioned {
|
68c69353-9bc1,0 |
})
|
2020-10-13 13:05:19 0 [Note] WSREP: save pc into disk |
2020-10-13 13:05:19 0 [Note] WSREP: forgetting 68c69353-9bc1 (tcp://[****:8400:4:2c3:216:3eff:fe73:266e]:4567) |
2020-10-13 13:05:19 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 1 (db1-c2m3p) |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 0 (db1-c2m2p) |
2020-10-13 13:05:19 0 [Note] WSREP: STATE EXCHANGE: got state msg: fb14cfe3-0d43-11eb-af77-078aeb4bcaf4 from 2 (db1-c2m2d) |
2020-10-13 13:05:19 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 66, |
members = 2/3 (joined/total), |
act_id = 3286866015, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:05:19 0 [Warning] WSREP: Donor 68c69353-09a8-11eb-9bc1-3376b5cfc3ea is no longer in the group. State transfer cannot be completed, need to abort. Aborting... |
2020-10-13 13:05:19 0 [Note] WSREP: /usr/sbin/mysqld: Terminated. |
2020-10-13 13:05:37 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340 |
2020-10-13 13:05:37 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so' |
2020-10-13 13:05:37 0 [Note] WSREP: wsrep_load(): Galera 26.4.5(rb3764ab6) by Codership Oy <info@codership.com> loaded successfully. |
2020-10-13 13:05:37 0 [Note] WSREP: CRC-32C: using hardware acceleration. |
2020-10-13 13:05:37 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0 |
2020-10-13 13:05:37 0 [Note] WSREP: GCache DEBUG: opened preamble: |
Version: 2 |
UUID: 77d86ba4-c581-11e8-b013-173e23a1017d
|
Seqno: -1 - -1 |
Offset: -1 |
Synced: 0 |
2020-10-13 13:05:37 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 77d86ba4-c581-11e8-b013-173e23a1017d, offset: -1 |
2020-10-13 13:05:37 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete. |
2020-10-13 13:05:39 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete. |
2020-10-13 13:05:39 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 2892554740-2893434340 |
2020-10-13 13:05:39 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/2146704032 bytes) complete. |
2020-10-13 13:05:39 0 [Note] WSREP: GCache::RingBuffer unused buffers scan...100.0% (2146704032/2146704032 bytes) complete. |
2020-10-13 13:05:39 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): found 513/880114 locked buffers |
2020-10-13 13:05:39 0 [Note] WSREP: GCache DEBUG: RingBuffer::recover(): free space: 84281592/2147483648 |
2020-10-13 13:05:40 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = db1-c2m2d.os.acsc2012; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.keepalive_period = PT3S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /opt/db1/tmp/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = |
2020-10-13 13:05:40 0 [Note] WSREP: Start replication |
2020-10-13 13:05:40 0 [Note] WSREP: Connecting with bootstrap option: 0 |
2020-10-13 13:05:40 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1 |
2020-10-13 13:05:40 0 [Note] WSREP: protonet asio version 0 |
2020-10-13 13:05:40 0 [Note] WSREP: Using CRC-32C for message checksums. |
2020-10-13 13:05:40 0 [Note] WSREP: backend: asio |
2020-10-13 13:05:40 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 |
2020-10-13 13:05:40 0 [Note] WSREP: restore pc from disk successfully |
2020-10-13 13:05:40 0 [Note] WSREP: GMCast version 0 |
2020-10-13 13:05:40 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') listening at tcp://[::]:4567 |
2020-10-13 13:05:40 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') multicast: , ttl: 1 |
2020-10-13 13:05:40 0 [Note] WSREP: EVS version 1 |
2020-10-13 13:05:40 0 [Note] WSREP: gcomm: connecting to group 'db1-c2mXp', peer 'db1-c2m3p.os:' |
2020-10-13 13:05:40 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') connection established to c7ca3602-9def tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 |
2020-10-13 13:05:40 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 |
2020-10-13 13:05:40 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') connection established to b51bd9d6-8d21 tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 |
2020-10-13 13:05:43 0 [Note] WSREP: gcomm: connected |
2020-10-13 13:05:43 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 |
2020-10-13 13:05:43 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) |
2020-10-13 13:05:43 0 [Note] WSREP: Opened channel 'db1-c2mXp' |
2020-10-13 13:05:43 2 [Note] WSREP: Starting rollbacker thread 2 |
2020-10-13 13:05:43 1 [Note] WSREP: Starting applier thread 1 |
2020-10-13 13:05:43 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.19859S), skipping check |
2020-10-13 13:05:44 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:05:50 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 |
2020-10-13 13:05:51 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') reconnecting to b51bd9d6-8d21 (tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567), attempt 0 |
2020-10-13 13:05:51 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') reconnecting to c7ca3602-9def (tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567), attempt 0 |
2020-10-13 13:05:51 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') connection established to b51bd9d6-8d21 tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 |
2020-10-13 13:05:51 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') connection established to c7ca3602-9def tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 |
2020-10-13 13:05:53 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') reconnecting to b51bd9d6-8d21 (tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567), attempt 0 |
2020-10-13 13:05:53 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') reconnecting to c7ca3602-9def (tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567), attempt 0 |
2020-10-13 13:05:53 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') connection established to b51bd9d6-8d21 tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 |
2020-10-13 13:05:53 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') connection established to c7ca3602-9def tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 |
2020-10-13 13:05:56 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:06:21 0 [Note] WSREP: EVS version upgrade 0 -> 1 |
2020-10-13 13:06:21 0 [Note] WSREP: declaring b51bd9d6-8d21 at tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 stable |
2020-10-13 13:06:21 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:06:21 0 [Note] WSREP: PC protocol upgrade 0 -> 1 |
2020-10-13 13:06:21 0 [Note] WSREP: Node b51bd9d6-8d21 state prim |
2020-10-13 13:06:21 0 [Note] WSREP: view(view_id(PRIM,b51bd9d6-8d21,159) memb { |
b51bd9d6-8d21,0 |
c7ca3602-9def,0 |
f7b801a1-a850,0 |
} joined {
|
} left {
|
} partitioned {
|
})
|
2020-10-13 13:06:21 0 [Note] WSREP: save pc into disk |
2020-10-13 13:06:21 0 [Note] WSREP: clear restored view |
2020-10-13 13:06:21 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Warning] WSREP: FLOW message from member 1 in non-primary configuration. Ignored. |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 0 (db1-c2m2p) |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 2 (db1-c2m2d) |
2020-10-13 13:06:21 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2058e43e-0d44-11eb-8787-1f99ecd795c6 from 1 (db1-c2m3p) |
2020-10-13 13:06:21 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 67, |
members = 2/3 (joined/total), |
act_id = 3286866099, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:06:21 0 [Note] WSREP: Flow-control interval: [28, 28] |
2020-10-13 13:06:21 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 3286866100) |
2020-10-13 13:06:21 1 [Note] WSREP: ####### processing CC 3286866100, local, ordered |
2020-10-13 13:06:21 1 [Note] WSREP: Process first view: 77d86ba4-c581-11e8-b013-173e23a1017d my uuid: f7b801a1-0d43-11eb-a850-878d2f0020e7 |
2020-10-13 13:06:21 1 [Note] WSREP: Server db1-c2m2d connected to cluster at position 77d86ba4-c581-11e8-b013-173e23a1017d:3286866100 with ID f7b801a1-0d43-11eb-a850-878d2f0020e7 |
2020-10-13 13:06:21 1 [Note] WSREP: Server status change disconnected -> connected |
2020-10-13 13:06:22 1 [Note] WSREP: ####### My UUID: f7b801a1-0d43-11eb-a850-878d2f0020e7 |
2020-10-13 13:06:22 1 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes |
2020-10-13 13:06:22 0 [Note] WSREP: Service thread queue flushed. |
2020-10-13 13:06:22 1 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1 |
2020-10-13 13:06:22 1 [Note] WSREP: State transfer required: |
Group state: 77d86ba4-c581-11e8-b013-173e23a1017d:3286866100 |
Local state: 00000000-0000-0000-0000-000000000000:-1 |
2020-10-13 13:06:22 1 [Note] WSREP: Server status change connected -> joiner |
2020-10-13 13:06:22 0 [Note] WSREP: Joiner monitor thread started to monitor |
2020-10-13 13:06:22 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'db1-c2m2d.os.acsc2012' --datadir '/var/lib/mysql/' --parent '32460' --binlog 'mysql-bin' --binlog-index 'mysql-bin.index' --mysqld-args --wsrep_start_position=77d86ba4-c581-11e8-b013-173e23a1017d:2893434340' |
WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20201013 13:06:22.960) |
2020-10-13 13:06:23 1 [Note] WSREP: Prepared SST request: mariabackup|db1-c2m2d.os.acsc2012:4444/xtrabackup_sst//1 |
2020-10-13 13:06:23 1 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 3286866100, STRv: 3 |
2020-10-13 13:06:23 1 [Note] WSREP: IST receiver addr using tcp://db1-c2m2d.os.acsc2012:4568 |
2020-10-13 13:06:23 1 [Note] WSREP: Prepared IST receiver for 0-3286866100, listening at: tcp://[****:1d64:e1cb:2e4:216:3eff:fe7d:6501]:4568 |
2020-10-13 13:06:29 0 [Note] WSREP: Member 2.0 (db1-c2m2d) requested state transfer from '*any*'. Selected 0.0 (db1-c2m2p)(SYNCED) as donor. |
2020-10-13 13:06:29 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3286866166) |
2020-10-13 13:06:29 1 [Note] WSREP: Requesting state transfer: success, donor: 0 |
2020-10-13 13:06:29 1 [Note] WSREP: Resetting GCache seqno map due to different histories. |
2020-10-13 13:06:29 1 [Note] WSREP: GCache history reset: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340 -> 77d86ba4-c581-11e8-b013-173e23a1017d:3286866100 |
2020-10-13 13:06:29 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567 |
2020-10-13 13:06:29 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:06:29 0 [Note] WSREP: forgetting b51bd9d6-8d21 (tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567) |
2020-10-13 13:06:29 0 [Note] WSREP: (f7b801a1-a850, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:06:29 0 [Note] WSREP: Node c7ca3602-9def state prim |
2020-10-13 13:06:29 0 [Note] WSREP: view(view_id(PRIM,c7ca3602-9def,160) memb { |
c7ca3602-9def,0 |
f7b801a1-a850,0 |
} joined {
|
} left {
|
} partitioned {
|
b51bd9d6-8d21,0 |
})
|
2020-10-13 13:06:29 0 [Note] WSREP: save pc into disk |
2020-10-13 13:06:29 0 [Note] WSREP: forgetting b51bd9d6-8d21 (tcp://[****:8400:4:2c3:216:3eff:fe7d:6424]:4567) |
2020-10-13 13:06:30 1 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): discarded 2063201848 bytes |
2020-10-13 13:06:30 1 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): found 1/68 locked buffers |
2020-10-13 13:06:30 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 |
2020-10-13 13:06:30 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. |
2020-10-13 13:06:30 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 253a3cb3-0d44-11eb-bf81-b6c941f0a1d5 |
2020-10-13 13:06:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 253a3cb3-0d44-11eb-bf81-b6c941f0a1d5 from 0 (db1-c2m3p) |
2020-10-13 13:06:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 253a3cb3-0d44-11eb-bf81-b6c941f0a1d5 from 1 (db1-c2m2d) |
2020-10-13 13:06:30 0 [Note] WSREP: Quorum results: |
version = 6, |
component = PRIMARY,
|
conf_id = 68, |
members = 1/2 (joined/total), |
act_id = 3286866279, |
last_appl. = 3286865942, |
protocols = 2/10/4 (gcs/repl/appl), |
vote policy= 0, |
group UUID = 77d86ba4-c581-11e8-b013-173e23a1017d
|
2020-10-13 13:06:30 0 [Warning] WSREP: Donor b51bd9d6-09a7-11eb-8d21-9aa8b7de5bcf is no longer in the group. State transfer cannot be completed, need to abort. Aborting... |
2020-10-13 13:06:30 0 [Note] WSREP: /usr/sbin/mysqld: Terminated. |
2020-10-13 13:06:46 0 [Note] WSREP: Loading provider /usr/lib/libgalera_smm.so initial position: 77d86ba4-c581-11e8-b013-173e23a1017d:2893434340 |
2020-10-13 13:06:46 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so' |
2020-10-13 13:06:46 0 [Note] WSREP: wsrep_load(): Galera 26.4.5(rb3764ab6) by Codership Oy <info@codership.com> loaded successfully. |
2020-10-13 13:06:46 0 [Note] WSREP: CRC-32C: using hardware acceleration. |
2020-10-13 13:06:46 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 0 |
2020-10-13 13:06:46 0 [Note] WSREP: GCache DEBUG: opened preamble: |
Version: 2 |
UUID: 77d86ba4-c581-11e8-b013-173e23a1017d
|
Seqno: -1 - -1 |
Offset: -1 |
Synced: 0 |
2020-10-13 13:06:46 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 77d86ba4-c581-11e8-b013-173e23a1017d, offset: -1 |
2020-10-13 13:06:46 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete. |
2020-10-13 13:06:46 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete. |
2020-10-13 13:06:46 0 [Note] WSREP: Recovering GCache ring buffer: didn't recover any events. |
2020-10-13 13:06:46 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = db1-c2m2d.os.acsc2012; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.keepalive_period = PT3S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /opt/db1/tmp/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = |
2020-10-13 13:06:46 0 [Note] WSREP: Start replication |
2020-10-13 13:06:46 0 [Note] WSREP: Connecting with bootstrap option: 0 |
2020-10-13 13:06:46 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1 |
2020-10-13 13:06:46 0 [Note] WSREP: protonet asio version 0 |
2020-10-13 13:06:46 0 [Note] WSREP: Using CRC-32C for message checksums. |
2020-10-13 13:06:46 0 [Note] WSREP: backend: asio |
2020-10-13 13:06:46 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 |
2020-10-13 13:06:46 0 [Note] WSREP: restore pc from disk successfully |
2020-10-13 13:06:46 0 [Note] WSREP: GMCast version 0 |
2020-10-13 13:06:46 0 [Note] WSREP: (f7b801a1-a851, 'tcp://[::]:4567') listening at tcp://[::]:4567 |
2020-10-13 13:06:46 0 [Note] WSREP: (f7b801a1-a851, 'tcp://[::]:4567') multicast: , ttl: 1 |
2020-10-13 13:06:46 0 [Note] WSREP: EVS version 1 |
2020-10-13 13:06:46 0 [Note] WSREP: gcomm: connecting to group 'db1-c2mXp', peer 'db1-c2m3p.os:' |
2020-10-13 13:06:46 0 [Note] WSREP: (f7b801a1-a851, 'tcp://[::]:4567') connection established to c7ca3602-9def tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 |
2020-10-13 13:06:46 0 [Note] WSREP: (f7b801a1-a851, 'tcp://[::]:4567') turning message relay requesting on, nonlive peers: |
2020-10-13 13:06:47 0 [Note] WSREP: gcomm: connected |
2020-10-13 13:06:47 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 |
2020-10-13 13:06:47 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) |
2020-10-13 13:06:47 0 [Note] WSREP: Opened channel 'db1-c2mXp' |
2020-10-13 13:06:47 1 [Note] WSREP: Starting rollbacker thread 1 |
2020-10-13 13:06:47 2 [Note] WSREP: Starting applier thread 2 |
2020-10-13 13:06:50 0 [Note] WSREP: (f7b801a1-a851, 'tcp://[::]:4567') turning message relay requesting off |
2020-10-13 13:07:32 0 [Note] WSREP: EVS version upgrade 0 -> 1 |
2020-10-13 13:07:32 0 [Note] WSREP: declaring c7ca3602-9def at tcp://[****:8400:4:2c3:216:3eff:fef0:d954]:4567 stable |
2020-10-13 13:07:32 0 [Note] WSREP: PC protocol upgrade 0 -> 1 |
2020-10-13 13:07:32 0 [Note] WSREP: view(view_id(NON_PRIM,c7ca3602-9def,162) memb { |
c7ca3602-9def,0 |
f7b801a1-a851,0 |
} joined {
|
} left {
|
} partitioned {
|
f7b801a1-a850,0 |
})
|
2020-10-13 13:07:32 0 [Warning] WSREP: node uuid: c7ca3602-9def last_prim(type: 3, uuid: c7ca3602-9def) is inconsistent to restored view(type: V_NON_PRIM, uuid: c7ca3602-9def |
2020-10-13 13:07:32 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 2 |
2020-10-13 13:07:32 0 [Note] WSREP: Flow-control interval: [23, 23] |
2020-10-13 13:07:32 0 [Note] WSREP: Received NON-PRIMARY. |
2020-10-13 13:07:32 2 [Note] WSREP: ================================================ |
View:
|
id: 00000000-0000-0000-0000-000000000000:-1 |
status: non-primary
|
protocol_version: -1 |
capabilities:
|
final: no |
own_index: 1 |
members(2): |
0: c7ca3602-0cc5-11eb-9def-3a8a6cbcfe44, unspecified |
1: f7b801a1-0d43-11eb-a851-878d2f0020e7, db1-c2m2d |
=================================================
|
2020-10-13 13:07:32 2 [Note] WSREP: Non-primary view |
2020-10-13 13:07:32 2 [Note] WSREP: Server status change disconnected -> connected |
|
Attachments
Issue Links
- relates to
-
-
- Open
-
Activity
Based on the above comment, I don't think this issue is a blocker for our release. Thus I'll move it down to critical for now.
the problem, it's happened with miss config with iptables and fucked my cluster of production =) (the goal was to refresh a server of dev faster)
The intruder must be already inside the security zone, to get access to cluster replication ports. So the problem can be mitigated by proper firewalling
Security deployment should always use TLS for cluster replication, and in such installation, the intruder should have also had cluster TLS certificate to penetrate
We document that should use SSL. If one doesn't — it's insecure config, nothing we can fix short term
without SSL one can also request SST and get a full copy of the server's data without any authentication
this one is kind of "by design"
10.6 task list has feature requests for hardening cluster node authentication and whitelisting cluster nodes., either one of these features would block this attack vector.