2021-11-26 14:13:04.450 INFO: CRC-32C: using 64-bit x86 acceleration. 2021-11-26 14:13:04.450 INFO: Read config: daemon: 0 name: garb address: gcomm://10.16.103.30,10.16.103.31,10.15.103.30,10.15.103.31 group: trv_acc sst: trivial donor: options: evs.keepalive_period=PT3S;evs.suspect_timeout=PT30S;evs.inactive_timeout=PT1M;evs.install_timeout=PT1M;gcs.fc_limit=20;gcs.fc_factor=0.8;socket.ssl_key=/etc/garbd/aicetrvdba010-key.pem;socket.ssl_cert=/etc/garbd/aicetrvdba010-cert.pem;socket.ssl_ca=/etc/garbd/ca-cert.pem;gmcast.segment=1;socket.ssl_cipher=AES128-SHA256; gcs.fc_limit=9999999; gcs.fc_factor=1.0; gcs.fc_master_slave=yes cfg: log: /var/log/garbd.log 2021-11-26 14:13:04.451 INFO: protonet asio version 0 2021-11-26 14:13:04.451 INFO: Using CRC-32C for message checksums. 2021-11-26 14:13:04.451 INFO: backend: asio 2021-11-26 14:13:04.451 INFO: gcomm thread scheduling priority set to other:0 2021-11-26 14:13:04.452 WARN: access file(./gvwstate.dat) failed(No such file or directory) 2021-11-26 14:13:04.452 INFO: restore pc from disk failed 2021-11-26 14:13:04.452 INFO: GMCast version 0 2021-11-26 14:13:04.452 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2021-11-26 14:13:04.452 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2021-11-26 14:13:04.453 INFO: EVS version 1 2021-11-26 14:13:04.453 INFO: gcomm: connecting to group 'trv_acc', peer '10.16.103.30:,10.16.103.31:,10.15.103.30:,10.15.103.31:' 2021-11-26 14:13:07.455 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 19426 rttvar: 9713 rto: 220000 lost: 0 last_data_recv: 7486194 cwnd: 10 last_queued_since: 2981807939 last_delivered_since: 2981807939 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:07.455 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 18836 rttvar: 9418 rto: 219000 lost: 0 last_data_recv: 7486194 cwnd: 10 last_queued_since: 2982402325 last_delivered_since: 2982402325 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:07.455 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.16.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 16396 rttvar: 8198 rto: 217000 lost: 0 last_data_recv: 7486194 cwnd: 10 last_queued_since: 2984901514 last_delivered_since: 2984901514 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:07.455 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.16.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 16857 rttvar: 8428 rto: 217000 lost: 0 last_data_recv: 7486194 cwnd: 10 last_queued_since: 2984496670 last_delivered_since: 2984496670 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:07.456 INFO: EVS version upgrade 0 -> 1 2021-11-26 14:13:07.456 INFO: PC protocol upgrade 0 -> 1 2021-11-26 14:13:07.456 WARN: no nodes coming from prim view, prim not possible 2021-11-26 14:13:07.456 INFO: view(view_id(NON_PRIM,96eb2dbf-836c,1) memb { 96eb2dbf-836c,1 } joined { } left { } partitioned { }) 2021-11-26 14:13:07.956 WARN: last inactive check more than PT1.5S ago (PT3.50352S), skipping check 2021-11-26 14:13:11.456 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5624 rttvar: 2812 rto: 217000 lost: 0 last_data_recv: 7490195 cwnd: 10 last_queued_since: 2994624323 last_delivered_since: 2994624323 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:14.456 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 216000 lost: 0 last_data_recv: 7493195 cwnd: 10 last_queued_since: 1793347229 last_delivered_since: 1793347229 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:17.457 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5106 rttvar: 2553 rto: 216000 lost: 0 last_data_recv: 7496196 cwnd: 10 last_queued_since: 2994913902 last_delivered_since: 2994913902 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:20.458 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5144 rttvar: 2572 rto: 215000 lost: 0 last_data_recv: 7499196 cwnd: 10 last_queued_since: 2994978793 last_delivered_since: 2994978793 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:23.958 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 6478 rttvar: 3239 rto: 214000 lost: 0 last_data_recv: 7502697 cwnd: 10 last_queued_since: 3493242685 last_delivered_since: 3493242685 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:27.459 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5934 rttvar: 2967 rto: 213000 lost: 0 last_data_recv: 7506198 cwnd: 10 last_queued_since: 3494029027 last_delivered_since: 3494029027 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:30.459 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4651 rttvar: 2325 rto: 213000 lost: 0 last_data_recv: 7509198 cwnd: 10 last_queued_since: 2995572067 last_delivered_since: 2995572067 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:33.460 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5214 rttvar: 2607 rto: 212000 lost: 0 last_data_recv: 7512199 cwnd: 10 last_queued_since: 2995066712 last_delivered_since: 2995066712 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:36.461 INFO: (96eb2dbf-836c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4471 rttvar: 2235 rto: 212000 lost: 0 last_data_recv: 7515200 cwnd: 10 last_queued_since: 2995658983 last_delivered_since: 2995658983 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:13:37.466 INFO: PC protocol downgrade 1 -> 0 2021-11-26 14:13:37.466 INFO: view((empty)) 2021-11-26 14:13:37.467 ERROR: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at /home/buildbot/buildbot/build/gcomm/src/pc.cpp:connect():160 2021-11-26 14:13:37.467 ERROR: /home/buildbot/buildbot/build/gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out) 2021-11-26 14:13:37.467 ERROR: /home/buildbot/buildbot/build/gcs/src/gcs.cpp:gcs_open():1633: Failed to open channel 'trv_acc' at 'gcomm://10.16.103.30,10.16.103.31,10.15.103.30,10.15.103.31': -110 (Connection timed out) 2021-11-26 14:13:37.467 INFO: Shifting CLOSED -> DESTROYED (TO: 0) 2021-11-26 14:13:37.467 FATAL: Exception in creating receive loop: Failed to open connection to group: 110 (Connection timed out) at /home/buildbot/buildbot/build/garb/garb_gcs.cpp:Gcs():35 2021-11-26 14:41:29.265 INFO: CRC-32C: using 64-bit x86 acceleration. 2021-11-26 14:41:29.265 INFO: Read config: daemon: 0 name: garb address: gcomm://10.16.103.30,10.16.103.31,10.15.103.30,10.15.103.31 group: trv_acc sst: trivial donor: options: evs.keepalive_period=PT3S;evs.suspect_timeout=PT30S;evs.inactive_timeout=PT1M;evs.install_timeout=PT1M;gcs.fc_limit=20;gcs.fc_factor=0.8;socket.ssl_key=/etc/garbd/aicetrvdba010-key.pem;socket.ssl_cert=/etc/garbd/aicetrvdba010-cert.pem;socket.ssl_ca=/etc/garbd/ca-cert.pem;gmcast.segment=1;socket.ssl_cipher=AES128-SHA256; gcs.fc_limit=9999999; gcs.fc_factor=1.0; gcs.fc_master_slave=yes cfg: log: /var/log/garbd.log 2021-11-26 14:41:29.266 INFO: protonet asio version 0 2021-11-26 14:41:29.266 INFO: Using CRC-32C for message checksums. 2021-11-26 14:41:29.266 INFO: backend: asio 2021-11-26 14:41:29.267 INFO: gcomm thread scheduling priority set to other:0 2021-11-26 14:41:29.267 WARN: access file(./gvwstate.dat) failed(No such file or directory) 2021-11-26 14:41:29.267 INFO: restore pc from disk failed 2021-11-26 14:41:29.267 INFO: GMCast version 0 2021-11-26 14:41:29.268 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2021-11-26 14:41:29.268 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2021-11-26 14:41:29.268 INFO: EVS version 1 2021-11-26 14:41:29.268 INFO: gcomm: connecting to group 'trv_acc', peer '10.16.103.30:,10.16.103.31:,10.15.103.30:,10.15.103.31:' 2021-11-26 14:41:32.270 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 33088 rttvar: 16544 rto: 234000 lost: 0 last_data_recv: 9191009 cwnd: 10 last_queued_since: 2967990770 last_delivered_since: 2967990770 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:32.271 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 21002 rttvar: 10501 rto: 222000 lost: 0 last_data_recv: 9191009 cwnd: 10 last_queued_since: 2980111602 last_delivered_since: 2980111602 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:32.271 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.16.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 15030 rttvar: 7515 rto: 216000 lost: 0 last_data_recv: 9191010 cwnd: 10 last_queued_since: 2986030656 last_delivered_since: 2986030656 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:32.271 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.16.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 15718 rttvar: 7859 rto: 216000 lost: 0 last_data_recv: 9191010 cwnd: 10 last_queued_since: 2985241076 last_delivered_since: 2985241076 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:32.271 INFO: EVS version upgrade 0 -> 1 2021-11-26 14:41:32.271 INFO: PC protocol upgrade 0 -> 1 2021-11-26 14:41:32.271 WARN: no nodes coming from prim view, prim not possible 2021-11-26 14:41:32.271 INFO: view(view_id(NON_PRIM,8f115646-993d,1) memb { 8f115646-993d,1 } joined { } left { } partitioned { }) 2021-11-26 14:41:32.771 WARN: last inactive check more than PT1.5S ago (PT3.50325S), skipping check 2021-11-26 14:41:36.772 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 6735 rttvar: 3367 rto: 229000 lost: 0 last_data_recv: 9195511 cwnd: 10 last_queued_since: 3493824521 last_delivered_since: 3493824521 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:40.272 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4186 rttvar: 2093 rto: 218000 lost: 0 last_data_recv: 9199011 cwnd: 10 last_queued_since: 3495435969 last_delivered_since: 3495435969 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:43.273 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4627 rttvar: 2313 rto: 226000 lost: 0 last_data_recv: 9202012 cwnd: 10 last_queued_since: 2995827382 last_delivered_since: 2995827382 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:46.774 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5294 rttvar: 2647 rto: 217000 lost: 0 last_data_recv: 9205513 cwnd: 10 last_queued_since: 3494632447 last_delivered_since: 3494632447 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:49.774 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4775 rttvar: 2387 rto: 223000 lost: 0 last_data_recv: 9208513 cwnd: 10 last_queued_since: 2995383884 last_delivered_since: 2995383884 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:53.275 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4410 rttvar: 2205 rto: 215000 lost: 0 last_data_recv: 9212014 cwnd: 10 last_queued_since: 3495514381 last_delivered_since: 3495514381 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:56.776 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4554 rttvar: 2277 rto: 221000 lost: 0 last_data_recv: 9215514 cwnd: 10 last_queued_since: 3495562496 last_delivered_since: 3495562496 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:41:59.776 INFO: (8f115646-993d, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.15.103.31:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5082 rttvar: 2541 rto: 214000 lost: 0 last_data_recv: 9218515 cwnd: 10 last_queued_since: 2994854631 last_delivered_since: 2994854631 send_queue_length: 0 send_queue_bytes: 0 2021-11-26 14:42:02.281 INFO: PC protocol downgrade 1 -> 0 2021-11-26 14:42:02.282 INFO: view((empty)) 2021-11-26 14:42:02.282 ERROR: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at /home/buildbot/buildbot/build/gcomm/src/pc.cpp:connect():160 2021-11-26 14:42:02.282 ERROR: /home/buildbot/buildbot/build/gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out) 2021-11-26 14:42:02.282 ERROR: /home/buildbot/buildbot/build/gcs/src/gcs.cpp:gcs_open():1633: Failed to open channel 'trv_acc' at 'gcomm://10.16.103.30,10.16.103.31,10.15.103.30,10.15.103.31': -110 (Connection timed out) 2021-11-26 14:42:02.282 INFO: Shifting CLOSED -> DESTROYED (TO: 0) 2021-11-26 14:42:02.282 FATAL: Exception in creating receive loop: Failed to open connection to group: 110 (Connection timed out) at /home/buildbot/buildbot/build/garb/garb_gcs.cpp:Gcs():35