Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.4.8
-
None
-
CentOS 7 virtual machine.
Description
Using MariaDB to do SST but node stuck in joining state even though in the logs it seems to have finished syncing. The name of the node with the issue is HQGaleraDBProd1.
2019-11-08 1:28:15 0 [Note] InnoDB: To roll back: 2 transactions, 9456 rows
2019-11-08 1:28:30 0 [Note] InnoDB: To roll back: 2 transactions, 7574 rows
2019-11-08 1:28:45 0 [Note] InnoDB: To roll back: 2 transactions, 5466 rows
2019-11-08 1:29:00 0 [Note] InnoDB: To roll back: 2 transactions, 2516 rows
2019-11-08 1:29:07 0 [Note] InnoDB: Rolled back recovered transaction 1161492834
2019-11-08 1:29:07 0 [Note] InnoDB: Rolled back recovered transaction 1161492832
2019-11-08 1:29:07 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2019-11-08 1:30:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) desyncs itself from group
2019-11-08 1:30:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) resyncs itself to group
2019-11-08 1:30:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) synced with group.
2019-11-08 1:34:05 0 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.20.11.46:57668 local endpoint ssl://10.1.30.60:4567 cipher: ECDHE-RSA-AES256-GCM-SHA384 compression: none
2019-11-08 1:34:05 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') connection established to 0602cb8a ssl://10.20.11.46:4567
2019-11-08 1:34:05 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2019-11-08 1:34:08 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting off
2019-11-08 1:35:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) desyncs itself from group
2019-11-08 1:35:18 0 [Note] WSREP: Member 1.0 (Galeradbprod02) resyncs itself to group
2019-11-08 1:35:18 0 [Note] WSREP: Member 1.0 (Galeradbprod02) synced with group.
2019-11-08 1:40:27 0 [Note] WSREP: SSL handshake successful, remote endpoint ssl://172.25.1.10:51004 local endpoint ssl://10.1.30.60:4567 cipher: ECDHE-RSA-AES256-GCM-SHA384 compression: none
2019-11-08 1:40:27 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://172.25.1.10:4567
2019-11-08 1:40:27 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') connection established to a67447ac ssl://172.25.1.10:4567
2019-11-08 1:40:27 0 [Note] WSREP: declaring 0602cb8a at ssl://10.20.11.46:4567 stable
2019-11-08 1:40:27 0 [Note] WSREP: declaring 678b492c at ssl://10.1.30.61:4567 stable
2019-11-08 1:40:27 0 [Note] WSREP: declaring a67447ac at ssl://172.25.1.10:4567 stable
2019-11-08 1:40:27 0 [Note] WSREP: declaring c4669934 at ssl://10.20.11.45:4567 stable
2019-11-08 1:40:28 0 [Note] WSREP: Node 0602cb8a state prim
2019-11-08 1:40:28 0 [Note] WSREP: view(view_id(PRIM,0602cb8a,152) memb
joined {
} left {
} partitioned {
})
2019-11-08 1:40:28 0 [Note] WSREP: save pc into disk
2019-11-08 1:40:28 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 4, memb_num = 5
2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: sent state msg: a74acf01-01f2-11ea-942b-5393bb976bf1
2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 0 (FLLGaleraDBColo2)
2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 1 (Galeradbprod02)
2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 4 (HQGaleraDBProd1)
2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 3 (GaleraDBColo1)
2019-11-08 1:40:29 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 2 (garb)
2019-11-08 1:40:29 0 [Note] WSREP: Quorum results:
version = 5,
component = PRIMARY,
conf_id = 4,
members = 3/5 (joined/total),
act_id = 107831078,
last_appl. = 0,
protocols = 1/10/4 (gcs/repl/appl),
vote policy= 0,
group UUID = 1cdf2a6b-ce84-11e8-88a7-93f3d3b4ed22
2019-11-08 1:40:29 0 [Note] WSREP: Writing down CC checksum: 5223b1c7 177d6ffe a531ea80 e7e58843 at offset 384
2019-11-08 1:40:29 0 [Note] WSREP: Flow-control interval: [286, 358]
2019-11-08 1:40:29 0 [Note] WSREP: Trying to continue unpaused monitor
2019-11-08 1:40:29 0 [Note] WSREP: Member 2.2 (garb) requested state transfer from 'any'. Selected 3.1 (GaleraDBColo1)(SYNCED) as donor.
2019-11-08 1:40:29 0 [Note] WSREP: 2.2 (garb): State transfer from 3.1 (GaleraDBColo1) complete.
2019-11-08 1:40:29 0 [Note] WSREP: 3.1 (GaleraDBColo1): State transfer to 2.2 (garb) complete.
2019-11-08 1:40:29 0 [Note] WSREP: Member 2.2 (garb) synced with group.
2019-11-08 1:40:29 0 [Note] WSREP: Member 3.1 (GaleraDBColo1) synced with group.
2019-11-08 1:40:30 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting off
2019-11-08 1:41:12 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
2019-11-08 1:41:12 0 [Note] WSREP: Shutdown replication
2019-11-08 1:41:12 0 [Note] WSREP: Server status change joined -> disconnecting
2019-11-08 1:41:12 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-11-08 1:41:12 0 [Note] WSREP: Closing send monitor...
2019-11-08 1:41:12 0 [Note] WSREP: Closed send monitor.
2019-11-08 1:41:12 0 [Note] WSREP: gcomm: terminating thread
2019-11-08 1:41:12 0 [Note] WSREP: gcomm: joining thread
2019-11-08 1:41:12 0 [Note] WSREP: gcomm: closing backend
2019-11-08 1:41:13 0 [Note] WSREP: view(view_id(NON_PRIM,0602cb8a,152) memb
joined {
} left {
} partitioned
)
2019-11-08 1:41:13 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2019-11-08 1:41:13 0 [Note] WSREP: view((empty))
2019-11-08 1:41:13 0 [Note] WSREP: gcomm: closed
2019-11-08 1:41:13 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2019-11-08 1:41:13 0 [Note] WSREP: Writing down CC checksum: 7151e494 5f3cf73f 364e791b 7ef788ff at offset 128
2019-11-08 1:41:13 0 [Note] WSREP: Flow-control interval: [128, 160]
2019-11-08 1:41:13 0 [Note] WSREP: Trying to continue unpaused monitor
2019-11-08 1:41:13 0 [Note] WSREP: Received NON-PRIMARY.
2019-11-08 1:41:13 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 107831079)
2019-11-08 1:41:13 0 [Note] WSREP: Received self-leave message.
2019-11-08 1:41:13 0 [Note] WSREP: Writing down CC checksum: 4b2707b5 8f7d8a17 d69ecd9d 73b5096e at offset 64
2019-11-08 1:41:13 0 [Note] WSREP: Flow-control interval: [0, 0]
2019-11-08 1:41:13 0 [Note] WSREP: Trying to continue unpaused monitor
2019-11-08 1:41:13 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2019-11-08 1:41:13 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
2019-11-08 1:41:13 135 [Warning] Aborted connection 135 to db: 'unconnected' user: 'maxscale' host: '10.1.10.219' (Got an error reading communication packets)
2019-11-08 1:41:13 0 [Note] WSREP: RECV thread exiting 0: Success
2019-11-08 1:41:13 0 [Note] WSREP: recv_thread() joined.
2019-11-08 1:41:13 0 [Note] WSREP: Closing replication queue.
2019-11-08 1:41:13 0 [Note] WSREP: Closing slave action queue.
2019-11-08 1:41:13 202 [Warning] Aborted connection 202 to db: 'unconnected' user: 'percona_mon' host: 'localhost' (Got an error writing communication packets)
Attachments
Issue Links
- relates to
-
MDEV-21002 Galera Cluster Node During IST Goes from "Synced" to "Joining: receiving State transfer" (stuck, requires kill -9)
- Closed