Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21008

Node Stuck in joining State

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.4.8
    • N/A
    • Galera, Server
    • None
    • CentOS 7 virtual machine.

    Description

      Using MariaDB to do SST but node stuck in joining state even though in the logs it seems to have finished syncing. The name of the node with the issue is HQGaleraDBProd1.

      2019-11-08 1:28:15 0 [Note] InnoDB: To roll back: 2 transactions, 9456 rows
      2019-11-08 1:28:30 0 [Note] InnoDB: To roll back: 2 transactions, 7574 rows
      2019-11-08 1:28:45 0 [Note] InnoDB: To roll back: 2 transactions, 5466 rows
      2019-11-08 1:29:00 0 [Note] InnoDB: To roll back: 2 transactions, 2516 rows
      2019-11-08 1:29:07 0 [Note] InnoDB: Rolled back recovered transaction 1161492834
      2019-11-08 1:29:07 0 [Note] InnoDB: Rolled back recovered transaction 1161492832
      2019-11-08 1:29:07 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      2019-11-08 1:30:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) desyncs itself from group
      2019-11-08 1:30:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) resyncs itself to group
      2019-11-08 1:30:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) synced with group.
      2019-11-08 1:34:05 0 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.20.11.46:57668 local endpoint ssl://10.1.30.60:4567 cipher: ECDHE-RSA-AES256-GCM-SHA384 compression: none
      2019-11-08 1:34:05 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') connection established to 0602cb8a ssl://10.20.11.46:4567
      2019-11-08 1:34:05 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
      2019-11-08 1:34:08 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting off
      2019-11-08 1:35:17 0 [Note] WSREP: Member 1.0 (Galeradbprod02) desyncs itself from group
      2019-11-08 1:35:18 0 [Note] WSREP: Member 1.0 (Galeradbprod02) resyncs itself to group
      2019-11-08 1:35:18 0 [Note] WSREP: Member 1.0 (Galeradbprod02) synced with group.
      2019-11-08 1:40:27 0 [Note] WSREP: SSL handshake successful, remote endpoint ssl://172.25.1.10:51004 local endpoint ssl://10.1.30.60:4567 cipher: ECDHE-RSA-AES256-GCM-SHA384 compression: none
      2019-11-08 1:40:27 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://172.25.1.10:4567
      2019-11-08 1:40:27 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') connection established to a67447ac ssl://172.25.1.10:4567
      2019-11-08 1:40:27 0 [Note] WSREP: declaring 0602cb8a at ssl://10.20.11.46:4567 stable
      2019-11-08 1:40:27 0 [Note] WSREP: declaring 678b492c at ssl://10.1.30.61:4567 stable
      2019-11-08 1:40:27 0 [Note] WSREP: declaring a67447ac at ssl://172.25.1.10:4567 stable
      2019-11-08 1:40:27 0 [Note] WSREP: declaring c4669934 at ssl://10.20.11.45:4567 stable
      2019-11-08 1:40:28 0 [Note] WSREP: Node 0602cb8a state prim
      2019-11-08 1:40:28 0 [Note] WSREP: view(view_id(PRIM,0602cb8a,152) memb

      { 0602cb8a,1 678b492c,0 a67447ac,2 c4669934,1 f19a4d7f,0 }

      joined {
      } left {
      } partitioned {
      })
      2019-11-08 1:40:28 0 [Note] WSREP: save pc into disk
      2019-11-08 1:40:28 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 4, memb_num = 5
      2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: sent state msg: a74acf01-01f2-11ea-942b-5393bb976bf1
      2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 0 (FLLGaleraDBColo2)
      2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 1 (Galeradbprod02)
      2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 4 (HQGaleraDBProd1)
      2019-11-08 1:40:28 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 3 (GaleraDBColo1)
      2019-11-08 1:40:29 0 [Note] WSREP: STATE EXCHANGE: got state msg: a74acf01-01f2-11ea-942b-5393bb976bf1 from 2 (garb)
      2019-11-08 1:40:29 0 [Note] WSREP: Quorum results:
      version = 5,
      component = PRIMARY,
      conf_id = 4,
      members = 3/5 (joined/total),
      act_id = 107831078,
      last_appl. = 0,
      protocols = 1/10/4 (gcs/repl/appl),
      vote policy= 0,
      group UUID = 1cdf2a6b-ce84-11e8-88a7-93f3d3b4ed22
      2019-11-08 1:40:29 0 [Note] WSREP: Writing down CC checksum: 5223b1c7 177d6ffe a531ea80 e7e58843 at offset 384
      2019-11-08 1:40:29 0 [Note] WSREP: Flow-control interval: [286, 358]
      2019-11-08 1:40:29 0 [Note] WSREP: Trying to continue unpaused monitor
      2019-11-08 1:40:29 0 [Note] WSREP: Member 2.2 (garb) requested state transfer from 'any'. Selected 3.1 (GaleraDBColo1)(SYNCED) as donor.
      2019-11-08 1:40:29 0 [Note] WSREP: 2.2 (garb): State transfer from 3.1 (GaleraDBColo1) complete.
      2019-11-08 1:40:29 0 [Note] WSREP: 3.1 (GaleraDBColo1): State transfer to 2.2 (garb) complete.
      2019-11-08 1:40:29 0 [Note] WSREP: Member 2.2 (garb) synced with group.
      2019-11-08 1:40:29 0 [Note] WSREP: Member 3.1 (GaleraDBColo1) synced with group.
      2019-11-08 1:40:30 0 [Note] WSREP: (f19a4d7f, 'ssl://0.0.0.0:4567') turning message relay requesting off
      2019-11-08 1:41:12 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
      2019-11-08 1:41:12 0 [Note] WSREP: Shutdown replication
      2019-11-08 1:41:12 0 [Note] WSREP: Server status change joined -> disconnecting
      2019-11-08 1:41:12 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2019-11-08 1:41:12 0 [Note] WSREP: Closing send monitor...
      2019-11-08 1:41:12 0 [Note] WSREP: Closed send monitor.
      2019-11-08 1:41:12 0 [Note] WSREP: gcomm: terminating thread
      2019-11-08 1:41:12 0 [Note] WSREP: gcomm: joining thread
      2019-11-08 1:41:12 0 [Note] WSREP: gcomm: closing backend
      2019-11-08 1:41:13 0 [Note] WSREP: view(view_id(NON_PRIM,0602cb8a,152) memb

      { f19a4d7f,0 }

      joined {
      } left {
      } partitioned

      { 0602cb8a,1 678b492c,0 a67447ac,2 c4669934,1 }

      )
      2019-11-08 1:41:13 0 [Note] WSREP: PC protocol downgrade 1 -> 0
      2019-11-08 1:41:13 0 [Note] WSREP: view((empty))
      2019-11-08 1:41:13 0 [Note] WSREP: gcomm: closed
      2019-11-08 1:41:13 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      2019-11-08 1:41:13 0 [Note] WSREP: Writing down CC checksum: 7151e494 5f3cf73f 364e791b 7ef788ff at offset 128
      2019-11-08 1:41:13 0 [Note] WSREP: Flow-control interval: [128, 160]
      2019-11-08 1:41:13 0 [Note] WSREP: Trying to continue unpaused monitor
      2019-11-08 1:41:13 0 [Note] WSREP: Received NON-PRIMARY.
      2019-11-08 1:41:13 0 [Note] WSREP: Shifting JOINER -> OPEN (TO: 107831079)
      2019-11-08 1:41:13 0 [Note] WSREP: Received self-leave message.
      2019-11-08 1:41:13 0 [Note] WSREP: Writing down CC checksum: 4b2707b5 8f7d8a17 d69ecd9d 73b5096e at offset 64
      2019-11-08 1:41:13 0 [Note] WSREP: Flow-control interval: [0, 0]
      2019-11-08 1:41:13 0 [Note] WSREP: Trying to continue unpaused monitor
      2019-11-08 1:41:13 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
      2019-11-08 1:41:13 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: -1)
      2019-11-08 1:41:13 135 [Warning] Aborted connection 135 to db: 'unconnected' user: 'maxscale' host: '10.1.10.219' (Got an error reading communication packets)
      2019-11-08 1:41:13 0 [Note] WSREP: RECV thread exiting 0: Success
      2019-11-08 1:41:13 0 [Note] WSREP: recv_thread() joined.
      2019-11-08 1:41:13 0 [Note] WSREP: Closing replication queue.
      2019-11-08 1:41:13 0 [Note] WSREP: Closing slave action queue.
      2019-11-08 1:41:13 202 [Warning] Aborted connection 202 to db: 'unconnected' user: 'percona_mon' host: 'localhost' (Got an error writing communication packets)

      Attachments

        Issue Links

          Activity

            People

              jplindst Jan Lindström (Inactive)
              obissick Oren Bissick (Inactive)
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.