[MDEV-8967] If wsrep_on=OFF on a node, there might still be some Galera Cluster communications Created: 2015-10-19  Updated: 2015-10-27  Resolved: 2015-10-27

Status: Closed
Project: MariaDB Server
Component/s: Documentation, Galera
Affects Version/s: 10.0.21-galera
Fix Version/s: 10.0.22-galera

Type: Bug Priority: Major
Reporter: Geoff Montee (Inactive) Assignee: Nirbhay Choubey (Inactive)
Resolution: Not a Bug Votes: 0
Labels: galera


 Description   

It seems that if you set wsrep_on=OFF, some Galera Cluster communications can still happen.

To demonstrate this, let's start with a 2-node cluster.

MariaDB [(none)]> show global status where Variable_name IN('wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_incoming_addresses');
 
+--------------------------+-------------------------------------+
 
| Variable_name            | Value                               |
 
+--------------------------+-------------------------------------+
 
| wsrep_incoming_addresses | 172.31.25.43:3306,172.31.25.44:3306 |
 
| wsrep_cluster_size       | 2                                   |
 
| wsrep_cluster_status     | Primary                             |
 
+--------------------------+-------------------------------------+
 
3 rows in set (0.00 sec)

Then let's kill the mysqld process on one node:

sudo kill -9 $(sudo pidof mysqld)

This will cause the cluster to become a 1-node cluster, and it will be non-Primary, assuming that the nodes use the default values of most wsrep_provider_option settings.

MariaDB [(none)]> show global status where Variable_name IN('wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_incoming_addresses');
 
+--------------------------+-------------------+
 
| Variable_name            | Value             |
 
+--------------------------+-------------------+
 
| wsrep_incoming_addresses | 172.31.25.44:3306 |
 
| wsrep_cluster_size       | 1                 |
 
| wsrep_cluster_status     | non-Primary       |
 
+--------------------------+-------------------+
 
3 rows in set (0.00 sec)

On the remaining node, let's set:

SET GLOBAL wsrep_on=OFF

Now, let's bring the killed node back online:

sudo service mysql start

Even though wsrep_on=OFF, a cluster does seem to get formed:

MariaDB [(none)]> show global status where Variable_name IN('wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_incoming_addresses');
 
+--------------------------+-------------------------------------+
 
| Variable_name            | Value                               |
 
+--------------------------+-------------------------------------+
 
| wsrep_incoming_addresses | 172.31.25.43:3306,172.31.25.44:3306 |
 
| wsrep_cluster_size       | 2                                   |
 
| wsrep_cluster_status     | Primary                             |
 
+--------------------------+-------------------------------------+
 
3 rows in set (0.00 sec)

The log contents for the node with wsrep_on=OFF, are:

151019 12:51:09 mysqld_safe mysqld from pid file /var/lib/mysql/ip-172-31-25-44.us-west-2.compute.internal.pid ended
 
151019 12:51:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
 
151019 12:51:12 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.GXlNPF' --pid-file='/var/lib/mysql/ip-172-31-25-44.us-west-2.compute.internal-recover.pid'
 
151019 12:51:12 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-wsrep) starting as process 2659 ...
 
151019 12:51:15 mysqld_safe WSREP: Recovered position 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:51:15 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-wsrep) starting as process 2694 ...
 
151019 12:51:15 [Note] WSREP: Read nil XID from storage engines, skipping position init
 
151019 12:51:15 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
 
151019 12:51:15 [Note] WSREP: wsrep_load(): Galera 25.3.9(r3387) by Codership Oy <info@codership.com> loaded successfully.
 
151019 12:51:15 [Note] WSREP: CRC-32C: using hardware acceleration.
 
151019 12:51:15 [Note] WSREP: Found saved state: 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:51:15 [Note] WSREP: Passing config to GCS: base_host = 172.31.25.44; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recover
 
151019 12:51:15 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:15 [Note] WSREP: Assign initial position for certification: 3, protocol version: -1
 
151019 12:51:15 [Note] WSREP: wsrep_sst_grab()
 
151019 12:51:15 [Note] WSREP: Start replication
 
151019 12:51:15 [Note] WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster
 
151019 12:51:15 [Note] WSREP: Setting initial position to 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:51:15 [Note] WSREP: protonet asio version 0
 
151019 12:51:15 [Note] WSREP: Using CRC-32C for message checksums.
 
151019 12:51:15 [Note] WSREP: backend: asio
 
151019 12:51:15 [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory)
 
151019 12:51:15 [Note] WSREP: restore pc from disk failed
 
151019 12:51:15 [Note] WSREP: GMCast version 0
 
151019 12:51:15 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
 
151019 12:51:15 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
 
151019 12:51:15 [Note] WSREP: EVS version 0
 
151019 12:51:15 [Note] WSREP: gcomm: bootstrapping new group 'my_wsrep_cluster'
 
151019 12:51:15 [Note] WSREP: start_prim is enabled, turn off pc_recovery
 
151019 12:51:15 [Note] WSREP: Node 9c6a5002 state prim
 
151019 12:51:15 [Note] WSREP: view(view_id(PRIM,9c6a5002,1) memb {
 
        9c6a5002,0
 
} joined {
 
} left {
 
} partitioned {
 
})
 
151019 12:51:15 [Note] WSREP: save pc into disk
 
151019 12:51:15 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.25.42:4567
 
151019 12:51:15 [Note] WSREP: discarding pending addr proto entry 0x7f284a4d82c0
 
151019 12:51:15 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.25.43:4567
 
151019 12:51:15 [Note] WSREP: discarding pending addr proto entry 0x7f284a4d8500
 
151019 12:51:15 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.25.44:4567
 
151019 12:51:15 [Note] WSREP: discarding pending addr proto entry 0x7f284a4d85c0
 
151019 12:51:15 [Note] WSREP: gcomm: connected
 
151019 12:51:15 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
 
151019 12:51:15 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
 
151019 12:51:15 [Note] WSREP: Opened channel 'my_wsrep_cluster'
 
151019 12:51:15 [Note] WSREP: Waiting for SST to complete.
 
151019 12:51:15 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
 
151019 12:51:15 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 9c6ac139-7681-11e5-a736-1b0dd4380317
 
151019 12:51:15 [Note] WSREP: STATE EXCHANGE: sent state msg: 9c6ac139-7681-11e5-a736-1b0dd4380317
 
151019 12:51:15 [Note] WSREP: STATE EXCHANGE: got state msg: 9c6ac139-7681-11e5-a736-1b0dd4380317 from 0 (ip-172-31-25-44.us-west-2.compute.internal)
 
151019 12:51:15 [Note] WSREP: Quorum results:
 
        version    = 3,
 
        component  = PRIMARY,
 
        conf_id    = 0,
 
        members    = 1/1 (joined/total),
 
        act_id     = 3,
 
        last_appl. = -1,
 
        protocols  = 0/7/3 (gcs/repl/appl),
 
        group UUID = 0c761b78-767e-11e5-9a39-3be96e837b66
 
151019 12:51:15 [Note] WSREP: Flow-control interval: [16, 16]
 
151019 12:51:15 [Note] WSREP: Restored state OPEN -> JOINED (3)
 
151019 12:51:15 [Note] WSREP: Member 0.0 (ip-172-31-25-44.us-west-2.compute.internal) synced with group.
 
151019 12:51:15 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 3)
 
151019 12:51:15 [Note] WSREP: New cluster view: global state: 0c761b78-767e-11e5-9a39-3be96e837b66:3, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
 
151019 12:51:15 [Note] WSREP: SST complete, seqno: 3
 
151019 12:51:15 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
151019 12:51:15 [Note] InnoDB: The InnoDB memory heap is disabled
 
151019 12:51:15 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
151019 12:51:15 [Note] InnoDB: Memory barrier is not used
 
151019 12:51:15 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
151019 12:51:15 [Note] InnoDB: Using Linux native AIO
 
151019 12:51:15 [Note] InnoDB: Using CPU crc32 instructions
 
151019 12:51:15 [Note] InnoDB: Initializing buffer pool, size = 128.0M
 
151019 12:51:15 [Note] InnoDB: Completed initialization of buffer pool
 
151019 12:51:15 [Note] InnoDB: Highest supported file format is Barracuda.
 
151019 12:51:15 [Note] InnoDB: 128 rollback segment(s) are active.
 
151019 12:51:15 [Note] InnoDB: Waiting for purge to start
 
151019 12:51:15 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.25-73.1 started; log sequence number 1622291
 
151019 12:51:15 [Note] Plugin 'FEEDBACK' is disabled.
 
151019 12:51:15 [Note] Server socket created on IP: '::'.
 
151019 12:51:15 [Note] Event Scheduler: Loaded 0 events
 
151019 12:51:15 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.0.21-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r4144
 
151019 12:51:15 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:51:15 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
151019 12:51:15 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:15 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
 
151019 12:51:15 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:15 [Note] WSREP: Synchronized with group, ready for connections
 
151019 12:51:15 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:51:30 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
 
151019 12:51:30 [Note] WSREP: declaring 2a4ff567 at tcp://172.31.25.43:4567 stable
 
151019 12:51:30 [Note] WSREP: Node 9c6a5002 state prim
 
151019 12:51:30 [Note] WSREP: view(view_id(PRIM,2a4ff567,9) memb {
 
        2a4ff567,0
 
        9c6a5002,0
 
} joined {
 
} left {
 
} partitioned {
 
})
 
151019 12:51:30 [Note] WSREP: save pc into disk
 
151019 12:51:30 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
 
151019 12:51:30 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
151019 12:51:31 [Note] WSREP: STATE EXCHANGE: sent state msg: a5fa0a0f-7681-11e5-8d08-0e195450fcbe
 
151019 12:51:31 [Note] WSREP: STATE EXCHANGE: got state msg: a5fa0a0f-7681-11e5-8d08-0e195450fcbe from 0 (ip-172-31-25-43.us-west-2.compute.internal)
 
151019 12:51:31 [Note] WSREP: STATE EXCHANGE: got state msg: a5fa0a0f-7681-11e5-8d08-0e195450fcbe from 1 (ip-172-31-25-44.us-west-2.compute.internal)
 
151019 12:51:31 [Note] WSREP: Quorum results:
 
        version    = 3,
 
        component  = PRIMARY,
 
        conf_id    = 1,
 
        members    = 2/2 (joined/total),
 
        act_id     = 3,
 
        last_appl. = 0,
 
        protocols  = 0/7/3 (gcs/repl/appl),
 
        group UUID = 0c761b78-767e-11e5-9a39-3be96e837b66
 
151019 12:51:31 [Note] WSREP: Flow-control interval: [23, 23]
 
151019 12:51:31 [Note] WSREP: New cluster view: global state: 0c761b78-767e-11e5-9a39-3be96e837b66:3, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3
 
151019 12:51:31 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:51:31 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
151019 12:51:31 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:31 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
 
151019 12:51:31 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:31 [Note] WSREP: Member 0.0 (ip-172-31-25-43.us-west-2.compute.internal) synced with group.
 
151019 12:51:33 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
151019 12:53:12 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.25.43:4567
 
151019 12:53:13 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') reconnecting to 2a4ff567 (tcp://172.31.25.43:4567), attempt 0
 
151019 12:53:16 [Note] WSREP: evs::proto(9c6a5002, OPERATIONAL, view_id(REG,2a4ff567,9)) suspecting node: 2a4ff567
 
151019 12:53:16 [Note] WSREP: evs::proto(9c6a5002, OPERATIONAL, view_id(REG,2a4ff567,9)) suspected node without join message, declaring inactive
 
151019 12:53:17 [Note] WSREP: view(view_id(NON_PRIM,2a4ff567,9) memb {
 
        9c6a5002,0
 
} joined {
 
} left {
 
} partitioned {
 
        2a4ff567,0
 
})
 
151019 12:53:17 [Note] WSREP: view(view_id(NON_PRIM,9c6a5002,10) memb {
 
        9c6a5002,0
 
} joined {
 
} left {
 
} partitioned {
 
        2a4ff567,0
 
})
 
151019 12:53:17 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
151019 12:53:17 [Note] WSREP: Flow-control interval: [16, 16]
 
151019 12:53:17 [Note] WSREP: Received NON-PRIMARY.
 
151019 12:53:17 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 3)
 
151019 12:53:17 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
 
151019 12:53:17 [Note] WSREP: Flow-control interval: [16, 16]
 
151019 12:53:17 [Note] WSREP: Received NON-PRIMARY.
 
151019 12:53:17 [Note] WSREP: New cluster view: global state: 0c761b78-767e-11e5-9a39-3be96e837b66:3, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
 
151019 12:53:17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:53:17 [Note] WSREP: New cluster view: global state: 0c761b78-767e-11e5-9a39-3be96e837b66:3, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
 
151019 12:53:17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:53:58 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') reconnecting to 2a4ff567 (tcp://172.31.25.43:4567), attempt 30
 
151019 12:54:43 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') reconnecting to 2a4ff567 (tcp://172.31.25.43:4567), attempt 60
 
151019 12:54:55 [Note] WSREP: declaring 2a4ff567 at tcp://172.31.25.43:4567 stable
 
151019 12:54:55 [Note] WSREP: re-bootstrapping prim from partitioned components
 
151019 12:54:55 [Note] WSREP: view(view_id(PRIM,2a4ff567,11) memb {
 
        2a4ff567,0
 
        9c6a5002,0
 
} joined {
 
} left {
 
} partitioned {
 
})
 
151019 12:54:55 [Note] WSREP: save pc into disk
 
151019 12:54:55 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
 
151019 12:54:55 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
 
151019 12:54:56 [Note] WSREP: STATE EXCHANGE: sent state msg: 20212a96-7682-11e5-abbd-4e0226d29d1c
 
151019 12:54:56 [Note] WSREP: STATE EXCHANGE: got state msg: 20212a96-7682-11e5-abbd-4e0226d29d1c from 0 (ip-172-31-25-43.us-west-2.compute.internal)
 
151019 12:54:56 [Note] WSREP: STATE EXCHANGE: got state msg: 20212a96-7682-11e5-abbd-4e0226d29d1c from 1 (ip-172-31-25-44.us-west-2.compute.internal)
 
151019 12:54:56 [Warning] WSREP: Quorum: No node with complete state:
 
 
 
 
 
        Version      : 3
 
        Flags        : 0x1
 
        Protocols    : 0 / 7 / 3
 
        State        : NON-PRIMARY
 
        Prim state   : NON-PRIMARY
 
        Prim UUID    : 00000000-0000-0000-0000-000000000000
 
        Prim  seqno  : -1
 
        First seqno  : -1
 
        Last  seqno  : 3
 
        Prim JOINED  : 0
 
        State UUID   : 20212a96-7682-11e5-abbd-4e0226d29d1c
 
        Group UUID   : 0c761b78-767e-11e5-9a39-3be96e837b66
 
        Name         : 'ip-172-31-25-43.us-west-2.compute.internal'
 
        Incoming addr: '172.31.25.43:3306'
 
        Version      : 3
 
        Flags        : 0x2
 
        Protocols    : 0 / 7 / 3
 
        State        : NON-PRIMARY
 
        Prim state   : SYNCED
 
        Prim UUID    : a5fa0a0f-7681-11e5-8d08-0e195450fcbe
 
        Prim  seqno  : 2
 
        First seqno  : -1
 
        Last  seqno  : 3
 
        Prim JOINED  : 2
 
        State UUID   : 20212a96-7682-11e5-abbd-4e0226d29d1c
 
        Group UUID   : 0c761b78-767e-11e5-9a39-3be96e837b66
 
        Name         : 'ip-172-31-25-44.us-west-2.compute.internal'
 
        Incoming addr: '172.31.25.44:3306'
 
 
 
151019 12:54:56 [Note] WSREP: Partial re-merge of primary a5fa0a0f-7681-11e5-8d08-0e195450fcbe found: 1 of 2.
 
151019 12:54:56 [Note] WSREP: Quorum results:
 
        version    = 3,
 
        component  = PRIMARY,
 
        conf_id    = 2,
 
        members    = 2/2 (joined/total),
 
        act_id     = 3,
 
        last_appl. = 0,
 
        protocols  = 0/7/3 (gcs/repl/appl),
 
        group UUID = 0c761b78-767e-11e5-9a39-3be96e837b66
 
151019 12:54:56 [Note] WSREP: Flow-control interval: [23, 23]
 
151019 12:54:56 [Note] WSREP: Restored state OPEN -> SYNCED (3)
 
151019 12:54:56 [Note] WSREP: New cluster view: global state: 0c761b78-767e-11e5-9a39-3be96e837b66:3, view# 3: Primary, number of nodes: 2, my index: 1, protocol version 3
 
151019 12:54:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:54:56 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
151019 12:54:56 [Note] WSREP: Service thread queue flushed.
 
151019 12:54:56 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
 
151019 12:54:56 [Note] WSREP: Service thread queue flushed.
 
151019 12:54:56 [Note] WSREP: Synchronized with group, ready for connections
 
151019 12:54:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:54:56 [Note] WSREP: Member 0.0 (ip-172-31-25-43.us-west-2.compute.internal) synced with group.
 
151019 12:54:58 [Note] WSREP: (9c6a5002, 'tcp://0.0.0.0:4567') turning message relay requesting off

The log contents for the second node are:

151019 12:51:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
 
151019 12:51:27 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.8OcDPr' --pid-file='/var/lib/mysql/ip-172-31-25-43.us-west-2.compute.internal-recover.pid'
 
151019 12:51:27 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-wsrep) starting as process 5713 ...
 
151019 12:51:30 mysqld_safe WSREP: Recovered position 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:51:30 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-wsrep) starting as process 5748 ...
 
151019 12:51:30 [Note] WSREP: Read nil XID from storage engines, skipping position init
 
151019 12:51:30 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
 
151019 12:51:30 [Note] WSREP: wsrep_load(): Galera 25.3.9(r3387) by Codership Oy <info@codership.com> loaded successfully.
 
151019 12:51:30 [Note] WSREP: CRC-32C: using hardware acceleration.
 
151019 12:51:30 [Note] WSREP: Found saved state: 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:51:30 [Note] WSREP: Passing config to GCS: base_host = 172.31.25.43; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recover
 
151019 12:51:30 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:30 [Note] WSREP: Assign initial position for certification: 3, protocol version: -1
 
151019 12:51:30 [Note] WSREP: wsrep_sst_grab()
 
151019 12:51:30 [Note] WSREP: Start replication
 
151019 12:51:30 [Note] WSREP: Setting initial position to 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:51:30 [Note] WSREP: protonet asio version 0
 
151019 12:51:30 [Note] WSREP: Using CRC-32C for message checksums.
 
151019 12:51:30 [Note] WSREP: backend: asio
 
151019 12:51:30 [Note] WSREP: restore pc from disk successfully
 
151019 12:51:30 [Note] WSREP: GMCast version 0
 
151019 12:51:30 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
 
151019 12:51:30 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
 
151019 12:51:30 [Note] WSREP: EVS version 0
 
151019 12:51:30 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '172.31.25.44:,172.31.25.43:,172.31.25.42:'
 
151019 12:51:30 [Warning] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') address 'tcp://172.31.25.43:4567' points to own listening address, blacklisting
 
151019 12:51:30 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') address 'tcp://172.31.25.43:4567' pointing to uuid 2a4ff567 is blacklisted, skipping
 
151019 12:51:30 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
 
151019 12:51:30 [Note] WSREP: declaring 9c6a5002 at tcp://172.31.25.44:4567 stable
 
151019 12:51:30 [Note] WSREP: Node 9c6a5002 state prim
 
151019 12:51:30 [Note] WSREP: view(view_id(PRIM,2a4ff567,9) memb {
 
        2a4ff567,0
 
        9c6a5002,0
 
} joined {
 
} left {
 
} partitioned {
 
})
 
151019 12:51:30 [Note] WSREP: save pc into disk
 
151019 12:51:30 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.25.42:4567
 
151019 12:51:30 [Note] WSREP: clear restored view
 
151019 12:51:31 [Note] WSREP: gcomm: connected
 
151019 12:51:31 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
 
151019 12:51:31 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
 
151019 12:51:31 [Note] WSREP: Opened channel 'my_wsrep_cluster'
 
151019 12:51:31 [Note] WSREP: Waiting for SST to complete.
 
151019 12:51:31 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
 
151019 12:51:31 [Note] WSREP: STATE_EXCHANGE: sent state UUID: a5fa0a0f-7681-11e5-8d08-0e195450fcbe
 
151019 12:51:31 [Note] WSREP: STATE EXCHANGE: sent state msg: a5fa0a0f-7681-11e5-8d08-0e195450fcbe
 
151019 12:51:31 [Note] WSREP: STATE EXCHANGE: got state msg: a5fa0a0f-7681-11e5-8d08-0e195450fcbe from 0 (ip-172-31-25-43.us-west-2.compute.internal)
 
151019 12:51:31 [Note] WSREP: STATE EXCHANGE: got state msg: a5fa0a0f-7681-11e5-8d08-0e195450fcbe from 1 (ip-172-31-25-44.us-west-2.compute.internal)
 
151019 12:51:31 [Note] WSREP: Quorum results:
 
        version    = 3,
 
        component  = PRIMARY,
 
        conf_id    = 1,
 
        members    = 2/2 (joined/total),
 
        act_id     = 3,
 
        last_appl. = -1,
 
        protocols  = 0/7/3 (gcs/repl/appl),
 
        group UUID = 0c761b78-767e-11e5-9a39-3be96e837b66
 
151019 12:51:31 [Note] WSREP: Flow-control interval: [23, 23]
 
151019 12:51:31 [Note] WSREP: Restored state OPEN -> JOINED (3)
 
151019 12:51:31 [Note] WSREP: New cluster view: global state: 0c761b78-767e-11e5-9a39-3be96e837b66:3, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
 
151019 12:51:31 [Note] WSREP: SST complete, seqno: 3
 
151019 12:51:31 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
151019 12:51:31 [Note] InnoDB: The InnoDB memory heap is disabled
 
151019 12:51:31 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
151019 12:51:31 [Note] InnoDB: Memory barrier is not used
 
151019 12:51:31 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
151019 12:51:31 [Note] InnoDB: Using Linux native AIO
 
151019 12:51:31 [Note] InnoDB: Using CPU crc32 instructions
 
151019 12:51:31 [Note] InnoDB: Initializing buffer pool, size = 128.0M
 
151019 12:51:31 [Note] WSREP: Member 0.0 (ip-172-31-25-43.us-west-2.compute.internal) synced with group.
 
151019 12:51:31 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 3)
 
151019 12:51:31 [Note] InnoDB: Completed initialization of buffer pool
 
151019 12:51:31 [Note] InnoDB: Highest supported file format is Barracuda.
 
151019 12:51:31 [Note] InnoDB: 128 rollback segment(s) are active.
 
151019 12:51:31 [Note] InnoDB: Waiting for purge to start
 
151019 12:51:31 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.25-73.1 started; log sequence number 1622809
 
151019 12:51:31 [Note] Plugin 'FEEDBACK' is disabled.
 
151019 12:51:31 [Note] Server socket created on IP: '::'.
 
151019 12:51:31 [Note] Event Scheduler: Loaded 0 events
 
151019 12:51:31 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.0.21-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r4144
 
151019 12:51:31 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:51:31 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
151019 12:51:31 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:31 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
 
151019 12:51:31 [Note] WSREP: Service thread queue flushed.
 
151019 12:51:31 [Note] WSREP: Synchronized with group, ready for connections
 
151019 12:51:31 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:51:33 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
151019 12:53:12 mysqld_safe Number of processes running now: 0
 
151019 12:53:12 mysqld_safe WSREP: not restarting wsrep node automatically
 
151019 12:53:12 mysqld_safe mysqld from pid file /var/lib/mysql/ip-172-31-25-43.us-west-2.compute.internal.pid ended
 
151019 12:54:52 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
 
151019 12:54:52 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.R2ZH5R' --pid-file='/var/lib/mysql/ip-172-31-25-43.us-west-2.compute.internal-recover.pid'
 
151019 12:54:52 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-wsrep) starting as process 5916 ...
 
151019 12:54:55 mysqld_safe WSREP: Recovered position 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:54:55 [Note] /usr/sbin/mysqld (mysqld 10.0.21-MariaDB-wsrep) starting as process 5952 ...
 
151019 12:54:55 [Note] WSREP: Read nil XID from storage engines, skipping position init
 
151019 12:54:55 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
 
151019 12:54:55 [Note] WSREP: wsrep_load(): Galera 25.3.9(r3387) by Codership Oy <info@codership.com> loaded successfully.
 
151019 12:54:55 [Note] WSREP: CRC-32C: using hardware acceleration.
 
151019 12:54:55 [Note] WSREP: Found saved state: 0c761b78-767e-11e5-9a39-3be96e837b66:-1
 
151019 12:54:55 [Note] WSREP: Passing config to GCS: base_host = 172.31.25.43; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recover
 
151019 12:54:55 [Note] WSREP: Service thread queue flushed.
 
151019 12:54:55 [Note] WSREP: Assign initial position for certification: 3, protocol version: -1
 
151019 12:54:55 [Note] WSREP: wsrep_sst_grab()
 
151019 12:54:55 [Note] WSREP: Start replication
 
151019 12:54:55 [Note] WSREP: Setting initial position to 0c761b78-767e-11e5-9a39-3be96e837b66:3
 
151019 12:54:55 [Note] WSREP: protonet asio version 0
 
151019 12:54:55 [Note] WSREP: Using CRC-32C for message checksums.
 
151019 12:54:55 [Note] WSREP: backend: asio
 
151019 12:54:55 [Note] WSREP: restore pc from disk successfully
 
151019 12:54:55 [Note] WSREP: GMCast version 0
 
151019 12:54:55 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
 
151019 12:54:55 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
 
151019 12:54:55 [Note] WSREP: EVS version 0
 
151019 12:54:55 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '172.31.25.44:,172.31.25.43:,172.31.25.42:'
 
151019 12:54:55 [Warning] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') address 'tcp://172.31.25.43:4567' points to own listening address, blacklisting
 
151019 12:54:55 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') address 'tcp://172.31.25.43:4567' pointing to uuid 2a4ff567 is blacklisted, skipping
 
151019 12:54:55 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
 
151019 12:54:55 [Note] WSREP: declaring 9c6a5002 at tcp://172.31.25.44:4567 stable
 
151019 12:54:55 [Note] WSREP: re-bootstrapping prim from partitioned components
 
151019 12:54:55 [Note] WSREP: view(view_id(PRIM,2a4ff567,11) memb {
 
        2a4ff567,0
 
        9c6a5002,0
 
} joined {
 
} left {
 
} partitioned {
 
})
 
151019 12:54:55 [Note] WSREP: save pc into disk
 
151019 12:54:55 [Note] WSREP: discarding pending addr without UUID: tcp://172.31.25.42:4567
 
151019 12:54:55 [Note] WSREP: clear restored view
 
151019 12:54:56 [Note] WSREP: gcomm: connected
 
151019 12:54:56 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
 
151019 12:54:56 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
 
151019 12:54:56 [Note] WSREP: Opened channel 'my_wsrep_cluster'
 
151019 12:54:56 [Note] WSREP: Waiting for SST to complete.
 
151019 12:54:56 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
 
151019 12:54:56 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 20212a96-7682-11e5-abbd-4e0226d29d1c
 
151019 12:54:56 [Note] WSREP: STATE EXCHANGE: sent state msg: 20212a96-7682-11e5-abbd-4e0226d29d1c
 
151019 12:54:56 [Note] WSREP: STATE EXCHANGE: got state msg: 20212a96-7682-11e5-abbd-4e0226d29d1c from 0 (ip-172-31-25-43.us-west-2.compute.internal)
 
151019 12:54:56 [Note] WSREP: STATE EXCHANGE: got state msg: 20212a96-7682-11e5-abbd-4e0226d29d1c from 1 (ip-172-31-25-44.us-west-2.compute.internal)
 
151019 12:54:56 [Warning] WSREP: Quorum: No node with complete state:
 
 
 
 
 
        Version      : 3
 
        Flags        : 0x1
 
        Protocols    : 0 / 7 / 3
 
        State        : NON-PRIMARY
 
        Prim state   : NON-PRIMARY
 
        Prim UUID    : 00000000-0000-0000-0000-000000000000
 
        Prim  seqno  : -1
 
        First seqno  : -1
 
        Last  seqno  : 3
 
        Prim JOINED  : 0
 
        State UUID   : 20212a96-7682-11e5-abbd-4e0226d29d1c
 
        Group UUID   : 0c761b78-767e-11e5-9a39-3be96e837b66
 
        Name         : 'ip-172-31-25-43.us-west-2.compute.internal'
 
        Incoming addr: '172.31.25.43:3306'
 
 
 
        Version      : 3
 
        Flags        : 0x2
 
        Protocols    : 0 / 7 / 3
 
        State        : NON-PRIMARY
 
        Prim state   : SYNCED
 
        Prim UUID    : a5fa0a0f-7681-11e5-8d08-0e195450fcbe
 
        Prim  seqno  : 2
 
        First seqno  : -1
 
        Last  seqno  : 3
 
        Prim JOINED  : 2
 
        State UUID   : 20212a96-7682-11e5-abbd-4e0226d29d1c
 
        Group UUID   : 0c761b78-767e-11e5-9a39-3be96e837b66
 
        Name         : 'ip-172-31-25-44.us-west-2.compute.internal'
 
        Incoming addr: '172.31.25.44:3306'
 
 
 
151019 12:54:56 [Note] WSREP: Partial re-merge of primary a5fa0a0f-7681-11e5-8d08-0e195450fcbe found: 1 of 2.
 
151019 12:54:56 [Note] WSREP: Quorum results:
 
        version    = 3,
 
        component  = PRIMARY,
 
        conf_id    = 2,
 
        members    = 2/2 (joined/total),
 
        act_id     = 3,
 
        last_appl. = -1,
 
        protocols  = 0/7/3 (gcs/repl/appl),
 
        group UUID = 0c761b78-767e-11e5-9a39-3be96e837b66
 
151019 12:54:56 [Note] WSREP: Flow-control interval: [23, 23]
 
151019 12:54:56 [Note] WSREP: Restored state OPEN -> JOINED (3)
 
151019 12:54:56 [Note] WSREP: New cluster view: global state: 0c761b78-767e-11e5-9a39-3be96e837b66:3, view# 3: Primary, number of nodes: 2, my index: 0, protocol version 3
 
151019 12:54:56 [Note] WSREP: SST complete, seqno: 3
 
151019 12:54:56 [Note] InnoDB: Using mutexes to ref count buffer pool pages
 
151019 12:54:56 [Note] InnoDB: The InnoDB memory heap is disabled
 
151019 12:54:56 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
151019 12:54:56 [Note] InnoDB: Memory barrier is not used
 
151019 12:54:56 [Note] InnoDB: Compressed tables use zlib 1.2.7
 
151019 12:54:56 [Note] InnoDB: Using Linux native AIO
 
151019 12:54:56 [Note] InnoDB: Using CPU crc32 instructions
 
151019 12:54:56 [Note] InnoDB: Initializing buffer pool, size = 128.0M
 
151019 12:54:56 [Note] WSREP: Member 0.0 (ip-172-31-25-43.us-west-2.compute.internal) synced with group.
 
151019 12:54:56 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 3)
 
151019 12:54:56 [Note] InnoDB: Completed initialization of buffer pool
 
151019 12:54:56 [Note] InnoDB: Highest supported file format is Barracuda.
 
151019 12:54:56 [Note] InnoDB: 128 rollback segment(s) are active.
 
151019 12:54:56 [Note] InnoDB: Waiting for purge to start
 
151019 12:54:56 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.25-73.1 started; log sequence number 1622983
 
151019 12:54:56 [Note] Plugin 'FEEDBACK' is disabled.
 
151019 12:54:56 [Note] Server socket created on IP: '::'.
 
151019 12:54:56 [Note] Event Scheduler: Loaded 0 events
 
151019 12:54:56 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.0.21-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r4144
 
151019 12:54:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:54:56 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
151019 12:54:56 [Note] WSREP: Service thread queue flushed.
 
151019 12:54:56 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
 
151019 12:54:56 [Note] WSREP: Service thread queue flushed.
 
151019 12:54:56 [Note] WSREP: Synchronized with group, ready for connections
 
151019 12:54:56 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
151019 12:54:58 [Note] WSREP: (2a4ff567, 'tcp://0.0.0.0:4567') turning message relay requesting off



 Comments   
Comment by Nirbhay Choubey (Inactive) [ 2015-10-27 ]

This is an expected behavior. I have updates the KB entry to clarify this.
https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/#wsrep_on

Generated at Thu Feb 08 07:31:08 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.