Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.2.9
-
None
Description
We are running a 3 (A,B,C) nodes (8cores, 16GB) mariadb galera cluster on production. The applications only connect to A node with read/write account. Recently we found the B and C were of sync with A and only the schema change and some databases can be syncd to each other. (Use InnoDB engine).
From the SHOW STATUS LIKE 'wsrep%' for each node , both 3 nodes has the same wsrep_cluster_state_uuid and wsrep_local_state_uuid , the wsrep_local_state_comment was labeled as 'synced'.
How to trouble shooting and resolve this kind of issue ?
Node A:
wsrep_apply_oooe 0.039301 |
wsrep_apply_oool 0.000004 |
wsrep_apply_window 1.075744 |
wsrep_causal_reads 0 |
wsrep_cert_deps_distance 47.902862 |
wsrep_cert_index_size 202 |
wsrep_cert_interval 0.086835 |
wsrep_cluster_conf_id 5 |
wsrep_cluster_size 3 |
wsrep_cluster_state_uuid a6cb55bc-5d73-11e9-9715-5b7096acd7d4 |
wsrep_cluster_status Primary
|
wsrep_commit_oooe 0.000000 |
wsrep_commit_oool 0.000000 |
wsrep_commit_window 1.036447 |
wsrep_connected ON
|
wsrep_desync_count 0 |
wsrep_evs_delayed
|
wsrep_evs_evict_list
|
wsrep_evs_repl_latency 0.000386854/0.000641471/0.00101774/0.000170883/41 |
wsrep_evs_state OPERATIONAL
|
wsrep_flow_control_paused 0.000121 |
wsrep_flow_control_paused_ns 1499988210973 |
wsrep_flow_control_recv 19783 |
wsrep_flow_control_sent 0 |
wsrep_gcomm_uuid 0dfcdc42-1801-11ea-8ca9-76317559752c |
wsrep_incoming_addresses 192.168.10.12:3306,192.168.10.13:3306,192.168.10.11:3306 |
wsrep_last_committed 49328708 |
wsrep_local_bf_aborts 1 |
wsrep_local_cached_downto 49320131 |
wsrep_local_cert_failures 5 |
wsrep_local_commits 17453848 |
wsrep_local_index 0 |
wsrep_local_recv_queue 0 |
wsrep_local_recv_queue_avg 0.001017 |
wsrep_local_recv_queue_max 2 |
wsrep_local_recv_queue_min 0 |
wsrep_local_replays 4 |
wsrep_local_send_queue 0 |
wsrep_local_send_queue_avg 0.011968 |
wsrep_local_send_queue_max 215 |
wsrep_local_send_queue_min 0 |
wsrep_local_state 4 |
wsrep_local_state_comment Synced
|
wsrep_local_state_uuid a6cb55bc-5d73-11e9-9715-5b7096acd7d4 |
wsrep_protocol_version 7 |
wsrep_provider_name Galera
|
wsrep_provider_vendor Codership Oy <info@codership.com> |
wsrep_provider_version 25.3.20(r3703) |
wsrep_ready ON
|
wsrep_received 212459 |
wsrep_received_bytes 2386091 |
wsrep_repl_data_bytes 137697330836 |
wsrep_repl_keys 231049449 |
wsrep_repl_keys_bytes 2251998858 |
wsrep_repl_other_bytes 0 |
wsrep_replicated 17455391 |
wsrep_replicated_bytes 157381443713 |
wsrep_thread_count 2 |
Node C:
wsrep_apply_oooe 0.000000 |
wsrep_apply_oool 0.000000 |
wsrep_apply_window 1.000000 |
wsrep_causal_reads 0 |
wsrep_cert_deps_distance 47.901373 |
wsrep_cert_index_size 936 |
wsrep_cert_interval 0.086837 |
wsrep_cluster_conf_id 5 |
wsrep_cluster_size 3 |
wsrep_cluster_state_uuid a6cb55bc-5d73-11e9-9715-5b7096acd7d4 |
wsrep_cluster_status Primary
|
wsrep_commit_oooe 0.000000 |
wsrep_commit_oool 0.000000 |
wsrep_commit_window 1.000000 |
wsrep_connected ON
|
wsrep_desync_count 0 |
wsrep_evs_delayed
|
wsrep_evs_evict_list
|
wsrep_evs_repl_latency 0/0/0/0/0 |
wsrep_evs_state OPERATIONAL
|
wsrep_flow_control_paused 0.000121 |
wsrep_flow_control_paused_ns 1499874188022 |
wsrep_flow_control_recv 19783 |
wsrep_flow_control_sent 18513 |
wsrep_gcomm_uuid 5bc64b98-1802-11ea-86f3-3f1bc40b6f50 |
wsrep_incoming_addresses 192.168.10.12:3306,192.168.10.13:3306,192.168.10.11:3306 |
wsrep_last_committed 49329044 |
wsrep_local_bf_aborts 18 |
wsrep_local_cached_downto 49320471 |
wsrep_local_cert_failures 0 |
wsrep_local_commits 21 |
wsrep_local_index 1 |
wsrep_local_recv_queue 0 |
wsrep_local_recv_queue_avg 0.182513 |
wsrep_local_recv_queue_max 1179 |
wsrep_local_recv_queue_min 0 |
wsrep_local_replays 0 |
wsrep_local_send_queue 0 |
wsrep_local_send_queue_avg 0.000005 |
wsrep_local_send_queue_max 2 |
wsrep_local_send_queue_min 0 |
wsrep_local_state 4 |
wsrep_local_state_comment Synced
|
wsrep_local_state_uuid a6cb55bc-5d73-11e9-9715-5b7096acd7d4 |
wsrep_protocol_version 7 |
wsrep_provider_name Galera
|
wsrep_provider_vendor Codership Oy <info@codership.com> |
wsrep_provider_version 25.3.20(r3703) |
wsrep_ready ON
|
wsrep_received 17668083 |
wsrep_received_bytes 157389215710 |
wsrep_repl_data_bytes 658359 |
wsrep_repl_keys 2770 |
wsrep_repl_keys_bytes 22649 |
wsrep_repl_other_bytes 0 |
wsrep_replicated 21 |
wsrep_replicated_bytes 682352 |
wsrep_thread_count 2 |