[MDEV-22382] Cluster nodes out of sync without error/warning message Created: 2020-04-28  Updated: 2020-11-19  Resolved: 2020-11-19

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.2.9
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: eason Assignee: Jan Lindström (Inactive)
Resolution: Incomplete Votes: 0
Labels: None


 Description   

We are running a 3 (A,B,C) nodes (8cores, 16GB) mariadb galera cluster on production. The applications only connect to A node with read/write account. Recently we found the B and C were of sync with A and only the schema change and some databases can be syncd to each other. (Use InnoDB engine).

From the SHOW STATUS LIKE 'wsrep%' for each node , both 3 nodes has the same wsrep_cluster_state_uuid and wsrep_local_state_uuid , the wsrep_local_state_comment was labeled as 'synced'.

How to trouble shooting and resolve this kind of issue ?

Node A:

wsrep_apply_oooe	0.039301
 
wsrep_apply_oool	0.000004
 
wsrep_apply_window	1.075744
 
wsrep_causal_reads	0
 
wsrep_cert_deps_distance	47.902862
 
wsrep_cert_index_size	202
 
wsrep_cert_interval	0.086835
 
wsrep_cluster_conf_id	5
 
wsrep_cluster_size	3
 
wsrep_cluster_state_uuid	a6cb55bc-5d73-11e9-9715-5b7096acd7d4
 
wsrep_cluster_status	Primary
 
wsrep_commit_oooe	0.000000
 
wsrep_commit_oool	0.000000
 
wsrep_commit_window	1.036447
 
wsrep_connected	ON
 
wsrep_desync_count	0
 
wsrep_evs_delayed	
wsrep_evs_evict_list	
wsrep_evs_repl_latency	0.000386854/0.000641471/0.00101774/0.000170883/41
 
wsrep_evs_state	OPERATIONAL
 
wsrep_flow_control_paused	0.000121
 
wsrep_flow_control_paused_ns	1499988210973
 
wsrep_flow_control_recv	19783
 
wsrep_flow_control_sent	0
 
wsrep_gcomm_uuid	0dfcdc42-1801-11ea-8ca9-76317559752c
 
wsrep_incoming_addresses	192.168.10.12:3306,192.168.10.13:3306,192.168.10.11:3306
 
wsrep_last_committed	49328708
 
wsrep_local_bf_aborts	1
 
wsrep_local_cached_downto	49320131
 
wsrep_local_cert_failures	5
 
wsrep_local_commits	17453848
 
wsrep_local_index	0
 
wsrep_local_recv_queue	0
 
wsrep_local_recv_queue_avg	0.001017
 
wsrep_local_recv_queue_max	2
 
wsrep_local_recv_queue_min	0
 
wsrep_local_replays	4
 
wsrep_local_send_queue	0
 
wsrep_local_send_queue_avg	0.011968
 
wsrep_local_send_queue_max	215
 
wsrep_local_send_queue_min	0
 
wsrep_local_state	4
 
wsrep_local_state_comment	Synced
 
wsrep_local_state_uuid	a6cb55bc-5d73-11e9-9715-5b7096acd7d4
 
wsrep_protocol_version	7
 
wsrep_provider_name	Galera
 
wsrep_provider_vendor	Codership Oy <info@codership.com>
 
wsrep_provider_version	25.3.20(r3703)
 
wsrep_ready	ON
 
wsrep_received	212459
 
wsrep_received_bytes	2386091
 
wsrep_repl_data_bytes	137697330836
 
wsrep_repl_keys	231049449
 
wsrep_repl_keys_bytes	2251998858
 
wsrep_repl_other_bytes	0
 
wsrep_replicated	17455391
 
wsrep_replicated_bytes	157381443713
 
wsrep_thread_count	2

Node C:

wsrep_apply_oooe	0.000000
 
wsrep_apply_oool	0.000000
 
wsrep_apply_window	1.000000
 
wsrep_causal_reads	0
 
wsrep_cert_deps_distance	47.901373
 
wsrep_cert_index_size	936
 
wsrep_cert_interval	0.086837
 
wsrep_cluster_conf_id	5
 
wsrep_cluster_size	3
 
wsrep_cluster_state_uuid	a6cb55bc-5d73-11e9-9715-5b7096acd7d4
 
wsrep_cluster_status	Primary
 
wsrep_commit_oooe	0.000000
 
wsrep_commit_oool	0.000000
 
wsrep_commit_window	1.000000
 
wsrep_connected	ON
 
wsrep_desync_count	0
 
wsrep_evs_delayed	
wsrep_evs_evict_list	
wsrep_evs_repl_latency	0/0/0/0/0
 
wsrep_evs_state	OPERATIONAL
 
wsrep_flow_control_paused	0.000121
 
wsrep_flow_control_paused_ns	1499874188022
 
wsrep_flow_control_recv	19783
 
wsrep_flow_control_sent	18513
 
wsrep_gcomm_uuid	5bc64b98-1802-11ea-86f3-3f1bc40b6f50
 
wsrep_incoming_addresses	192.168.10.12:3306,192.168.10.13:3306,192.168.10.11:3306
 
wsrep_last_committed	49329044
 
wsrep_local_bf_aborts	18
 
wsrep_local_cached_downto	49320471
 
wsrep_local_cert_failures	0
 
wsrep_local_commits	21
 
wsrep_local_index	1
 
wsrep_local_recv_queue	0
 
wsrep_local_recv_queue_avg	0.182513
 
wsrep_local_recv_queue_max	1179
 
wsrep_local_recv_queue_min	0
 
wsrep_local_replays	0
 
wsrep_local_send_queue	0
 
wsrep_local_send_queue_avg	0.000005
 
wsrep_local_send_queue_max	2
 
wsrep_local_send_queue_min	0
 
wsrep_local_state	4
 
wsrep_local_state_comment	Synced
 
wsrep_local_state_uuid	a6cb55bc-5d73-11e9-9715-5b7096acd7d4
 
wsrep_protocol_version	7
 
wsrep_provider_name	Galera
 
wsrep_provider_vendor	Codership Oy <info@codership.com>
 
wsrep_provider_version	25.3.20(r3703)
 
wsrep_ready	ON
 
wsrep_received	17668083
 
wsrep_received_bytes	157389215710
 
wsrep_repl_data_bytes	658359
 
wsrep_repl_keys	2770
 
wsrep_repl_keys_bytes	22649
 
wsrep_repl_other_bytes	0
 
wsrep_replicated	21
 
wsrep_replicated_bytes	682352
 
wsrep_thread_count	2



 Comments   
Comment by Jan Lindström (Inactive) [ 2020-11-19 ]

Can you please upgrade your server to more recent version of MariaDB and Galera library and if this is still reproducible, please include full error logs to a new bug report.

Generated at Thu Feb 08 09:14:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.