Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.4.18
-
None
-
CentOS 7.8
Description
Hi,
I have a setup with 3 Galera nodes on MariaDB 10.4.18.
I have blocking issue with MariaDB stability during network fluctuations (this usually happens for me when my client performs network interventions of it has hardware issues on the network).
The issues is the following:
1. After few ups and downs of the network, on single node, Galera cluster goes into a complete freeze.
2. The node that lost connection is OK, as it doe not receive traffic anymore
3 The other 2 nodes are accessible via mysql connection with root account but when I check the process list in MariaDB I have always the following query hanged updating: DELETE FROM mysql.wsrep_cluster.
- I see other queries after this, but they all have the time lower than "DELETE FROM mysql.wsrep_cluster", which makes me thing that this query somehow locks the whole database and the other queries are waiting for it. Unfortunately it never ends.
Logs analysis:
1. We do not see any slow log for the query that hangs
2. We do not see any error in MariaDB / Galera error log
In order to replicate this we perform the following test:
1. Start Galera on 3 nodes
2. We have a webservice that connects to Galera start a tool that sends queries continuously to the database
3. We perform successive ifup / ifdown at about 60 seconds interval on single node
4. After 10 -15 tries, the cluster hangs in the situation above
I'm attaching the script that we use for ifup / ifdown testing and also I'm putting here he Galera configuration:
#Galera Provider Configuration
|
wsrep_on=ON
|
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so |
|
#Galera Cluster Configuration
|
wsrep_cluster_name="<EDITED>" |
wsrep_cluster_address="gcomm://<EDITED>" |
|
#Galera Synchronization Configuration
|
wsrep_sst_method=rsync
|
|
#Galera Node Configuration
|
wsrep_node_address="<EDITED>" |
wsrep_node_name="<EDITED>" |
wsrep_slave_threads = 8 |
|
wsrep_provider_options="gcache.size = 2G; gcache.page_size = 1G; gcs.fc_limit = 256; gcs.fc_factor = 0.99; cert.log_conflicts = ON;" |
|
#read consistency
|
wsrep_sync_wait=3 |
wsrep_retry_autocommit = 1 |
|
wsrep_debug = 1 |
wsrep_log_conflicts = 1 |
Attachments
Issue Links
- duplicates
-
MDEV-26760 Galera hangs during concurrent updates
- Closed
- relates to
-
MDEV-23379 Deprecate and ignore options for InnoDB concurrency throttling
- Closed