Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25883

Galera Cluster hangs while "DELETE FROM mysql.wsrep_cluster"

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 10.4.18
    • 10.4.22, 10.5.13
    • Galera
    • None
    • CentOS 7.8

    Description

      Hi,

      I have a setup with 3 Galera nodes on MariaDB 10.4.18.
      I have blocking issue with MariaDB stability during network fluctuations (this usually happens for me when my client performs network interventions of it has hardware issues on the network).

      The issues is the following:
      1. After few ups and downs of the network, on single node, Galera cluster goes into a complete freeze.
      2. The node that lost connection is OK, as it doe not receive traffic anymore
      3 The other 2 nodes are accessible via mysql connection with root account but when I check the process list in MariaDB I have always the following query hanged updating: DELETE FROM mysql.wsrep_cluster.

      • I see other queries after this, but they all have the time lower than "DELETE FROM mysql.wsrep_cluster", which makes me thing that this query somehow locks the whole database and the other queries are waiting for it. Unfortunately it never ends.

      Logs analysis:
      1. We do not see any slow log for the query that hangs
      2. We do not see any error in MariaDB / Galera error log

      In order to replicate this we perform the following test:
      1. Start Galera on 3 nodes
      2. We have a webservice that connects to Galera start a tool that sends queries continuously to the database
      3. We perform successive ifup / ifdown at about 60 seconds interval on single node
      4. After 10 -15 tries, the cluster hangs in the situation above

      I'm attaching the script that we use for ifup / ifdown testing and also I'm putting here he Galera configuration:

      #Galera Provider Configuration
      wsrep_on=ON
      wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
       
      #Galera Cluster Configuration
      wsrep_cluster_name="<EDITED>"
      wsrep_cluster_address="gcomm://<EDITED>"
       
      #Galera Synchronization Configuration
      wsrep_sst_method=rsync
       
      #Galera Node Configuration
      wsrep_node_address="<EDITED>"
      wsrep_node_name="<EDITED>"
      wsrep_slave_threads = 8
       
      wsrep_provider_options="gcache.size = 2G; gcache.page_size = 1G; gcs.fc_limit = 256; gcs.fc_factor = 0.99; cert.log_conflicts = ON;"
       
      #read consistency
      wsrep_sync_wait=3
      wsrep_retry_autocommit = 1
       
      wsrep_debug = 1
      wsrep_log_conflicts = 1
      

      Attachments

        1. Galera Log - ETH-UP-Down-FAIL.txt
          7 kB
          Andras Marinescu
        2. Galera Log - ETH-UP-Down-OK.txt
          5 kB
          Andras Marinescu
        3. galera-stability.php
          4 kB
          Andras Marinescu
        4. galera-stability.sql
          2 kB
          Andras Marinescu
        5. ifdownupmain.sh
          0.6 kB
          Andras Marinescu
        6. MariaDB-Log.png
          24 kB
          Andras Marinescu
        7. MariaDB-server.cnf
          4 kB
          Andras Marinescu
        8. MariaDB validation diagram.png
          65 kB
          Andras Marinescu
        9. threads-tool.php
          0.9 kB
          Andras Marinescu

        Issue Links

          Activity

            Transition Time In Source Status Execution Times
            Ramesh Sivaraman made transition -
            Open Confirmed
            34d 1h 9m 1
            Mario Karuza (Inactive) made transition -
            Confirmed In Progress
            77d 7h 43m 1
            Mario Karuza (Inactive) made transition -
            In Progress In Review
            9s 1
            Jan Lindström (Inactive) made transition -
            In Review Closed
            1d 18h 34m 1

            People

              jplindst Jan Lindström (Inactive)
              ionut.andras Andras Marinescu
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.