Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24294

MariaDB - Cluster freezes if node hangs

    XMLWordPrintable

    Details

      Description

      Currently I have a recurring problem. Our database cluster, consisting of three nodes, currently fails almost daily. The reason is repeated that one of the three nodes hangs and thus somehow hangs the whole cluster. But... we have the cluster to protect us against failures.

      The problem behaves in such a way that every connection attempt is timed out. I connect via ssh to each of the nodes and execute the command "mariadb" or "mysql". So far it was always the case that the command worked on 2 of 3 nodes, one node (the hanging one) is not responding. If I now restart the hanging node via "reboot -f", the cluster is healthy again after a few seconds.

      A reboot without "-f" does not work because the MariaDB service cannot be stopped. Even after several hours the frozen node is not removed from the cluster.

      So far once the first and twice the third node hangs. Each time the whole cluster was no longer usable.

      The command "mysqlcheck -A -e" displays "OK" for all tables. So i hope that no one is corrupted.

      Before we upgraded to version 10.5.8, we did not have this problem. I don't know if this problem is related to the new version, so I'm reporting it here.

      We have two tables with 3 to 5 millions data records. The other tables (about 10 more) have 1 to 60.000 data records. The database is accessed about 20-100 times a second.

      I'm desperate about this, because the database has always been very stable.

      Does anyone have an idea?

      Following the configration:

      The innodb_buffer_pool_size is set to 22G and the max connections to 800 (up to now, a maximum of 120 were used simultaneously).

      # /etc/mysql/mariadb.conf.d/60-galera.cnf
      #
      # * Galera-related settings
      #
      # See the examples of server wsrep.cnf files in /usr/share/mysql
       
      [galera]
      # Mandatory settings
      wsrep_on=ON
      wsrep_provider=/usr/lib/galera/libgalera_smm.so
      wsrep_cluster_address="gcomm://10.0.0.3,10.0.0.4,10.0.0.2"
      binlog_format=row
      default_storage_engine=InnoDB
      innodb_autoinc_lock_mode=2
       
      # Allow server to accept connections on all interfaces.
      bind-address=0.0.0.0
       
      # Optional settings
      #wsrep_slave_threads=1
      #innodb_flush_log_at_trx_commit=0
       
      wsrep_cluster_name="mariadb-galera-cluster"
      wsrep_sst_method=rsync
       
      # Cluster node configuration
      wsrep_node_address="10.0.0.3"
      wsrep_node_name="db-1"
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              seppo Seppo Jaakola
              Reporter:
              BC-M Malte Bastian
              Votes:
              11 Vote for this issue
              Watchers:
              21 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration