[MDEV-26760] Galera hangs during concurrent updates Created: 2021-10-04  Updated: 2021-12-07  Resolved: 2021-10-21

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.18
Fix Version/s: 10.4.22, 10.5.13

Type: Bug Priority: Blocker
Reporter: Andras Marinescu Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS 7.8


Attachments: PNG File MariaDB validation diagram.png     File MariaDB-server.cnf     File galera-lock-replication.php     File galera-lock-replication.sql     File threads-tool.php    
Issue Links:
Duplicate
is duplicated by MDEV-25883 Galera Cluster hangs while "DELETE FR... Closed

 Description   

Hi,

I have a setup with 3 Galera nodes on MariaDB 10.4.18.
I have blocking issue with MariaDB stability during concurrent updates on same row (this usually happens for me when multiple client applications tries to modify a record in the database in the same time).

The issues is the following:
1. Multiple customers starts the application and tries to perform an update on the same record.
2. Galera handles a few concurrencies but eventually hangs.

Now, I understand that we need to retry the transaction in the case of a deadlock, the issue is that Galera hangs before the application retries that.

In order to replicate this we perform the following test:
1. Start Galera on 3 nodes
2. Have a ProxySQL with 3 nodes as writers
3. We have a tool that connects to Galera start a tool that sends queries continuously to the database

I'm attaching the script that we use testing and also I'm putting here he Galera configuration:

  • MariaDB validation diagram.png - This is the network topology for testing
  • MariaDB-server.cnf
  • galera-lock-replication.sql - SQLs to create the database and table for testing
  • galera-lock-replication.php - the tool that connects to database and send sql update
  • threads-tool.php - the tool that starts "galera-lock-replication.php" in multithreading mode to simulate concurrency

Issue replication

1. Create the database using the galera-lock-replication.sql script provided in attach

2. Copy the 2 php files (galera-lock-replication.php and threads-tool.php) provided in attach in /galera-test/ folder under apache root

3. Start a first terminal with:

# php /path/to/galera-test/threads-tool.php

  • this will start bulks of 100 paralel calls to the frontend webservice in a loop:

    http://127.0.0.1/galera-test/galera-lock-replication.php
    

4. Wait few seconds (between 5 and 20)

5. Galera cluster should start refuse connections



 Comments   
Comment by Andras Marinescu [ 2021-10-21 ]

It is possible to download a nightly build or a release candidate before release next week?

Generated at Thu Feb 08 09:47:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.