Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13322

Cluster hangs from highly concurrent sysbench session to multiple nodes

    XMLWordPrintable

Details

    • Task
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Cannot Reproduce
    • N/A
    • Galera
    • None

    Description

      Hi,

      For this exercise I used virtualbox, 3 nodes and maxscale on 1 node. I do not think maxscale has to do with it but it is an easy way to make it run on multiple nodes. I used the training VM's from https://downloads.mariadb.com/training/courses/mariadb-enterprise-cluster/OS-Images/.

      Steps to reproduce:

      • Install a cluster. I used a 3-node cluster
      • Install maxscale using the configuration attached. I used a readconnroute router that only routes to the slave. This means that the traffic will be round-robinned between the nodes that do not have wsrep_local_index set to 1.
      • Install sysbench
      • Run sysbench to mascale

      In most cases the cluster locked up after 10 seconds, in another case it took a couple of minutes. The sysbench command used:
      sysbench --db-driver=mysql --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --mysql-user=galera --mysql-password=galera --mysql-db=test --oltp-table-size=25000 --report-interval=5 --max-requests=0 --time=300 --num-threads=5 run --mysql-host=galera4 --mysql-port=4006

      In the processlist of galera2 you will find a wsrep thread that is locked by something:
      2 | system user | | NULL | Sleep | 3317 | Update_rows_log_event::find_row(129056) | UPDATE sbtest1 SET c='21925224570-87741851440-47847350341-42585377753-88509468277-36281519091-414733 | 0.000 |

      On the other nodes you see threads waiting on COMMIT.

      I am still waiting for the error logs, is there any other information you need?

      Thanks,
      Michaël

      Attachments

        1. processlist_galera3.txt
          2 kB
        2. processlist_galera2.txt
          3 kB
        3. processlist_galera1.txt
          3 kB
        4. my.cnf_galera3.txt
          2 kB
        5. my.cnf_galera2.txt
          2 kB
        6. my.cnf_galera1.txt
          2 kB
        7. maxscale.cnf.txt
          2 kB
        8. global_vars_status_galera3.txt
          32 kB
        9. global_vars_status_galera2.txt
          32 kB
        10. global_vars_status_galera1.txt
          32 kB

        Activity

          People

            anikitin Andrii Nikitin (Inactive)
            michaeldg Michaël de groot
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.