Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33185

Connection timeout/slowdown upon increased load

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 11.2.2
    • None
    • None
    •  Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-91-generic x86_64)

    Description

      Once I got Server version: 11.2.2-MariaDB-1:11.2.2+maria~ubu2204-log mariadb.org binary distribution installed I have started noticing periodic reports about connection timeout to Master DB and at times connection timeout to Replica DB (1 master , 2 replicas).

      We had similar (but more severe) problem reported at MDEV-31139 where it was fixed and everything was working more or less fine for at least a couple next releases.

      Since that time we have figured out that server would recover/not hang up if we stop using it once we see connection time going above 50ms (a usual connection time is under 1ms). With 11.1.2 this was not happening so often or at least much less often to get noticed as 11.2.2 slows down drastically 10-12 times a day where we have to stop using Master (or some replica) to let it recover.

      The problem occurs when intense writing starts with about 500+ inserts per second and there are 8000+ QPS in SELECTs.

      Master server is given a number 12 here and connection time is reported 142 ms (and a switch away from it happens as replicas have 0.3ms connection time and application starts using them instead of a slow master for reading)

          "connections_info": {
              "server_id": 12,
              "visible": true,
              "writable": true,
              "readable": false,
              "master": true,
              "is_local_network": true,
              "slave_io_running": "",
              "slave_sql_running": "",
              "seconds_behind_master": 0,
              "slave_io_state": "",
              "slave_sql_state": "",
              "mysqli_exception": "",
              "last_error": "",
              "last_io_error": "",
              "last_sql_error": "",
              "connection_time": 142.21,
              "write_time": 3.8,
              "threads_connected": 144,
              "max_connections": 1200,
              "remote_clients": [
                  {
                      "server_id": 13,
                      "visible": true,
                      "writable": false,
                      "readable": true,
                      "master": false,
                      "is_local_network": true,
                      "slave_io_running": "Yes",
                      "slave_sql_running": "Yes",
                      "seconds_behind_master": 0,
                      "slave_io_state": "Waiting for master to send event",
                      "slave_sql_state": "Slave has read all relay log; waiting for more updates",
                      "mysqli_exception": "",
                      "last_error": "",
                      "last_io_error": "",
                      "last_sql_error": "",
                      "connection_time": 0.38,
                      "write_time": 0,
                      "threads_connected": 30,
                      "max_connections": 1200,
                      "remote_clients": []
                  },
                  {
                      "server_id": 11,
                      "visible": true,
                      "writable": false,
                      "readable": true,
                      "master": false,
                      "is_local_network": true,
                      "slave_io_running": "Yes",
                      "slave_sql_running": "Yes",
                      "seconds_behind_master": 0,
                      "slave_io_state": "Waiting for master to send event",
                      "slave_sql_state": "Slave has read all relay log; waiting for more updates",
                      "mysqli_exception": "",
                      "last_error": "",
                      "last_io_error": "",
                      "last_sql_error": "",
                      "connection_time": 0.31,
                      "write_time": 0,
                      "threads_connected": 19,
                      "max_connections": 1200,
                      "remote_clients": []
                  }
              ]
          }
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Erland22 Erlandas
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.