Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14131

Burst in connections may show: Lost connection to MySQL server at 'handshake: reading inital communication packet', system error: 22

Details

    • Bug
    • Status: Confirmed (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.2.9, 10.1(EOL)
    • 10.2(EOL)
    • Server
    • None

    Description

      Often happens in load test

      Attachments

        Issue Links

          Activity

            anikitin Andrii Nikitin (Inactive) added a comment - - edited

            The problem is the easiest to reproduce on freshly started server, but it may happen after some uptime (the more idle server had, the easiest to reproduce) as well.

            Start server with max_connections=600 and use script with following content to generate burst of 600 simultaneous connections:

            set -e
            # make sure that all spawned processes are cleaned up
            trap "exit" INT TERM
            trap "kill 0" EXIT
             
            # clean up previous logs
            rm -f test14131*.log
             
            for i in {1..600} ; do
              ( mysql -e "select 1" &> test14131-$BASHPID.log ) &
            done
             
            sleep 1
            mysql -e 'show global status like "Aborted%"'
            echo number of erros: "$(grep 'system error: 22' test14131*.log  | wc -l)"
            

            a@UBINTI:~/env1$ m7-system2/shutdown.sh 
            a@UBINTI:~/env1$ m7-system2/startup.sh 
            171026 16:35:36 mysqld_safe Logging to '/home/a/env1/m7-system2/dt/error.log'.
            171026 16:35:36 mysqld_safe Starting mysqld daemon with databases from /home/a/env1/m7-system2/dt
            mysqld is alive
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 0
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 0
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 4     |
            +------------------+-------+
            number of errors: 46
            a@UBINTI:~/env1$ m7-system2/shutdown.sh 
            a@UBINTI:~/env1$ m7-system2/startup.sh 
            171026 16:36:12 mysqld_safe Logging to '/home/a/env1/m7-system2/dt/error.log'.
            171026 16:36:12 mysqld_safe Starting mysqld daemon with databases from /home/a/env1/m7-system2/dt
            mysqld is alive
            a@UBINTI:~/env1$ pidof bash
            5757 5030 3777
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 250
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 0
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 0
            a@UBINTI:~/env1$ m7-system2/shutdown.sh 
            a@UBINTI:~/env1$ m7-system2/startup.sh 
            171026 16:36:49 mysqld_safe Logging to '/home/a/env1/m7-system2/dt/error.log'.
            171026 16:36:49 mysqld_safe Starting mysqld daemon with databases from /home/a/env1/m7-system2/dt
            mysqld is alive
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 161
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 0
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 49
            a@UBINTI:~/env1$ bash spawn_sql.sh
            +------------------+-------+
            | Variable_name    | Value |
            +------------------+-------+
            | Aborted_clients  | 0     |
            | Aborted_connects | 0     |
            +------------------+-------+
            number of errors: 0
            

            anikitin Andrii Nikitin (Inactive) added a comment - - edited The problem is the easiest to reproduce on freshly started server, but it may happen after some uptime (the more idle server had, the easiest to reproduce) as well. Start server with max_connections=600 and use script with following content to generate burst of 600 simultaneous connections: set -e # make sure that all spawned processes are cleaned up trap "exit" INT TERM trap "kill 0" EXIT   # clean up previous logs rm -f test14131*.log   for i in {1..600} ; do ( mysql -e "select 1" &> test14131-$BASHPID.log ) & done   sleep 1 mysql -e 'show global status like "Aborted%"' echo number of erros: "$(grep 'system error: 22' test14131*.log | wc -l)" a@UBINTI:~/env1$ m7-system2/shutdown.sh a@UBINTI:~/env1$ m7-system2/startup.sh 171026 16:35:36 mysqld_safe Logging to '/home/a/env1/m7-system2/dt/error.log'. 171026 16:35:36 mysqld_safe Starting mysqld daemon with databases from /home/a/env1/m7-system2/dt mysqld is alive a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 0 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 0 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 4 | +------------------+-------+ number of errors: 46 a@UBINTI:~/env1$ m7-system2/shutdown.sh a@UBINTI:~/env1$ m7-system2/startup.sh 171026 16:36:12 mysqld_safe Logging to '/home/a/env1/m7-system2/dt/error.log'. 171026 16:36:12 mysqld_safe Starting mysqld daemon with databases from /home/a/env1/m7-system2/dt mysqld is alive a@UBINTI:~/env1$ pidof bash 5757 5030 3777 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 250 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 0 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 0 a@UBINTI:~/env1$ m7-system2/shutdown.sh a@UBINTI:~/env1$ m7-system2/startup.sh 171026 16:36:49 mysqld_safe Logging to '/home/a/env1/m7-system2/dt/error.log'. 171026 16:36:49 mysqld_safe Starting mysqld daemon with databases from /home/a/env1/m7-system2/dt mysqld is alive a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 161 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 0 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 49 a@UBINTI:~/env1$ bash spawn_sql.sh +------------------+-------+ | Variable_name | Value | +------------------+-------+ | Aborted_clients | 0 | | Aborted_connects | 0 | +------------------+-------+ number of errors: 0

            People

              serg Sergei Golubchik
              anikitin Andrii Nikitin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.