[MXS-1109] Routing many prepares statements gives "#HY000Lost connection to backend server" Created: 2017-01-27 Updated: 2017-06-07 Resolved: 2017-03-14 |
|
| Status: | Closed |
| Project: | MariaDB MaxScale |
| Component/s: | mariadbbackend |
| Affects Version/s: | 2.0.3 |
| Fix Version/s: | 2.1.1 |
| Type: | Bug | Priority: | Major |
| Reporter: | Johan Nilsson | Assignee: | markus makela |
| Resolution: | Fixed | Votes: | 2 |
| Labels: | None | ||
| Environment: |
Linux CentOS 6.8, MySQL 5.6.19, php 5.3.3 |
||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Sprint: | 2017-27, 2017-28, 2017-29 | ||||||||
| Description |
|
We have in production one max scale, routing and connection pooling queries between one web portal and a mysql server. When running multiple prepare statements in parallell on the web-portal, maxscale closes the client connection with "#HY000Lost connection to backend server". |
| Comments |
| Comment by markus makela [ 2017-01-30 ] | ||||||
|
Can you upload the maxscale.cnf and the error logs, both with all the sensitive information removed? Also, if possible, please give an example set of SQL commands that causes this problem. | ||||||
| Comment by Johan Nilsson [ 2017-01-31 ] | ||||||
|
Uploaded maxscale.cnf and the max scale-logfile with info and debug enabled. Also, uploaded a tcpdump-file containing two tcp-streams, one for webserver < The statements where we see the problem is prepared statement-calls, like | ||||||
| Comment by markus makela [ 2017-02-06 ] | ||||||
|
I've been looking at the TCP dump and it does seem like the client on port 27313 is executing a prepared statement while MaxScale is still streaming a resultset back to the client. That in itself is not in any way wrong or unexpected but it might be a clue as to why it might not work. What sort of a connector are you using? | ||||||
| Comment by Johan Nilsson [ 2017-02-06 ] | ||||||
|
The setup when the trace was taken: So, the MaxScale installed on the web-server is used to pool connections towards the second MaxScale, which is used to be able to easy switch primary database backend server. We have tried to exclude either of the MaxScale-instances, but without any reduction in the frequency of session disconnects. | ||||||
| Comment by markus makela [ 2017-02-08 ] | ||||||
|
One thing that would be good to rule out is the use of persistent connections. If you remove both the persistpoolmax and persistmaxtime parameters from all servers, do the disconnections still happen? | ||||||
| Comment by Johan Nilsson [ 2017-02-08 ] | ||||||
|
Removed persistpoolmax and persistmaxtime from both instances, and the problem disappeared. | ||||||
| Comment by markus makela [ 2017-02-14 ] | ||||||
|
I can confirm that this is caused by the persistent connections. This only happens with the 2.0 version of MaxScale and the upcoming 2.1 version of MaxScale has fixed this. I'll investigate why if fails with 2.0 and if a fix for the 2.0 version can be created. | ||||||
| Comment by markus makela [ 2017-02-14 ] | ||||||
|
I've managed to pinpoint the problem to the use of prepared statements with persistent connections. With normal queries, the persistent connections work. When the text protocol query is switched to a binary protocol prepared statement, the errors again appear. | ||||||
| Comment by markus makela [ 2017-02-16 ] | ||||||
|
Can you try if the 2.1 beta release of MaxScale fixes this problem? | ||||||
| Comment by Johan Nilsson [ 2017-02-17 ] | ||||||
|
We installed the 2.1 beta-release, but were unable to test. tcp-dumping showed traffic from maxscale to mysql. So it almost seems like there's some incompatibility with php in 2.1.0-beta... | ||||||
| Comment by markus makela [ 2017-02-17 ] | ||||||
|
Does the MaxScale error log provide any information as to why it fails? | ||||||
| Comment by Johan Nilsson [ 2017-02-17 ] | ||||||
|
Unfortunately not. It only states when connection is opened/closed | ||||||
| Comment by markus makela [ 2017-02-27 ] | ||||||
|
Is the /var/lib/mysql/mysql.sock configured in maxscale.cnf and is the file created when MaxScale starts? Stopping MaxScale, removing any stale sockets and restarting MaxScale should help solve any problems caused by old socket files. | ||||||
| Comment by Johan Nilsson [ 2017-03-01 ] | ||||||
|
Yes, the socket in configured in maxscale.cnf, and created when MaxScale starts. | ||||||
| Comment by markus makela [ 2017-03-02 ] | ||||||
|
In the configuration you posted in the issue, the socket is defined with a different name:
Is this still configured the same? If so, please change it to /var/lib/mysql/mysql.sock. | ||||||
| Comment by Johan Nilsson [ 2017-03-02 ] | ||||||
|
Well, the socket is only used on the web-frontends, and I believe that I posted the configuration for the central server (that isn't recieving any traffic on the socket) | ||||||
| Comment by markus makela [ 2017-03-14 ] | ||||||
|
This seems to have been fixed by the 2.1 version of MaxScale. |