[MXS-2710] max_connections problem Created: 2019-10-03  Updated: 2020-01-13  Resolved: 2020-01-08

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 2.3.11
Fix Version/s: 2.3.16, 2.4.6

Type: Bug Priority: Major
Reporter: Sergey Smirnov Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Problem/Incident
is caused by MXS-2645 Session client count is not always de... Closed

 Description   

After few hours any connection attempts through maxscale finaly:
ERROR 1040 (HY000): Too many connections
Direct connect without maxscale working fine.

config example:

[maxscale]
 
threads = auto
 
log_warning=1
 
log_debug=1
 
 
 
[mysql07]
 
type = server
 
protocol = MariaDBBackend
 
address = 192.168.4.106
 
port = 3307
 
 
 
[mysql08]
 
type = server
 
protocol = MariaDBBackend
 
address = 192.168.4.107
 
port = 3307
 
 
 
[Service-for-cluster_78]
 
user = maxscale
 
password = pass
 
filters = Hint
 
max_slave_replication_lag=300
 
max_sescmd_history=1500
 
prune_sescmd_history=true
 
type = service
 
router = readwritesplit
 
servers = mysql07, mysql08
 
master_accept_reads = true
 
max_connections = 800
 
 
 
[MariaDB-Monitor-for-cluster_78]
 
type = monitor
 
module = mariadbmon
 
user = replica
 
password = pass
 
monitor_interval = 5000
 
failover_timeout = 10
 
auto_failover = false
 
detect_replication_lag = true
 
handle_events = false
 
servers = mysql07, mysql08
 
 
 
[Listener-for-cluster_78]
 
address = 0.0.0.0
 
type = listener
 
service = Service-for-cluster_78
 
protocol = MariaDBClient
 
port = 3306
 
 
 
[MaxAdmin-Service]
 
type = service
 
router = cli
 
 
 
[MaxAdmin-Listener]
 
type = listener
 
service = MaxAdmin-Service
 
protocol = maxscaled
 
socket = default
 
 
 
[Hint]
 
type = filter
 
module = hintfilter

MySQL server variables:

| max_connect_errors                            | 1000            |
 
| max_connections                               | 1000            |
 
| max_user_connections                          | 800             |

MySQL server status:

| Max_used_connections                          | 193                 |
 
| Max_used_connections_time                     | 2019-10-03 06:40:06 |

maxctrl show servers "connections": 0 (Statistics section)
maxctrl show services fragment:

{
 
	"connections":500,
 
	"current_connections":1,
 
	"queries":1500,
 
	"route_master":0,
 
	"route_slave":1500,
 
	"route_all":500,
 
	"rw_transactions":0,
 
	"ro_transactions":0,
 
	"replayed_transactions":0,
 
	"server_query_statistics":[
 
		{
 
			"id":"mysql07",
 
			"total":2000,
 
			"read":2000,
 
			"write":0,
 
			"avg_sess_duration":"74.4405ms",
 
			"avg_sess_active_pct":98.50054979380471,
 
			"avg_selects_per_session":2
 
		},
 
		{
 
			"id":"mysql08",
 
			"total":500,
 
			"read":500,
 
			"write":0,
 
			"avg_sess_duration":"74.422ms",
 
			"avg_sess_active_pct":0,
 
			"avg_selects_per_session":0
 
		}
 
	]
 
}



 Comments   
Comment by markus makela [ 2019-10-04 ]

Does removing max_connections = 800 in maxscale.cnf remove the problem?

Comment by Sergey Smirnov [ 2019-10-07 ]

Yes, after remove "max_connections" from maxscale working fine.
When maxscale return "Too many connections", direct connection working normal.

Comment by markus makela [ 2019-10-07 ]

Does the output of SHOW PROCESSLIST keep growing when the parameter is not enabled? If yes, then it might be a client-side problem where the connections aren't closed properly.

Comment by Sergey Smirnov [ 2019-10-09 ]

Explain it, please. I do not understand. On task description maxctrl output with "connections:0", direct connection working fine, why it is client side problem? My status variable mean what I have 193 connection maximum
Threads_connected < Max_used_connections
Threads_connected = row count in show processlist

Comment by markus makela [ 2019-10-09 ]

So when you removed the max_connections = 800 from maxscale.cnf the value of Threads_connected was the same? If that's the case perhaps the connection count mechanism has a problem.

Comment by markus makela [ 2019-10-09 ]

I think that this will be fixed with MXS-2645.

Comment by Sergey Smirnov [ 2019-10-10 ]

I am waiting release 2.3.13. Can't find release notes / branch on github / package in repository ..

Comment by markus makela [ 2019-10-10 ]

It's not released yet.

Comment by Sergey Smirnov [ 2019-11-01 ]

I reproduce the problem on 2.3.13. I use the config from the task description. The problem is reproduced in a few hours of waiting without load.

Comment by markus makela [ 2020-01-02 ]

I managed to find a case where the connection count would only be decremented which caused integer underflow to occur. This in turn would eventually cause the whole instance to lock up once enough of these cases would occur.

Generated at Thu Feb 08 04:16:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.