[#MXS-810] Error log is too verbose

[MXS-810] Error log is too verbose Created: 2016-07-26 Updated: 2016-10-17 Resolved: 2016-10-17
Status:	Closed
Project:	MariaDB MaxScale
Component/s:	Core
Affects Version/s:	None
Fix Version/s:	2.0.0

Type:

Bug

Priority:

Minor

Reporter:

Guillaume Lefranc

Assignee:

Johan Wikman

Resolution:

Fixed

Votes:

Labels:

None

Issue Links:

Duplicate
duplicates	~~MXS-443~~	Network outages cause highly repetiti...	Closed

Description

Error log is really too verbose, resulting in hundreds of messages being written per second and flooding the error log, also causing IO spikes.

For example let's take this error where maxscale reaches max connections, it will log an error for each connection error, in my humble opinion the rate of error write should be limited.

2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Splitter Service: Refresh rate limit exceeded for load of users' table.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Splitter Service: Refresh rate limit exceeded for load of users' table.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Splitter Service: Refresh rate limit exceeded for load of users' table.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Splitter Service: Refresh rate limit exceeded for load of users' table.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Invalid authentication message from backend. Error : 42000, Msg : User 'prddun' has exceeded the 'max_user_connections' resource (current value: 1000)
2016-07-26 04:42:28 error : Splitter Service: Refresh rate limit exceeded for load of users' table.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.
2016-07-26 04:42:28 error : Backend server didn't accept authentication for user prddun.

Comments

Comment by Johan Wikman [ 2016-08-03 ]

The initial idea is that you will be able to have the following kind of entry

log_throttling = X, Y, Z

in the configuration file, where X denotes the number of times a particular error message can occur in a time window of length Y, before it is suppressed for Z milliseconds.

For instance, an entry like

log_throttling = 10, 1000, 10000

would mean that if an error occurs 10 times within 1 second, then that error will thereafter be suppressed for 10 seconds.

Comment by Johan Wikman [ 2016-08-08 ]

The symptom - the log being flooded - is handled by item ~~MXS-823~~.

This issue is now downgraded to minor since ~~MXS-823~~ will prevent the the log from being flooded. However, MaxScale should explicitly deal with max connections having been reached and do something intelligent instead of just writing to the log. That is, deal with the situation in a way that does not involve writing the same error message over and over again.

Comment by Johan Wikman [ 2016-08-11 ]

Closed and reopened to get the status from STALLED to OPEN.

The immediate problem is handled by ~~MXS-823~~.

Comment by Johan Wikman [ 2016-10-17 ]

The real problem is covered by ~~MXS-824~~.

Generated at Thu Feb 08 04:02:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.

[MXS-810] Error log is too verbose Created: 2016-07-26 Updated: 2016-10-17 Resolved: 2016-10-17