[MDEV-21456] TCP health checks to port 3306 quickly exceed max_connect_errors Created: 2020-01-10 Updated: 2023-01-15 Resolved: 2023-01-15 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Server |
| Affects Version/s: | 10.5.0, 10.4 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | Marcos Bjoerkelund | Assignee: | Sergei Golubchik |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | None | ||
| Environment: |
64-bit Debian 9, MariaDB 10.4.11 |
||
| Issue Links: |
|
||||||||||||||||
| Description |
|
Context We have a MariaDB Galera setup with load balancing configured to each of the MariaDB nodes in the cluster. We want to achieve a simple production-like experience. In order for the load balancer to work properly, it is using a TCP health check to port 3306 for all nodes to check if they are up, succeeding if it is open. We're using Azure's health check service for TCP ports, and in our case this is performed each 5 seconds. Description of the issue Since 10.4, each TCP health check to port 3306 is causing a warning to be added to the logs, i.e.:
This can be reproduced with the following command in the instance running the MariaDB 10.4 server:
This log disappears when setting "log_warning=1", which used to be the default value for the parameter prior to the 10.2.4 release (see "Configuring the error log verbosity"). This warning was implemented as part of We ask if you can consider increasing the required "log_warning" level for the warning to be triggered, to a value such as 3, since TCP port checks are a rather common action for a warning to appear each time. |
| Comments |
| Comment by Dominik Leiner [ 2021-08-05 ] |
|
There is a significant difference between a TCP health check and a login attempt with invalid credentials. I would like to second Marcos that it would be good to allow for warnings (log_warning=2) without having the "This connection closed normally without authentication" in the logfiles. |
| Comment by marnik [ 2022-01-24 ] |
|
We have a similar problem using a TCP health check. However, if I'm not mistaken, this not only generates a warning in the log, but it also increases some counter in the database, and it blocks the host from which the TCP check originates after max_connect_errors (https://mariadb.com/docs/reference/mdb/system-variables/max_connect_errors/), which defaults to 100 (counter is reset after successful login). We're running the connection check from the same hosts as the database interactions, and our hosts were getting blocked with the error "Host 'IP' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'" because in some cases there were too many TCP health checks without actual database interaction (hence not resetting the counter on a succesful login). Our current workaround is to set max_connect_errors to the max value (which effectively disables this feature and has some risk too) since I couldn't find a way to turn it off. |
| Comment by Daniel Black [ 2022-11-18 ] |
|
Fixed in |
| Comment by Daniel Black [ 2023-01-04 ] |
|
Reopened and with this issue I aim to fix the max_connect_errors errors marnik mentions. |
| Comment by Daniel Black [ 2023-01-05 ] |
|
Option 1: https://github.com/MariaDB/server/pull/2413 |
| Comment by Sergei Golubchik [ 2023-01-15 ] |
|
max_connect_errors was added, by design, to block hosts that disconnect pre-auth. If someone intentionally does it and doesn't want hosts to be blocked, the solution is to set max_connect_errors to the max value |