[MDEV-25411] server won't stop, seems to loop endlessly on an "aborted connection" error Created: 2021-04-13 Updated: 2021-04-15 |
|
| Status: | Open |
| Project: | MariaDB Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | Michael Lawrence | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Environment: |
mariadb server, 10.5.5 CentOS 7 |
||
| Description |
|
Hello - I have encountered a server instance that seems to no longer be able to shutdown normally. It looks to be trying to endlessly kill an aborted connection, with messages that repeat like these: 2021-04-13 10:19:17 0 [Warning] Aborted connection 5 to db: 'unconnected' user: 'unauthenticated' host: '' (KILLED) There is nothing connected to the instance when this happens, and there are no aborted connections reported in the "Aborted_connects" value from "show status", it is set to 0. This has persisted across restarts and a reboot of the server itself. The server process keeps running, but it is no longer reachable on port 3306 when it gets in to that loop. It will stay that way endlessly, it needs to be sent a "kill -9" to end the process. |
| Comments |
| Comment by Daniel Black [ 2021-04-14 ] |
|
it looks like its taking its time to perform recovery. Try increasing time out https://mariadb.com/kb/en/systemd/#configuring-the-systemd-service-timeout and including more log messages from journalctl If more time doesn't help look at including the stack trace from the running process - https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/#getting-backtraces-from-a-running-mysqld-process-with-gdb-on-linux |
| Comment by Michael Lawrence [ 2021-04-14 ] |
|
Hi Daniel, thank you for the response. We don't experience any issues with start up. This instance will start pretty quickly and work normally. It is used as a replica and the replication is working fine. Overnight it has the replicated data copied to another instance and there anonymized for use by developers. I have not seen any messages about it doing a recovery, and I set the verbosity of the error log to 9. That was when I saw the errors I posted to Jira: [Warning] Aborted connection 5 to db: 'unconnected' user: 'unauthenticated' host: '' (KILLED) Always on "connection 5". Those seemingly are happening in an infinite loop. The first time it looks to have happened was this past Saturday night and was still in that loop on Monday morning. Prior to raising the error log verbosity, all I would see was a message that the server was shutting down, and then nothing else. The mysqld process would not go away though, and it was no longer responsive on its listener port. I will check in to the stack trace process and see if I can generate one at the time the shutdown problem occurs. This has been worked around temporarily by issuing a shutdown command, waiting several minutes, and then doing a kill -9. But obviously I would prefer to not have to do that and want to know why it's stopped shutting down normally. Thanks - Mike On 4/14/21, 4:49 AM, "Daniel Black (Jira)" <jira@mariadb.org> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Daniel Black commented on MDEV-25411: it looks like its taking its time to perform recovery. Try increasing time out https://mariadb.com/kb/en/systemd/#configuring-the-systemd-service-timeout and including more log messages from journalctl If more time doesn't help look at including the stack trace from the running process - https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/#getting-backtraces-from-a-running-mysqld-process-with-gdb-on-linux > server won't stop, seems to loop endlessly on an "aborted connection" error – "NOTICE: This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. The integrity and security of this message cannot be guaranteed on the Internet. |
| Comment by Michael Lawrence [ 2021-04-14 ] |
|
Re: using gdb: "On Linux, once you have debugging symbols for your mysqld binary, you can use the gdb utility to get backtraces," If I understand that correctly, I would need to replace my current install, which is from yum, with a custom-compiled install that enables debugging symbols for the compiled binary? I can take a crack at compiling a binary for 10.5.5 but I'm not sure I will really be able to duplicate what's going on with the standard version I have from the yum packages. I'll see what happens. On 4/14/21, 4:49 AM, "Daniel Black (Jira)" <jira@mariadb.org> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Daniel Black commented on MDEV-25411: it looks like its taking its time to perform recovery. Try increasing time out https://mariadb.com/kb/en/systemd/#configuring-the-systemd-service-timeout and including more log messages from journalctl If more time doesn't help look at including the stack trace from the running process - https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/#getting-backtraces-from-a-running-mysqld-process-with-gdb-on-linux > server won't stop, seems to loop endlessly on an "aborted connection" error – "NOTICE: This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. The integrity and security of this message cannot be guaranteed on the Internet. |
| Comment by Daniel Black [ 2021-04-15 ] |
|
> If I understand that correctly, I would need to replace my current install, which is from yum, with a custom-compiled install that enables debugging symbols for the compiled binary Eek, no! Guess that kb page needs to be cleaned up. Install the already existing debuginfo package (which doesn't change the server binaries, it just provides the info to resolve stack traces). So no production impacts. |