Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.0.17
-
None
-
FreeBSD 10.1-RELEASE-p7 / x86-64 / MariaDB 10.0.17
Description
I have a production MariaDB server which is occasionally experiencing a hang; it refuses new connections, does not create any more threads, and stays at 0% cpu until forced to restart.
I'm using threadpool:
thread_handling=pool-of-threads
thread_pool_size=48
thread_pool_max_threads=128
thread_pool_idle_timeout=30
and xtrabackup. The issue always occurs around the same time of day (once every week or two) so I suspect it may be related to xtrabackup's pausing the server to backup.
Captured a gdb backtrace in this state, but imposed a workaround of restarting the process rather than try to debug further, since the server is being used.
Attachments
Activity
astrange, any news? Did this ever occur again? As I said, thread_pool_max_threads is too low, this makes, the chances of deadlock higher, at least theoretically, but in this case there would be some output in the error log.
The issue has not occurred since I increased max threads to 1000. I moved the xtrabackup to a separate replicated server and will check it hasn't been seeing any problems shortly. I'll close this if it hasn't.
astrange, any news. It's been a while since last comment. Did it ever occur since?
Rarely. It doesn't track xtrabackup, but does track other scheduled backups on the server that don't touch the database. I think this is a FreeBSD issue… harder to debug one of those of course.
This can be closed.
I'd say the max threads is set too low, perhaps 1000 or so should be about right.
From the stacks provided it is hard to guess what happened (the Nov gdb dump it looks like threadpool has nothing to do because clients would be idle, and May dump shows one thread that is active still)
If you are still able to login with a new user, show processlist output would be a nice to have.