Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Duplicate
-
10.11.10
-
None
-
Debian 12.9 (VMWare)
Description
Hi,
For the past week, some of our database servers randomly stop to respond.
What we observe is that the buffer pool is approaching 100% of max usage, and even simple SELECT queries hangs, the database do not respond anymore.
Connections stack up until max_connection is reached. Connections are not freed, even if the client process does not exists anymore.
MariaDB does not respond to restart command, and we need to "kill -9" for it to start working again.
It occurs multiple times a day, for multiple of our servers, in different environments, in different times of the day, with different applications, with or without load on the system (server metrics are fine in all cases), but nothing unusual. It even occurs on replica servers
We are struggling to find common behavior that could explain this issue on our side. The only common thing is that these servers were upgraded from Debian 12.8 to Debian 12.9 shortly before the issue started to appear, but all servers upgraded do not seams to have this issue.
We observe the following error in the log file :
[Warning] InnoDB: Could not free any blocks in the buffer pool! 2725888 blocks are in use and 0 free. Consider increasing innodb_buffer_pool_size.
|
Sometimes we observe this error message :
[ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
|
[ERROR] mysqld got signal 6 ;
|
Sorry, we probably made a mistake, and this is a bug.
|
|
|
Your assistance in bug reporting will enable us to fix this for the next release.
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.11.10-MariaDB-deb12-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646
|
key_buffer_size=10485760
|
read_buffer_size=131072
|
max_used_connections=201
|
max_threads=202
|
thread_count=201
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 455168 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x560707a0815e]
|
/usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x5607074d34a9]
|
2025-01-16 17:49:42 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
|
libc_sigaction.c:0(__restore_rt)[0x7f1b6a65b050]
|
nptl/pthread_kill.c:44(__pthread_kill_implementation)[0x7f1b6a6a9ebc]
|
posix/raise.c:27(__GI_raise)[0x7f1b6a65afb2]
|
stdlib/abort.c:81(__GI_abort)[0x7f1b6a645472]
|
/usr/sbin/mariadbd(+0x6c2bcd)[0x5607070ebbcd]
|
/usr/sbin/mariadbd(+0x6ba800)[0x5607070e3800]
|
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x38)[0x56070799e038]
|
/usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2f)[0x56070799ea6f]
|
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x57)[0x56070799ccc7]
|
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7f1b6a4d44a3]
|
nptl/pthread_create.c:442(start_thread)[0x7f1b6a6a81c4]
|
x86_64/clone3.S:83(clone3)[0x7f1b6a72885c]
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
(We are working on the procedure described here https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ to get the full stack trace)
If you need additional information, please ask.
Attachments
Issue Links
- duplicates
-
-
- Closed
-
- links to