Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Cannot Reproduce
-
10.6.12
-
FreeBSD 12.3-RELEASE-p10 amd64 GENERIC
Virtual Machine with 22GB of RAM + 8 vCPU
Description
I already reported about "memory leaks" we have seen after upgrade from 10.3 to 10.6 in comments of bugs MDEV-30096 and MDEV-29988 but it seems there are still another one.
Our servers run with the same applications and the same workload for a long time but since we upgraded MariaDB to 10.6 we see OOM crashes too often and even if we already doubled the RAM sizes of affected VMs.
This machine has configuration where it can eat up max 17GB of memory (if this calculation is right and if there is 100 connections)
:mysql> SELECT @@key_buffer_size / 1048576,
|
-> @@query_cache_size / 1048576,
|
-> @@innodb_buffer_pool_size / 1048576,
|
-> @@innodb_log_buffer_size / 1048576,
|
-> @@max_connections,
|
-> @@read_buffer_size / 1048576,
|
-> @@read_rnd_buffer_size / 1048576,
|
-> @@sort_buffer_size / 1048576,
|
-> @@join_buffer_size / 1048576,
|
-> @@binlog_cache_size / 1048576,
|
-> @@thread_stack / 1048576,
|
-> @@tmp_table_size / 1048576,
|
-> ( @@key_buffer_size
|
-> + @@query_cache_size
|
-> + @@innodb_buffer_pool_size
|
-> + @@innodb_log_buffer_size
|
-> + @@max_connections * (
|
-> @@read_buffer_size
|
-> + @@read_rnd_buffer_size
|
-> + @@sort_buffer_size
|
-> + @@join_buffer_size
|
-> + @@binlog_cache_size
|
-> + @@thread_stack
|
-> + @@tmp_table_size )
|
-> ) / (1024 * 1024 * 1024) AS MAX_MEMORY_GB\G
|
*************************** 1. row ***************************
|
@@key_buffer_size / 1048576: 128.0000
|
@@query_cache_size / 1048576: 1.0000
|
@@innodb_buffer_pool_size / 1048576: 1024.0000
|
@@innodb_log_buffer_size / 1048576: 8.0000
|
@@max_connections: 100
|
@@read_buffer_size / 1048576: 1.0000
|
@@read_rnd_buffer_size / 1048576: 2.0000
|
@@sort_buffer_size / 1048576: 8.0000
|
@@join_buffer_size / 1048576: 2.0000
|
@@binlog_cache_size / 1048576: 1.0000
|
@@thread_stack / 1048576: 0.2852
|
@@tmp_table_size / 1048576: 150.0000
|
MAX_MEMORY_GB: 17.1773
|
(is this calculation above right?)
But even if there is no more than 5 connections (listed by SHOW PROCESSLIST or tool mytop) MariaDB eats more than 18GB and than was killed by OOM.
Top usage taken 40 seconds before OOM kill
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
|
1632 mysql 37 20 0 21G 18G select 3 44:45 62.89% /usr/local/libexec/mariadbd --defaults-extra-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/local --datadir=/var/db/mysql --plugin-dir=/usr/local/lib/mysql/plugin --wsrep_on=1 --wsrep_provider=none --wsrep_on=1 --wsrep_provider=none --log-error=/var/db/mysql/mysql.err --pid-file=/var/run/mysql/mysqld.pid --socket=/var/run/mysql/mysql.sock --port=3306
|
If I will take the maximum connections seen today = 14 * 164MB per thread + 1161 MB global it should not take more than 3.5GB!
We have graphs in monitoring where we see connections (max 14) and queries per second (49 avg, 330 max)
We see this behavior on other machines with MariaDB 10.6.12 too. The more RAM we add to the VM the more MariaDB uses and crashes often.
What is the right way to identify what in MariaDB eats so much memory and not release it? (There are no long running persistent connections but periodic tasks running from cron)
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-