Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.1.21
-
None
-
None
-
None
-
OS: CentOS 7
Physical/Virtual: Can reproduce on both vm and hardware, but primarily on physical. 1TB memory, dual 12x core CPU.
Description
I have a 720GB database, 808GB innodb_buffer_cache, 1TB system memory and 10 tables. For all I can figure we shouldn’t be running out of memory, however, the mysqld process steadily increases in memory consumption until OOM killer rightfully claims it (we now restart it before that happens). After finding some relevant memory leak fixes released in 10.1.21 we upgraded to that from 10.1.13, but unfortunately it didn’t help.
Running mysqltuner.pl (https://github.com/major/MySQLTuner-perl) seems to agree that we shouldn’t be using more than 809 GB.
|
mysqltuner snippet |
[--] Total buffers: 806.4G global + 2.9M per thread (1000 max threads)
|
[OK] Maximum reached memory usage: 808.2G (80.21% of installed RAM)
|
[OK] Maximum possible memory usage: 809.2G (80.31% of installed RAM)
|
At the time this data was gathered, the mysqld process was at 980GB. (gathered with `ps -eo pid,comm,rss`), leaving the available system memory at 17GB.
I can consistently reproduce this with similar workload on a scaled down test instance.
I have attached a binlog from my test instance, show variables, and a few grafana graphs.
Thanks for taking a look!
