[MDEV-6300] memory leak in 5.5.37-galera Created: 2014-06-04  Updated: 2020-06-02  Resolved: 2020-06-02

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 5.5.37-galera, 5.5.39-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Daniel Black Assignee: Seppo Jaakola
Resolution: Won't Fix Votes: 0
Labels: galera
Environment:

Ubuntu 12.04.4 LTS (GNU/Linux 2.6.32-28-server x86_64)

Attachments: PNG File memory-month-g1.png     PNG File memory-month-repl.png     Text File status.txt     File valgrind.mysqld.7706     Text File variables.txt    

 Description   

package versions:
mariadb-galera-server-5.5-5.5.37+maria-1~precise
galera-25.3.5-wheezy

3 or 4 servers have a similar memory leak profile to the memory-month-g1 image attached. Variables and settings attached are from the g1 server. The dates that an increase in memory usage occurs is the same on all server. The 4th server, memory-month-repl1 is only used as read only and has significantly lower traffic and lower memory usage overall.

The restart at the beginning of the graph was an upgrade from an upgrade from 5.5.32. Looking at the year graphs it appears there was a leak there too that I hadn't noticed before.

Client has another regional database on 5.5.37-MariaDB-1~precise-log (non-galera) with similar structured databases and code usage. If this non-galera instance has a leak is much slower.

top - 09:14:36 up 172 days,  7:56,  2 users,  load average: 0.07, 0.02, 0.00
 
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
 
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
 
Mem:   7970076k total,  7873132k used,    96944k free,   126140k buffers
 
Swap:  1073144k total,     6408k used,  1066736k free,  2927628k cached
 
 
 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                           
 7637 mysql     20   0 12.7g 5.5g 1.3g S    0 72.0 505:21.12 mysqld                                                                            
13405 syslog    20   0  254m 4540 1040 S    0  0.1   3:04.63 rsyslogd                                                                          
25822 openquer  20   0 17292 1428 1072 R    0  0.0   0:00.03 top                                                                               
    1 root      20   0 24356 1556  696 S    0  0.0   0:00.56 init

I've restarted repl with no query cache and lower usage settings as it was overallocated.

There is a UAT environment with this that I can test out patches with.

Would trying the patch from MDEV-4974 be of use?

 Comments   
Comment by Daniel Black [ 2014-06-19 ]

MDEV-6300-leaks-track.sql.gz pushed to private ftp as the the output from the psergey-mdev4974-xpl1.diff patch in MDEV-4974.

graphing showed small increase of memory usage over a day on a test server. I'm not 100 I've captured this.

Happy to try other patches and test stuff out.

Comment by Nirbhay Choubey (Inactive) [ 2014-06-19 ]

I will take a look at the uploaded data. BTW, the fix for MDEV#4974 is already in mariadb-galera-5.5.37.
So, the leak could be because of some other issue.

Comment by Daniel Black [ 2014-06-20 ]

ack. going to try valgrind running on running process today.

Comment by Nirbhay Choubey (Inactive) [ 2014-07-24 ]

Valgrind report (picked from MDEV-6368).

Comment by Daniel Black [ 2014-07-24 ]

i'm not sure that valgrind report is going to help.

I did attempt to get a valgrind on a UAT instance. Unfortunately there was insufficient RAM On the UAT instance to successfully run valgrind even with innodb_buffer_size set abnormally low.

This week the UAT instance RAM has been upgraded and I'm running valgrind on the instance as I type. Apologies for the delay and lack of communication.

Comment by Nirbhay Choubey (Inactive) [ 2014-07-25 ]

Besides some seemingly acceptable valgrind errors, I am still not able to reproduce this. Also tried to populate tables from the uploaded script with some sample data but no luck so far.

Comment by Jan Lindström (Inactive) [ 2017-03-27 ]

Could you try with 5.5.54-galera and see if this repeats?

Comment by Daniel Black [ 2017-03-29 ]

Sorry I've no access to this former employeer's client any more. arjen can say if its still a problem.

MDEV-12205 has a prototype for debugging memory leaks that shouldn't be too invasive.

Comment by Ralf Gebhardt [ 2020-06-02 ]

Unsupported version of Galera Cluster

Generated at Thu Feb 08 07:10:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.