[MDEV-11811] dual master with parallel replication memory leak in write master Created: 2017-01-16  Updated: 2017-01-17  Resolved: 2017-01-17

Status: Closed
Project: MariaDB Server
Component/s: Replication
Affects Version/s: 10.0, 10.1, 10.2
Fix Version/s: 10.0.30, 10.1.21

Type: Bug Priority: Critical
Reporter: wy Assignee: Kristian Nielsen
Resolution: Fixed Votes: 0
Labels: replication
Environment:

MariaDB 10.1.8 or MariaDB 10.1.18 on debina 8

Attachments: File rpl_mdev11811.test    

 Description   

dual master with parallel replication memory leak in write master
(slave_parallel_threads=8)

master(active)<----->master(standby)

if write in master(active) frequently, master(active) use memory more and more and then physical memory give out,2G physical memory per day,and then swap give up，the OS OOM Killer killed mysqld process.

if you stop slave on master(active) before OOM Killer killed,the OS reclaim a lot of physical memory

 Comments   
Comment by wy [ 2017-01-16 ]

https://github.com/MariaDB/server/blob/74d52ded721d8d39bfb25d18a0b0b8ea8a7af884/sql/log_event.cc

Gtid_list_log_event::do_apply_event function not use free_root(thd->mem_root, MYF(MY_KEEP_PREALLOC));

i don't ensure.

Comment by Elena Stepanova [ 2017-01-16 ]

knielsen, would you want to handle this?

Comment by Kristian Nielsen [ 2017-01-16 ]

Hm, that's a good point actually.

Gtid_list_log_event::do_apply_event() is one of the few functions that run
on the active master in circular replication. And it modifies tables, so
could be using thd->mem_root. And on standby slave,
Query_log_event::do_apply_event() will run all the time to free the
thd->mem_root - but not on the active master.

I'll try to come up with a patch.

Comment by Kristian Nielsen [ 2017-01-17 ]

You are absolutely right, Gtid_list_log_event::do_apply_event() is missing free_root(), and this is causing a memory leak.

Very nice catch, thanks!

I was able to reproduce the leak with the attached (crude) mysql-test-run.pl script. It is fixed by this patch, which just adds the missing free_root():

http://lists.askmonty.org/pipermail/commits/2017-January/010445.html

Comment by Kristian Nielsen [ 2017-01-17 ]

I asked Monty to review the patch

Comment by Kristian Nielsen [ 2017-01-17 ]

Pushed to 10.0

Generated at Thu Feb 08 07:52:50 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.