bugzillaId-736: Memory leak while doing read/write splitting

This is imported from bugzilla item: http://bugs.mariadb.com/show_bug.cgi?id=736

Yves Trudeau 2015-02-19 16:33:04 UTC
I found what seems to be a memory leak in MaxScale while doing Read/write splitting. Just running the following load:

yves@yves-desktop:~/src/$ ( echo 'drop table if exists simpleinsert; create table simpleinsert (id int not null auto_increment, val tinyint not null, primary key (id)) engine=innodb;'; while true; do echo 'rollback;set autocommit=1;set autocommit=0;insert into simpleinsert (val) values (0);commit;'; sleep 0.1; done ) | mysql -h 10.3.1.120 -u tpcc -ptpcc test

is sufficient. Below is the rate of increase of the RSS.

root@radian6-maxscale-1:~/git/MaxScale/build# while true; do ps faxu | grep maxscale |grep -v grep; sleep 15; done
root 8084 1.5 14.0 638896 106632 ? Ssl 10:55 0:22 /usr/local/skysql/maxscale/bin/maxscale -c /usr/local/skysql/maxscale-0 -f /usr/local/skysql/maxscale-0/etc/MaxScale.cnf -lfile
root 8084 1.6 15.2 648848 115872 ? Ssl 10:55 0:23 /usr/local/skysql/maxscale/bin/maxscale -c /usr/local/skysql/maxscale-0 -f /usr/local/skysql/maxscale-0/etc/MaxScale.cnf -lfile
root 8084 1.7 16.4 658936 125112 ? Ssl 10:55 0:25 /usr/local/skysql/maxscale/bin/maxscale -c /usr/local/skysql/maxscale-0 -f /usr/local/skysql/maxscale-0/etc/MaxScale.cnf -lfile
root 8084 1.7 17.6 668888 134088 ? Ssl 10:55 0:26 /usr/local/skysql/maxscale/bin/maxscale -c /usr/local/skysql/maxscale-0 -f /usr/local/skysql/maxscale-0/etc/MaxScale.cnf -lfile

Going from about 106MB to 134MB in 1min. The relevant parts of my config:

[maxscale]

threads=1

log_debug=0

log_trace=0

[MySQL Monitor]

type=monitor

module=mysqlmon

servers=server1,server2,server3

user=maxuser

passwd=maxpasswd

detect_replication_lag=1

detect_stale_master=1

monitor_interval=2000

[RW Split Router]

type=service

router=readwritesplit

servers=server1,server2,server3

router_options=slave_selection_criteria=LEAST_CURRENT_OPERATIONS

max_slave_connections=1

max_slave_replication_lag=30

use_sql_variables_in=all

user=maxuser

passwd=maxpasswd

[CLI] 

type=service 

router=cli

[RW Split Listener]

type=listener

service=RW Split Router

protocol=MySQLClient

port=4006

Something interesting is that the memory is reused, if I restart the insert script, the memory for about the time it took originally to reach the current memory usage and then it will rise again. Valgrind shows many entries like:

==8131== 25,906,560 bytes in 6,288 blocks are still reachable in loss record 160 of 162
==8131== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8131== by 0x794BB1: my_malloc (my_malloc.c:41)
==8131== by 0x78EF58: reset_root_defaults (my_alloc.c:129)
==8131== by 0x669781: THD::init_for_queries() (sql_class.cc:1345)
==8131== by 0x572C08: create_embedded_thd (lib_sql.cc:685)
==8131== by 0x192FA09D: ???
==8131== by 0x192F9E74: ???
==8131== by 0x192F9BD4: ???
==8131== by 0x190DB445: ???
==8131== by 0x190DAE5E: ???
==8131== by 0x1F3A30EB: ???
==8131== by 0x19F1EF3E: ???
==8131==
==8131== 51,662,208 bytes in 6,288 blocks are still reachable in loss record 161 of 162
==8131== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8131== by 0x794BB1: my_malloc (my_malloc.c:41)
==8131== by 0x78EF58: reset_root_defaults (my_alloc.c:129)
==8131== by 0x669767: THD::init_for_queries() (sql_class.cc:1342)
==8131== by 0x572C08: create_embedded_thd (lib_sql.cc:685)
==8131== by 0x192FA09D: ???
==8131== by 0x192F9E74: ???
==8131== by 0x192F9BD4: ???
==8131== by 0x190DB445: ???
==8131== by 0x190DAE5E: ???
==8131== by 0x1F3A30EB: ???
==8131== by 0x19F1EF3E: ???
==8131==
==8131== 115,497,984 bytes in 6,288 blocks are still reachable in loss record 162 of 162
==8131== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8131== by 0x794BB1: my_malloc (my_malloc.c:41)
==8131== by 0x572ACD: create_embedded_thd (sql_list.h:629)
==8131== by 0x192FA09D: ???
==8131== by 0x192F9E74: ???
==8131== by 0x192F9BD4: ???
==8131== by 0x190DB445: ???
==8131== by 0x190DAE5E: ???
==8131== by 0x1F3A30EB: ???
==8131== by 0x19F1EF3E: ???
==8131== by 0x19F1D1A9: ???
==8131== by 0x55CA70: process_pollq (poll.c:858) }}

Regards,

Yves