Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.0(EOL), 10.1(EOL)
-
None
-
Debian 8
Description
When MySQLdb (python-mysqldb) uses the MariaDB libmysqlclient.so.18 libraries, it seems to leak memory with each failed connection. A simple contrived loop (w/ mariadb stopped, generating client 2003 errors) leads to a few GB RSS for the associated python process in a few seconds when using the MariaDB 10.1 client packages:
#!/usr/bin/env python
|
|
|
import resource |
import time |
|
|
import MySQLdb |
|
|
print("Client version: %s" % MySQLdb.get_client_info()) |
|
|
max_allowed_rss = 512*1024 |
failed_connects = 0 |
initial_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss |
|
|
for _ in xrange(50000): |
try: |
MySQLdb.connect(user='root', host='127.0.0.1') |
except MySQLdb.Error: |
failed_connects += 1 |
|
|
print("maxrss initial was: %sKB, final is %sKB; %.2fMiB change after %d failed connections." % |
(
|
initial_rss,
|
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
|
(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - initial_rss) / 1024.0, |
failed_connects
|
)
|
)
|
The output of running this is as follows:
$ apt-get install libmysqlclient18=5.5.50-0+deb8u1
|
$ python mdev10455_mysqldb_memleak.py
|
Client version: 5.5.50
|
maxrss initial was: 8364KB, final is 10044KB; 1.64MiB change after 50000 failed connections.
|
|
|
$ apt-get install libmysqlclient18=5.6.30-1~bpo8+1
|
$ python mdev10455_mysqldb_memleak.py
|
Client version: 5.6.30
|
maxrss initial was: 8536KB, final is 10216KB; 1.64MiB change after 50000 failed connections.
|
|
|
$ apt-get install libmysqlclient18=10.1.16+maria-1~jessie
|
$ python mdev10455_mysqldb_memleak.py
|
Client version: 10.1.16-MariaDB
|
maxrss initial was: 9272KB, final is 437896KB; 418.58MiB change after 50000 failed connections.
|
|
Removing libmariadbclient18 and reverting to the libmysqlclient18 provided by either MySQL 5.5 (5.5.50-0+deb8u1) (debian/jessie main) or MySQL 5.6 (5.6.30-1~bpo8+1 from jessie-backports), the memory leak goes away and memory use stays constant over time.
I see MySQL-python does not call mysql_close() on a failed connection which seems to resolve this behavior with MariaDB - so perhaps this is an incorrect API usage bug. However the MariaDB behavior does seem to differ from the MySQL (5.5.50, 5.6.30) behavior, so I am opening this issue.
Attachments
Issue Links
- relates to
-
MDEV-10546 Make unit tests run with valgrind
-
- Open
-
-
-
- Open
-
Activity
pure C test do not leak memory, so probably python uses some preparation and then do not clean...
and BTW I did not call mysql_close()... but also did not init a new connection... actually without mysql_close() it can just leak allocated MYSQL object (who knows what that python id doing and how call mysql_init())...
I do not see obvious flows in client connection attributes initialization. So it is probably how python uses the clientlibrary. Which is a bit difficult for me to test because I can't repeat it with build libraries (it looks like python uses something different then LD_LIBRARY_PATH and LD_PRELOAD to get the library).
BTW that libmariadbclient18 is libmysqld supplied with server or connectors-C? Is it debian?
This libmariadbclient18 is libmysqld supplied I believe - not the connector/C variant. I see Debian ships what I think is an older version of the MariaDB Connector/C (libmariadb2 -> v2.0.0) and this does not exhibit the same "leaking" behavior (using the naive python script above):
# apt-get install libmariadb2
|
# LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmariadb.so.2 python mdev10455_mysqldb_memleak.py
|
Client version: 5.5.0
|
maxrss initial was: 9184KB, final is 10868KB; 1.64MiB change after 50000 failed connections.
|
Simply removing LD_PRELOAD shows different behavior:
# python mdev10455_mysqldb_memleak.py
|
Client version: 10.1.16-MariaDB
|
maxrss initial was: 9256KB, final is 437884KB; 418.58MiB change after 50000 failed connections.
|
FWIW, the following "pure C" program exhibits the same behavior. This should be similar to what MySQL-python does from reading the code. As I understand it from my reading of the source, MySQL-python allocates a struct with a "MYSQL connection" member, calls mysql_init a series of mysql_option calls and mysql_real_connect. If mysql_real_connect fails, mysql_errno / mysql_error are called, but mysql_close is not.
#include <stdio.h>
|
#include <sys/time.h>
|
#include <sys/resource.h>
|
|
|
#include "mysql.h"
|
#include "mysqld_error.h"
|
|
|
int main(int argc, char **argv) { |
int i = 0xdeadbeef; |
int connect_errors = 0, closed_connections = 0; |
struct rusage usage; |
for(i = 0;i < 50000;i++) { |
MYSQL connection, *c = NULL;
|
|
|
c = mysql_init(&connection);
|
c = mysql_real_connect(&connection, "localhost", "invalid", "invalid", "", 3306, "/var/run/mysqld/bad.sock", 0); |
if (c == NULL) { |
/* fprintf(stderr, "[%d] %s\n", mysql_errno(&connection), mysql_error(&connection)); */ |
connect_errors++;
|
}
|
/* explicitly skip mysql_close */ |
if (argc > 1) { |
mysql_close(&connection);
|
closed_connections++;
|
}
|
}
|
fprintf(stderr, "%d total connect errors (%d closed connections)\n", connect_errors, closed_connections); |
|
|
if(getrusage(RUSAGE_SELF, &usage) == 0) { |
fprintf(stderr, "max_rss: %d\n", usage.ru_maxrss); |
}
|
}
|
Here's some output from my environment - still using Debian8 w/ libmariadbclient18 + libmariadbclient-dev from 10.1.16~maria-1~jessie packages.
$ gcc mdev10455.c $(mysql_config --include --libs_r) -o mdev10455_memleak
|
|
|
$ ldd mdev10455_memleak | grep libmysqlclient
|
libmysqlclient.so.18 => /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18 (0x00007fa87e66a000)
|
|
|
|
|
# Mimic MySQL-python
|
$ ./mdev10455_memleak
|
50000 total connect errors (0 closed connections)
|
max_rss: 259172
|
|
|
# Test with calling mysql_close after each failure attempt.
|
$ ./mdev10455_memleak --close
|
50000 total connect errors (50000 closed connections)
|
max_rss: 4696
|
|
|
# Test by forcing MariaDB Connector/C without closing connections
|
$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmariadb.so.2 ./mdev10455_memleak
|
50000 total connect errors (0 closed connections)
|
max_rss: 4236
|
|
|
# Test by forcing MariaDB Connector/C with closing connections
|
$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmariadb.so.2 ./mdev10455_memleak --close
|
50000 total connect errors (50000 closed connections)
|
max_rss: 4180
|
|
|
# Test by installing mysql-5.5.50 libmysqlclient
|
$ sudo apt-get install libmysqlclient18=5.5.50-0+deb8u1
|
$ ./mdev10455_memleak
|
50000 total connect errors (0 closed connections)
|
max_rss: 4520
|
|
I also ran valgrind output here, if it's helpful at all and this does seem to indicate a leak:
$ valgrind --tool=memcheck --leak-check=full ./mdev10455_memleak
|
==984== Memcheck, a memory error detector
|
==984== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
|
==984== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
|
==984== Command: ./mdev10455_memleak
|
==984==
|
50000 total connect errors (0 closed connections)
|
max_rss: 488972
|
==984==
|
==984== HEAP SUMMARY:
|
==984== in use at exit: 434,000,392 bytes in 350,003 blocks
|
==984== total heap usage: 450,056 allocs, 100,053 frees, 1,278,805,913 bytes allocated
|
==984==
|
==984== 32 bytes in 1 blocks are possibly lost in loss record 1 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E9ABD3: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E5AC84: mysql_server_init (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E614D6: mysql_init (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400842: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 160 bytes in 1 blocks are possibly lost in loss record 7 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E99FD9: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E6CF1D: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E5AC8B: mysql_server_init (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E614D6: mysql_init (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400842: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 2,352 bytes in 14 blocks are possibly lost in loss record 10 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63AF8: mysql_options (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63F2A: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 22,456 bytes in 401 blocks are possibly lost in loss record 12 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E99D84: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63D10: mysql_options4 (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E64057: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 25,600 bytes in 400 blocks are possibly lost in loss record 13 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E99D84: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63D10: mysql_options4 (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63FD9: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 25,664 bytes in 401 blocks are possibly lost in loss record 14 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E99D84: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63D10: mysql_options4 (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E64015: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 28,944 bytes in 402 blocks are possibly lost in loss record 15 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E99D84: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63D10: mysql_options4 (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63FBB: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 29,016 bytes in 403 blocks are possibly lost in loss record 16 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E99D84: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63D10: mysql_options4 (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63FF7: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 3,232,680 bytes in 395 blocks are possibly lost in loss record 20 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E930EC: alloc_dynamic (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E97FC8: ??? (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63D77: mysql_options4 (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63FBB: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== 430,615,680 (8,396,808 direct, 422,218,872 indirect) bytes in 49,981 blocks are definitely lost in loss record 24 of 24
|
==984== at 0x4C28C20: malloc (vg_replace_malloc.c:296)
|
==984== by 0x4E9EB49: my_malloc (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63AF8: mysql_options (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x4E63F2A: mysql_real_connect (in /usr/lib/libmysqlclient.so.18.0.0)
|
==984== by 0x400877: main (in /root/mdev10455_memleak)
|
==984==
|
==984== LEAK SUMMARY:
|
==984== definitely lost: 8,396,808 bytes in 49,981 blocks
|
==984== indirectly lost: 422,218,872 bytes in 297,590 blocks
|
==984== possibly lost: 3,366,904 bytes in 2,418 blocks
|
==984== still reachable: 17,808 bytes in 14 blocks
|
==984== suppressed: 0 bytes in 0 blocks
|
==984== Reachable blocks (those to which a pointer was found) are not shown.
|
==984== To see them, rerun with: --leak-check=full --show-leak-kinds=all
|
==984==
|
==984== For counts of detected and suppressed errors, rerun with: -v
|
==984== ERROR SUMMARY: 10 errors from 10 contexts (suppressed: 0 from 0)
|
And again with libmysqlclient18_5.5.50-0+deb8u1_amd64.deb
$ valgrind --tool=memcheck --leak-check=full ./mdev10455_memleak
|
==1143== Memcheck, a memory error detector
|
==1143== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
|
==1143== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
|
==1143== Command: ./mdev10455_memleak
|
==1143==
|
50000 total connect errors (0 closed connections)
|
max_rss: 63848
|
==1143==
|
==1143== HEAP SUMMARY:
|
==1143== in use at exit: 360 bytes in 3 blocks
|
==1143== total heap usage: 100,058 allocs, 100,055 frees, 843,606,225 bytes allocated
|
==1143==
|
==1143== LEAK SUMMARY:
|
==1143== definitely lost: 0 bytes in 0 blocks
|
==1143== indirectly lost: 0 bytes in 0 blocks
|
==1143== possibly lost: 0 bytes in 0 blocks
|
==1143== still reachable: 360 bytes in 3 blocks
|
==1143== suppressed: 0 bytes in 0 blocks
|
==1143== Reachable blocks (those to which a pointer was found) are not shown.
|
==1143== To see them, rerun with: --leak-check=full --show-leak-kinds=all
|
==1143==
|
==1143== For counts of detected and suppressed errors, rerun with: -v
|
==1143== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
|
Also, as I follow my curiosity, this issue also seems to affect other libmysqlclient based connectors which seem to assume they either need not or should not call mysql_close after mysql_real_connect fails.
For another example, see this trivial perl program using DBD-mysql.
#!/usr/bin/perl
|
|
|
use DBI; |
|
|
for(my $i = 0; $i < 50000;$i++) { |
my $dbh = DBI->connect("DBI:mysql:database=test;host=localhost", "user", "pw", { 'PrintError' => 0 }); |
}
|
|
valgrind shows a similar leak report here - which goes away with either MariaDB Connector/C or the MySQL 5.5 / 5.6 libmysqlclient libraries.
support added, problem fixed, now trying to put the test into regression test test suite (if possible)...
revision-id: 66ac894c40ad089175aaf6d4922f7250c23b9b3d (mariadb-10.0.26-17-g66ac894)
parent(s): b5fb2a685b6ec67d37033b020a8145d1aac1fc93
committer: Oleksandr Byelkin
timestamp: 2016-08-11 17:50:21 +0200
message:
MDEV-10455: libmariadbclient18 + MySQL-python leaks memory on failed connections
Support of CLIENT_REMEMBER_OPTIONS and freeing options added.
—
Thanks for the report.
It started happening with the following revision in 10.0.5:
commit b803b46dac40d2417e41c778938d97951ce4ec88Author: unknown <sanja@askmonty.org>Date: Thu Oct 3 18:00:44 2013 +0300Client attributes