[MXS-2183] Excessive memory use, but not growing endlessly Created: 2018-11-23  Updated: 2020-08-25  Resolved: 2018-11-28

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 2.2.15
Fix Version/s: 2.2.18

Type: Bug Priority: Major
Reporter: Hartmut Holzgraefe Assignee: markus makela
Resolution: Fixed Votes: 1
Labels: None

Attachments: PNG File RAM_usage.PNG    
Sprint: MXS-SPRINT-71

 Description   

After upgrade from MaxScale 2.0.4 to 2.2.15 the process size has been growing up to ~22GB (~75% of total RAM) over the course of six days (growing faster at day than at night), then remained steady at slightly below 75% for the following five days.

The previous 2.0.4 instance showed a constant RAM usage of around 10%, so ~3GB, only.



 Comments   
Comment by markus makela [ 2018-11-27 ]

Found an odd memory leak that appeared to happen when MaxScale was idle:

sending command leak_check definiteleak any to pid 24294
==24294== 699 (112 direct, 587 indirect) bytes in 1 blocks are definitely lost in loss record 1,415 of 1,634
==24294==    at 0x4C2EBAB: malloc (vg_replace_malloc.c:299)
==24294==    by 0x4EC6551: mxs_malloc (alloc.cc:32)
==24294==    by 0x4ECA509: gwbuf_clone_portion(gwbuf*, unsigned long, unsigned long) (buffer.cc:386)
==24294==    by 0x4ECA87F: gwbuf_split (buffer.cc:443)
==24294==    by 0x4F0EA90: modutil_get_next_MySQL_packet (modutil.cc:540)
==24294==    by 0xBE0C520: route_by_statement(session*, unsigned long, gwbuf**) (mysql_client.cc:1639)
==24294==    by 0xBE0AA66: gw_read_finish_processing(dcb*, gwbuf*, unsigned long) (mysql_client.cc:1151)
==24294==    by 0xBE0A803: gw_read_normal_data(dcb*, gwbuf*, int) (mysql_client.cc:1092)
==24294==    by 0xBE08E8B: gw_read_client_event(dcb*) (mysql_client.cc:539)
==24294==    by 0x4EEBDB2: dcb_process_poll_events(dcb*, unsigned int) (dcb.cc:3070)
==24294==    by 0x4EEC157: dcb_handler(dcb*, unsigned int) (dcb.cc:3155)
==24294==    by 0x4EEC204: dcb_poll_handler(mxs_poll_data*, int, unsigned int) (dcb.cc:3191)
==24294== 
==24294== 936 bytes in 39 blocks are definitely lost in loss record 1,442 of 1,634
==24294==    at 0x4C2EBAB: malloc (vg_replace_malloc.c:299)
==24294==    by 0x4EC6551: mxs_malloc (alloc.cc:32)
==24294==    by 0xA6FEAA9: server_command_init(server_command_st*, mxs_mysql_cmd_t) (mysql_common.cc:488)
==24294==    by 0xA6FEE24: protocol_add_srv_command (mysql_common.cc:598)
==24294==    by 0xB3E320A: prepare_for_write (mysql_backend.c:420)
==24294==    by 0xB3E3775: gw_read_backend_event (mysql_backend.c:539)
==24294==    by 0x4EEBDB2: dcb_process_poll_events(dcb*, unsigned int) (dcb.cc:3070)
==24294==    by 0x4EEC157: dcb_handler(dcb*, unsigned int) (dcb.cc:3155)
==24294==    by 0x4EEC204: dcb_poll_handler(mxs_poll_data*, int, unsigned int) (dcb.cc:3191)
==24294==    by 0x4F4DC18: maxscale::Worker::poll_waitevents() (worker.cc:1212)
==24294==    by 0x4F4CD17: maxscale::Worker::run() (worker.cc:892)
==24294==    by 0x409083: main (gateway.cc:2292)
==24294== 
==24294== 3,360 bytes in 140 blocks are definitely lost in loss record 1,547 of 1,634
==24294==    at 0x4C2EBAB: malloc (vg_replace_malloc.c:299)
==24294==    by 0x4EC6551: mxs_malloc (alloc.cc:32)
==24294==    by 0xA6FEAA9: server_command_init(server_command_st*, mxs_mysql_cmd_t) (mysql_common.cc:488)
==24294==    by 0xA6FEE24: protocol_add_srv_command (mysql_common.cc:598)
==24294==    by 0xB3E320A: prepare_for_write (mysql_backend.c:420)
==24294==    by 0xB3E3775: gw_read_backend_event (mysql_backend.c:539)
==24294==    by 0x4EEBDB2: dcb_process_poll_events(dcb*, unsigned int) (dcb.cc:3070)
==24294==    by 0x4EEC157: dcb_handler(dcb*, unsigned int) (dcb.cc:3155)
==24294==    by 0x4EEC204: dcb_poll_handler(mxs_poll_data*, int, unsigned int) (dcb.cc:3191)
==24294==    by 0x4F4DC18: maxscale::Worker::poll_waitevents() (worker.cc:1212)
==24294==    by 0x4F4CD17: maxscale::Worker::run() (worker.cc:892)
==24294==    by 0x4F4D799: maxscale::Worker::thread_main(void*) (worker.cc:1113)
==24294== 
==24294== 8,131 (1,568 direct, 6,563 indirect) bytes in 14 blocks are definitely lost in loss record 1,573 of 1,634
==24294==    at 0x4C2EBAB: malloc (vg_replace_malloc.c:299)
==24294==    by 0x4EC6551: mxs_malloc (alloc.cc:32)
==24294==    by 0x4EC9C0E: gwbuf_alloc (buffer.cc:65)
==24294==    by 0x4EE5529: dcb_basic_read(dcb*, int, int, int, int*) (dcb.cc:687)
==24294==    by 0x4EE51F9: dcb_read (dcb.cc:589)
==24294==    by 0xBE08D68: gw_read_client_event(dcb*) (mysql_client.cc:484)
==24294==    by 0x4EEBDB2: dcb_process_poll_events(dcb*, unsigned int) (dcb.cc:3070)
==24294==    by 0x4EEC157: dcb_handler(dcb*, unsigned int) (dcb.cc:3155)
==24294==    by 0x4EEC204: dcb_poll_handler(mxs_poll_data*, int, unsigned int) (dcb.cc:3191)
==24294==    by 0x4F4DC18: maxscale::Worker::poll_waitevents() (worker.cc:1212)
==24294==    by 0x4F4CD17: maxscale::Worker::run() (worker.cc:892)
==24294==    by 0x409083: main (gateway.cc:2292)
==24294== 
==24294== 29,049 (5,600 direct, 23,449 indirect) bytes in 50 blocks are definitely lost in loss record 1,611 of 1,634
==24294==    at 0x4C2EBAB: malloc (vg_replace_malloc.c:299)
==24294==    by 0x4EC6551: mxs_malloc (alloc.cc:32)
==24294==    by 0x4EC9C0E: gwbuf_alloc (buffer.cc:65)
==24294==    by 0x4EE5529: dcb_basic_read(dcb*, int, int, int, int*) (dcb.cc:687)
==24294==    by 0x4EE51F9: dcb_read (dcb.cc:589)
==24294==    by 0xBE08D68: gw_read_client_event(dcb*) (mysql_client.cc:484)
==24294==    by 0x4EEBDB2: dcb_process_poll_events(dcb*, unsigned int) (dcb.cc:3070)
==24294==    by 0x4EEC157: dcb_handler(dcb*, unsigned int) (dcb.cc:3155)
==24294==    by 0x4EEC204: dcb_poll_handler(mxs_poll_data*, int, unsigned int) (dcb.cc:3191)
==24294==    by 0x4F4DC18: maxscale::Worker::poll_waitevents() (worker.cc:1212)
==24294==    by 0x4F4CD17: maxscale::Worker::run() (worker.cc:892)
==24294==    by 0x4F4D799: maxscale::Worker::thread_main(void*) (worker.cc:1113)
==24294== 
==24294== LEAK SUMMARY:
==24294==    definitely lost: 11,576 bytes in 244 blocks
==24294==    indirectly lost: 30,599 bytes in 196 blocks
==24294==      possibly lost: 2,890,616 bytes in 3,008 blocks
==24294==    still reachable: 1,451,067 bytes in 11,654 blocks
==24294==         suppressed: 0 bytes in 0 blocks
==24294== Reachable blocks (those to which a pointer was found) are not shown.
==24294== To see them, add 'reachable any' args to leak_check
==24294== 

Generated at Thu Feb 08 04:12:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.