Debian Buster

Thanks for the report.

Could you please add high and low water marks as describe here: https://mariadb.com/kb/en/mariadb-maxscale-24-mariadb-maxscale-configuration-guide/#writeq_high_water

If there is a significant difference in the network speed between MaxScale and the servers on one hand, and MaxScale and the clients on the other, then MaxScale can end up buffering a lot of data if it can read data much faster than what it can write.

That may be the reason for your second graph, but probably not for your first.

Having water marks enabled will not have an adverse effect if MaxScale can read and write just as fast, but will prevent an excessive memory consumption if it for whatever reason can read faster than what it can write. For starters you could specify a high water mark of, say, 32 mega bytes and a low water mark of 16. Note that these are applied per connection.

jan.reges Can you say anything specific about the queries you make?

Do you use maxadmin or maxctrl (or the REST-API directly) for monitoring MaxScale? If so, what commands do you execute.

Hi Johan,

i added `writeq_high_water=32Mi` and `writeq_low_water=16Mi` into maxscale.cnf on server where was memory leak after import. I restarted MaxScale. We ran mysqldump&mysql and it didn't help - MaxScale has again allocated over 350 MB RAM. MaxScale is connected to MASTER and SLAVE DB (both MariaDB 10.3.22) through 1 GBit LAN L2 switch, so there is minimal latency. Also MaxScale client server is also connected to same switch.

I attached screenshot `db-master-slave-queries.png`, which contains query statistics from MASTER and SLAVE DB (relevant do maxscale-2.4.8-memory-leak.png). As you can see, there are just dozens of queries per second, primarily reads (selects). All servers (MaxScale, DB-MASTER and DB-SLAVE) are quite idle (load under 0.1, CPU idle, lot of free RAM, very low IO load on SSD drives). There are no complicated queries - just quite simple CRUD and few selects with joins.

In whole MariaDB there is only 1 database which is accessible by 1 user. Database contains 176 DB tables (InnoDB only) with about 25 millions records and 3,5 GB in total.

We do not yet have any form of MaxScale monitoring that would regularly call any MaxScale commands. Only occasionally I manually call commands like `maxctrl list servers`, ` maxctrl list sessions`. For now, we monitors just master-slave replication directly on master/slave MariaDB by `SHOW SLAVE STATUS` or similar queries.

If you repeatedly do the dump, does the memory keep growing?

Less than 1GB of total memory doesn't seem that much given that you load large database dumps though MaxScale. The way the memory allocation works is that memory is not released to the system unless large chunks of it are free. Does the memory usage ever grow past the 1GB mark?

Hi,

after few days allocated memory slowly decreased, so memory was no longer growing dramatically. I think, that this ticket can be closed.

Just for imagine, i attached graphs for MaxScale of 3 projects on separated servers, for last 21 days.

Project A - MaxScale 2.4.9 (RSS memory increases / decreases between 20 - 250 MB)
Project B - MaxScale 2.4.9 (RSS memory decreases slowly)
Project C - MaxScale 2.4.8 (RSS memory permamently about 350 MB). Today, we upgrade it from MaxScale 2.4.8 to MaxScale 2.4.9

350MB of memory seems like a pretty reasonable figure so it's possible that the memory growth might've been the query classifier cache warming up.

jan.reges thanks for monitoring MaxScale this long, let us know if the memory doesn't stop growing over the next few weeks or if it ever reaches the gigabyte range.

Ok, Marcus.

Thank you for your support.

Hi,

@jan.reges what did you do to fix this issue? Only upgraded to 2.4.9 or did you add also high and low water marks in your configuration?

Currently I use Maxscale 2.4.10 with readwritesplit and I have the same issue.

If I want to import a dumps of several GB (+400 tables), maxscale take more / less 4 / 5GB of RAM and once the import is done, I need to restart because the memory is not released and it starts to swap.

Hi @anthosz,

I added these two lines into maxscale.cnf:

writeq_high_water=32Mi
writeq_low_water=16Mi

After adding these rows, the memory continued to grow in the case of large imports. Memory is regularly released/freed (in context of days, not minutes/hours). Long term, our MaxScale instances allocate around 100-400 MB of memory, which is acceptable by us. Also @markus+makela wrote, that hundreds of MB is reasonable allocation.

So I can't confirm or deny that the watermark setting solved the memory problem or has any effect to reported high memory usage. At a time when problems arose, we introduced an internal rule that huge imports or restores of entire databases are performed directly over the MASTER server.

jan.reges do you have any numbers on what the peak memory usage for the MaxScale instance was? If it's less than a gigabyte I'd say it's possible that it's just normal behavior of how the memory allocator performs (it doesn't always free the memory it requests from the OS). If it's significantly larger than a gigabyte, some more details would be welcome.

Thx Jàn, I will test it

Hum I just try it and no effect on my side.

writeq_high_water=64Mi # To be aligned with max_allowed_packet

writeq_low_water=32Mi

When I import a dump of 15GB, Maxscale take 5GB RAM

Once the import is done, maxscale continue to use 5GB RAM...

Do you have an idea @Markus?