[MXS-2259] Maxscale consumes large amounts of memory even with buffer limits set. Created: 2019-01-11 Updated: 2020-08-25 Resolved: 2019-04-09 |
|
| Status: | Closed |
| Project: | MariaDB MaxScale |
| Component/s: | Core, mariadbclient |
| Affects Version/s: | 2.3.2 |
| Fix Version/s: | 2.3.3, 2.3.6 |
| Type: | Bug | Priority: | Major |
| Reporter: | Kyle Joiner (Inactive) | Assignee: | markus makela |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Sprint: | MXS-SPRINT-75, MXS-SPRINT-76, MXS-SPRINT-77, MXS-SPRINT-78, MXS-SPRINT-79 |
| Description |
|
With the high and low water limits set Maxscale will consume more memory that allowed by the limits: With the following set a loading a mysqldump file uses 15G: |
| Comments |
| Comment by markus makela [ 2019-01-16 ] | ||||||||||||||||||
|
I wasn't able to reproduce this problem but I did find a bug that could possibly explain why it wouldn't work. | ||||||||||||||||||
| Comment by markus makela [ 2019-01-28 ] | ||||||||||||||||||
|
Wasn't fixed by 2.3.3. | ||||||||||||||||||
| Comment by markus makela [ 2019-03-04 ] | ||||||||||||||||||
|
Possibly caused by the added caching of query parsing results. Update: Testing of this hypothesis seems to point towards the query classifier but not to the query classifier cache. The following valgrind output seems to reveal at least one source of memory use:
Further testing with readconnroute seems to point towards the query classifier as the culprit. To verify this, the dump should be done via readconnroute and the memory use should be measured. | ||||||||||||||||||
| Comment by markus makela [ 2019-04-05 ] | ||||||||||||||||||
|
During the investigation into how MaxScale behaves when a large amount of data is sent through it, it was discovered that the client protocol module would perform unbounded reads from the client socket resulting in excessive use of memory. The network throttling was working correctly but by the time it was engaged, the damage was already done: everything the client read on the first go was already stored in memory. To prevent this, the client protocol module should read at most one packet (roughly 16MB worth of data) at a time and then proceed with the routing. With this change, the throttling limitation is respected more accurately and the maximum overshoot in memory usage will be limited to about 16MB. |