[MXS-3822] MaxScale Global Memory Use Indicator Created: 2021-10-20 Updated: 2023-03-21 Resolved: 2022-09-27 |
|
| Status: | Closed |
| Project: | MariaDB MaxScale |
| Component/s: | Core, maxctrl, REST-API |
| Affects Version/s: | None |
| Fix Version/s: | 22.08.2 |
| Type: | New Feature | Priority: | Major |
| Reporter: | Juan | Assignee: | Johan Wikman |
| Resolution: | Fixed | Votes: | 2 |
| Labels: | None | ||
| Issue Links: |
|
||||||||||||
| Sprint: | MXS-SPRINT-159, MXS-SPRINT-160, MXS-SPRINT-161, MXS-SPRINT-166, MXS-SPRINT-167 | ||||||||||||
| Description |
|
Some of our customers are frustrated by the inability to monitor global memory usage from MaxScale using any internal reporting such as status variables. Occasionally MaxScale uses a larger than expected amount of memory, be it because large transactions are in-flight or because the history of commands gets unexpectedly large, for example, or due to memory leaks in a current version of a MaxScale service or supporting tool such as SQLite. The desire is to have a global variable in MaxScale reporting on the current cumulative memory use off all MaxScale services, including the SQLite footprint, so that automated monitoring could become aware of potential problem situations and address them proactively. |
| Comments |
| Comment by Ivan Zlatoustov (Inactive) [ 2022-01-18 ] | |||||||||||||||||||||||||||||||||||||||||
|
Coordinate with monitoring team addition of the same metric to MaxScale exporter | |||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2022-07-06 ] | |||||||||||||||||||||||||||||||||||||||||
|
Some preliminary work was done on this. In C++17 the concept polymorphic memory resource is introduced. It provides a way for using different memory allocators with template types, without making the instantiated types different. What this means is that different allocators can be used with different variables of the same type, without making those variables incompatible. Further, allocators can be arranged in a hierarchy where an allocator uses another allocator as its backing store. In the case of MaxScale this could mean that there would be one global allocator and then each routing worker would have an allocator of its own that uses the global allocator as its backing store. Each session in turn would have its own allocator that uses the routing worker's allocator as its backing store. With this functionality it would basically be possible to internally track down to the last byte just how much memory a particular session is using, how much memory each routing worker is using and how much memory MaxScale as a whole is using, and make this information available via the REST-API. Unfortunately, it seems that although this functionality is part of C++ 17, the support is currently experimental on some platforms, even if the compiler on the platform claims to support C++ 17. Consequently, this feature has to be put on hold until the required functionality is fully supported on all MaxScale platforms. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Johan Wikman [ 2022-09-27 ] | |||||||||||||||||||||||||||||||||||||||||
|
The object returned by the REST-API endpoint /maxscale/threads has been extended. In the attributes object there is now a field like
where query_classifier is the memory used by the thread-specific query classifier, sessions is the memory used by the sessions handled by the thread, zombies is the memory used by connections ready to be closed but that have not yet been closed, and total is simply the sum of the previous. NOTE: Currently the numbers, especially sessions, are pretty rough, as exact numbers are too costly to obtain, and not all memory usage is included, and should only be used for monitoring how the situation evolves over time, but not as a direct measure of the amount of memory used. maxctrl show threads has been extended to display this information.
The value shown for each thread, is the memory.total value from above. A new column Total shows the sum. In addition, there is a new REST-API endpoint /maxscale/memory that only reports the memory usage for each thread and the process.
The value data.attributes.process.total provides a number for the memory usage of the MaxScale process. However, it must be stressed, again, that the value currently does not show the absolute memory usage of a thread or the process, but provides a value that can be used for following how the situation evolves over time. |