[MCOL-5459] Use of jemalloc's hooks to control memory consumption Created: 2023-03-21 Updated: 2023-07-01 |
|
| Status: | Stalled |
| Project: | MariaDB ColumnStore |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | Icebox |
| Type: | Task | Priority: | Major |
| Reporter: | Sergey Zefirov | Assignee: | Sergey Zefirov |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | None | ||
| Description |
|
There are extent hooks for jemalloc's arena operations: https://jemalloc.net/jemalloc.3.html#arena.i.extent_hooks These hooks allocate and deallocate space for arenas and provide further services to manage arena's space. These are not frequently called functions, as most of jemalloc's work is done inside arenas's space. We can safely monitor size of arenas created even in a critical section. Some more research leads to this: https://github.com/jemalloc/jemalloc/issues/1031 As one can see, we can set only partiially defined hooks. If we are not interested in the jemalloc's extents' merging other functionality, we can omit these (as they appear to not change allocated set). Jemalloc allocates big objects with arenas too: ??opt.oversize_threshold (size_t) r- The threshold in bytes of which requests are considered oversize. Allocation requests with greater sizes are fulfilled from a dedicated arena (automatically managed, however not within narenas), in order to reduce fragmentation by not mixing huge allocations with small ones. It appears we can restrict remalloc's memory use and do it in a performant manner. We definitely should investigate that because it can bring the stability to the program as a whole. |