[MCOL-5459] Use of jemalloc's hooks to control memory consumption Created: 2023-03-21  Updated: 2023-07-01

Status: Stalled
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: None
Fix Version/s: Icebox

Type: Task Priority: Major
Reporter: Sergey Zefirov Assignee: Sergey Zefirov
Resolution: Unresolved Votes: 1
Labels: None


 Description   

There are extent hooks for jemalloc's arena operations: https://jemalloc.net/jemalloc.3.html#arena.i.extent_hooks

These hooks allocate and deallocate space for arenas and provide further services to manage arena's space.

These are not frequently called functions, as most of jemalloc's work is done inside arenas's space. We can safely monitor size of arenas created even in a critical section.

Some more research leads to this: https://github.com/jemalloc/jemalloc/issues/1031

As one can see, we can set only partiially defined hooks. If we are not interested in the jemalloc's extents' merging other functionality, we can omit these (as they appear to not change allocated set).

Jemalloc allocates big objects with arenas too:

??opt.oversize_threshold (size_t) r-

The threshold in bytes of which requests are considered oversize. Allocation requests with greater sizes are fulfilled from a dedicated arena (automatically managed, however not within narenas), in order to reduce fragmentation by not mixing huge allocations with small ones.
??

It appears we can restrict remalloc's memory use and do it in a performant manner.

We definitely should investigate that because it can bring the stability to the program as a whole.


Generated at Thu Feb 08 02:58:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.