Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-5459

Use of jemalloc's hooks to control memory consumption

    XMLWordPrintable

Details

    • Task
    • Status: Stalled (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • Icebox
    • None
    • None

    Description

      There are extent hooks for jemalloc's arena operations: https://jemalloc.net/jemalloc.3.html#arena.i.extent_hooks

      These hooks allocate and deallocate space for arenas and provide further services to manage arena's space.

      These are not frequently called functions, as most of jemalloc's work is done inside arenas's space. We can safely monitor size of arenas created even in a critical section.

      Some more research leads to this: https://github.com/jemalloc/jemalloc/issues/1031

      As one can see, we can set only partiially defined hooks. If we are not interested in the jemalloc's extents' merging other functionality, we can omit these (as they appear to not change allocated set).

      Jemalloc allocates big objects with arenas too:

      ??opt.oversize_threshold (size_t) r-

      The threshold in bytes of which requests are considered oversize. Allocation requests with greater sizes are fulfilled from a dedicated arena (automatically managed, however not within narenas), in order to reduce fragmentation by not mixing huge allocations with small ones.
      ??

      It appears we can restrict remalloc's memory use and do it in a performant manner.

      We definitely should investigate that because it can bring the stability to the program as a whole.

      Attachments

        Activity

          People

            sergey.zefirov Sergey Zefirov
            sergey.zefirov Sergey Zefirov
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.