Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29432

innodb huge pages reclaim

Details

    Description

      MDEV-25341 added some memory reclaim however large page places the kernel in a difficult predicament unless the entire page is freed. While madvise(MADV_FREE) succeeds and VirtualAlloc(MEM_RESET) doesn't explicitly prohibit large pages.

      This task serves to act as a defrag.

      In MDEV-25342 the chunk (as a collection of pages) sizes are a multiple of a large page size. If pages are re-organised to allow for a chunk to be freed that provides a defrag mechanism.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34863 RAM Usage Changed Significantly Between 10.11 Releases

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-24670 avoid OOM by linux kernel co-operative memory management

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-25341 innodb buffer pool soft decommit of memory

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-29445 reorganise innodb buffer pool (and remove buffer pool chunks)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            marko Marko Mäkelä added a comment -

            For MDEV-34863, I have a fix that depends on MDEV-29445. This fix would rewrite buf_pool_t::garbage_collect() in such a way that a large contiguous chunk of the buffer pool would be released in one call.

            Based on my testing, madvise(MADV_DONTNEED) (which is a more immediate variant of madvise(MADV_FREE)) would not release any HugeTLB pages to Linux. I tested it with both 1GiB and 2MiB pages on AMD64. That is why my proposed version of MDEV-29445 would disable the interface as well as prohibit SET GLOBAL innodb_buffer_pool_size when the server has been started with --large-pages.

            marko Marko Mäkelä added a comment - For MDEV-34863 , I have a fix that depends on MDEV-29445 . This fix would rewrite buf_pool_t::garbage_collect() in such a way that a large contiguous chunk of the buffer pool would be released in one call. Based on my testing, madvise(MADV_DONTNEED) (which is a more immediate variant of madvise(MADV_FREE) ) would not release any HugeTLB pages to Linux. I tested it with both 1GiB and 2MiB pages on AMD64. That is why my proposed version of MDEV-29445 would disable the interface as well as prohibit SET GLOBAL innodb_buffer_pool_size when the server has been started with --large-pages .
            marko Marko Mäkelä added a comment -

            Is there anything that can be done about this after MDEV-29445 was implemented? Based on my experiments, it is not possible to release large_pages memory allocations to the operating system between mmap() and munmap().

            marko Marko Mäkelä added a comment - Is there anything that can be done about this after MDEV-29445 was implemented? Based on my experiments, it is not possible to release large_pages memory allocations to the operating system between mmap() and munmap() .

            People

              danblack Daniel Black
              danblack Daniel Black
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.