Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40652

Preserve InnoDB buffer pool between restarts (systemd)

    XMLWordPrintable

Details

    Description

      systemd file descriptor store + memfd creates an opportunity for the entire innodb buffer pool to remain intack between restarts.

      The recovery of these pages would be much faster than full log recovery.

      Use cases are:

      • changing fixed read-only system variables/option
      • offline for maintenance
      • minor upgrades
      • OOM killed process
      • fatal assertion/crash

      The caveats around getting the mmap of a buffer pool back are:

      • This is just a memory, there's no mutex state accounted for, clean up is required
      • need to account for changed options on buffer pool size
      • need some compatibility marker that the memory is good (between upgrades of differing MariaDB versions)

      notes from discussion:

      • Some extra care would be needed to allow upgrades. We’d need some data structure version identifier, so that if an incompatible build of the server is starting up, it’d reinitialize the buffer pool.
      • We do have the block descriptors in the same contiguous virtual address range, and there is a fixed mapping between block descriptors and page frames. We’d only have to traverse the memory and populate buf_pool.page_hash based on it. Log-based crash recovery (including a full scan of the log) is actually necessary if there exist any page that was being modified by an incomplete mini-transaction and had already been modified since the latest checkpoint but not written back to the file system. We might be able to consult the buf_page_t::oldest_modification field to determine this.
      • mem resize plausible. Either: the state of the resize is discoverable, potentially with local variables in the same memory location; or just unstore the fd, and re-store post resize.
      • mysys/my_virtual_mem.c - just add the fd argument to mmap.
      • we’d have to determine also the size of the old buffer pool, in case the new server is being started with a smaller innodb_buffer_pool_size_max. Some ftruncate of the memfd might be needed as well.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              danblack Daniel Black
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 10d
                  10d
                  Remaining:
                  Remaining Estimate - 10d
                  10d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.