Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31956

SSD based InnoDB buffer pool extension

Details

    • Server 12.1 dev sprint

    Description

      In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed.

      It could make sense to use this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache would be separate from the persistent storage of data files and ib_logfile0.

      Such a local cache would avoid page reads and writes on slow network or HDD storage. On the persistent store, only ib_logfile0 would be written, until a write-back is needed for other reasons.

      We can simply treat the combination of the buffer pool and the local disk as one unit. Persistent pages could be freely moved between the local disk and the buffer pool. We must keep track of which pages in this combined "virtual buffer pool" are dirty, that is, will have to be written back to the persistent storage. Any durability guarantees would only apply to the persistent storage.

      Attachments

        Issue Links

          Activity

            maxmether Max Mether added a comment - - edited

            Wouldn't the benefit of this local SSD only be to enlarge the size of the buffer pool?

            Another idea would be to have the REDO log on local SSD storage for fast writes but the actual data files on remote storage... ?

            maxmether Max Mether added a comment - - edited Wouldn't the benefit of this local SSD only be to enlarge the size of the buffer pool? Another idea would be to have the REDO log on local SSD storage for fast writes but the actual data files on remote storage... ?

            maxmether you still cannot treat local SSD as persistent storage in this "cloud" scenario where containers are independently moved or stopped. You need all Log writes to be flushed to the network storage device.

            jagsr123 Jags (Inactive) added a comment - maxmether you still cannot treat local SSD as persistent storage in this "cloud" scenario where containers are independently moved or stopped. You need all Log writes to be flushed to the network storage device.

            Right, it is the write-ahead log that determines what is durable. If we archived all log from the database creation to the current time, the data files could be reconstructed for any past point of time.

            marko Marko Mäkelä added a comment - Right, it is the write-ahead log that determines what is durable. If we archived all log from the database creation to the current time, the data files could be reconstructed for any past point of time.
            maxmether Max Mether added a comment -

            My thinking was that by putting the WAL on the SSD we would significantly increase the write throughput (as the log entry of every commit has to be written to disk for the transaction to be durable). Of course if we cannot guarantee that the disk is durable then that of course doesn't work.

            maxmether Max Mether added a comment - My thinking was that by putting the WAL on the SSD we would significantly increase the write throughput (as the log entry of every commit has to be written to disk for the transaction to be durable). Of course if we cannot guarantee that the disk is durable then that of course doesn't work.

            High level design.

            External buffer pool file.

            Fixed size external buffer pool file is created on server start and should be ignored during backup and recovery. The file contains plain buffer pool pages, any information about what external buffer pool page corresponds to what space and page is in-memory. We might also think about external buffer pool file encryption.

            Buffer pool integrating

            Flushed to external buffer pool file pages are represented with special objects in buffer pool page hash. The objects should have some field that distinguishes it from an in-memory block descriptor near the start. Also either separate LRU list should be used for external buffer pool to evict page from it, or existing LRU should be segmented. When some page is evicted from external buffer pool file because the file is full and the page itself is at the end of LRU list, its description is just removed from page hash, there is no need to change anything in the external buffer pool file.

            Storing and loading pages in external buffer pool file

            A page is stored in external buffer pool file when it's clean and evicted in page cleaner thread, or when it's dirty and flushed to space. Before reading a page from a space it's checked if external buffer pool file contains it, if yes, the page is read from external buffer pool file, otherwise it's read from data file.

            vlad.lesin Vladislav Lesin added a comment - High level design. External buffer pool file. Fixed size external buffer pool file is created on server start and should be ignored during backup and recovery. The file contains plain buffer pool pages, any information about what external buffer pool page corresponds to what space and page is in-memory. We might also think about external buffer pool file encryption. Buffer pool integrating Flushed to external buffer pool file pages are represented with special objects in buffer pool page hash. The objects should have some field that distinguishes it from an in-memory block descriptor near the start. Also either separate LRU list should be used for external buffer pool to evict page from it, or existing LRU should be segmented. When some page is evicted from external buffer pool file because the file is full and the page itself is at the end of LRU list, its description is just removed from page hash, there is no need to change anything in the external buffer pool file. Storing and loading pages in external buffer pool file A page is stored in external buffer pool file when it's clean and evicted in page cleaner thread, or when it's dirty and flushed to space. Before reading a page from a space it's checked if external buffer pool file contains it, if yes, the page is read from external buffer pool file, otherwise it's read from data file.

            Some update. I worked on this task during the last features development sprint. The general question I tried to solve is how to implement pages flushing to external buffer pool file and pages reading from it. As a result - I implemented the variant where write io requests to external buffer pool file are submitted along with write io requests to data files, and the page is unlocked only when both requests are completed. It passed smoke testing, but there are crashes in some mtr tests, which I plan to fix during the next features development sprint. I pushed my work in ES repository.

            vlad.lesin Vladislav Lesin added a comment - Some update. I worked on this task during the last features development sprint. The general question I tried to solve is how to implement pages flushing to external buffer pool file and pages reading from it. As a result - I implemented the variant where write io requests to external buffer pool file are submitted along with write io requests to data files, and the page is unlocked only when both requests are completed. It passed smoke testing, but there are crashes in some mtr tests, which I plan to fix during the next features development sprint. I pushed my work in ES repository.

            People

              vlad.lesin Vladislav Lesin
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.