Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31956

SSD based InnoDB buffer pool extension

Details

    • Q2/2025 Development

    Description

      In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed.

      It could make sense to use this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache would be separate from the persistent storage of data files and ib_logfile0.

      Such a local cache would avoid page reads and writes on slow network or HDD storage. On the persistent store, only ib_logfile0 would be written, until a write-back is needed for other reasons.

      We can simply treat the combination of the buffer pool and the local disk as one unit. Persistent pages could be freely moved between the local disk and the buffer pool. We must keep track of which pages in this combined "virtual buffer pool" are dirty, that is, will have to be written back to the persistent storage. Any durability guarantees would only apply to the persistent storage.

      Attachments

        Issue Links

          Activity

            marko Marko Mäkelä created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            Summary SSD based InnoDB buffer pool cache SSD based InnoDB buffer pool extension
            serg Sergei Golubchik made changes -
            Description In a cloud setting, it could make sense to use ephemeral local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache would be separate from the persistent storage of data files and {{ib_logfile0}}.

            Such a local cache would avoid page reads and writes on slow network or HDD storage. On the persistent store, only {{ib_logfile0}} would be written, until [a write-back is needed for other reasons|https://smalldatum.blogspot.com/2022/10/reasons-for-writeback-with-update-in.html].

            We can simply treat the combination of the buffer pool and the local disk as one unit. Persistent pages could be freely moved between the local disk and the buffer pool. We must keep track of which pages in this combined "virtual buffer pool" are dirty, that is, will have to be written back to the persistent storage. Any durability guarantees would only apply to the persistent storage.
            In one of the practical cloud MariaDB setups, a server node accesses its datadir over the network, but also has a fast local SSD storage for temporary data. The content of such temporary storage is lost when the server container is destroyed.

            It could make sense to use this ephemeral fast local storage (SSD) as an extension of the portion of InnoDB buffer pool (DRAM) that caches persistent data pages. This cache would be separate from the persistent storage of data files and {{ib_logfile0}}.

            Such a local cache would avoid page reads and writes on slow network or HDD storage. On the persistent store, only {{ib_logfile0}} would be written, until [a write-back is needed for other reasons|https://smalldatum.blogspot.com/2022/10/reasons-for-writeback-with-update-in.html].

            We can simply treat the combination of the buffer pool and the local disk as one unit. Persistent pages could be freely moved between the local disk and the buffer pool. We must keep track of which pages in this combined "virtual buffer pool" are dirty, that is, will have to be written back to the persistent storage. Any durability guarantees would only apply to the persistent storage.
            serg Sergei Golubchik made changes -
            Assignee Marko Mäkelä [ marko ] Vladislav Lesin [ vlad.lesin ]
            serg Sergei Golubchik made changes -
            Component/s Storage Engine - InnoDB [ 10129 ]
            Component/s Storage Engine - InnoDB [ 14986 ]
            Fix Version/s 11.3 [ 28565 ]
            Fix Version/s 10.6 [ 24027 ]
            Key MENT-1723 MDEV-31956
            Affects Version/s N/A [ 23617 ]
            Project MariaDB Enterprise [ 11500 ] MariaDB Server [ 10000 ]
            serg Sergei Golubchik made changes -
            Labels SkySQL performance performance
            vlad.lesin Vladislav Lesin made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            vlad.lesin Vladislav Lesin made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.4 [ 29301 ]
            Fix Version/s 11.3 [ 28565 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.5 [ 29506 ]
            Fix Version/s 11.4 [ 29301 ]
            vlad.lesin Vladislav Lesin made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            JIraAutomate JiraAutomate made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            vlad.lesin Vladislav Lesin made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            maxmether Max Mether made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            vlad.lesin Vladislav Lesin made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.6 [ 29515 ]
            Fix Version/s 11.5 [ 29506 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 11.7 [ 29815 ]
            Fix Version/s 11.6 [ 29515 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.8 [ 29921 ]
            Fix Version/s 11.7 [ 29815 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 11.9 [ 29945 ]
            Fix Version/s 11.8 [ 29921 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 12.1 [ 29992 ]
            Fix Version/s 12.0 [ 29945 ]
            marko Marko Mäkelä made changes -
            Sprint Server 12.1 dev sprint [ 793 ]
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            serg Sergei Golubchik made changes -
            Priority Major [ 3 ] Critical [ 2 ]

            People

              vlad.lesin Vladislav Lesin
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.