Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19514

Defer change buffer merge until pages are requested

Details

    Description

      For MDEV-14481, we must defer the change buffer merge to the moment when the secondary index leaf page is requested by a user thread.

      This would also simplify MDEV-16526, because the change buffer I/O would not have to be treated as a special case.

      The change buffer format will not be changed as part of this task. That could remain part of MDEV-11634.

      As part of this, the counter innodb_ibuf_merge_usec will be removed from information_schema.innodb_metrics.

      Attachments

        1. MDEV-19514.ods
          57 kB
        2. MDEV-19514-2.ods
          74 kB
        3. MDEV-19514-3.ods
          63 kB

        Issue Links

          blocks

          Task - A task that needs to be done. MDEV-11634 Improve the InnoDB change buffer

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-14481 Execute InnoDB crash recovery in the background

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-16526 Overhaul the InnoDB page flushing

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18724 Replace buf_block_t::mutex with more std::atomic

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22090 Change buffer is not freed after dropping or rebuilding table

          • Major - Major loss of function.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20543 Latching order violation during B-tree operations

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20805 ibuf_add_free_page() is not initializing FIL_PAGE_TYPE first

          • Major - Major loss of function.
          • Closed
          is part of

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12700 Allow innodb_read_only startup without prior slow shutdown

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-16989 InnoDB hang on crash recovery: Waited for 10 seconds for 256 pending reads

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21030 MariaDB keepts crashing on start

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23380 InnoDB reads a page from disk despite parsing MLOG_INIT_FILE_PAGE2 record

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23973 Change buffer corruption when reallocating an recently freed page

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25783 CHECK TABLE harvests InnoDB: Index 'abdcef' contains 10001 entries, should be 10000

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26464 InnoDB: Failing assertion: UT_LIST_GET_LEN(buf_pool->flush_list) == 0 during shutdown

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-27199 Require ib_logfile0 to exist unless innodb_force_recovery=6

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-27734 Set innodb_change_buffering=none by default

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34759 buf_page_get_low() is unnecessarily acquiring exclusive latch on secondary index pages

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-18698 Show InnoDB's internal background threads in SHOW ENGINE INNODB STATUS

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27153 ibdata1 file Leaking ? (errno 135) - using innodb_undo_tablespaces

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31621 Remove ibuf_read_merge_pages() call from ibuf_insert_low()

          • Major - Major loss of function.
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          Activity

            Ascending order - Click to sort in descending order
            View 14 older comments
            mdcallag Mark Callaghan added a comment -

            It isn't possible to show the perf impact of having the change buffer enabled and then disabled for a workload by using MariaDB 11, assuming that MariaDB 11 no longer supports the change buffer.

            mdcallag Mark Callaghan added a comment - It isn't possible to show the perf impact of having the change buffer enabled and then disabled for a workload by using MariaDB 11, assuming that MariaDB 11 no longer supports the change buffer.
            marko Marko Mäkelä added a comment - - edited

            mdcallag, there are 7 major versions of MariaDB Server (10.5 through 10.11) where not much has been changed with regard to the change buffer. It was disabled by default in MDEV-27734 (10.5), deprecated in MDEV-27735 (10.9), and removed 3 major releases later (MDEV-29694). Apart from MDEV-30009 and other data corruption bugs covered in my FOSDEM 2023 talk, we have at least MDEV-30134 that I will have to analyze and fix.

            marko Marko Mäkelä added a comment - - edited mdcallag , there are 7 major versions of MariaDB Server (10.5 through 10.11) where not much has been changed with regard to the change buffer. It was disabled by default in MDEV-27734 (10.5), deprecated in MDEV-27735 (10.9), and removed 3 major releases later ( MDEV-29694 ). Apart from MDEV-30009 and other data corruption bugs covered in my FOSDEM 2023 talk , we have at least MDEV-30134 that I will have to analyze and fix.
            mdcallag Mark Callaghan added a comment -

            My workload (3 secondary indexes, uniform random access, database about 8X larger than memory) is closer to a worst case for showing how bad things can get without the change buffer. Your workload (1 secondary index, database not much larger than memory) is closer to a best case. If your workload cached all, or most, secondary index leaf pages then you will obviously not see a benefit from the change buffer.

            mdcallag Mark Callaghan added a comment - My workload (3 secondary indexes, uniform random access, database about 8X larger than memory) is closer to a worst case for showing how bad things can get without the change buffer. Your workload (1 secondary index, database not much larger than memory) is closer to a best case. If your workload cached all, or most, secondary index leaf pages then you will obviously not see a benefit from the change buffer.
            mihaQ MikaH added a comment -

            Thank you Mark Callaghan for sharing your experiences related to innodb change buffering. I am playing with dataset sizes 4-8x more than available RAM. We stay on SW-level 10.5.6 until MariaDB & Codership is able to publish newer software (including wsrep) that can beat the performance and stability of the 10.5.6, and we have verified it on our own performance and stability tests. I will share our results but it takes time.

            mihaQ MikaH added a comment - Thank you Mark Callaghan for sharing your experiences related to innodb change buffering. I am playing with dataset sizes 4-8x more than available RAM. We stay on SW-level 10.5.6 until MariaDB & Codership is able to publish newer software (including wsrep) that can beat the performance and stability of the 10.5.6, and we have verified it on our own performance and stability tests. I will share our results but it takes time.
            marko Marko Mäkelä added a comment -

            mihaQ, I assume that you experienced performance regressions related to some page flushing changes in MariaDB 10.5.7. They should be mostly addressed in later releases of the 10.5 series. In the 10.6 series, as you can read in MDEV-30628, there is a regression that I am currently working on. Preliminary results related to MDEV-26055 and MDEV-26827 are very promising. Once that is tackled, I will move on to MDEV-29401.

            marko Marko Mäkelä added a comment - mihaQ , I assume that you experienced performance regressions related to some page flushing changes in MariaDB 10.5.7. They should be mostly addressed in later releases of the 10.5 series. In the 10.6 series, as you can read in MDEV-30628 , there is a regression that I am currently working on. Preliminary results related to MDEV-26055 and MDEV-26827 are very promising. Once that is tackled, I will move on to MDEV-29401 .

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.