Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16526

Overhaul the InnoDB page flushing

Details

    Description

      The writing of modified InnoDB data pages to data files should be overhauled. See some of the comments in MDEV-15058.

      Attachments

        Issue Links

          includes

          Task - A task that needs to be done. MDEV-12227 Defer writes to the InnoDB temporary tablespace

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-23754 Replace buf_pool.flush_list with a priority queue

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-23756 Implement event-driven innodb_adaptive_flushing=OFF that ignores innodb_io_capacity

          • Major - Major loss of function.
          • Open
          is blocked by

          Task - A task that needs to be done. MDEV-15053 Reduce buf_pool_t::mutex contention

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-18115 Remove dummy tablespace for the redo log

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-19514 Defer change buffer merge until pages are requested

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23399 10.5 performance regression with IO-bound tpcc

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26827 Make page flushing even faster

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-11378 AliSQL: [Perf] Issue#23 MERGE INNODB AIO REQUEST

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-11384 AliSQL: [Feature] Issue#19 BUFFER POOL LIST SCAN OPTIMIZATION

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-15058 Remove multiple InnoDB buffer pool instances

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-16339 Upgrading to 10.1.32 shows innodb_empty_free_list_algorithm=BACKOFF as default when it should be 'LEGACY'

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-17481 mariadb service won't shutdown when it's running and the OS datetime updated backwards

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19356 Assertion 'space->free_limit == 0 || space->free_limit == free_limit'

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-21132 Remove buf_page_t::newest_modification

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-24854 Change innodb_flush_method=O_DIRECT by default

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-11916 Page compression - use smaller writes, avoid trimming/zeroing rest of the page if possible

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12226 Avoid writes of freed (garbage) pages to InnoDB data files

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13670 [Note] InnoDB: page_cleaner: 1000ms intended loop took XXXXms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Task - A task that needs to be done. MDEV-14425 Change the InnoDB redo log format to reduce write amplification

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-14550 Error log flood : "InnoDB: page_cleaner: 1000ms intended loop took N ms. The settings might not be optimal."

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15528 Avoid writing freed InnoDB pages

          • Major - Major loss of function.
          • Closed
          links to

          Web Link FOSDEM 2019: PostgreSQL vs. fsync

          Web Link Historical - InnoDB IO Performance

          Web Link MySQL Bug #94912 O_DIRECT_NO_FSYNC possible write hole

          Web Link PostgreSQL's fsync() surprise

          Web Link Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

          Activity

            Ascending order - Click to sort in descending order
            View 15 older comments
            marko Marko Mäkelä added a comment -

            It appears that XtraDB used to disable the single-page flushing when using a larger buffer pool, via the Boolean-disguised-as-enum parameter innodb_empty_free_list_algorithm=BACKOFF. The logic would sometimes fail, as reported in MDEV-16339.

            I think that we should consider removing the single-page flushing nevertheless. Perhaps there could be active signaling between the page cleaner and the buf_LRU_get_free_block(), instead of passive sleeping?

            marko Marko Mäkelä added a comment - It appears that XtraDB used to disable the single-page flushing when using a larger buffer pool, via the Boolean-disguised-as-enum parameter innodb_empty_free_list_algorithm=BACKOFF . The logic would sometimes fail, as reported in MDEV-16339 . I think that we should consider removing the single-page flushing nevertheless. Perhaps there could be active signaling between the page cleaner and the buf_LRU_get_free_block() , instead of passive sleeping?
            marko Marko Mäkelä added a comment -

            Now that most of the work has been completed in MDEV-23399 and MDEV-23855, I think that the main work to be done will be in MDEV-12227, MDEV-23756 and possibly other tickets that have been linked as related.

            Because this is mostly an umbrella task, having an estimate is not too meaningful.

            marko Marko Mäkelä added a comment - Now that most of the work has been completed in MDEV-23399 and MDEV-23855 , I think that the main work to be done will be in MDEV-12227 , MDEV-23756 and possibly other tickets that have been linked as related. Because this is mostly an umbrella task, having an estimate is not too meaningful.
            wlad Vladislav Vaintroub added a comment -

            I adjusted the priority to "Major", eventhough it is probably "minor" , for the rest of the things left there.

            wlad Vladislav Vaintroub added a comment - I adjusted the priority to "Major", eventhough it is probably "minor" , for the rest of the things left there.
            marko Marko Mäkelä added a comment -

            I think that most of this has already been done. One remaining tweak is MDEV-26827. We failed to observe any improvement with it; in fact, a small regression was observed. So, it will need some additional work.

            marko Marko Mäkelä added a comment - I think that most of this has already been done. One remaining tweak is MDEV-26827 . We failed to observe any improvement with it; in fact, a small regression was observed. So, it will need some additional work.
            marko Marko Mäkelä added a comment -

            After MDEV-26055 and MDEV-26827 have been completed, there is not much that can be improved in this area. I can only think of MDEV-11378, submitting scatter-gather write requests instead of multiple single-page write requests.

            marko Marko Mäkelä added a comment - After MDEV-26055 and MDEV-26827 have been completed, there is not much that can be improved in this area. I can only think of MDEV-11378 , submitting scatter-gather write requests instead of multiple single-page write requests.

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.