Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33487

Reduce the blocking time for writing binlogs of large transactions

Details

    Description

      During the commit stage of a large transaction, it may cost too much time to write binlog cache and block all subsequent transactions for a long time. One possible solution of this issue is to write the binlog of the large transaction to a temporary file and then rename it to the next new binlog file.

      We need to reserve some space for format description event, gtid list event, binlog checkpoint event and gtid event. After writing the binlog cache, we lock LOCK_log and write the aforementioned events. If the reserved space doesn't match the required size, we delete the temporary file and fallback to original commit path(Write binlog cache with LOCK_log locked).

      Attachments

        Issue Links

          Activity

            poempeng Yin Peng added a comment -

            I have created a pull request based on 374783c3d9a87170b313a05240dadaa813ec6034.
            Simple test:
            sysbench write only, 128 threads, 250 tables * 25000
            QPS changes during the commit of large transaction that generates 2GB binlog:

            non_blocking_binlog_threshold= default

            non_blocking_binlog_threshold= 256MB

            poempeng Yin Peng added a comment - I have created a pull request based on 374783c3d9a87170b313a05240dadaa813ec6034. Simple test: sysbench write only, 128 threads, 250 tables * 25000 QPS changes during the commit of large transaction that generates 2GB binlog: non_blocking_binlog_threshold= default non_blocking_binlog_threshold= 256MB
            danblack Daniel Black added a comment - For review: https://github.com/MariaDB/server/pull/3087 Sorry for delay poempeng .

            Generally the patch looks well thought out to me. I have a few minor notes that I'm holding off making, as a Zulip conversation (summarized here) mentions that knielsen may have some work which conflicts with this. So I'm re-assigning to Kristian for a design review.

            bnestere Brandon Nesterenko added a comment - Generally the patch looks well thought out to me. I have a few minor notes that I'm holding off making, as a Zulip conversation ( summarized here ) mentions that knielsen may have some work which conflicts with this. So I'm re-assigning to Kristian for a design review.
            knielsen Kristian Nielsen added a comment - Review: https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thread/6OUVSUBUUCU6AHUR4CYGYSSZZUKZ5FIZ/

            Closing as duplicate of MDEV-32014, which was added to the 11.7.0 preview.

            bnestere Brandon Nesterenko added a comment - Closing as duplicate of MDEV-32014 , which was added to the 11.7.0 preview.

            People

              bnestere Brandon Nesterenko
              poempeng Yin Peng
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.