Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11065

Compressed binary log

    XMLWordPrintable

Details

    Description

      We add new event types to support compress the binlog as follow:
      QUERY_COMPRESSED_EVENT,
      WRITE_ROWS_COMPRESSED_EVENT_V1,
      UPDATE_ROWS_COMPRESSED_EVENT_V1,
      DELETE_POWS_COMPRESSED_EVENT_V1,
      WRITE_ROWS_COMPRESSED_EVENT,
      UPDATE_ROWS_COMPRESSED_EVENT,
      DELETE_POWS_COMPRESSED_EVENT

      Of course, in mariadb 10.2, it's always the V1 version of compressed binlog row event.

      We introduce two option for this feature: "log_bin_compress " and "log_bin_compress_min_len", the
      former is a switch of whether the binlog should be compressed and the latter is the minimum length of
      sql statement(in statement mode) or record(in row mode) that can be compressed.
      In master, it can be described by the code:
      if binlog_format == statement

      { if log_bin_compress == true and query_len >= log_bin_compress_min_len create a Query_compressed_log_event; else create a Query_log_event; }

      if binlog_format == row

      { if log_bin_compress == true and record_len >= log_bin_compress_min_len create a Write_rows_compressed_log_event(when INSERT) else create a Write_log_event(when INSERT); }

      And in slave, the compressed binlog events would be converted to the uncompressed form in IO thread, such as QUERY_COMPRESSED_EVENT convert to QUERY_EVENT.
      So the SQL and worker threads can be stay unchanged. And the sql thread will be executed as fast as executing uncompressed binlog.

      Now, we use zlib as compressed algrithm.
      And the compressed record is
      Compressed Record(3 parts)
      Record Header: 1 Byte
      0 Bit: Always 1, mean compressed;
      1-3 Bit: Reversed, compressed algorithm。0 means zlib, It's always 0 by now.
      4-7 Bit: Bytes of "Record Original Length"
      Record Original Length: 1-4 Bytes
      Compressed Buf: the real compressed part.

      The feture can reduce 42% ~ 70% binlog capacity in our production environment.

      Attachments

        Issue Links

          Activity

            People

              knielsen Kristian Nielsen
              svoj Sergey Vojtovich
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.