Details

      Description

      We add new event types to support compress the binlog as follow:
      QUERY_COMPRESSED_EVENT,
      WRITE_ROWS_COMPRESSED_EVENT_V1,
      UPDATE_ROWS_COMPRESSED_EVENT_V1,
      DELETE_POWS_COMPRESSED_EVENT_V1,
      WRITE_ROWS_COMPRESSED_EVENT,
      UPDATE_ROWS_COMPRESSED_EVENT,
      DELETE_POWS_COMPRESSED_EVENT

      Of course, in mariadb 10.2, it's always the V1 version of compressed binlog row event.

      We introduce two option for this feature: "log_bin_compress " and "log_bin_compress_min_len", the
      former is a switch of whether the binlog should be compressed and the latter is the minimum length of
      sql statement(in statement mode) or record(in row mode) that can be compressed.
      In master, it can be described by the code:
      if binlog_format == statement

      { if log_bin_compress == true and query_len >= log_bin_compress_min_len create a Query_compressed_log_event; else create a Query_log_event; }

      if binlog_format == row

      { if log_bin_compress == true and record_len >= log_bin_compress_min_len create a Write_rows_compressed_log_event(when INSERT) else create a Write_log_event(when INSERT); }

      And in slave, the compressed binlog events would be converted to the uncompressed form in IO thread, such as QUERY_COMPRESSED_EVENT convert to QUERY_EVENT.
      So the SQL and worker threads can be stay unchanged. And the sql thread will be executed as fast as executing uncompressed binlog.

      Now, we use zlib as compressed algrithm.
      And the compressed record is
      Compressed Record(3 parts)
      Record Header: 1 Byte
      0 Bit: Always 1, mean compressed;
      1-3 Bit: Reversed, compressed algorithm。0 means zlib, It's always 0 by now.
      4-7 Bit: Bytes of "Record Original Length"
      Record Original Length: 1-4 Bytes
      Compressed Buf: the real compressed part.

      The feture can reduce 42% ~ 70% binlog capacity in our production environment.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                knielsen Kristian Nielsen
                Reporter:
                svoj Sergey Vojtovich
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: