[MDEV-11065] Compressed binary log Created: 2016-10-15  Updated: 2017-01-22  Resolved: 2016-11-03

Status: Closed
Project: MariaDB Server
Component/s: Replication
Fix Version/s: 10.2.3

Type: Task Priority: Major
Reporter: Sergey Vojtovich Assignee: Kristian Nielsen
Resolution: Fixed Votes: 1
Labels: contribution, foundation


 Description   

We add new event types to support compress the binlog as follow:
QUERY_COMPRESSED_EVENT,
WRITE_ROWS_COMPRESSED_EVENT_V1,
UPDATE_ROWS_COMPRESSED_EVENT_V1,
DELETE_POWS_COMPRESSED_EVENT_V1,
WRITE_ROWS_COMPRESSED_EVENT,
UPDATE_ROWS_COMPRESSED_EVENT,
DELETE_POWS_COMPRESSED_EVENT

Of course, in mariadb 10.2, it's always the V1 version of compressed binlog row event.

We introduce two option for this feature: "log_bin_compress " and "log_bin_compress_min_len", the
former is a switch of whether the binlog should be compressed and the latter is the minimum length of
sql statement(in statement mode) or record(in row mode) that can be compressed.
In master, it can be described by the code:
if binlog_format == statement

{ if log_bin_compress == true and query_len >= log_bin_compress_min_len create a Query_compressed_log_event; else create a Query_log_event; }

if binlog_format == row

{ if log_bin_compress == true and record_len >= log_bin_compress_min_len create a Write_rows_compressed_log_event(when INSERT) else create a Write_log_event(when INSERT); }

And in slave, the compressed binlog events would be converted to the uncompressed form in IO thread, such as QUERY_COMPRESSED_EVENT convert to QUERY_EVENT.
So the SQL and worker threads can be stay unchanged. And the sql thread will be executed as fast as executing uncompressed binlog.

Now, we use zlib as compressed algrithm.
And the compressed record is
Compressed Record(3 parts)
Record Header: 1 Byte
0 Bit: Always 1, mean compressed;
1-3 Bit: Reversed, compressed algorithm。0 means zlib, It's always 0 by now.
4-7 Bit: Bytes of "Record Original Length"
Record Original Length: 1-4 Bytes
Compressed Buf: the real compressed part.

The feture can reduce 42% ~ 70% binlog capacity in our production environment.



 Comments   
Comment by Kristian Nielsen [ 2016-11-03 ]

Thanks!
Pushed into 10.2.3.

Comment by Elena Stepanova [ 2016-11-05 ]

Apparently the link to the pull request is wrong, should be GitHub Pull Request #247

Generated at Thu Feb 08 07:47:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.