Type:
Task
Priority:
Major
Resolution:
Fixed
We add new event types to support compress the binlog as follow:
QUERY_COMPRESSED_EVENT,
WRITE_ROWS_COMPRESSED_EVENT_V1,
UPDATE_ROWS_COMPRESSED_EVENT_V1,
DELETE_POWS_COMPRESSED_EVENT_V1,
WRITE_ROWS_COMPRESSED_EVENT,
UPDATE_ROWS_COMPRESSED_EVENT,
DELETE_POWS_COMPRESSED_EVENT
Of course, in mariadb 10.2, it's always the V1 version of compressed binlog row event.
We introduce two option for this feature: "log_bin_compress " and "log_bin_compress_min_len", the
former is a switch of whether the binlog should be compressed and the latter is the minimum length of
sql statement(in statement mode) or record(in row mode) that can be compressed.
In master, it can be described by the code:
if binlog_format == statement
{
if log_bin_compress == true and query_len >= log_bin_compress_min_len
create a Query_compressed_log_event;
else
create a Query_log_event;
}
if binlog_format == row
{
if log_bin_compress == true and record_len >= log_bin_compress_min_len
create a Write_rows_compressed_log_event(when INSERT)
else
create a Write_log_event(when INSERT);
}
And in slave, the compressed binlog events would be converted to the uncompressed form in IO thread, such as QUERY_COMPRESSED_EVENT convert to QUERY_EVENT.
So the SQL and worker threads can be stay unchanged. And the sql thread will be executed as fast as executing uncompressed binlog.
Now, we use zlib as compressed algrithm.
And the compressed record is
Compressed Record(3 parts)
Record Header: 1 Byte
0 Bit: Always 1, mean compressed;
1-3 Bit: Reversed, compressed algorithm。0 means zlib, It's always 0 by now.
4-7 Bit: Bytes of "Record Original Length"
Record Original Length: 1-4 Bytes
Compressed Buf: the real compressed part.
The feture can reduce 42% ~ 70% binlog capacity in our production environment.
{"report":{"fcp":1277.2000000476837,"ttfb":264.7000000476837,"pageVisibility":"visible","entityId":58510,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":32,"apdex":0.5,"journeyId":"385b420b-41e1-4408-9693-11c3d2aa9bf4","navigationType":0,"readyForUser":1356.7000000476837,"redirectCount":0,"resourceLoadedEnd":1627.4000000953674,"resourceLoadedStart":279.7000000476837,"resourceTiming":[{"duration":393.09999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":279.7000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":279.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":672.7999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":393.2999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":279.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":279.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":673.2000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":440.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":280.10000014305115,"connectEnd":280.10000014305115,"connectStart":280.10000014305115,"domainLookupEnd":280.10000014305115,"domainLookupStart":280.10000014305115,"fetchStart":280.10000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":280.10000014305115,"responseEnd":720.7000000476837,"responseStart":720.7000000476837,"secureConnectionStart":280.10000014305115},{"duration":564.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":280.2999999523163,"connectEnd":280.2999999523163,"connectStart":280.2999999523163,"domainLookupEnd":280.2999999523163,"domainLookupStart":280.2999999523163,"fetchStart":280.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":280.2999999523163,"responseEnd":844.5,"responseStart":844.5,"secureConnectionStart":280.2999999523163},{"duration":568,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":280.40000009536743,"connectEnd":280.40000009536743,"connectStart":280.40000009536743,"domainLookupEnd":280.40000009536743,"domainLookupStart":280.40000009536743,"fetchStart":280.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":280.40000009536743,"responseEnd":848.4000000953674,"responseStart":848.4000000953674,"secureConnectionStart":280.40000009536743},{"duration":568.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":280.60000014305115,"connectEnd":280.60000014305115,"connectStart":280.60000014305115,"domainLookupEnd":280.60000014305115,"domainLookupStart":280.60000014305115,"fetchStart":280.60000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":280.60000014305115,"responseEnd":848.7999999523163,"responseStart":848.7999999523163,"secureConnectionStart":280.60000014305115},{"duration":568.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":280.7999999523163,"connectEnd":280.7999999523163,"connectStart":280.7999999523163,"domainLookupEnd":280.7999999523163,"domainLookupStart":280.7999999523163,"fetchStart":280.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":280.7999999523163,"responseEnd":849.2000000476837,"responseStart":849.2000000476837,"secureConnectionStart":280.7999999523163},{"duration":618.7000000476837,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":280.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":280.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":899.6000001430511,"responseStart":0,"secureConnectionStart":0},{"duration":568.5,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":281.10000014305115,"connectEnd":281.10000014305115,"connectStart":281.10000014305115,"domainLookupEnd":281.10000014305115,"domainLookupStart":281.10000014305115,"fetchStart":281.10000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":281.10000014305115,"responseEnd":849.6000001430511,"responseStart":849.6000001430511,"secureConnectionStart":281.10000014305115},{"duration":618.8000001907349,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":281.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":281.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":900.1000001430511,"responseStart":0,"secureConnectionStart":0},{"duration":568.6000001430511,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":281.5,"connectEnd":281.5,"connectStart":281.5,"domainLookupEnd":281.5,"domainLookupStart":281.5,"fetchStart":281.5,"redirectEnd":0,"redirectStart":0,"requestStart":281.5,"responseEnd":850.1000001430511,"responseStart":850.1000001430511,"secureConnectionStart":281.5},{"duration":1146.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":282.2999999523163,"connectEnd":282.2999999523163,"connectStart":282.2999999523163,"domainLookupEnd":282.2999999523163,"domainLookupStart":282.2999999523163,"fetchStart":282.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":282.2999999523163,"responseEnd":1428.7000000476837,"responseStart":1428.7000000476837,"secureConnectionStart":282.2999999523163},{"duration":1345.1000001430511,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":282.2999999523163,"connectEnd":282.2999999523163,"connectStart":282.2999999523163,"domainLookupEnd":282.2999999523163,"domainLookupStart":282.2999999523163,"fetchStart":282.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":282.2999999523163,"responseEnd":1627.4000000953674,"responseStart":1627.4000000953674,"secureConnectionStart":282.2999999523163},{"duration":483.30000019073486,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":945.7999999523163,"connectEnd":945.7999999523163,"connectStart":945.7999999523163,"domainLookupEnd":945.7999999523163,"domainLookupStart":945.7999999523163,"fetchStart":945.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":945.7999999523163,"responseEnd":1429.1000001430511,"responseStart":1429.1000001430511,"secureConnectionStart":945.7999999523163},{"duration":384,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1268.4000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1268.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1652.4000000953674,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":123,"responseStart":265,"responseEnd":275,"domLoading":278,"domInteractive":1672,"domContentLoadedEventStart":1672,"domContentLoadedEventEnd":1751,"domComplete":2543,"loadEventStart":2543,"loadEventEnd":2544,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1631},{"name":"bigPipe.sidebar-id.end","time":1631.9000000953674},{"name":"bigPipe.activity-panel-pipe-id.start","time":1632.1000001430511},{"name":"bigPipe.activity-panel-pipe-id.end","time":1633.1000001430511},{"name":"activityTabFullyLoaded","time":1766.4000000953674}],"measures":[],"correlationId":"fb48620cc68d3c","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":91,"dbReadsTimeInMs":10,"dbConnsTimeInMs":18,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Thanks!
Pushed into 10.2.3.