After upgrading to 10.1.20, replication now intermittently stops (randomly and not all slaves of a master at the same time) with the following errors:
2017-10-01 20:38:20 140079966804736 [ERROR] Master 'trades': Error reading packet from server: bogus data in log event; the first event 'mysql-bin.038184' at 156161357, the last event read from 'mysql-bin.038185' at 60018003, the last byte read from 'mysql-bin.038185' at 60018022. (server_errno=1236)
2017-10-01 20:38:20 140079966804736 [ERROR] Master 'trades': Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'bogus data in log event; the first event 'mysql-bin.038184' at 156161357, the last event read from 'mysql-bin.038185' at 60018003, the last byte read from 'mysql-bin.038185' at 60018022.', Internal MariaDB error code: 1236
We can fix this by issuing a START SLAVE and everything works without issue but the regular replication failures occurs.
It looks like upstream bug : https://bugs.mysql.com/bug.php?id=84752
With setting slave_compressed_protocol=1 and change the sync_binlog value to off, it's working fine. This is not easily reproducible. As per the comment in upstream bug, able to reproduce only with high load on master server and multiple slaves (like 5 or 6)
relates to
MDEV-19689
Replication I/O thread stops daily: "bogus data in log event"
Closed
links to
Elena Stepanova
made changes -
2017-10-06 10:02
Field
Original Value
New Value
Labels
upstream
Elena Stepanova
made changes -
2017-10-07 12:57
Fix Version/s
10.1
[ 16100
]
Assignee
Andrei Elkin
[ elkin
]
Sergei Golubchik
made changes -
2017-11-04 08:22
Description
After upgrading to 10.1.20, replication now intermittently stops (randomly and not all slaves of a master at the same time) with the following errors:
2017-10-01 20:38:20 140079966804736 [ERROR] Master 'trades': Error reading packet from server: bogus data in log event; the first event 'mysql-bin.038184' at 156161357, the last event read from 'mysql-bin.038185' at 60018003, the last byte read from 'mysql-bin.038185' at 60018022. (server_errno=1236)
2017-10-01 20:38:20 140079966804736 [ERROR] Master 'trades': Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'bogus data in log event; the first event 'mysql-bin.038184' at 156161357, the last event read from 'mysql-bin.038185' at 60018003, the last byte read from 'mysql-bin.038185' at 60018022.', Internal MariaDB error code: 1236
We can fix this by issuing a START SLAVE and everything works without issue but the regular replication failures occurs.
It looks like upstream bug : https://bugs.mysql.com/bug.php?id=84752
With setting slave_compressed_protocol=1 and change the sync_binlog value to off, it's working fine. This is not easily reproducible. As per the comment in upstream bug, able to reproduce only with high load on master server and multiple slaves (like 5 or 6)
After upgrading to 10.1.20, replication now intermittently stops (randomly and not all slaves of a master at the same time) with the following errors:
{noformat}
2017-10-01 20:38:20 140079966804736 [ERROR] Master 'trades': Error reading packet from server: bogus data in log event; the first event 'mysql-bin.038184' at 156161357, the last event read from 'mysql-bin.038185' at 60018003, the last byte read from 'mysql-bin.038185' at 60018022. (server_errno=1236)
2017-10-01 20:38:20 140079966804736 [ERROR] Master 'trades': Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'bogus data in log event; the first event 'mysql-bin.038184' at 156161357, the last event read from 'mysql-bin.038185' at 60018003, the last byte read from 'mysql-bin.038185' at 60018022.', Internal MariaDB error code: 1236
{noformat}
We can fix this by issuing a START SLAVE and everything works without issue but the regular replication failures occurs.
It looks like upstream bug : https://bugs.mysql.com/bug.php?id=84752
With setting slave_compressed_protocol=1 and change the sync_binlog value to off, it's working fine. This is not easily reproducible. As per the comment in upstream bug, able to reproduce only with high load on master server and multiple slaves (like 5 or 6)
Sergei Golubchik
made changes -
2017-11-04 08:23
Remote Link
This issue links to "Bug #84752 Multi-Slave Replication Fail: bogus data in log event (Web Link)"
[ 28201
]
Sergei Golubchik
made changes -
2017-11-29 15:33
Priority
Major
[ 3
]
Critical
[ 2
]
Andrei Elkin
made changes -
2018-02-02 18:16
Status
Open
[ 1
]
In Progress
[ 3
]
Andrei Elkin
made changes -
2018-02-07 12:35
Sprint
10.1.30
[ 215
]
10.1.30, 10.3.5-1
[ 215, 229
]
Sergei Golubchik
made changes -
2018-03-06 08:29
Sprint
10.1.30, 10.3.5-1
[ 215, 229
]
Andrei Elkin
made changes -
2018-03-14 17:38
Sprint
10.2.14
[ 229
]
10.2.14, 10.1.32
[ 229, 235
]
Andrei Elkin
made changes -
2018-03-19 14:09
Assignee
Andrei Elkin
[ elkin
]
Sergei Golubchik
[ serg
]
Status
In Progress
[ 3
]
In Review
[ 10002
]
Sergei Golubchik
made changes -
2018-04-06 09:55
Assignee
Sergei Golubchik
[ serg
]
Andrei Elkin
[ elkin
]
Status
In Review
[ 10002
]
Stalled
[ 10000
]
Andrei Elkin
made changes -
2018-04-11 09:42
Assignee
Andrei Elkin
[ elkin
]
Sergei Golubchik
[ serg
]
Status
Stalled
[ 10000
]
In Review
[ 10002
]
Sergei Golubchik
made changes -
2018-06-11 22:48
Assignee
Sergei Golubchik
[ serg
]
Andrei Elkin
[ elkin
]
Status
In Review
[ 10002
]
Stalled
[ 10000
]
Andrei Elkin
made changes -
2018-06-12 17:57
Status
Stalled
[ 10000
]
In Progress
[ 3
]
Andrei Elkin
made changes -
2018-06-12 17:58
Assignee
Andrei Elkin
[ elkin
]
Sergei Golubchik
[ serg
]
Status
In Progress
[ 3
]
In Review
[ 10002
]
Andrei Elkin
made changes -
2018-06-13 15:36
Assignee
Sergei Golubchik
[ serg
]
Andrei Elkin
[ elkin
]
Andrei Elkin
made changes -
2018-06-13 15:37
Status
In Review
[ 10002
]
Stalled
[ 10000
]
Julien Fritsch
made changes -
2018-06-14 08:55
Assignee
Andrei Elkin
[ elkin
]
Sergei Golubchik
[ serg
]
Sergei Golubchik
made changes -
2018-06-18 16:30
Labels
upstream
need_feedback upstream
Sergei Golubchik
made changes -
2018-07-01 18:53
Fix Version/s
10.1.35
[ 23116
]
Fix Version/s
10.2.17
[ 23111
]
Fix Version/s
10.3.8
[ 23113
]
Fix Version/s
10.2
[ 14601
]
Fix Version/s
10.1
[ 16100
]
Resolution
Fixed
[ 1
]
Status
Stalled
[ 10000
]
Closed
[ 6
]
Julien Fritsch
made changes -
2020-12-08 15:38
Labels
need_feedback upstream
upstream
Sergei Golubchik
made changes -
2021-12-06 21:45
Workflow
MariaDB v3
[ 82945
]
MariaDB v4
[ 152947
]
{"report":{"fcp":1414.2000000476837,"ttfb":390,"pageVisibility":"visible","entityId":63505,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"af5f631a-b7b8-4137-b305-e3d73e741f59","navigationType":0,"readyForUser":1483.2000000476837,"redirectCount":0,"resourceLoadedEnd":2128.3000000715256,"resourceLoadedStart":395.10000002384186,"resourceTiming":[{"duration":544.8999999761581,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":395.10000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":395.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":940,"responseStart":0,"secureConnectionStart":0},{"duration":545,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":395.3000000715256,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":395.3000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":940.3000000715256,"responseStart":0,"secureConnectionStart":0},{"duration":553.8000000715256,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":395.5,"connectEnd":395.5,"connectStart":395.5,"domainLookupEnd":395.5,"domainLookupStart":395.5,"fetchStart":395.5,"redirectEnd":0,"redirectStart":0,"requestStart":395.5,"responseEnd":949.3000000715256,"responseStart":949.3000000715256,"secureConnectionStart":395.5},{"duration":596.3999999761581,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":395.7000000476837,"connectEnd":395.7000000476837,"connectStart":395.7000000476837,"domainLookupEnd":395.7000000476837,"domainLookupStart":395.7000000476837,"fetchStart":395.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":395.7000000476837,"responseEnd":992.1000000238419,"responseStart":992.1000000238419,"secureConnectionStart":395.7000000476837},{"duration":599.8999999761581,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":395.90000009536743,"connectEnd":395.90000009536743,"connectStart":395.90000009536743,"domainLookupEnd":395.90000009536743,"domainLookupStart":395.90000009536743,"fetchStart":395.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":395.90000009536743,"responseEnd":995.8000000715256,"responseStart":995.8000000715256,"secureConnectionStart":395.90000009536743},{"duration":600.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":396,"connectEnd":396,"connectStart":396,"domainLookupEnd":396,"domainLookupStart":396,"fetchStart":396,"redirectEnd":0,"redirectStart":0,"requestStart":396,"responseEnd":996.2000000476837,"responseStart":996.2000000476837,"secureConnectionStart":396},{"duration":600.2999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":396.2000000476837,"connectEnd":396.2000000476837,"connectStart":396.2000000476837,"domainLookupEnd":396.2000000476837,"domainLookupStart":396.2000000476837,"fetchStart":396.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":396.2000000476837,"responseEnd":996.5,"responseStart":996.5,"secureConnectionStart":396.2000000476837},{"duration":693.7999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":396.40000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":396.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1090.2000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":693.8000000715256,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":396.60000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":396.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1090.4000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":600.3999999761581,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":396.60000002384186,"connectEnd":396.60000002384186,"connectStart":396.60000002384186,"domainLookupEnd":396.60000002384186,"domainLookupStart":396.60000002384186,"fetchStart":396.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":396.60000002384186,"responseEnd":997,"responseStart":997,"secureConnectionStart":396.60000002384186},{"duration":600.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":396.90000009536743,"connectEnd":396.90000009536743,"connectStart":396.90000009536743,"domainLookupEnd":396.90000009536743,"domainLookupStart":396.90000009536743,"fetchStart":396.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":396.90000009536743,"responseEnd":997.5,"responseStart":997.5,"secureConnectionStart":396.90000009536743},{"duration":853.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":397.7000000476837,"connectEnd":397.7000000476837,"connectStart":397.7000000476837,"domainLookupEnd":397.7000000476837,"domainLookupStart":397.7000000476837,"fetchStart":397.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":397.7000000476837,"responseEnd":1250.9000000953674,"responseStart":1250.9000000953674,"secureConnectionStart":397.7000000476837},{"duration":1725.8000000715256,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":402.5,"connectEnd":402.5,"connectStart":402.5,"domainLookupEnd":402.5,"domainLookupStart":402.5,"fetchStart":402.5,"redirectEnd":0,"redirectStart":0,"requestStart":402.5,"responseEnd":2128.3000000715256,"responseStart":2128.3000000715256,"secureConnectionStart":402.5},{"duration":152.10000002384186,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1101.8000000715256,"connectEnd":1101.8000000715256,"connectStart":1101.8000000715256,"domainLookupEnd":1101.8000000715256,"domainLookupStart":1101.8000000715256,"fetchStart":1101.8000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":1101.8000000715256,"responseEnd":1253.9000000953674,"responseStart":1253.9000000953674,"secureConnectionStart":1101.8000000715256}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":145,"responseStart":390,"responseEnd":401,"domLoading":393,"domInteractive":2162,"domContentLoadedEventStart":2162,"domContentLoadedEventEnd":2212,"domComplete":2927,"loadEventStart":2927,"loadEventEnd":2927,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":2130},{"name":"bigPipe.sidebar-id.end","time":2131},{"name":"bigPipe.activity-panel-pipe-id.start","time":2131.2000000476837},{"name":"bigPipe.activity-panel-pipe-id.end","time":2137.5},{"name":"activityTabFullyLoaded","time":2232.100000023842}],"measures":[],"correlationId":"f80558f6d4083a","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":182,"dbReadsTimeInMs":41,"dbConnsTimeInMs":54,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}