For context, this is a galera cluster of 3 nodes with around 4TB of data and using mariabackup as SST method and daily cluster backup.
When the SST process is started, it successfully starts to transmit data. However after a few hours passes, data transmission stops with following error in the mariabackup.backup.log:
[01] 2025-03-26 04:42:32 Streaming ./app_user_5054672/report.frm to <STDOUT>
[01] 2025-03-26 04:42:32 ...done
[01] 2025-03-26 04:42:32 Streaming ./app_user_5054672/sms_statistic.frm to <STDOUT>
[01] 2025-03-26 04:42:32 ...done
[01] 2025-03-26 04:42:32 Streaming ./app_user_5054672/email_statistic_opened.frm to <STDOUT>
[01] 2025-03-26 04:42:32 ...done
[00] 2025-03-26 04:42:32 Finished backing up non-InnoDB tables and files
[00] 2025-03-26 04:42:32 Waiting for log copy thread to read lsn 257758688468141
[00] 2025-03-26 04:42:32 Retrying read of log at LSN=257754432284012
[00] 2025-03-26 04:42:33 Retrying read of log at LSN=257754432284012
[00] 2025-03-26 04:42:34 Retrying read of log at LSN=257754432284012
[00] 2025-03-26 04:42:35 Retrying read of log at LSN=257754432284012
[00] 2025-03-26 04:42:37 Retrying read of log at LSN=257754432284012
[00] 2025-03-26 04:42:37 Was only able to copy log from 257737152711197 to 257754432284012, not 257758688468141; try increasing innodb_log_file_size
mariabackup: Stopping log copying thread[00] 2025-03-26 04:42:37 Retrying read of log at LSN=257754432284012
We've already tested increasing the innodb_log_file_size from original 90GB to current 220GB. This did not help with SST but the daily backup was able to finish.
During the time we've ran the SSTs, the traffic was not high enough to fill the log.
We've noticed that the process waits for
[00] 2025-03-26 04:42:32 Waiting for log copy thread to read lsn 25775868846814
to be read with a timeout of 5 seconds, when it runs out the SST fails. Please notice that lsn it is waiting for is from the future
We've tried to increase the timeout by adding
[mariabackup]
ftwrl-wait-timeout=60
But this did not change anything even after adding it to the [SST] section as well.
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
0.3
0
{"report":{"fcp":925,"ttfb":240.40000009536743,"pageVisibility":"visible","key":"jira.issue.nav-detail","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"9d0dd274-12c0-4d7a-9202-a98e4ef467c1","navigationType":0,"readyForUser":2135,"redirectCount":0,"resourceLoadedEnd":518.8000001907349,"resourceLoadedStart":248.5,"resourceTiming":[{"duration":79.60000038146973,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":248.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":248.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":328.1000003814697,"responseStart":0,"secureConnectionStart":0},{"duration":85.09999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/1478ee9518804fac4b35845dc4ce556a-CDN/lu2cib/820016/12ta74/97a9a6b13407d4702f4a258757d43f81/_/download/contextbatch/css/jira.view.issue,viewissue.standalone,jira.navigator.kickass,jira.general,jira.global,jira.navigator,atl.general,jira.navigator.simple,jira.navigator.advanced,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":248.80000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":248.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":333.90000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":86.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/3.0.4/_/download/batch/com.atlassian.jira.jira-tzdetect-plugin:tzdetect-banner-component/com.atlassian.jira.jira-tzdetect-plugin:tzdetect-banner-component.css","startTime":249.10000038146973,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":249.10000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":335.7000002861023,"responseStart":0,"secureConnectionStart":0},{"duration":86.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/a7c82c2f9b4c8fd63530eb9f2a1d712a-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.filter.deletion.warning:styles/jira.filter.deletion.warning:styles.css","startTime":249.30000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":249.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":335.80000019073486,"responseStart":0,"secureConnectionStart":0},{"duration":205.7000002861023,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":249.40000009536743,"connectEnd":249.40000009536743,"connectStart":249.40000009536743,"domainLookupEnd":249.40000009536743,"domainLookupStart":249.40000009536743,"fetchStart":249.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":339.7000002861023,"responseEnd":455.1000003814697,"responseStart":355.80000019073486,"secureConnectionStart":249.40000009536743},{"duration":269.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/3e25a0b0c47560558d0a6707b047e06f-CDN/lu2cib/820016/12ta74/97a9a6b13407d4702f4a258757d43f81/_/download/contextbatch/js/jira.view.issue,viewissue.standalone,jira.navigator.kickass,jira.general,jira.global,jira.navigator,atl.general,jira.navigator.simple,jira.navigator.advanced,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":249.60000038146973,"connectEnd":249.60000038146973,"connectStart":249.60000038146973,"domainLookupEnd":249.60000038146973,"domainLookupStart":249.60000038146973,"fetchStart":249.60000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":339.6000003814697,"responseEnd":518.8000001907349,"responseStart":363.30000019073486,"secureConnectionStart":249.60000038146973},{"duration":107.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":249.80000019073486,"connectEnd":249.80000019073486,"connectStart":249.80000019073486,"domainLookupEnd":249.80000019073486,"domainLookupStart":249.80000019073486,"fetchStart":249.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":340.40000009536743,"responseEnd":357,"responseStart":355.2000002861023,"secureConnectionStart":249.80000019073486},{"duration":110.2000002861023,"initiatorType":"script","name":"https://jira.mariadb.org/s/aa6a6199803063cf5eed6122dfa3f13b-CDN/lu2cib/820016/12ta74/3.0.4/_/download/batch/com.atlassian.jira.jira-tzdetect-plugin:tzdetect-banner-component/com.atlassian.jira.jira-tzdetect-plugin:tzdetect-banner-component.js?locale=en","startTime":250.40000009536743,"connectEnd":250.40000009536743,"connectStart":250.40000009536743,"domainLookupEnd":250.40000009536743,"domainLookupStart":250.40000009536743,"fetchStart":250.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":341.30000019073486,"responseEnd":360.6000003814697,"responseStart":358.1000003814697,"secureConnectionStart":250.40000009536743},{"duration":116.7999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/3.0.4/_/download/batch/com.atlassian.jira.jira-tzdetect-plugin:tzdetect-lib/com.atlassian.jira.jira-tzdetect-plugin:tzdetect-lib.js","startTime":250.60000038146973,"connectEnd":250.60000038146973,"connectStart":250.60000038146973,"domainLookupEnd":250.60000038146973,"domainLookupStart":250.60000038146973,"fetchStart":250.60000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":347.1000003814697,"responseEnd":367.40000009536743,"responseStart":365.1000003814697,"secureConnectionStart":250.60000038146973},{"duration":117.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":250.80000019073486,"connectEnd":250.80000019073486,"connectStart":250.80000019073486,"domainLookupEnd":250.80000019073486,"domainLookupStart":250.80000019073486,"fetchStart":250.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":347.30000019073486,"responseEnd":367.90000009536743,"responseStart":366.30000019073486,"secureConnectionStart":250.80000019073486},{"duration":116.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":251,"connectEnd":251,"connectStart":251,"domainLookupEnd":251,"domainLookupStart":251,"fetchStart":251,"redirectEnd":0,"redirectStart":0,"requestStart":347.5,"responseEnd":367.5,"responseStart":365.7000002861023,"secureConnectionStart":251},{"duration":95.19999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":251.2000002861023,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":251.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":346.40000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":134.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":251.30000019073486,"connectEnd":251.30000019073486,"connectStart":251.30000019073486,"domainLookupEnd":251.30000019073486,"domainLookupStart":251.30000019073486,"fetchStart":251.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":353.2000002861023,"responseEnd":385.7000002861023,"responseStart":384.1000003814697,"secureConnectionStart":251.30000019073486},{"duration":252.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":258.6000003814697,"connectEnd":258.6000003814697,"connectStart":258.6000003814697,"domainLookupEnd":258.6000003814697,"domainLookupStart":258.6000003814697,"fetchStart":258.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":361.40000009536743,"responseEnd":510.80000019073486,"responseStart":509.1000003814697,"secureConnectionStart":258.6000003814697},{"duration":254.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":258.80000019073486,"connectEnd":258.80000019073486,"connectStart":258.80000019073486,"domainLookupEnd":258.80000019073486,"domainLookupStart":258.80000019073486,"fetchStart":258.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":454.90000009536743,"responseEnd":513.2000002861023,"responseStart":512,"secureConnectionStart":258.80000019073486},{"duration":387.7000002861023,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":706.4000000953674,"connectEnd":706.4000000953674,"connectStart":706.4000000953674,"domainLookupEnd":706.4000000953674,"domainLookupStart":706.4000000953674,"fetchStart":706.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":1061.5,"responseEnd":1094.1000003814697,"responseStart":1093.5,"secureConnectionStart":706.4000000953674},{"duration":379.7000002861023,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":907.4000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":907.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1287.1000003814697,"responseStart":0,"secureConnectionStart":0},{"duration":551.5999999046326,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/api/2/filter/favourite?expand=subscriptions[-5:]&_=1743273854367","startTime":1110.3000001907349,"connectEnd":1110.3000001907349,"connectStart":1110.3000001907349,"domainLookupEnd":1110.3000001907349,"domainLookupStart":1110.3000001907349,"fetchStart":1110.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":1629.6000003814697,"responseEnd":1661.9000000953674,"responseStart":1661.3000001907349,"secureConnectionStart":1110.3000001907349},{"duration":869.3000001907349,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/issueNav/latest/preferredSearchLayout","startTime":1191,"connectEnd":1191,"connectStart":1191,"domainLookupEnd":1191,"domainLookupStart":1191,"fetchStart":1191,"redirectEnd":0,"redirectStart":0,"requestStart":2030,"responseEnd":2060.300000190735,"responseStart":2059.800000190735,"secureConnectionStart":1191},{"duration":830.5999999046326,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/issueNav/1/issueTable/stable","startTime":1243.2000002861023,"connectEnd":1243.2000002861023,"connectStart":1243.2000002861023,"domainLookupEnd":1243.2000002861023,"domainLookupStart":1243.2000002861023,"fetchStart":1243.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":2032.9000000953674,"responseEnd":2073.800000190735,"responseStart":2073.1000003814697,"secureConnectionStart":1243.2000002861023},{"duration":812.7999997138977,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/menu/latest/appswitcher?_=1743273854368","startTime":1250.6000003814697,"connectEnd":1250.6000003814697,"connectStart":1250.6000003814697,"domainLookupEnd":1250.6000003814697,"domainLookupStart":1250.6000003814697,"fetchStart":1250.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":2033.3000001907349,"responseEnd":2063.4000000953674,"responseStart":2062.6000003814697,"secureConnectionStart":1250.6000003814697}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":50,"responseStart":240,"responseEnd":248,"domLoading":246,"domInteractive":1064,"domContentLoadedEventStart":1064,"domContentLoadedEventEnd":1276,"domComplete":2420,"loadEventStart":2420,"loadEventEnd":2421,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[],"measures":[],"correlationId":"1649864e08c61c","effectiveType":"4g","downlink":9.6,"rtt":0,"serverDuration":113,"dbConnsTimeInMs":8,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Major