Type:
Bug
Priority:
Major
Resolution:
Duplicate
Affects Version/s:
10.2.16 , 10.2.18 , 10.1(EOL) , 10.2(EOL) , 10.3(EOL)
When Galera is enabled, MariaDB's systemd service executes the "galera_recovery" script as an ExecStartPre operation. See the following:
https://github.com/MariaDB/server/blob/ce8716a1ed786ff971b5e15c88385d50b649ec7f/support-files/mariadb.service.in#L71
The MariaDB systemd service has a default TimeoutStartSec value of 90 seconds, so if this ExecStartPre step takes longer than that, then this can cause startup to fail. For example, see the following failure from a syslog:
Sep 13 15:48:28 server1 systemd[1]: Starting MariaDB 10.2.16 database server...
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Start-pre operation timed out. Terminating.
Sep 13 15:49:58 server1 systemd[1]: Failed to start MariaDB 10.2.16 database server.
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Unit entered failed state.
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Failed with result 'timeout'.
galera_recovery has to perform server startup, so this step can take a while, especially if the server previously crashed, and it has to perform crash recovery. However, it looks like systemd timeouts should have been extended during server startup as part of MDEV-14705 . Despite that, server versions with the fix for MDEV-14705 still see timeouts during ExecStartPre. Is it likely that important long-running startup functions were missed?
See also MDEV-17571 as another case where systemd timeout extensions didn't seem to work as intended.
Elena Stepanova
made changes -
2018-12-07 23:56
Fix Version/s
10.1
[ 16100
]
Fix Version/s
10.2
[ 14601
]
Fix Version/s
10.3
[ 22126
]
Fix Version/s
10.4
[ 22408
]
Assignee
Rasmus Johansson
[ ratzpo
]
Geoff Montee (Inactive)
made changes -
2018-12-07 23:59
Description
When Galera is enabled, MariaDB's systemd service executes the "galera_recovery" script as an ExecStartPre operation. See the following:
https://github.com/MariaDB/server/blob/ce8716a1ed786ff971b5e15c88385d50b649ec7f/support-files/mariadb.service.in#L71
The MariaDB systemd service has a default TimeoutStartSec value of 90 seconds, so if this ExecStartPre step takes longer than that, then this can cause startup to fail. For example, see the following failure from a syslog:
{noformat}
Sep 13 15:48:28 server1 systemd[1]: Starting MariaDB 10.2.16 database server...
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Start-pre operation timed out. Terminating.
Sep 13 15:49:58 server1 systemd[1]: Failed to start MariaDB 10.2.16 database server.
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Unit entered failed state.
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Failed with result 'timeout'.
{noformat}
galera_recovery has to perform server startup, so this step can take a while, especially if the server previously crashed, and it has to perform crash recovery. However, it looks like systemd timeouts should have been extended during server startup as part of MDEV-14705 . Despite that, server versions with the fix for MDEV-14705 still see timeouts during ExecStartPre. Is it likely that important long-running startup functions were mixed?
See also MDEV-17571 as another case where systemd timeout extensions didn't seem to work as intended.
When Galera is enabled, MariaDB's systemd service executes the "galera_recovery" script as an ExecStartPre operation. See the following:
https://github.com/MariaDB/server/blob/ce8716a1ed786ff971b5e15c88385d50b649ec7f/support-files/mariadb.service.in#L71
The MariaDB systemd service has a default TimeoutStartSec value of 90 seconds, so if this ExecStartPre step takes longer than that, then this can cause startup to fail. For example, see the following failure from a syslog:
{noformat}
Sep 13 15:48:28 server1 systemd[1]: Starting MariaDB 10.2.16 database server...
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Start-pre operation timed out. Terminating.
Sep 13 15:49:58 server1 systemd[1]: Failed to start MariaDB 10.2.16 database server.
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Unit entered failed state.
Sep 13 15:49:58 server1 systemd[1]: mariadb.service: Failed with result 'timeout'.
{noformat}
galera_recovery has to perform server startup, so this step can take a while, especially if the server previously crashed, and it has to perform crash recovery. However, it looks like systemd timeouts should have been extended during server startup as part of MDEV-14705 . Despite that, server versions with the fix for MDEV-14705 still see timeouts during ExecStartPre. Is it likely that important long-running startup functions were missed?
See also MDEV-17571 as another case where systemd timeout extensions didn't seem to work as intended.
Axel Schwenke
made changes -
2019-12-05 11:00
Fix Version/s
N/A
[ 14700
]
Fix Version/s
10.2
[ 14601
]
Fix Version/s
10.1
[ 16100
]
Fix Version/s
10.3
[ 22126
]
Fix Version/s
10.4
[ 22408
]
Resolution
Duplicate
[ 3
]
Status
Stalled
[ 10000
]
Closed
[ 6
]
Sergei Golubchik
made changes -
2021-12-06 21:48
Workflow
MariaDB v3
[ 91110
]
MariaDB v4
[ 155321
]
{"report":{"fcp":1051.6999998092651,"ttfb":293,"pageVisibility":"visible","entityId":71288,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"f801b80a-22aa-4d1a-b088-c825119ca571","navigationType":0,"readyForUser":1131.8999998569489,"redirectCount":0,"resourceLoadedEnd":1442.8999998569489,"resourceLoadedStart":301.39999985694885,"resourceTiming":[{"duration":243.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":301.39999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":301.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":544.8999998569489,"responseStart":0,"secureConnectionStart":0},{"duration":243.70000004768372,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":301.69999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":301.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":545.3999998569489,"responseStart":0,"secureConnectionStart":0},{"duration":274.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":301.89999985694885,"connectEnd":301.89999985694885,"connectStart":301.89999985694885,"domainLookupEnd":301.89999985694885,"domainLookupStart":301.89999985694885,"fetchStart":301.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":301.89999985694885,"responseEnd":576.6999998092651,"responseStart":576.6999998092651,"secureConnectionStart":301.89999985694885},{"duration":320.60000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":302.69999980926514,"connectEnd":302.69999980926514,"connectStart":302.69999980926514,"domainLookupEnd":302.69999980926514,"domainLookupStart":302.69999980926514,"fetchStart":302.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":302.69999980926514,"responseEnd":623.2999999523163,"responseStart":623.2999999523163,"secureConnectionStart":302.69999980926514},{"duration":324.39999985694885,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":303,"connectEnd":303,"connectStart":303,"domainLookupEnd":303,"domainLookupStart":303,"fetchStart":303,"redirectEnd":0,"redirectStart":0,"requestStart":303,"responseEnd":627.3999998569489,"responseStart":627.3999998569489,"secureConnectionStart":303},{"duration":324.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":303.09999990463257,"connectEnd":303.09999990463257,"connectStart":303.09999990463257,"domainLookupEnd":303.09999990463257,"domainLookupStart":303.09999990463257,"fetchStart":303.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":303.09999990463257,"responseEnd":627.8999998569489,"responseStart":627.8999998569489,"secureConnectionStart":303.09999990463257},{"duration":324.89999985694885,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":303.2999999523163,"connectEnd":303.2999999523163,"connectStart":303.2999999523163,"domainLookupEnd":303.2999999523163,"domainLookupStart":303.2999999523163,"fetchStart":303.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":303.2999999523163,"responseEnd":628.1999998092651,"responseStart":628.1999998092651,"secureConnectionStart":303.2999999523163},{"duration":418,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":303.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":303.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":721.5,"responseStart":0,"secureConnectionStart":0},{"duration":325,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":303.69999980926514,"connectEnd":303.69999980926514,"connectStart":303.69999980926514,"domainLookupEnd":303.69999980926514,"domainLookupStart":303.69999980926514,"fetchStart":303.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":303.69999980926514,"responseEnd":628.6999998092651,"responseStart":628.6999998092651,"secureConnectionStart":303.69999980926514},{"duration":417.7999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":303.7999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":303.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":721.5999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":325.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":304,"connectEnd":304,"connectStart":304,"domainLookupEnd":304,"domainLookupStart":304,"fetchStart":304,"redirectEnd":0,"redirectStart":0,"requestStart":304,"responseEnd":629.1999998092651,"responseStart":629.1999998092651,"secureConnectionStart":304},{"duration":1124.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":306.5,"connectEnd":306.5,"connectStart":306.5,"domainLookupEnd":306.5,"domainLookupStart":306.5,"fetchStart":306.5,"redirectEnd":0,"redirectStart":0,"requestStart":306.5,"responseEnd":1430.5999999046326,"responseStart":1430.5999999046326,"secureConnectionStart":306.5},{"duration":1119.7000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":311.89999985694885,"connectEnd":311.89999985694885,"connectStart":311.89999985694885,"domainLookupEnd":311.89999985694885,"domainLookupStart":311.89999985694885,"fetchStart":311.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":311.89999985694885,"responseEnd":1431.5999999046326,"responseStart":1431.5999999046326,"secureConnectionStart":311.89999985694885},{"duration":196.39999985694885,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":735.2999999523163,"connectEnd":735.2999999523163,"connectStart":735.2999999523163,"domainLookupEnd":735.2999999523163,"domainLookupStart":735.2999999523163,"fetchStart":735.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":735.2999999523163,"responseEnd":931.6999998092651,"responseStart":931.6999998092651,"secureConnectionStart":735.2999999523163},{"duration":429.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1004.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1004.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1433.7999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":431.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1005.2999999523163,"connectEnd":1005.2999999523163,"connectStart":1005.2999999523163,"domainLookupEnd":1005.2999999523163,"domainLookupStart":1005.2999999523163,"fetchStart":1005.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":1005.2999999523163,"responseEnd":1436.5,"responseStart":1436.5,"secureConnectionStart":1005.2999999523163},{"duration":425.39999985694885,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1045.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1045.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1470.8999998569489,"responseStart":0,"secureConnectionStart":0},{"duration":437.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/097ae97cb8fbec7d6ea4bbb1f26955b9-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":1005.6999998092651,"connectEnd":1005.6999998092651,"connectStart":1005.6999998092651,"domainLookupEnd":1005.6999998092651,"domainLookupStart":1005.6999998092651,"fetchStart":1005.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":1005.6999998092651,"responseEnd":1442.8999998569489,"responseStart":1442.8999998569489,"secureConnectionStart":1005.6999998092651}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":48,"responseStart":293,"responseEnd":312,"domLoading":297,"domInteractive":1475,"domContentLoadedEventStart":1475,"domContentLoadedEventEnd":1525,"domComplete":2209,"loadEventStart":2209,"loadEventEnd":2210,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1444.2999999523163},{"name":"bigPipe.sidebar-id.end","time":1445.0999999046326},{"name":"bigPipe.activity-panel-pipe-id.start","time":1445.2999999523163},{"name":"bigPipe.activity-panel-pipe-id.end","time":1449.3999998569489},{"name":"activityTabFullyLoaded","time":1547.8999998569489}],"measures":[],"correlationId":"aca024c288440","effectiveType":"4g","downlink":9.7,"rtt":0,"serverDuration":170,"dbReadsTimeInMs":40,"dbConnsTimeInMs":51,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}