Hi MariaDB-team,
we have a five node MariaDB Galera Cluster.
After taking one node out of service an clearing data directories and start MariaDB the transfer starts but after about 90% it fails with:
...
|
Sep 27 09:20:58 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:20:58 0 [Note] WSREP: REPORTING SST PROGRESS: '{ "from": 1, "to": 3, "total": 19935177778, "done": 19935177778, "indefinite": -1 }'
|
Sep 27 09:21:04 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:04 0 [Note] WSREP: REPORTING SST PROGRESS: '{ "from": 1, "to": 3, "total": 19947753124, "done": 19947753124, "indefinite": -1 }'
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Warning] WSREP: 0.0 (maria-muc-1): State transfer to 3.0 (maria-ham-3) failed: -22 (Invalid argument)
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [ERROR] WSREP: ./gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1217: Will never receive state. Need to abort.
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: terminating thread
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: joining thread
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closing backend
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view(view_id(NON_PRIM,0abc8cb4-9ae0,609) memb {
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: a591b16e-9113,0
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: } joined {
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: } left {
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: } partitioned {
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 0abc8cb4-9ae0,0
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 29958514-86e6,0
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 7150c82e-8f59,0
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: dfc13b8b-8838,0
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: })
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: PC protocol downgrade 1 -> 0
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view((empty))
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closed
|
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: /usr/sbin/mariadbd: Terminated.
|
Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
|
Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Failed with result 'signal'.
|
Sep 27 09:21:08 maria-ham-3 systemd[1]: Failed to start MariaDB 10.11.9 database server.
|
This error comes again at each try to restart the transfer.
Versions:
# dpkg -l galera-4 mariadb-server
|
Desired=Unknown/Install/Remove/Purge/Hold
|
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
|
||/ Name Version Architecture Description
|
+++-==============-=======================-============-====================================================
|
ii galera-4 26.4.19-ubu2204 amd64 Replication framework for transactional applications
|
ii mariadb-server 1:10.11.9+maria~ubu2204 amd64 MariaDB database server binaries
|
I set the wsrep_sst_donor on node maria-ham-3 to maria-muc-1 to make debugging easier.
The filesystems have enough space and I can't find any reason why the transfer stops after it started successfully.
Can you give me any hint where to look or what to do for further debugging?
Greetings
Lars
- duplicates
-
MDEV-35023
WSREP: State transfer failed: -22 (Invalid argument)
-
-
Closed
No workflow transitions have been executed yet.
{"report":{"fcp":1346.3000001907349,"ttfb":459.7000002861023,"pageVisibility":"visible","entityId":130957,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"61beb8c1-ef5e-4be7-9f83-e17351707855","navigationType":0,"readyForUser":1447.8000001907349,"redirectCount":0,"resourceLoadedEnd":1842,"resourceLoadedStart":465.2000002861023,"resourceTiming":[{"duration":343.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":465.2000002861023,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":465.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":808.7000002861023,"responseStart":0,"secureConnectionStart":0},{"duration":343.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":465.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":465.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":808.9000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":351,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":465.6000003814697,"connectEnd":465.6000003814697,"connectStart":465.6000003814697,"domainLookupEnd":465.6000003814697,"domainLookupStart":465.6000003814697,"fetchStart":465.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":465.6000003814697,"responseEnd":816.6000003814697,"responseStart":816.6000003814697,"secureConnectionStart":465.6000003814697},{"duration":400.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":465.90000009536743,"connectEnd":465.90000009536743,"connectStart":465.90000009536743,"domainLookupEnd":465.90000009536743,"domainLookupStart":465.90000009536743,"fetchStart":465.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":465.90000009536743,"responseEnd":866.4000000953674,"responseStart":866.4000000953674,"secureConnectionStart":465.90000009536743},{"duration":403.7999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":466.1000003814697,"connectEnd":466.1000003814697,"connectStart":466.1000003814697,"domainLookupEnd":466.1000003814697,"domainLookupStart":466.1000003814697,"fetchStart":466.1000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":466.1000003814697,"responseEnd":869.9000000953674,"responseStart":869.9000000953674,"secureConnectionStart":466.1000003814697},{"duration":404.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":466.2000002861023,"connectEnd":466.2000002861023,"connectStart":466.2000002861023,"domainLookupEnd":466.2000002861023,"domainLookupStart":466.2000002861023,"fetchStart":466.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":466.2000002861023,"responseEnd":870.4000000953674,"responseStart":870.4000000953674,"secureConnectionStart":466.2000002861023},{"duration":404.2000002861023,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":466.5,"connectEnd":466.5,"connectStart":466.5,"domainLookupEnd":466.5,"domainLookupStart":466.5,"fetchStart":466.5,"redirectEnd":0,"redirectStart":0,"requestStart":466.5,"responseEnd":870.7000002861023,"responseStart":870.7000002861023,"secureConnectionStart":466.5},{"duration":512.0999999046326,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":466.6000003814697,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":466.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":978.7000002861023,"responseStart":0,"secureConnectionStart":0},{"duration":405,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":466.7000002861023,"connectEnd":466.7000002861023,"connectStart":466.7000002861023,"domainLookupEnd":466.7000002861023,"domainLookupStart":466.7000002861023,"fetchStart":466.7000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":466.7000002861023,"responseEnd":871.7000002861023,"responseStart":871.7000002861023,"secureConnectionStart":466.7000002861023},{"duration":512,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":466.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":466.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":978.9000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":405.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":467.1000003814697,"connectEnd":467.1000003814697,"connectStart":467.1000003814697,"domainLookupEnd":467.1000003814697,"domainLookupStart":467.1000003814697,"fetchStart":467.1000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":467.1000003814697,"responseEnd":872.3000001907349,"responseStart":872.3000001907349,"secureConnectionStart":467.1000003814697},{"duration":695.8000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":472.40000009536743,"connectEnd":472.40000009536743,"connectStart":472.40000009536743,"domainLookupEnd":472.40000009536743,"domainLookupStart":472.40000009536743,"fetchStart":472.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":472.40000009536743,"responseEnd":1168.2000002861023,"responseStart":1168.2000002861023,"secureConnectionStart":472.40000009536743},{"duration":1356.8000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":472.40000009536743,"connectEnd":472.40000009536743,"connectStart":472.40000009536743,"domainLookupEnd":472.40000009536743,"domainLookupStart":472.40000009536743,"fetchStart":472.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":472.40000009536743,"responseEnd":1829.2000002861023,"responseStart":1829.2000002861023,"secureConnectionStart":472.40000009536743},{"duration":179.09999990463257,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1010.2000002861023,"connectEnd":1010.2000002861023,"connectStart":1010.2000002861023,"domainLookupEnd":1010.2000002861023,"domainLookupStart":1010.2000002861023,"fetchStart":1010.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":1010.2000002861023,"responseEnd":1189.3000001907349,"responseStart":1189.3000001907349,"secureConnectionStart":1010.2000002861023},{"duration":568.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1273.8000001907349,"connectEnd":1273.8000001907349,"connectStart":1273.8000001907349,"domainLookupEnd":1273.8000001907349,"domainLookupStart":1273.8000001907349,"fetchStart":1273.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":1273.8000001907349,"responseEnd":1842,"responseStart":1842,"secureConnectionStart":1273.8000001907349},{"duration":555.5,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1339.8000001907349,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1339.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1895.3000001907349,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":299,"responseStart":460,"responseEnd":465,"domLoading":463,"domInteractive":1850,"domContentLoadedEventStart":1850,"domContentLoadedEventEnd":1895,"domComplete":2274,"loadEventStart":2274,"loadEventEnd":2275,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1831.5},{"name":"bigPipe.sidebar-id.end","time":1832.3000001907349},{"name":"bigPipe.activity-panel-pipe-id.start","time":1832.4000000953674},{"name":"bigPipe.activity-panel-pipe-id.end","time":1832.9000000953674},{"name":"activityTabFullyLoaded","time":1902.7000002861023}],"measures":[],"correlationId":"acca7a5f9b5820","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":97,"dbReadsTimeInMs":15,"dbConnsTimeInMs":23,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Critical