Hi team,
we have a 5 node Galera Cluster 26.4.19 with MariaDB Server 10.11.9.
After a test update of MariaDB on one node (maria-ham-3) I switched back to the versions above. I cleared data directory and wanted to sync and join (what worked fine in the past) with the node:
{{
- rm -rf /GALERANFS_HAM_DATA/data/mariadb/* /GALERANFS_HAM_DATA/InnoDB/ibdata1 /GALERANFS_HAM_LOG/ib_log/ib_logfile0
- systemctl start mariadb
}}
This leads to a transfer starting but the later failing:
{{
Sep 27 09:20:52 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:20:52 0 [Note] WSREP: REPORTING SST PROGRESS: '
{ "from": 1, "to": 3, "total": 19922598377, "done": 19922598377, "indefinite": -1 }
'
Sep 27 09:20:58 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:20:58 0 [Note] WSREP: REPORTING SST PROGRESS: '
{ "from": 1, "to": 3, "total": 19935177778, "done": 19935177778, "indefinite": -1 }
'
Sep 27 09:21:04 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:04 0 [Note] WSREP: REPORTING SST PROGRESS: '
{ "from": 1, "to": 3, "total": 19947753124, "done": 19947753124, "indefinite": -1 }
'
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Warning] WSREP: 0.0 (maria-muc-1): State transfer to 3.0 (maria-ham-3) failed: -22 (Invalid argument)
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [ERROR] WSREP: ./gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1217: Will never receive state. Need to abort.
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: terminating thread
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: joining thread
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closing backend
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view(view_id(NON_PRIM,0abc8cb4-9ae0,609) memb
{
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: a591b16e-9113,0
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }
joined
{
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }
left
{
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }
partitioned
{
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 0abc8cb4-9ae0,0
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 29958514-86e6,0
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 7150c82e-8f59,0
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: dfc13b8b-8838,0
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }
)
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: PC protocol downgrade 1 -> 0
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view((empty))
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closed
Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: /usr/sbin/mariadbd: Terminated.
Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Failed with result 'signal'.
Sep 27 09:21:08 maria-ham-3 systemd[1]: Failed to start MariaDB 10.11.9 database server.
}}
I configured the wsrep_sst_donor to one node to ease the debugging.
The logs of both sides are attached.
Filesystems are not full.
I cannot find out what stopped the transfer. Can you give me any hint of what to do for further debugging?
Thanks in advance
Lars
- is duplicated by
-
MDEV-35024
WSREP: State transfer failed: -22 (Invalid argument)
-
-
Open
{"report":{"fcp":1048.4000000059605,"ttfb":192.7000000178814,"pageVisibility":"visible","entityId":130956,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"7f981299-64d1-4df6-a972-ab9ab569b94f","navigationType":0,"readyForUser":1121.6000000238419,"redirectCount":0,"resourceLoadedEnd":1358.6000000238419,"resourceLoadedStart":197.80000001192093,"resourceTiming":[{"duration":446.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":197.80000001192093,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":197.80000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":644.3000000119209,"responseStart":0,"secureConnectionStart":0},{"duration":446.59999999403954,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":198.10000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":198.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":644.7000000178814,"responseStart":0,"secureConnectionStart":0},{"duration":455.80000001192093,"initiatorType":"script","name":"https://jira.mariadb.org/s/e9b27a47da5fb0f74a35acd57e9847fb-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":198.30000001192093,"connectEnd":198.30000001192093,"connectStart":198.30000001192093,"domainLookupEnd":198.30000001192093,"domainLookupStart":198.30000001192093,"fetchStart":198.30000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":198.30000001192093,"responseEnd":654.1000000238419,"responseStart":654.1000000238419,"secureConnectionStart":198.30000001192093},{"duration":480.30000001192093,"initiatorType":"script","name":"https://jira.mariadb.org/s/c32eb0da7ad9831253f8397e6cc26afd-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":198.40000000596046,"connectEnd":198.40000000596046,"connectStart":198.40000000596046,"domainLookupEnd":198.40000000596046,"domainLookupStart":198.40000000596046,"fetchStart":198.40000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":198.40000000596046,"responseEnd":678.7000000178814,"responseStart":678.7000000178814,"secureConnectionStart":198.40000000596046},{"duration":483.40000000596046,"initiatorType":"script","name":"https://jira.mariadb.org/s/bc0bcb146314416123c992714ee00ff7-CDN/lu2bv2/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":198.7000000178814,"connectEnd":198.7000000178814,"connectStart":198.7000000178814,"domainLookupEnd":198.7000000178814,"domainLookupStart":198.7000000178814,"fetchStart":198.7000000178814,"redirectEnd":0,"redirectStart":0,"requestStart":198.7000000178814,"responseEnd":682.1000000238419,"responseStart":682.1000000238419,"secureConnectionStart":198.7000000178814},{"duration":483.80000001192093,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":198.90000000596046,"connectEnd":198.90000000596046,"connectStart":198.90000000596046,"domainLookupEnd":198.90000000596046,"domainLookupStart":198.90000000596046,"fetchStart":198.90000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":198.90000000596046,"responseEnd":682.7000000178814,"responseStart":682.7000000178814,"secureConnectionStart":198.90000000596046},{"duration":484.10000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":199,"connectEnd":199,"connectStart":199,"domainLookupEnd":199,"domainLookupStart":199,"fetchStart":199,"redirectEnd":0,"redirectStart":0,"requestStart":199,"responseEnd":683.1000000238419,"responseStart":683.1000000238419,"secureConnectionStart":199},{"duration":600.0999999940395,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bv2/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":199.2000000178814,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":199.2000000178814,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":799.3000000119209,"responseStart":0,"secureConnectionStart":0},{"duration":484.30000001192093,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":199.30000001192093,"connectEnd":199.30000001192093,"connectStart":199.30000001192093,"domainLookupEnd":199.30000001192093,"domainLookupStart":199.30000001192093,"fetchStart":199.30000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":199.30000001192093,"responseEnd":683.6000000238419,"responseStart":683.6000000238419,"secureConnectionStart":199.30000001192093},{"duration":599.9000000059605,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":199.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":199.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":799.4000000059605,"responseStart":0,"secureConnectionStart":0},{"duration":484.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/719848dd97ebe0663199f49a3936487a-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":199.60000002384186,"connectEnd":199.60000002384186,"connectStart":199.60000002384186,"domainLookupEnd":199.60000002384186,"domainLookupStart":199.60000002384186,"fetchStart":199.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":199.60000002384186,"responseEnd":684.1000000238419,"responseStart":684.1000000238419,"secureConnectionStart":199.60000002384186},{"duration":1155.199999988079,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":200.60000002384186,"connectEnd":200.60000002384186,"connectStart":200.60000002384186,"domainLookupEnd":200.60000002384186,"domainLookupStart":200.60000002384186,"fetchStart":200.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":200.60000002384186,"responseEnd":1355.800000011921,"responseStart":1355.800000011921,"secureConnectionStart":200.60000002384186},{"duration":1158,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":200.60000002384186,"connectEnd":200.60000002384186,"connectStart":200.60000002384186,"domainLookupEnd":200.60000002384186,"domainLookupStart":200.60000002384186,"fetchStart":200.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":200.60000002384186,"responseEnd":1358.6000000238419,"responseStart":1358.6000000238419,"secureConnectionStart":200.60000002384186},{"duration":357.60000002384186,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":879,"connectEnd":879,"connectStart":879,"domainLookupEnd":879,"domainLookupStart":879,"fetchStart":879,"redirectEnd":0,"redirectStart":0,"requestStart":879,"responseEnd":1236.6000000238419,"responseStart":1236.6000000238419,"secureConnectionStart":879},{"duration":328.09999999403954,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1042.9000000059605,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1042.9000000059605,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1371,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":3,"responseStart":193,"responseEnd":195,"domLoading":196,"domInteractive":1372,"domContentLoadedEventStart":1372,"domContentLoadedEventEnd":1407,"domComplete":2218,"loadEventStart":2218,"loadEventEnd":2220,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1360.300000011921},{"name":"bigPipe.sidebar-id.end","time":1361},{"name":"bigPipe.activity-panel-pipe-id.start","time":1361.1000000238419},{"name":"bigPipe.activity-panel-pipe-id.end","time":1363.2000000178814},{"name":"activityTabFullyLoaded","time":1413.7000000178814}],"measures":[],"correlationId":"bd2f592d879470","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":131,"dbReadsTimeInMs":10,"dbConnsTimeInMs":19,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}