cluster got frozen after joiner node hit the following error:
2023-07-10 2:20:50 10 [ERROR] Slave SQL: Could not execute Write_rows_compressed_v1 event on table db.tablex; Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos 1583, Internal MariaDB error code: 1213
|
2023-07-10 2:20:50 103 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table db.tablex; Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos 772, Internal MariaDB error code: 1213
|
2023-07-10 2:20:50 10 [Warning] WSREP: Event 3 Write_rows_compressed_v1 apply failed: 149, seqno 45474182196
|
2023-07-10 2:20:50 103 [Warning] WSREP: Event 3 Write_rows_v1 apply failed: 149, seqno 45474182201
|
2023-07-10 2:20:50 0 [Note] WSREP: Member 0(fx114_db3) initiates vote on e4e16f9c-8779-11eb-b356-c3718f5f11e0:45474182196,8b5f5fd24a23838b: Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213;
|
2023-07-10 2:20:50 0 [Note] WSREP: Member 2(fx114_db2) responds to vote on e4e16f9c-8779-11eb-b356-c3718f5f11e0:45474182196,0000000000000000: Success
|
2023-07-10 2:20:50 0 [Warning] WSREP: Received bogus VOTE message: 45474182196.0, from node f3b6f9e7-1ea8-11ee-bb0a-5bd84ee8291e, expected > 45474389658. Ignoring.
|
2023-07-10 2:20:50 0 [Note] WSREP: Member 1(fx114_db1) responds to vote on e4e16f9c-8779-11eb-b356-c3718f5f11e0:45474182196,0000000000000000: Success
|
2023-07-10 2:20:50 0 [Warning] WSREP: Received bogus VOTE message: 45474182196.0, from node cc3e6460-1ea6-11ee-88f2-4a01f1fdd917, expected > 45474389637. Ignoring.
|
2023-07-10 2:24:49 0 [Note] /usr/local/libexec/mariadbd (initiated by: unknown): Normal shutdown
|
2023-07-10 2:24:49 0 [Note] WSREP: Shutdown replication
|
2023-07-10 2:24:49 0 [Note] WSREP: Server status change joined -> disconnecting
|
An user noted that the following command didn't work in that state and Graceful shutdown executed on each node have no effect. Only possible way to stop the nodes is with kill -9.
SET GLOBAL wsrep_provider_options='pc.bootstrap=YES' ;
|
SET GLOBAL wsrep_provider_options="evs.evict=1de97dad-f609-11e5-8a50-ce2e621b0c42";
|
{"report":{"fcp":712.0999999046326,"ttfb":159.39999985694885,"pageVisibility":"visible","entityId":123341,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"7cdc1b12-ccb8-4f73-86f9-36bcef04f75e","navigationType":0,"readyForUser":785.5,"redirectCount":0,"resourceLoadedEnd":730.5,"resourceLoadedStart":168.09999990463257,"resourceTiming":[{"duration":12.200000047683716,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":168.09999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":168.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":180.29999995231628,"responseStart":0,"secureConnectionStart":0},{"duration":12.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":168.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":168.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":181,"responseStart":0,"secureConnectionStart":0},{"duration":70.60000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":168.69999980926514,"connectEnd":168.69999980926514,"connectStart":168.69999980926514,"domainLookupEnd":168.69999980926514,"domainLookupStart":168.69999980926514,"fetchStart":168.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":168.69999980926514,"responseEnd":239.29999995231628,"responseStart":239.29999995231628,"secureConnectionStart":168.69999980926514},{"duration":127.70000004768372,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":169.19999980926514,"connectEnd":169.19999980926514,"connectStart":169.19999980926514,"domainLookupEnd":169.19999980926514,"domainLookupStart":169.19999980926514,"fetchStart":169.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":169.19999980926514,"responseEnd":296.89999985694885,"responseStart":296.89999985694885,"secureConnectionStart":169.19999980926514},{"duration":131.29999995231628,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":169.39999985694885,"connectEnd":169.39999985694885,"connectStart":169.39999985694885,"domainLookupEnd":169.39999985694885,"domainLookupStart":169.39999985694885,"fetchStart":169.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":169.39999985694885,"responseEnd":300.69999980926514,"responseStart":300.69999980926514,"secureConnectionStart":169.39999985694885},{"duration":131.79999995231628,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":169.39999985694885,"connectEnd":169.39999985694885,"connectStart":169.39999985694885,"domainLookupEnd":169.39999985694885,"domainLookupStart":169.39999985694885,"fetchStart":169.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":169.39999985694885,"responseEnd":301.19999980926514,"responseStart":301.19999980926514,"secureConnectionStart":169.39999985694885},{"duration":132.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":169.5,"connectEnd":169.5,"connectStart":169.5,"domainLookupEnd":169.5,"domainLookupStart":169.5,"fetchStart":169.5,"redirectEnd":0,"redirectStart":0,"requestStart":169.5,"responseEnd":301.59999990463257,"responseStart":301.59999990463257,"secureConnectionStart":169.5},{"duration":132.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":169.69999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":169.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":302.09999990463257,"responseStart":0,"secureConnectionStart":0},{"duration":132.79999995231628,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":169.79999995231628,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":169.79999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":302.59999990463257,"responseStart":0,"secureConnectionStart":0},{"duration":132.29999995231628,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":169.79999995231628,"connectEnd":169.79999995231628,"connectStart":169.79999995231628,"domainLookupEnd":169.79999995231628,"domainLookupStart":169.79999995231628,"fetchStart":169.79999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":169.79999995231628,"responseEnd":302.09999990463257,"responseStart":302.09999990463257,"secureConnectionStart":169.79999995231628},{"duration":132.79999995231628,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":169.89999985694885,"connectEnd":169.89999985694885,"connectStart":169.89999985694885,"domainLookupEnd":169.89999985694885,"domainLookupStart":169.89999985694885,"fetchStart":169.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":169.89999985694885,"responseEnd":302.69999980926514,"responseStart":302.69999980926514,"secureConnectionStart":169.89999985694885},{"duration":361.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":179.89999985694885,"connectEnd":179.89999985694885,"connectStart":179.89999985694885,"domainLookupEnd":179.89999985694885,"domainLookupStart":179.89999985694885,"fetchStart":179.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":179.89999985694885,"responseEnd":541.2999999523163,"responseStart":541.2999999523163,"secureConnectionStart":179.89999985694885},{"duration":431.10000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":180.89999985694885,"connectEnd":180.89999985694885,"connectStart":180.89999985694885,"domainLookupEnd":180.89999985694885,"domainLookupStart":180.89999985694885,"fetchStart":180.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":180.89999985694885,"responseEnd":612,"responseStart":612,"secureConnectionStart":180.89999985694885},{"duration":122.79999995231628,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":470.09999990463257,"connectEnd":470.09999990463257,"connectStart":470.09999990463257,"domainLookupEnd":470.09999990463257,"domainLookupStart":470.09999990463257,"fetchStart":470.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":470.09999990463257,"responseEnd":592.8999998569489,"responseStart":592.8999998569489,"secureConnectionStart":470.09999990463257},{"duration":74.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":656.0999999046326,"connectEnd":656.0999999046326,"connectStart":656.0999999046326,"domainLookupEnd":656.0999999046326,"domainLookupStart":656.0999999046326,"fetchStart":656.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":656.0999999046326,"responseEnd":730.5,"responseStart":730.5,"secureConnectionStart":656.0999999046326},{"duration":162.79999995231628,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":687.3999998569489,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":687.3999998569489,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":850.1999998092651,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":29,"responseStart":159,"responseEnd":180,"domLoading":163,"domInteractive":851,"domContentLoadedEventStart":851,"domContentLoadedEventEnd":894,"domComplete":1461,"loadEventStart":1461,"loadEventEnd":1462,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":828.1999998092651},{"name":"bigPipe.sidebar-id.end","time":829},{"name":"bigPipe.activity-panel-pipe-id.start","time":829.0999999046326},{"name":"bigPipe.activity-panel-pipe-id.end","time":831.5},{"name":"activityTabFullyLoaded","time":903.7999999523163}],"measures":[],"correlationId":"5ee6bef0f2a9e6","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":73,"dbReadsTimeInMs":13,"dbConnsTimeInMs":20,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
MDEV-32122 Node killing could cause applier to fail on write rows event leading inconsistency
