GTID sequence drifts between Write node and appliers node by issueing some KILL QUERY statements on a Galera node which is operating user's transactions.
Data consistency is maintained and wsrep_last_committed is in sync.
NOTE: It's not always sufficient to issue one single KILL QUERY, I should do more fine grained research to determine what are the specific circumstances, in my current tests I reproduced it by killing some random mysqlslap queries.
GTID sequence de-alignment breaks among others maxscale auto-failover.
To make things more complicated in the current attached pictures test Writer is behind but in other tests Writer went ahead(see attached txt for 2nd test)
is part of
MDEV-29293
MariaDB stuck on starting commit state (waiting on commit order critical section)
Closed
Claudio Nanni
made changes -
2023-04-18 10:26
Field
Original Value
New Value
Attachment
2nd_Test-galera_gtid_drift_kill_query.txt
[ 69569
]
Claudio Nanni
made changes -
2023-04-18 10:27
Attachment
2nd_Test-galera_gtid_drift_kill_query.txt
[ 69569
]
Julien Fritsch
made changes -
2023-04-18 16:39
Fix Version/s
10.4
[ 22408
]
Fix Version/s
10.5
[ 23123
]
Fix Version/s
10.6
[ 24028
]
Fix Version/s
10.3
[ 22126
]
Julien Fritsch
made changes -
2023-04-19 06:48
Fix Version/s
10.3
[ 22126
]
Fix Version/s
10.4
[ 22408
]
Fix Version/s
10.6
[ 24028
]
Claudio Nanni
made changes -
2023-04-19 07:44
Description
GTID sequence drifts between Write node and appliers node by issueing some KILL QUERY statements on a Galera node which is operating user's transactions.
Data consistency is maintained and wsrep_last_committed is in sync.
GTID sequence de-alignment breaks among others maxscale auto-failover.
To make things more complicated in the current attached pictures test Writer is behind but in other tests Writer went ahead(see attached txt for 2nd test)
GTID sequence drifts between Write node and appliers node by issueing some KILL QUERY statements on a Galera node which is operating user's transactions.
Data consistency is maintained and wsrep_last_committed is in sync.
NOTE: It's not always sufficient to issue one single KILL QUERY, I should do more fine grained research to determine what are the specific circumstances, in my current tests I reproduced it by killing some random mysqlslap queries.
GTID sequence de-alignment breaks among others maxscale auto-failover.
To make things more complicated in the current attached pictures test Writer is behind but in other tests Writer went ahead(see attached txt for 2nd test)
Ralf Gebhardt
made changes -
2023-04-20 17:11
Component/s
Galera
[ 10124
]
Assignee
Jan Lindström
[ JIRAUSER53125
]
Jan Lindström
made changes -
2023-04-21 05:27
Status
Open
[ 1
]
In Progress
[ 3
]
Julien Fritsch
made changes -
2023-04-26 09:33
Priority
Major
[ 3
]
Critical
[ 2
]
Julien Fritsch
made changes -
2023-05-30 12:55
Status
In Progress
[ 3
]
Needs Feedback
[ 10501
]
Claudio Nanni
made changes -
2023-05-30 15:47
Status
Needs Feedback
[ 10501
]
Open
[ 1
]
Jan Lindström
made changes -
2023-06-06 08:55
Fix Version/s
10.5.21
[ 28913
]
Fix Version/s
10.6.14
[ 28914
]
Fix Version/s
10.9.7
[ 28916
]
Fix Version/s
10.10.5
[ 28917
]
Fix Version/s
10.11.4
[ 28918
]
Fix Version/s
10.5
[ 23123
]
Resolution
Fixed
[ 1
]
Status
Open
[ 1
]
Closed
[ 6
]
Sergei Golubchik
made changes -
2023-06-07 15:12
Fix Version/s
10.5.22
[ 29011
]
Fix Version/s
10.6.15
[ 29013
]
Fix Version/s
10.9.8
[ 29015
]
Fix Version/s
10.10.6
[ 29017
]
Fix Version/s
10.11.5
[ 29019
]
Fix Version/s
10.5.21
[ 28913
]
Fix Version/s
10.6.14
[ 28914
]
Fix Version/s
10.9.7
[ 28916
]
Fix Version/s
10.10.5
[ 28917
]
Fix Version/s
10.11.4
[ 28918
]
{"report":{"fcp":1232.3000001907349,"ttfb":620.4000000953674,"pageVisibility":"visible","entityId":121153,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":32,"apdex":0.5,"journeyId":"fb915630-564b-4692-8d99-cea74d760c28","navigationType":0,"readyForUser":1307,"redirectCount":0,"resourceLoadedEnd":1536.6000003814697,"resourceLoadedStart":635.6000003814697,"resourceTiming":[{"duration":114.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":635.6000003814697,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":635.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":750.2000002861023,"responseStart":0,"secureConnectionStart":0},{"duration":114.69999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":635.8000001907349,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":635.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":750.5,"responseStart":0,"secureConnectionStart":0},{"duration":133.7000002861023,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":636,"connectEnd":636,"connectStart":636,"domainLookupEnd":636,"domainLookupStart":636,"fetchStart":636,"redirectEnd":0,"redirectStart":0,"requestStart":636,"responseEnd":769.7000002861023,"responseStart":769.7000002861023,"secureConnectionStart":636},{"duration":209.90000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":636.3000001907349,"connectEnd":636.3000001907349,"connectStart":636.3000001907349,"domainLookupEnd":636.3000001907349,"domainLookupStart":636.3000001907349,"fetchStart":636.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":636.3000001907349,"responseEnd":846.2000002861023,"responseStart":846.2000002861023,"secureConnectionStart":636.3000001907349},{"duration":213.90000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":636.5,"connectEnd":636.5,"connectStart":636.5,"domainLookupEnd":636.5,"domainLookupStart":636.5,"fetchStart":636.5,"redirectEnd":0,"redirectStart":0,"requestStart":636.5,"responseEnd":850.4000000953674,"responseStart":850.4000000953674,"secureConnectionStart":636.5},{"duration":214.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":636.7000002861023,"connectEnd":636.7000002861023,"connectStart":636.7000002861023,"domainLookupEnd":636.7000002861023,"domainLookupStart":636.7000002861023,"fetchStart":636.7000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":636.7000002861023,"responseEnd":851.1000003814697,"responseStart":851.1000003814697,"secureConnectionStart":636.7000002861023},{"duration":215.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":636.9000000953674,"connectEnd":636.9000000953674,"connectStart":636.9000000953674,"domainLookupEnd":636.9000000953674,"domainLookupStart":636.9000000953674,"fetchStart":636.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":636.9000000953674,"responseEnd":852,"responseStart":852,"secureConnectionStart":636.9000000953674},{"duration":274.80000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":637,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":637,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":911.8000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":215.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":637.2000002861023,"connectEnd":637.2000002861023,"connectStart":637.2000002861023,"domainLookupEnd":637.2000002861023,"domainLookupStart":637.2000002861023,"fetchStart":637.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":637.2000002861023,"responseEnd":852.6000003814697,"responseStart":852.5,"secureConnectionStart":637.2000002861023},{"duration":274.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":637.4000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":637.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":911.9000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":215.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":637.6000003814697,"connectEnd":637.6000003814697,"connectStart":637.6000003814697,"domainLookupEnd":637.6000003814697,"domainLookupStart":637.6000003814697,"fetchStart":637.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":637.6000003814697,"responseEnd":853.1000003814697,"responseStart":853.1000003814697,"secureConnectionStart":637.6000003814697},{"duration":428.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":638.4000000953674,"connectEnd":638.4000000953674,"connectStart":638.4000000953674,"domainLookupEnd":638.4000000953674,"domainLookupStart":638.4000000953674,"fetchStart":638.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":638.4000000953674,"responseEnd":1066.8000001907349,"responseStart":1066.8000001907349,"secureConnectionStart":638.4000000953674},{"duration":826.8000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":638.5,"connectEnd":638.5,"connectStart":638.5,"domainLookupEnd":638.5,"domainLookupStart":638.5,"fetchStart":638.5,"redirectEnd":0,"redirectStart":0,"requestStart":638.5,"responseEnd":1465.3000001907349,"responseStart":1465.3000001907349,"secureConnectionStart":638.5},{"duration":372.7000002861023,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":924,"connectEnd":924,"connectStart":924,"domainLookupEnd":924,"domainLookupStart":924,"fetchStart":924,"redirectEnd":0,"redirectStart":0,"requestStart":924,"responseEnd":1296.7000002861023,"responseStart":1296.7000002861023,"secureConnectionStart":924},{"duration":276.30000019073486,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1223,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1223,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1499.3000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":190.10000038146973,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1346,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1346,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1536.1000003814697,"responseStart":0,"secureConnectionStart":0},{"duration":182.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1347,"connectEnd":1347,"connectStart":1347,"domainLookupEnd":1347,"domainLookupStart":1347,"fetchStart":1347,"redirectEnd":0,"redirectStart":0,"requestStart":1347,"responseEnd":1529.4000000953674,"responseStart":1529.4000000953674,"secureConnectionStart":1347},{"duration":189.10000038146973,"initiatorType":"script","name":"https://jira.mariadb.org/s/097ae97cb8fbec7d6ea4bbb1f26955b9-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":1347.5,"connectEnd":1347.5,"connectStart":1347.5,"domainLookupEnd":1347.5,"domainLookupStart":1347.5,"fetchStart":1347.5,"redirectEnd":0,"redirectStart":0,"requestStart":1347.5,"responseEnd":1536.6000003814697,"responseStart":1536.6000003814697,"secureConnectionStart":1347.5}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":100,"responseStart":620,"responseEnd":626,"domLoading":632,"domInteractive":1550,"domContentLoadedEventStart":1550,"domContentLoadedEventEnd":1599,"domComplete":1827,"loadEventStart":1827,"loadEventEnd":1828,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1467.2000002861023},{"name":"bigPipe.sidebar-id.end","time":1468},{"name":"bigPipe.activity-panel-pipe-id.start","time":1468.1000003814697},{"name":"bigPipe.activity-panel-pipe-id.end","time":1476.2000002861023},{"name":"activityTabFullyLoaded","time":1619.5}],"measures":[],"correlationId":"8f45be503bd4d7","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":198,"dbReadsTimeInMs":30,"dbConnsTimeInMs":42,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
If there was some issue with KILL and GTID it was fixed on
MDEV-29293. However there is repeatable test case with INSERT IGNORE with duplicate key causing GTID sequence breakage and there is also problem with INSERT with at least one successful write and one duplicate key.