Sprint:
10.2.1-3, 10.2.1-4, 10.2.1-5, 10.0.26
From support issue.
A customer sporadically experiences an issue where the replication thread gets in state 'Table lock'. Other threads trying to access this table will get into the 'Waiting for table level lock' or 'Waiting for table lock' state. This lock seams to last indefinitely.
When this occurs, a normal shutdown of the server is no longer possible.
The setup is quite complicated, the same setup as described in the related issues:
Master A < - > Master B < - > Master C
Master B and C each have 3 slaves.
The table lock issue occured on master B. Master A receives most write traffic.
The SHOW OPEN TABLES command shows there is 1 table open:
Time: 2016-05-13 02:12:56
Database: customer_project
Table: LockedTableName
In_use: 1
Name_locked: 0
The processlist shows the SQL thread in 'Table lock' state:
18438 system user NULL Connect 4270 Table lock NULL 0.000
Sometimes the thread state is 'Waiting for table level lock' or it could be that the thread state changes from 'Waiting for table level lock' to 'Table lock' after it has been killed.
All tables are InnoDB. If a thread is waiting for a MDL the thread state is different.
Hopefully the next time we can generate a coredump. Adding the issue for now in case other people experience it as well. I don't expect this (for now) to give you enough information to work on the issue.
This issue occurs independently or possible before the related issues MDEV-9952 and MDEV-9670 occur. The issues do seam very much related (they occur in the same setup and usually not with much time in between). Hopefully this issue can provide some more ideas to the root cause.
{"report":{"fcp":989.7000002861023,"ttfb":207.80000019073486,"pageVisibility":"visible","entityId":56731,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":32,"apdex":0.5,"journeyId":"a2b20890-6c84-4d7d-a8de-0123b9fe03a8","navigationType":0,"readyForUser":1083.9000000953674,"redirectCount":0,"resourceLoadedEnd":1108.3000001907349,"resourceLoadedStart":213.2000002861023,"resourceTiming":[{"duration":284.8999996185303,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":213.2000002861023,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":213.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":498.09999990463257,"responseStart":0,"secureConnectionStart":0},{"duration":284.90000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":213.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":213.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":498.40000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":283.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/e9b27a47da5fb0f74a35acd57e9847fb-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":213.59999990463257,"connectEnd":213.59999990463257,"connectStart":213.59999990463257,"domainLookupEnd":213.59999990463257,"domainLookupStart":213.59999990463257,"fetchStart":213.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":213.59999990463257,"responseEnd":497,"responseStart":497,"secureConnectionStart":213.59999990463257},{"duration":381,"initiatorType":"script","name":"https://jira.mariadb.org/s/c32eb0da7ad9831253f8397e6cc26afd-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":213.80000019073486,"connectEnd":213.80000019073486,"connectStart":213.80000019073486,"domainLookupEnd":213.80000019073486,"domainLookupStart":213.80000019073486,"fetchStart":213.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":213.80000019073486,"responseEnd":594.8000001907349,"responseStart":594.8000001907349,"secureConnectionStart":213.80000019073486},{"duration":384.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/bc0bcb146314416123c992714ee00ff7-CDN/lu2bv2/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":214,"connectEnd":214,"connectStart":214,"domainLookupEnd":214,"domainLookupStart":214,"fetchStart":214,"redirectEnd":0,"redirectStart":0,"requestStart":214,"responseEnd":598.8000001907349,"responseStart":598.8000001907349,"secureConnectionStart":214},{"duration":385,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":214.2000002861023,"connectEnd":214.2000002861023,"connectStart":214.2000002861023,"domainLookupEnd":214.2000002861023,"domainLookupStart":214.2000002861023,"fetchStart":214.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":214.2000002861023,"responseEnd":599.2000002861023,"responseStart":599.2000002861023,"secureConnectionStart":214.2000002861023},{"duration":385.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":214.40000009536743,"connectEnd":214.40000009536743,"connectStart":214.40000009536743,"domainLookupEnd":214.40000009536743,"domainLookupStart":214.40000009536743,"fetchStart":214.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":214.40000009536743,"responseEnd":599.7000002861023,"responseStart":599.7000002861023,"secureConnectionStart":214.40000009536743},{"duration":446.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bv2/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":214.59999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":214.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":661.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":385.3999996185303,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":214.7000002861023,"connectEnd":214.7000002861023,"connectStart":214.7000002861023,"domainLookupEnd":214.7000002861023,"domainLookupStart":214.7000002861023,"fetchStart":214.7000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":214.7000002861023,"responseEnd":600.0999999046326,"responseStart":600.0999999046326,"secureConnectionStart":214.7000002861023},{"duration":446.30000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":214.90000009536743,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":214.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":661.2000002861023,"responseStart":0,"secureConnectionStart":0},{"duration":385.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/719848dd97ebe0663199f49a3936487a-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":215.09999990463257,"connectEnd":215.09999990463257,"connectStart":215.09999990463257,"domainLookupEnd":215.09999990463257,"domainLookupStart":215.09999990463257,"fetchStart":215.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":215.09999990463257,"responseEnd":600.9000000953674,"responseStart":600.9000000953674,"secureConnectionStart":215.09999990463257},{"duration":824.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":216.09999990463257,"connectEnd":216.09999990463257,"connectStart":216.09999990463257,"domainLookupEnd":216.09999990463257,"domainLookupStart":216.09999990463257,"fetchStart":216.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":216.09999990463257,"responseEnd":1040.5,"responseStart":1040.5,"secureConnectionStart":216.09999990463257},{"duration":824.6999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":216.2000002861023,"connectEnd":216.2000002861023,"connectStart":216.2000002861023,"domainLookupEnd":216.2000002861023,"domainLookupStart":216.2000002861023,"fetchStart":216.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":216.2000002861023,"responseEnd":1040.9000000953674,"responseStart":1040.9000000953674,"secureConnectionStart":216.2000002861023},{"duration":110,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":676.2000002861023,"connectEnd":676.2000002861023,"connectStart":676.2000002861023,"domainLookupEnd":676.2000002861023,"domainLookupStart":676.2000002861023,"fetchStart":676.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":676.2000002861023,"responseEnd":786.2000002861023,"responseStart":786.2000002861023,"secureConnectionStart":676.2000002861023},{"duration":179.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2bv2/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":928.9000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":928.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1108.3000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":141.2999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":929.8000001907349,"connectEnd":929.8000001907349,"connectStart":929.8000001907349,"domainLookupEnd":929.8000001907349,"domainLookupStart":929.8000001907349,"fetchStart":929.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":929.8000001907349,"responseEnd":1071.0999999046326,"responseStart":1071.0999999046326,"secureConnectionStart":929.8000001907349},{"duration":146.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/53a43b6764f587426c7bb9a150184c00-CDN/lu2bv2/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":930.2000002861023,"connectEnd":930.2000002861023,"connectStart":930.2000002861023,"domainLookupEnd":930.2000002861023,"domainLookupStart":930.2000002861023,"fetchStart":930.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":930.2000002861023,"responseEnd":1076.4000000953674,"responseStart":1076.4000000953674,"secureConnectionStart":930.2000002861023},{"duration":179,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":981.9000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":981.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1160.9000000953674,"responseStart":0,"secureConnectionStart":0}],"fetchStart":1,"domainLookupStart":1,"domainLookupEnd":1,"connectStart":1,"connectEnd":1,"requestStart":58,"responseStart":208,"responseEnd":211,"domLoading":212,"domInteractive":1205,"domContentLoadedEventStart":1205,"domContentLoadedEventEnd":1245,"domComplete":2026,"loadEventStart":2026,"loadEventEnd":2028,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1169.5999999046326},{"name":"bigPipe.sidebar-id.end","time":1170.4000000953674},{"name":"bigPipe.activity-panel-pipe-id.start","time":1170.5},{"name":"bigPipe.activity-panel-pipe-id.end","time":1173.3000001907349},{"name":"activityTabFullyLoaded","time":1250.9000000953674}],"measures":[],"correlationId":"a21a393e2c7831","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":94,"dbReadsTimeInMs":12,"dbConnsTimeInMs":20,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}