I have a setup with a read/write primary and two replicas which are not being used beyond replication monitoring.
All 3 crashed at different times within 2 months of an upgrade from 10.2.23 with very similar symptoms: innodb triggering an abort() (signal 6) because of a 600s lock acquisition timeout, which further devolved into a segfault (signal 11) because of a null-pointer dereference during diagnostic log production for the abort.
The upgrade for all processes happened simultaneously (all 10.2 stopped, then all 10.3 started, and no process restart until the crashes).
I did not witness the crashes happening on the replicates (we only found out once the replication monitoring notified us of them being unreachable), but I just witnessed a crash on the primary, and I could notice that there were two distinct phases, a first long (until I first noticed the issue) lock sequence, which somehow involved an apparent deadlock. In the attached mariadb-error log, this is the period from 2021-06-16 4:56:21 until 2021-06-16 6:20:24. Then thre is a second phase, eventually showing a similar deadlock, and which did lead to the abort(). This is hte period from 2021-06-16 8:11:08 to the abort() at 2021-06-16 8:23:00.
The deadlock appears first in the automatically-logged "show engine innodb status" at 2021-06-16 05:48:30. Everytime, the deadlock involve the same thread id (140189330171648), which at some point is visible as "Main thread ID" - but for whatever this means the "Main thread ID" changes throughout the incident EDIT: my bad, it stays the same. Still, the deadlocking thread is always the same one.
At 2021-06-16 06:20:26 is the last report of the first phase which still shows the deadlock: it somehow disappears without crash on the report at 2021-06-16 06:20:46, which is also the last report of the first phase.
Then, during the second phase, the deadlock reappears at in the 2021-06-16 08:19:06 report, and is present until the last report at 2021-06-16 08:22:46.
In all cases, the main thread was: Process ID=33661, Main thread ID=140189330171648, state: enforcing dict cache limit
Transition |
Time In Source Status |
Execution Times |
Open |
|
Closed |
|
132d 6h 45m
|
1
|
{"report":{"fcp":951.8000001907349,"ttfb":202.5999994277954,"pageVisibility":"visible","entityId":100617,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"04f7de03-b960-4155-863f-c5a7a5ec0157","navigationType":0,"readyForUser":1027.1999998092651,"redirectCount":0,"resourceLoadedEnd":1030.3999996185303,"resourceLoadedStart":214.19999980926514,"resourceTiming":[{"duration":265.69999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bsh/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":214.19999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":214.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":479.8999996185303,"responseStart":0,"secureConnectionStart":0},{"duration":265.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bsh/820016/12ta74/eb142f92e4bd16bd1ef8b08c1b9d5d56/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":214.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":214.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":480,"responseStart":0,"secureConnectionStart":0},{"duration":276.6000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/s/c54b129276d75dc2a3460e1d78f37913-CDN/lu2bsh/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":214.5999994277954,"connectEnd":214.5999994277954,"connectStart":214.5999994277954,"domainLookupEnd":214.5999994277954,"domainLookupStart":214.5999994277954,"fetchStart":214.5999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":214.5999994277954,"responseEnd":491.19999980926514,"responseStart":491.19999980926514,"secureConnectionStart":214.5999994277954},{"duration":362.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/f867843cd2fdb209d4d1d4f760f86346-CDN/lu2bsh/820016/12ta74/eb142f92e4bd16bd1ef8b08c1b9d5d56/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":214.89999961853027,"connectEnd":214.89999961853027,"connectStart":214.89999961853027,"domainLookupEnd":214.89999961853027,"domainLookupStart":214.89999961853027,"fetchStart":214.89999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":214.89999961853027,"responseEnd":577.3999996185303,"responseStart":577.3999996185303,"secureConnectionStart":214.89999961853027},{"duration":366.4000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/ffdb17665775c5b4d6f097f3974ee359-CDN/lu2bsh/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":215.0999994277954,"connectEnd":215.0999994277954,"connectStart":215.0999994277954,"domainLookupEnd":215.0999994277954,"domainLookupStart":215.0999994277954,"fetchStart":215.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":215.0999994277954,"responseEnd":581.5,"responseStart":581.3999996185303,"secureConnectionStart":215.0999994277954},{"duration":366.69999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":215.19999980926514,"connectEnd":215.19999980926514,"connectStart":215.19999980926514,"domainLookupEnd":215.19999980926514,"domainLookupStart":215.19999980926514,"fetchStart":215.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":215.19999980926514,"responseEnd":581.8999996185303,"responseStart":581.8999996185303,"secureConnectionStart":215.19999980926514},{"duration":367,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":215.30000019073486,"connectEnd":215.30000019073486,"connectStart":215.30000019073486,"domainLookupEnd":215.30000019073486,"domainLookupStart":215.30000019073486,"fetchStart":215.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":215.30000019073486,"responseEnd":582.3000001907349,"responseStart":582.3000001907349,"secureConnectionStart":215.30000019073486},{"duration":367.1000003814697,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bsh/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":215.69999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":215.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":582.8000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":367,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":215.80000019073486,"connectEnd":215.80000019073486,"connectStart":215.80000019073486,"domainLookupEnd":215.80000019073486,"domainLookupStart":215.80000019073486,"fetchStart":215.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":215.80000019073486,"responseEnd":582.8000001907349,"responseStart":582.8000001907349,"secureConnectionStart":215.80000019073486},{"duration":367.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bsh/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":215.89999961853027,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":215.89999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":583.3999996185303,"responseStart":0,"secureConnectionStart":0},{"duration":367.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/81b5d7c27af3ebc078cc4a36383678ba-CDN/lu2bsh/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":216.0999994277954,"connectEnd":216.0999994277954,"connectStart":216.0999994277954,"domainLookupEnd":216.0999994277954,"domainLookupStart":216.0999994277954,"fetchStart":216.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":216.0999994277954,"responseEnd":583.3999996185303,"responseStart":583.3999996185303,"secureConnectionStart":216.0999994277954},{"duration":571.8000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":231,"connectEnd":231,"connectStart":231,"domainLookupEnd":231,"domainLookupStart":231,"fetchStart":231,"redirectEnd":0,"redirectStart":0,"requestStart":231,"responseEnd":802.8000001907349,"responseStart":802.8000001907349,"secureConnectionStart":231},{"duration":774.7999992370605,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":237.30000019073486,"connectEnd":237.30000019073486,"connectStart":237.30000019073486,"domainLookupEnd":237.30000019073486,"domainLookupStart":237.30000019073486,"fetchStart":237.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":237.30000019073486,"responseEnd":1012.0999994277954,"responseStart":1012.0999994277954,"secureConnectionStart":237.30000019073486},{"duration":154.19999980926514,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":649,"connectEnd":649,"connectStart":649,"domainLookupEnd":649,"domainLookupStart":649,"fetchStart":649,"redirectEnd":0,"redirectStart":0,"requestStart":649,"responseEnd":803.1999998092651,"responseStart":803.1999998092651,"secureConnectionStart":649},{"duration":179.30000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2bsh/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":851.0999994277954,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":851.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1030.3999996185303,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":51,"responseStart":202,"responseEnd":238,"domLoading":206,"domInteractive":1082,"domContentLoadedEventStart":1082,"domContentLoadedEventEnd":1126,"domComplete":1529,"loadEventStart":1529,"loadEventEnd":1529,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1062},{"name":"bigPipe.sidebar-id.end","time":1062.6999998092651},{"name":"bigPipe.activity-panel-pipe-id.start","time":1062.8000001907349},{"name":"bigPipe.activity-panel-pipe-id.end","time":1063.6999998092651},{"name":"activityTabFullyLoaded","time":1141.8999996185303}],"measures":[],"correlationId":"b94ece4a595fb3","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":96,"dbReadsTimeInMs":16,"dbConnsTimeInMs":24,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}