I'm using mysqld_multi to run many instances of mysql on the same machine. Each instance is in a 3 node Galera cluster, so there are three physical boxes running a dozen instances each comprising a dozen clusters.
One instance will always crash given enough time. Sometimes it's less than a day, sometimes it runs for a week, but it always eventually crashes. That instance is no longer in a cluster as I'm trying to troubleshoot it but Galera is still loaded. I've tried not loading the provider as well with the same results.
It is just replicating and not serving any real queries/traffic. I haven't been able to narrow it down to a specific table or query unfortunately. Today was the first time I had another instance crash ever (it's been running for several months). What does stand out though, is that it's always this host that is actually being a slave that crashes. I have not had the other instances of the cluster crash when it was clustered. It's always the one acting as a slave. Someone on #maria IRC mentioned deadlocks with replication and show slave status but I can't confirm anything.
I was encouraged to submit a report and include my core dump traces. The 3rd dump labelled mysql-13 is the new crash I was referring to. The other two are from the same instance that has been repeatedly crashing. When it does crash, I rebuild the data fresh. I have the same data on a host that is running mariadb-server-5.5 (not galera) and has never crashed. It's actually what I use to rebuild/reseed this system from when it crashes.
{"report":{"fcp":1257.0999999046326,"ttfb":426.59999990463257,"pageVisibility":"visible","entityId":36605,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"797695a9-420d-4d1a-9666-f46e04fbc1c3","navigationType":0,"readyForUser":1334.3999998569489,"redirectCount":0,"resourceLoadedEnd":1400.2000000476837,"resourceLoadedStart":434.89999985694885,"resourceTiming":[{"duration":225.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":434.89999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":434.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":660.2999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":225.70000004768372,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":435.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":435.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":661,"responseStart":0,"secureConnectionStart":0},{"duration":291.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":436.2000000476837,"connectEnd":436.2000000476837,"connectStart":436.2000000476837,"domainLookupEnd":436.2000000476837,"domainLookupStart":436.2000000476837,"fetchStart":436.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":436.2000000476837,"responseEnd":728,"responseStart":728,"secureConnectionStart":436.2000000476837},{"duration":349,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":436.89999985694885,"connectEnd":436.89999985694885,"connectStart":436.89999985694885,"domainLookupEnd":436.89999985694885,"domainLookupStart":436.89999985694885,"fetchStart":436.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":436.89999985694885,"responseEnd":785.8999998569489,"responseStart":785.8999998569489,"secureConnectionStart":436.89999985694885},{"duration":352.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":437.2000000476837,"connectEnd":437.2000000476837,"connectStart":437.2000000476837,"domainLookupEnd":437.2000000476837,"domainLookupStart":437.2000000476837,"fetchStart":437.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":437.2000000476837,"responseEnd":790,"responseStart":790,"secureConnectionStart":437.2000000476837},{"duration":353,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":437.39999985694885,"connectEnd":437.39999985694885,"connectStart":437.39999985694885,"domainLookupEnd":437.39999985694885,"domainLookupStart":437.39999985694885,"fetchStart":437.39999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":437.39999985694885,"responseEnd":790.3999998569489,"responseStart":790.3999998569489,"secureConnectionStart":437.39999985694885},{"duration":353.2999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":437.5,"connectEnd":437.5,"connectStart":437.5,"domainLookupEnd":437.5,"domainLookupStart":437.5,"fetchStart":437.5,"redirectEnd":0,"redirectStart":0,"requestStart":437.5,"responseEnd":790.7999999523163,"responseStart":790.7999999523163,"secureConnectionStart":437.5},{"duration":435.7000000476837,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":437.7999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":437.7999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":873.5,"responseStart":0,"secureConnectionStart":0},{"duration":353.5,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":437.89999985694885,"connectEnd":437.89999985694885,"connectStart":437.89999985694885,"domainLookupEnd":437.89999985694885,"domainLookupStart":437.89999985694885,"fetchStart":437.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":437.89999985694885,"responseEnd":791.3999998569489,"responseStart":791.3999998569489,"secureConnectionStart":437.89999985694885},{"duration":435.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":438,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":438,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":873.5999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":353.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":438.2999999523163,"connectEnd":438.2999999523163,"connectStart":438.2999999523163,"domainLookupEnd":438.2999999523163,"domainLookupStart":438.2999999523163,"fetchStart":438.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":438.2999999523163,"responseEnd":791.8999998569489,"responseStart":791.8999998569489,"secureConnectionStart":438.2999999523163},{"duration":598.2999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":440.09999990463257,"connectEnd":440.09999990463257,"connectStart":440.09999990463257,"domainLookupEnd":440.09999990463257,"domainLookupStart":440.09999990463257,"fetchStart":440.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":440.09999990463257,"responseEnd":1038.3999998569489,"responseStart":1038.3999998569489,"secureConnectionStart":440.09999990463257},{"duration":803.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":440.7000000476837,"connectEnd":440.7000000476837,"connectStart":440.7000000476837,"domainLookupEnd":440.7000000476837,"domainLookupStart":440.7000000476837,"fetchStart":440.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":440.7000000476837,"responseEnd":1243.7999999523163,"responseStart":1243.7999999523163,"secureConnectionStart":440.7000000476837},{"duration":153.79999995231628,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":885.0999999046326,"connectEnd":885.0999999046326,"connectStart":885.0999999046326,"domainLookupEnd":885.0999999046326,"domainLookupStart":885.0999999046326,"fetchStart":885.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":885.0999999046326,"responseEnd":1038.8999998569489,"responseStart":1038.8999998569489,"secureConnectionStart":885.0999999046326},{"duration":96.69999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1153.7000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1153.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1250.3999998569489,"responseStart":0,"secureConnectionStart":0},{"duration":96.29999995231628,"initiatorType":"link","name":"https://jira.mariadb.org/s/50bc9be5bfead1a25e72c1a9338c94f6-CDN/lu2cib/820016/12ta74/e108c7645258ccb43280ed3404e3e949/_/download/contextbatch/css/com.atlassian.jira.plugins.jira-development-integration-plugin:0,-_super,-jira.view.issue,-jira.global,-jira.general,-jira.browse.project,-project.issue.navigator,-atl.general/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1154.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1154.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1250.5999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":245,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1155.2000000476837,"connectEnd":1155.2000000476837,"connectStart":1155.2000000476837,"domainLookupEnd":1155.2000000476837,"domainLookupStart":1155.2000000476837,"fetchStart":1155.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":1155.2000000476837,"responseEnd":1400.2000000476837,"responseStart":1400.2000000476837,"secureConnectionStart":1155.2000000476837},{"duration":251.90000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/097ae97cb8fbec7d6ea4bbb1f26955b9-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":1155.5999999046326,"connectEnd":1155.5999999046326,"connectStart":1155.5999999046326,"domainLookupEnd":1155.5999999046326,"domainLookupStart":1155.5999999046326,"fetchStart":1155.5999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":1155.5999999046326,"responseEnd":1407.5,"responseStart":1407.5,"secureConnectionStart":1155.5999999046326},{"duration":252.89999985694885,"initiatorType":"script","name":"https://jira.mariadb.org/s/e0bf5781d46ea69fb123572974cf39de-CDN/lu2cib/820016/12ta74/e108c7645258ccb43280ed3404e3e949/_/download/contextbatch/js/com.atlassian.jira.plugins.jira-development-integration-plugin:0,-_super,-jira.view.issue,-jira.global,-jira.general,-jira.browse.project,-project.issue.navigator,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":1156,"connectEnd":1156,"connectStart":1156,"domainLookupEnd":1156,"domainLookupStart":1156,"fetchStart":1156,"redirectEnd":0,"redirectStart":0,"requestStart":1156,"responseEnd":1408.8999998569489,"responseStart":1408.8999998569489,"secureConnectionStart":1156}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":275,"responseStart":426,"responseEnd":440,"domLoading":430,"domInteractive":1400,"domContentLoadedEventStart":1401,"domContentLoadedEventEnd":1462,"domComplete":2039,"loadEventStart":2039,"loadEventEnd":2040,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1371},{"name":"bigPipe.sidebar-id.end","time":1371.7999999523163},{"name":"bigPipe.activity-panel-pipe-id.start","time":1371.8999998569489},{"name":"bigPipe.activity-panel-pipe-id.end","time":1373.7999999523163},{"name":"activityTabFullyLoaded","time":1480.7000000476837}],"measures":[],"correlationId":"5be64f1f377490","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":95,"dbReadsTimeInMs":8,"dbConnsTimeInMs":16,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Hi,
Could you also attach here full unedited error log (at least one) from crashing server.
R: Jan