Hi ,
We got a crash of mysqld service after many events with semaphore waiting
InnoDB: ###### Diagnostic info printed to the standard error stream
|
2022-12-19 2:34:16 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
|
221219 2:34:16 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.4.26-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=8388608
|
max_used_connections=1839
|
max_threads=2002
|
thread_count=1845
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 82182626 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
dmesg had next records
[Mon Dec 19 02:38:09 2022] mysqld[11780]: segfault at 0 ip 000055835b30ca2e sp 00007e9454072a50 error 6 in mysqld[55835a50f000+146b000]
|
[Mon Dec 19 02:38:09 2022] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 57 59 78 ff 49 89 c7 49 39 c4 0f 84 bb 02 00 00 e8 e6 18 00 00 41 8b 4d 00 <89> 08 85 c9 74 1c 49 83 ff ff 0f 84 e2 01 00 00 4d 85 ff 75 27 83
|
Some variables
MariaDB [(none)]> show global variables like '%buffer_size' ;
|
+----------------------------+-----------+
|
| Variable_name | Value |
|
+----------------------------+-----------+
|
| aria_pagecache_buffer_size | 134217728 |
|
| aria_sort_buffer_size | 268434432 |
|
| bulk_insert_buffer_size | 8388608 |
|
| innodb_log_buffer_size | 33554432 |
|
| innodb_sort_buffer_size | 1048576 |
|
| join_buffer_size | 33554432 |
|
| key_buffer_size | 134217728 |
|
| mrr_buffer_size | 262144 |
|
| myisam_sort_buffer_size | 134216704 |
|
| preload_buffer_size | 32768 |
|
| read_buffer_size | 8388608 |
|
| read_rnd_buffer_size | 8388608 |
|
| sort_buffer_size | 33554432 |
|
+----------------------------+-----------+
|
13 rows in set (0.001 sec)
|
|
MariaDB [(none)]> select @@max_connections ;
|
+-------------------+
|
| @@max_connections |
|
+-------------------+
|
| 2000 |
|
+-------------------+
|
1 row in set (0.000 sec)
|
I found that the same issue was in MDEV-25955 , and described that it fixed at 10.4.18
but I have this with 10.4.26 version
We rapidly almost reached limit of connections , but strange that this is caused a lot of locks waiting and as result - segfault
I've uploaded archive with mysqld log and system variables - MDEV-30271.zip
to ftp://ftp.mariadb.org/private/
{"report":{"fcp":966.6999998092651,"ttfb":317.69999980926514,"pageVisibility":"visible","entityId":117783,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"e5ba398a-1694-4231-8e11-4b572385d36e","navigationType":0,"readyForUser":1055.5999994277954,"redirectCount":0,"resourceLoadedEnd":1501.1999998092651,"resourceLoadedStart":322.69999980926514,"resourceTiming":[{"duration":182.10000038146973,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":322.69999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":322.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":504.80000019073486,"responseStart":0,"secureConnectionStart":0},{"duration":182.19999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":323,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":323,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":505.19999980926514,"responseStart":0,"secureConnectionStart":0},{"duration":190.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":323.19999980926514,"connectEnd":323.19999980926514,"connectStart":323.19999980926514,"domainLookupEnd":323.19999980926514,"domainLookupStart":323.19999980926514,"fetchStart":323.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":323.19999980926514,"responseEnd":514,"responseStart":514,"secureConnectionStart":323.19999980926514},{"duration":272.6000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":323.3999996185303,"connectEnd":323.3999996185303,"connectStart":323.3999996185303,"domainLookupEnd":323.3999996185303,"domainLookupStart":323.3999996185303,"fetchStart":323.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":323.3999996185303,"responseEnd":596,"responseStart":596,"secureConnectionStart":323.3999996185303},{"duration":276.4000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":323.5999994277954,"connectEnd":323.5999994277954,"connectStart":323.5999994277954,"domainLookupEnd":323.5999994277954,"domainLookupStart":323.5999994277954,"fetchStart":323.5999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":323.5999994277954,"responseEnd":600,"responseStart":600,"secureConnectionStart":323.5999994277954},{"duration":276.5999994277954,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":323.80000019073486,"connectEnd":323.80000019073486,"connectStart":323.80000019073486,"domainLookupEnd":323.80000019073486,"domainLookupStart":323.80000019073486,"fetchStart":323.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":323.80000019073486,"responseEnd":600.3999996185303,"responseStart":600.3999996185303,"secureConnectionStart":323.80000019073486},{"duration":276.9000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":323.8999996185303,"connectEnd":323.8999996185303,"connectStart":323.8999996185303,"domainLookupEnd":323.8999996185303,"domainLookupStart":323.8999996185303,"fetchStart":323.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":323.8999996185303,"responseEnd":600.8000001907349,"responseStart":600.8000001907349,"secureConnectionStart":323.8999996185303},{"duration":334.20000076293945,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":324.0999994277954,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":324.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":658.3000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":276.8999996185303,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":324.30000019073486,"connectEnd":324.30000019073486,"connectStart":324.30000019073486,"domainLookupEnd":324.30000019073486,"domainLookupStart":324.30000019073486,"fetchStart":324.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":324.30000019073486,"responseEnd":601.1999998092651,"responseStart":601.1999998092651,"secureConnectionStart":324.30000019073486},{"duration":334,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":324.3999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":324.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":658.3999996185303,"responseStart":0,"secureConnectionStart":0},{"duration":277.20000076293945,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":324.5999994277954,"connectEnd":324.5999994277954,"connectStart":324.5999994277954,"domainLookupEnd":324.5999994277954,"domainLookupStart":324.5999994277954,"fetchStart":324.5999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":324.5999994277954,"responseEnd":601.8000001907349,"responseStart":601.8000001907349,"secureConnectionStart":324.5999994277954},{"duration":1169.9000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":329.8999996185303,"connectEnd":329.8999996185303,"connectStart":329.8999996185303,"domainLookupEnd":329.8999996185303,"domainLookupStart":329.8999996185303,"fetchStart":329.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":329.8999996185303,"responseEnd":1499.8000001907349,"responseStart":1499.8000001907349,"secureConnectionStart":329.8999996185303},{"duration":1171.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":330,"connectEnd":330,"connectStart":330,"domainLookupEnd":330,"domainLookupStart":330,"fetchStart":330,"redirectEnd":0,"redirectStart":0,"requestStart":330,"responseEnd":1501.1999998092651,"responseStart":1501.1999998092651,"secureConnectionStart":330},{"duration":619,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":670.3000001907349,"connectEnd":670.3000001907349,"connectStart":670.3000001907349,"domainLookupEnd":670.3000001907349,"domainLookupStart":670.3000001907349,"fetchStart":670.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":670.3000001907349,"responseEnd":1289.3000001907349,"responseStart":1289.3000001907349,"secureConnectionStart":670.3000001907349},{"duration":544,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":960.8000001907349,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":960.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1504.8000001907349,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":120,"responseStart":318,"responseEnd":324,"domLoading":321,"domInteractive":1520,"domContentLoadedEventStart":1520,"domContentLoadedEventEnd":1563,"domComplete":2715,"loadEventStart":2715,"loadEventEnd":2717,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1505.6999998092651},{"name":"bigPipe.sidebar-id.end","time":1506.5999994277954},{"name":"bigPipe.activity-panel-pipe-id.start","time":1506.8000001907349},{"name":"bigPipe.activity-panel-pipe-id.end","time":1509.8999996185303},{"name":"activityTabFullyLoaded","time":1571.5}],"measures":[],"correlationId":"db47658ef56e97","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":130,"dbReadsTimeInMs":22,"dbConnsTimeInMs":41,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}