Hi ,
We got a crash of mysqld service after many events with semaphore waiting
InnoDB: ###### Diagnostic info printed to the standard error stream
|
2022-12-19 2:34:16 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
|
221219 2:34:16 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.4.26-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=8388608
|
max_used_connections=1839
|
max_threads=2002
|
thread_count=1845
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 82182626 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x0
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000
|
dmesg had next records
[Mon Dec 19 02:38:09 2022] mysqld[11780]: segfault at 0 ip 000055835b30ca2e sp 00007e9454072a50 error 6 in mysqld[55835a50f000+146b000]
|
[Mon Dec 19 02:38:09 2022] Code: c7 45 00 00 00 00 00 8b 7d cc 4c 89 e2 4c 89 f6 e8 57 59 78 ff 49 89 c7 49 39 c4 0f 84 bb 02 00 00 e8 e6 18 00 00 41 8b 4d 00 <89> 08 85 c9 74 1c 49 83 ff ff 0f 84 e2 01 00 00 4d 85 ff 75 27 83
|
Some variables
MariaDB [(none)]> show global variables like '%buffer_size' ;
|
+----------------------------+-----------+
|
| Variable_name | Value |
|
+----------------------------+-----------+
|
| aria_pagecache_buffer_size | 134217728 |
|
| aria_sort_buffer_size | 268434432 |
|
| bulk_insert_buffer_size | 8388608 |
|
| innodb_log_buffer_size | 33554432 |
|
| innodb_sort_buffer_size | 1048576 |
|
| join_buffer_size | 33554432 |
|
| key_buffer_size | 134217728 |
|
| mrr_buffer_size | 262144 |
|
| myisam_sort_buffer_size | 134216704 |
|
| preload_buffer_size | 32768 |
|
| read_buffer_size | 8388608 |
|
| read_rnd_buffer_size | 8388608 |
|
| sort_buffer_size | 33554432 |
|
+----------------------------+-----------+
|
13 rows in set (0.001 sec)
|
|
MariaDB [(none)]> select @@max_connections ;
|
+-------------------+
|
| @@max_connections |
|
+-------------------+
|
| 2000 |
|
+-------------------+
|
1 row in set (0.000 sec)
|
I found that the same issue was in MDEV-25955 , and described that it fixed at 10.4.18
but I have this with 10.4.26 version
We rapidly almost reached limit of connections , but strange that this is caused a lot of locks waiting and as result - segfault
I've uploaded archive with mysqld log and system variables - MDEV-30271.zip
to ftp://ftp.mariadb.org/private/
Transition |
Time In Source Status |
Execution Times |
Open |
|
Closed |
|
62d 21h 32m
|
1
|
{"report":{"fcp":2160.600000143051,"ttfb":681.7000000476837,"pageVisibility":"visible","entityId":117783,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":32,"apdex":0.5,"journeyId":"37246bc2-13a8-4fca-9f11-baf4e2c28e46","navigationType":0,"readyForUser":2293.5,"redirectCount":0,"resourceLoadedEnd":2612.2000000476837,"resourceLoadedStart":700.7000000476837,"resourceTiming":[{"duration":724,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":700.7000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":700.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1424.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":724.4000000953674,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":701,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":701,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1425.4000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":787.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":701.2000000476837,"connectEnd":701.2000000476837,"connectStart":701.2000000476837,"domainLookupEnd":701.2000000476837,"domainLookupStart":701.2000000476837,"fetchStart":701.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":701.2000000476837,"responseEnd":1488.6000001430511,"responseStart":1488.6000001430511,"secureConnectionStart":701.2000000476837},{"duration":906.9000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":701.4000000953674,"connectEnd":701.4000000953674,"connectStart":701.4000000953674,"domainLookupEnd":701.4000000953674,"domainLookupStart":701.4000000953674,"fetchStart":701.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":701.4000000953674,"responseEnd":1608.3000001907349,"responseStart":1608.3000001907349,"secureConnectionStart":701.4000000953674},{"duration":910.6000001430511,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":701.5,"connectEnd":701.5,"connectStart":701.5,"domainLookupEnd":701.5,"domainLookupStart":701.5,"fetchStart":701.5,"redirectEnd":0,"redirectStart":0,"requestStart":701.5,"responseEnd":1612.1000001430511,"responseStart":1612.1000001430511,"secureConnectionStart":701.5},{"duration":911,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":701.8000001907349,"connectEnd":701.8000001907349,"connectStart":701.8000001907349,"domainLookupEnd":701.8000001907349,"domainLookupStart":701.8000001907349,"fetchStart":701.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":701.8000001907349,"responseEnd":1612.8000001907349,"responseStart":1612.8000001907349,"secureConnectionStart":701.8000001907349},{"duration":911.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":701.9000000953674,"connectEnd":701.9000000953674,"connectStart":701.9000000953674,"domainLookupEnd":701.9000000953674,"domainLookupStart":701.9000000953674,"fetchStart":701.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":701.9000000953674,"responseEnd":1613.3000001907349,"responseStart":1613.3000001907349,"secureConnectionStart":701.9000000953674},{"duration":961.4000000953674,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":702.2000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":702.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1663.6000001430511,"responseStart":0,"secureConnectionStart":0},{"duration":911.6000001430511,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":702.2000000476837,"connectEnd":702.2000000476837,"connectStart":702.2000000476837,"domainLookupEnd":702.2000000476837,"domainLookupStart":702.2000000476837,"fetchStart":702.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":702.2000000476837,"responseEnd":1613.8000001907349,"responseStart":1613.8000001907349,"secureConnectionStart":702.2000000476837},{"duration":961.5999999046326,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":702.4000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":702.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1664,"responseStart":0,"secureConnectionStart":0},{"duration":911.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":702.6000001430511,"connectEnd":702.6000001430511,"connectStart":702.6000001430511,"domainLookupEnd":702.6000001430511,"domainLookupStart":702.6000001430511,"fetchStart":702.6000001430511,"redirectEnd":0,"redirectStart":0,"requestStart":702.6000001430511,"responseEnd":1614.4000000953674,"responseStart":1614.4000000953674,"secureConnectionStart":702.6000001430511},{"duration":1907,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":703.4000000953674,"connectEnd":703.4000000953674,"connectStart":703.4000000953674,"domainLookupEnd":703.4000000953674,"domainLookupStart":703.4000000953674,"fetchStart":703.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":703.4000000953674,"responseEnd":2610.4000000953674,"responseStart":2610.4000000953674,"secureConnectionStart":703.4000000953674},{"duration":1908.7000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":703.5,"connectEnd":703.5,"connectStart":703.5,"domainLookupEnd":703.5,"domainLookupStart":703.5,"fetchStart":703.5,"redirectEnd":0,"redirectStart":0,"requestStart":703.5,"responseEnd":2612.2000000476837,"responseStart":2612.2000000476837,"secureConnectionStart":703.5},{"duration":406.30000019073486,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1707,"connectEnd":1707,"connectStart":1707,"domainLookupEnd":1707,"domainLookupStart":1707,"fetchStart":1707,"redirectEnd":0,"redirectStart":0,"requestStart":1707,"responseEnd":2113.300000190735,"responseStart":2113.300000190735,"secureConnectionStart":1707}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":504,"responseStart":681,"responseEnd":692,"domLoading":698,"domInteractive":2666,"domContentLoadedEventStart":2666,"domContentLoadedEventEnd":2729,"domComplete":3398,"loadEventStart":3398,"loadEventEnd":3399,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":2621.100000143051},{"name":"bigPipe.sidebar-id.end","time":2621.9000000953674},{"name":"bigPipe.activity-panel-pipe-id.start","time":2622.2000000476837},{"name":"bigPipe.activity-panel-pipe-id.end","time":2622.9000000953674},{"name":"activityTabFullyLoaded","time":2775.300000190735}],"measures":[],"correlationId":"2581607509609c","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":108,"dbReadsTimeInMs":17,"dbConnsTimeInMs":27,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}