elenst reports that her tests are hanging.
The reason seems to be the following combination of parameters:
innodb_write_io_threads=1 innodb_use_native_aio=0 innodb_doublewrite=1
To add confusion, InnoDB will claim that it is using native AIO (because innodb_use_native_aio=1 by default), but a little later, it will silently change to simulated asynchronous I/O (innodb_use_native_aio=0 ) due to io_setup() failure.
The hang is possible, because starting with MDEV-23855 buf_dblwr_t::flush_buffered_writes_completed() , up to 128 page writes may be submitted. But, there may already be 128 outstanding page writes. The thread would then hang here, because the maximum number of outstanding requests per thread is 256:
mariadb-10.5.8
#14 0x000055c699979745 in io_slots::acquire (this=0x55c69c565d50) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:103
#15 0x000055c699977bfe in os_aio (type=..., buf=0x7f9c7847c000, offset=4128768, n=16384) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:4211
#16 0x000055c699c14ce0 in fil_space_t::io (this=0x55c69cd565e8, type=..., offset=4128768, len=16384, buf=0x7f9c7847c000, bpage=0x7f9c78009ab0)
at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3431
#17 0x000055c699b9113c in buf_dblwr_t::flush_buffered_writes_completed (this=0x55c69b62b960 <buf_dblwr>, request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/buf/buf0dblwr.cc:678
#18 0x000055c699c14fb9 in fil_aio_callback (request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3466
#19 0x000055c699976827 in io_callback (cb=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:3918
#20 0x000055c699ca75dd in tpool::simulated_aio::simulated_aio_callback (param=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/tpool/aio_simulated.cc:162
A possible workaround could be to make innodb_write_io_threads=2 the minimum value.
Thanks to wlad for participating in the debugging.
mysql.err 2020-12-14 10:42 950 kB Elena Stepanova
{"report":{"fcp":774.7999992370605,"ttfb":187.89999961853027,"pageVisibility":"visible","entityId":94377,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":32,"apdex":1,"journeyId":"fd550d6c-4b4f-4b69-84bd-0c23c1449f22","navigationType":0,"readyForUser":857.6999998092651,"redirectCount":0,"resourceLoadedEnd":1189.6999998092651,"resourceLoadedStart":193.29999923706055,"resourceTiming":[{"duration":7.600000381469727,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":193.29999923706055,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":193.29999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":200.89999961853027,"responseStart":0,"secureConnectionStart":0},{"duration":7.699999809265137,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":193.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":193.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":201.19999980926514,"responseStart":0,"secureConnectionStart":0},{"duration":60.59999942779541,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":193.69999980926514,"connectEnd":193.69999980926514,"connectStart":193.69999980926514,"domainLookupEnd":193.69999980926514,"domainLookupStart":193.69999980926514,"fetchStart":193.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":193.69999980926514,"responseEnd":254.29999923706055,"responseStart":254.29999923706055,"secureConnectionStart":193.69999980926514},{"duration":166.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":193.89999961853027,"connectEnd":193.89999961853027,"connectStart":193.89999961853027,"domainLookupEnd":193.89999961853027,"domainLookupStart":193.89999961853027,"fetchStart":193.89999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":193.89999961853027,"responseEnd":360.19999980926514,"responseStart":360.19999980926514,"secureConnectionStart":193.89999961853027},{"duration":169.9000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":194.0999994277954,"connectEnd":194.0999994277954,"connectStart":194.0999994277954,"domainLookupEnd":194.0999994277954,"domainLookupStart":194.0999994277954,"fetchStart":194.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":194.0999994277954,"responseEnd":364,"responseStart":364,"secureConnectionStart":194.0999994277954},{"duration":170.20000076293945,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":194.29999923706055,"connectEnd":194.29999923706055,"connectStart":194.29999923706055,"domainLookupEnd":194.29999923706055,"domainLookupStart":194.29999923706055,"fetchStart":194.29999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":194.29999923706055,"responseEnd":364.5,"responseStart":364.5,"secureConnectionStart":194.29999923706055},{"duration":170.29999923706055,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":194.5,"connectEnd":194.5,"connectStart":194.5,"domainLookupEnd":194.5,"domainLookupStart":194.5,"fetchStart":194.5,"redirectEnd":0,"redirectStart":0,"requestStart":194.5,"responseEnd":364.79999923706055,"responseStart":364.79999923706055,"secureConnectionStart":194.5},{"duration":204.39999961853027,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":194.69999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":194.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":399.0999994277954,"responseStart":0,"secureConnectionStart":0},{"duration":170.5,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":194.79999923706055,"connectEnd":194.79999923706055,"connectStart":194.79999923706055,"domainLookupEnd":194.79999923706055,"domainLookupStart":194.79999923706055,"fetchStart":194.79999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":194.79999923706055,"responseEnd":365.29999923706055,"responseStart":365.29999923706055,"secureConnectionStart":194.79999923706055},{"duration":204.19999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":195,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":195,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":399.19999980926514,"responseStart":0,"secureConnectionStart":0},{"duration":170.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":195.0999994277954,"connectEnd":195.0999994277954,"connectStart":195.0999994277954,"domainLookupEnd":195.0999994277954,"domainLookupStart":195.0999994277954,"fetchStart":195.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":195.0999994277954,"responseEnd":365.8999996185303,"responseStart":365.8999996185303,"secureConnectionStart":195.0999994277954},{"duration":328.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":196.19999980926514,"connectEnd":196.19999980926514,"connectStart":196.19999980926514,"domainLookupEnd":196.19999980926514,"domainLookupStart":196.19999980926514,"fetchStart":196.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":196.19999980926514,"responseEnd":524.6999998092651,"responseStart":524.6999998092651,"secureConnectionStart":196.19999980926514},{"duration":993.4000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":196.29999923706055,"connectEnd":196.29999923706055,"connectStart":196.29999923706055,"domainLookupEnd":196.29999923706055,"domainLookupStart":196.29999923706055,"fetchStart":196.29999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":196.29999923706055,"responseEnd":1189.6999998092651,"responseStart":1189.6999998092651,"secureConnectionStart":196.29999923706055},{"duration":102,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":427.69999980926514,"connectEnd":427.69999980926514,"connectStart":427.69999980926514,"domainLookupEnd":427.69999980926514,"domainLookupStart":427.69999980926514,"fetchStart":427.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":427.69999980926514,"responseEnd":529.6999998092651,"responseStart":529.6999998092651,"secureConnectionStart":427.69999980926514}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":17,"responseStart":188,"responseEnd":192,"domLoading":191,"domInteractive":1208,"domContentLoadedEventStart":1208,"domContentLoadedEventEnd":1247,"domComplete":2160,"loadEventStart":2160,"loadEventEnd":2161,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1191.1999998092651},{"name":"bigPipe.sidebar-id.end","time":1192},{"name":"bigPipe.activity-panel-pipe-id.start","time":1192.0999994277954},{"name":"bigPipe.activity-panel-pipe-id.end","time":1194.3999996185303},{"name":"activityTabFullyLoaded","time":1250.0999994277954}],"measures":[],"correlationId":"232d5a87000334","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":113,"dbReadsTimeInMs":13,"dbConnsTimeInMs":21,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Blocker
MDEV-16264 Implement a common work queue for InnoDB background tasks
