elenst reports that her tests are hanging.
The reason seems to be the following combination of parameters:
innodb_write_io_threads=1 innodb_use_native_aio=0 innodb_doublewrite=1
To add confusion, InnoDB will claim that it is using native AIO (because innodb_use_native_aio=1 by default), but a little later, it will silently change to simulated asynchronous I/O (innodb_use_native_aio=0 ) due to io_setup() failure.
The hang is possible, because starting with MDEV-23855 buf_dblwr_t::flush_buffered_writes_completed() , up to 128 page writes may be submitted. But, there may already be 128 outstanding page writes. The thread would then hang here, because the maximum number of outstanding requests per thread is 256:
mariadb-10.5.8
#14 0x000055c699979745 in io_slots::acquire (this=0x55c69c565d50) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:103
#15 0x000055c699977bfe in os_aio (type=..., buf=0x7f9c7847c000, offset=4128768, n=16384) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:4211
#16 0x000055c699c14ce0 in fil_space_t::io (this=0x55c69cd565e8, type=..., offset=4128768, len=16384, buf=0x7f9c7847c000, bpage=0x7f9c78009ab0)
at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3431
#17 0x000055c699b9113c in buf_dblwr_t::flush_buffered_writes_completed (this=0x55c69b62b960 <buf_dblwr>, request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/buf/buf0dblwr.cc:678
#18 0x000055c699c14fb9 in fil_aio_callback (request=...) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/fil/fil0fil.cc:3466
#19 0x000055c699976827 in io_callback (cb=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/storage/innobase/os/os0file.cc:3918
#20 0x000055c699ca75dd in tpool::simulated_aio::simulated_aio_callback (param=0x55c69c565f40) at /home/mariadb/semaphore1/10.5-enterprise/tpool/aio_simulated.cc:162
A possible workaround could be to make innodb_write_io_threads=2 the minimum value.
Thanks to wlad for participating in the debugging.
mysql.err 2020-12-14 10:42 950 kB Elena Stepanova
{"report":{"fcp":809.3999996185303,"ttfb":150,"pageVisibility":"visible","entityId":94377,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"58242330-72e1-4882-a721-965a799d0d89","navigationType":0,"readyForUser":879,"redirectCount":0,"resourceLoadedEnd":1064.5,"resourceLoadedStart":155,"resourceTiming":[{"duration":179.60000038146973,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":155,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":155,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":334.6000003814697,"responseStart":0,"secureConnectionStart":0},{"duration":179.4000005722046,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":155.19999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":155.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":334.6000003814697,"responseStart":0,"secureConnectionStart":0},{"duration":188.39999961853027,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":155.30000019073486,"connectEnd":155.30000019073486,"connectStart":155.30000019073486,"domainLookupEnd":155.30000019073486,"domainLookupStart":155.30000019073486,"fetchStart":155.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":155.30000019073486,"responseEnd":343.69999980926514,"responseStart":343.69999980926514,"secureConnectionStart":155.30000019073486},{"duration":232.10000038146973,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":155.39999961853027,"connectEnd":155.39999961853027,"connectStart":155.39999961853027,"domainLookupEnd":155.39999961853027,"domainLookupStart":155.39999961853027,"fetchStart":155.39999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":155.39999961853027,"responseEnd":387.5,"responseStart":387.5,"secureConnectionStart":155.39999961853027},{"duration":236,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":155.60000038146973,"connectEnd":155.60000038146973,"connectStart":155.60000038146973,"domainLookupEnd":155.60000038146973,"domainLookupStart":155.60000038146973,"fetchStart":155.60000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":155.60000038146973,"responseEnd":391.6000003814697,"responseStart":391.6000003814697,"secureConnectionStart":155.60000038146973},{"duration":236.69999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":155.60000038146973,"connectEnd":155.60000038146973,"connectStart":155.60000038146973,"domainLookupEnd":155.60000038146973,"domainLookupStart":155.60000038146973,"fetchStart":155.60000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":155.60000038146973,"responseEnd":392.30000019073486,"responseStart":392.30000019073486,"secureConnectionStart":155.60000038146973},{"duration":237.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":155.69999980926514,"connectEnd":155.69999980926514,"connectStart":155.69999980926514,"domainLookupEnd":155.69999980926514,"domainLookupStart":155.69999980926514,"fetchStart":155.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":155.69999980926514,"responseEnd":393,"responseStart":393,"secureConnectionStart":155.69999980926514},{"duration":237.70000076293945,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":155.89999961853027,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":155.89999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":393.6000003814697,"responseStart":0,"secureConnectionStart":0},{"duration":237.70000076293945,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":155.89999961853027,"connectEnd":155.89999961853027,"connectStart":155.89999961853027,"domainLookupEnd":155.89999961853027,"domainLookupStart":155.89999961853027,"fetchStart":155.89999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":155.89999961853027,"responseEnd":393.6000003814697,"responseStart":393.5,"secureConnectionStart":155.89999961853027},{"duration":238.0999994277954,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":156.10000038146973,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":156.10000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":394.19999980926514,"responseStart":0,"secureConnectionStart":0},{"duration":238.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":156.10000038146973,"connectEnd":156.10000038146973,"connectStart":156.10000038146973,"domainLookupEnd":156.10000038146973,"domainLookupStart":156.10000038146973,"fetchStart":156.10000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":156.10000038146973,"responseEnd":394.30000019073486,"responseStart":394.30000019073486,"secureConnectionStart":156.10000038146973},{"duration":407,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":165.19999980926514,"connectEnd":165.19999980926514,"connectStart":165.19999980926514,"domainLookupEnd":165.19999980926514,"domainLookupStart":165.19999980926514,"fetchStart":165.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":165.19999980926514,"responseEnd":572.1999998092651,"responseStart":572.1999998092651,"secureConnectionStart":165.19999980926514},{"duration":899.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":165.30000019073486,"connectEnd":165.30000019073486,"connectStart":165.30000019073486,"domainLookupEnd":165.30000019073486,"domainLookupStart":165.30000019073486,"fetchStart":165.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":165.30000019073486,"responseEnd":1064.5,"responseStart":1064.5,"secureConnectionStart":165.30000019073486},{"duration":197.30000019073486,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":561.3000001907349,"connectEnd":561.3000001907349,"connectStart":561.3000001907349,"domainLookupEnd":561.3000001907349,"domainLookupStart":561.3000001907349,"fetchStart":561.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":561.3000001907349,"responseEnd":758.6000003814697,"responseStart":758.6000003814697,"secureConnectionStart":561.3000001907349}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":22,"responseStart":150,"responseEnd":168,"domLoading":153,"domInteractive":1087,"domContentLoadedEventStart":1087,"domContentLoadedEventEnd":1123,"domComplete":2071,"loadEventStart":2071,"loadEventEnd":2073,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1069.6999998092651},{"name":"bigPipe.sidebar-id.end","time":1070.5},{"name":"bigPipe.activity-panel-pipe-id.start","time":1070.6999998092651},{"name":"bigPipe.activity-panel-pipe-id.end","time":1073.1000003814697},{"name":"activityTabFullyLoaded","time":1130.1999998092651}],"measures":[],"correlationId":"8724cfad1d6fd1","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":72,"dbReadsTimeInMs":13,"dbConnsTimeInMs":20,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Blocker
MDEV-16264 Implement a common work queue for InnoDB background tasks
