MariaDB 10.6 has O_DIRECT value as default for innodb_flush_method instead of fsync. This lead to a performance regression in the default configuration, regardless of the filesystem used.
To benchmark the performance, I have used sysbench-0.4.8 oltp read test on btrfs (the most affected), ext4 and xfs.
Filesystem
threads
innodb_flush_method=fsync
innodb_flush_method=O_DIRECT
ext4
1
6.5s
7.0s
ext4
16
0.97s
1.15s
brtfs
1
6.5s
77s
brtfs
16
0.8s
3.1s
xfs
1
6.4s
7.2s
xfs
16
0.8s
1.1s
I have used a mount point in /abuild and the following configuration in /etc/my.cnf.d/sysbench_testing.cnf:
Note, that while I do support fsync (if innodb_buffer_pool_size is not configured), I think you're pushing this 20x a little bit too much. First, why would anyone consider btrfs at all - that's not the default, and its read speed as shown by your benchmark, is questionable. So for a database, maybe I'd stay away from btrfs pretty much.
Second, asking about how much buffer pool size to dedicate to database, might be considered the installer's or if you want, packager's responsibility . MariaDB's Windows MSI by default asks about 12.5% of RAM, and also lets user overwrite this, in both UI and silent mode. MSI has been doing it since 10 years, so maybe zypper also can consider asking user that one question.
Vladislav Vaintroub
added a comment - - edited Note, that while I do support fsync (if innodb_buffer_pool_size is not configured), I think you're pushing this 20x a little bit too much. First, why would anyone consider btrfs at all - that's not the default, and its read speed as shown by your benchmark, is questionable. So for a database, maybe I'd stay away from btrfs pretty much.
Second, asking about how much buffer pool size to dedicate to database, might be considered the installer's or if you want, packager's responsibility . MariaDB's Windows MSI by default asks about 12.5% of RAM, and also lets user overwrite this, in both UI and silent mode. MSI has been doing it since 10 years, so maybe zypper also can consider asking user that one question.
Can't we make MariaDB/InnoDB emit a warning or suggestion to users if it detects that the below is not the case?
If the working set fits in the default-size buffer pool, or if you have configured the buffer pool size to match the available RAM and buffer pool, it works well.
We can't rely on deb/rpm/msi packaging here. Many systems run installs/upgrades in an automatic and unattended way, and there is no human to tell things to and even less to ask them to decide on something. And anyway workloads might change over time long after the installation was done. Best place to decide on good defaults is MariaDB itself and best place to tell users something is wrong or suboptimal is by emitting warnings in the error logs of MariaDB at runtime.
Otto Kekäläinen
added a comment - Can't we make MariaDB/InnoDB emit a warning or suggestion to users if it detects that the below is not the case?
If the working set fits in the default-size buffer pool, or if you have configured the buffer pool size to match the available RAM and buffer pool, it works well.
We can't rely on deb/rpm/msi packaging here. Many systems run installs/upgrades in an automatic and unattended way, and there is no human to tell things to and even less to ask them to decide on something. And anyway workloads might change over time long after the installation was done. Best place to decide on good defaults is MariaDB itself and best place to tell users something is wrong or suboptimal is by emitting warnings in the error logs of MariaDB at runtime.
The 20x difference is on xfs. My benchmark shows regression on brtfs as well. I don't think making assumptions on the filesystem used by default is good idea either.
read speed as shown by your benchmark, is questionable
Why is it questionable? Can you please explain better?
Danilo Spinella
added a comment - First, why would anyone consider btrfs at all
The 20x difference is on xfs. My benchmark shows regression on brtfs as well. I don't think making assumptions on the filesystem used by default is good idea either.
read speed as shown by your benchmark, is questionable
Why is it questionable? Can you please explain better?
There exist monitoring tools that can build on existing instrumentation in the server to highlight problems. I do not think that such logic should be duplicated in the database server itself. The server can provide raw data. It cannot guess the intentions of the user.
My intuition suggests that any copy-on-write file system, such as xfs or btrfs, could be a bad match for a write-heavy InnoDB setup, or for any database that is based on B-trees and a circular write-ahead log.
Marko Mäkelä
added a comment - There exist monitoring tools that can build on existing instrumentation in the server to highlight problems. I do not think that such logic should be duplicated in the database server itself. The server can provide raw data. It cannot guess the intentions of the user.
My intuition suggests that any copy-on-write file system, such as xfs or btrfs, could be a bad match for a write-heavy InnoDB setup, or for any database that is based on B-trees and a circular write-ahead log.
For a future MariaDB LTS release, it could be beneficial to have fallback scenarios in case MDEV-19895 does not get implemented by then. The pool size default could be bumped to the mentioned 1GB, or reverting to fsync if innodb_buffer_pool_size not configured, maybe with an informational message suggesting to set the value. This would likely improve the out-of-the-box experience for many.
With O_DIRECT we generally see good results given the 1GB pool size (or higher). The plea for a warning message only comes from the fact that the default does not seem like an optimal compromise, and that the new behavior needs manual intervention compared to more "forgiving" fsync.
All in all, learning about these is an acceptable learning curve about MariaDB 10.6 and good documentation helps. O_DIRECT has been the correct default choice in general. Good (or at least ok) defaults would be always useful too, though.
Timo Jyrinki
added a comment - For a future MariaDB LTS release, it could be beneficial to have fallback scenarios in case MDEV-19895 does not get implemented by then. The pool size default could be bumped to the mentioned 1GB, or reverting to fsync if innodb_buffer_pool_size not configured, maybe with an informational message suggesting to set the value. This would likely improve the out-of-the-box experience for many.
With O_DIRECT we generally see good results given the 1GB pool size (or higher). The plea for a warning message only comes from the fact that the default does not seem like an optimal compromise, and that the new behavior needs manual intervention compared to more "forgiving" fsync.
All in all, learning about these is an acceptable learning curve about MariaDB 10.6 and good documentation helps. O_DIRECT has been the correct default choice in general. Good (or at least ok) defaults would be always useful too, though.
People
Unassigned
Danilo Spinella
Votes:
0Vote for this issue
Watchers:
6Start watching this issue
Dates
Created:
Updated:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":1107.3000001907349,"ttfb":333.90000009536743,"pageVisibility":"visible","entityId":107707,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"f0d3ea10-03cb-4ccb-818c-99f004363eef","navigationType":0,"readyForUser":1228.5999999046326,"redirectCount":0,"resourceLoadedEnd":809.5999999046326,"resourceLoadedStart":370.80000019073486,"resourceTiming":[{"duration":123.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":370.80000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":370.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":494.40000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":123.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":371.09999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":371.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":494.69999980926514,"responseStart":0,"secureConnectionStart":0},{"duration":292.8999996185303,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":371.30000019073486,"connectEnd":371.30000019073486,"connectStart":371.30000019073486,"domainLookupEnd":371.30000019073486,"domainLookupStart":371.30000019073486,"fetchStart":371.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":497.80000019073486,"responseEnd":664.1999998092651,"responseStart":519.5999999046326,"secureConnectionStart":371.30000019073486},{"duration":437.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":372,"connectEnd":372,"connectStart":372,"domainLookupEnd":372,"domainLookupStart":372,"fetchStart":372,"redirectEnd":0,"redirectStart":0,"requestStart":498.80000019073486,"responseEnd":809.5999999046326,"responseStart":521.9000000953674,"secureConnectionStart":372},{"duration":158.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":372.09999990463257,"connectEnd":372.09999990463257,"connectStart":372.09999990463257,"domainLookupEnd":372.09999990463257,"domainLookupStart":372.09999990463257,"fetchStart":372.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":499.59999990463257,"responseEnd":530.9000000953674,"responseStart":524.3000001907349,"secureConnectionStart":372.09999990463257},{"duration":158.7999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":372.30000019073486,"connectEnd":372.30000019073486,"connectStart":372.30000019073486,"domainLookupEnd":372.30000019073486,"domainLookupStart":372.30000019073486,"fetchStart":372.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":499.80000019073486,"responseEnd":531.0999999046326,"responseStart":525.9000000953674,"secureConnectionStart":372.30000019073486},{"duration":158.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":372.59999990463257,"connectEnd":372.59999990463257,"connectStart":372.59999990463257,"domainLookupEnd":372.59999990463257,"domainLookupStart":372.59999990463257,"fetchStart":372.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":499.90000009536743,"responseEnd":531.4000000953674,"responseStart":526.9000000953674,"secureConnectionStart":372.59999990463257},{"duration":123.2000002861023,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":372.69999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":372.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":495.90000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":163,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":372.90000009536743,"connectEnd":372.90000009536743,"connectStart":372.90000009536743,"domainLookupEnd":372.90000009536743,"domainLookupStart":372.90000009536743,"fetchStart":372.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":500,"responseEnd":535.9000000953674,"responseStart":528.6999998092651,"secureConnectionStart":372.90000009536743},{"duration":123.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":373,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":373,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":496.5,"responseStart":0,"secureConnectionStart":0},{"duration":164.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":373.30000019073486,"connectEnd":373.30000019073486,"connectStart":373.30000019073486,"domainLookupEnd":373.30000019073486,"domainLookupStart":373.30000019073486,"fetchStart":373.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":501.90000009536743,"responseEnd":537.5,"responseStart":531.6999998092651,"secureConnectionStart":373.30000019073486},{"duration":398.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":389.30000019073486,"connectEnd":389.30000019073486,"connectStart":389.30000019073486,"domainLookupEnd":389.30000019073486,"domainLookupStart":389.30000019073486,"fetchStart":389.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":758.5999999046326,"responseEnd":787.9000000953674,"responseStart":783.4000000953674,"secureConnectionStart":389.30000019073486},{"duration":399,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":389.40000009536743,"connectEnd":389.40000009536743,"connectStart":389.40000009536743,"domainLookupEnd":389.40000009536743,"domainLookupStart":389.40000009536743,"fetchStart":389.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":758.6999998092651,"responseEnd":788.4000000953674,"responseStart":786.1999998092651,"secureConnectionStart":389.40000009536743},{"duration":162.59999990463257,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":794.4000000953674,"connectEnd":794.4000000953674,"connectStart":794.4000000953674,"domainLookupEnd":794.4000000953674,"domainLookupStart":794.4000000953674,"fetchStart":794.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":910,"responseEnd":957,"responseStart":955.9000000953674,"secureConnectionStart":794.4000000953674}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":98,"responseStart":333,"responseEnd":388,"domLoading":337,"domInteractive":1295,"domContentLoadedEventStart":1295,"domContentLoadedEventEnd":1352,"domComplete":1872,"loadEventStart":1872,"loadEventEnd":1872,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1269.6999998092651},{"name":"bigPipe.sidebar-id.end","time":1270.5999999046326},{"name":"bigPipe.activity-panel-pipe-id.start","time":1270.6999998092651},{"name":"bigPipe.activity-panel-pipe-id.end","time":1272.4000000953674},{"name":"activityTabFullyLoaded","time":1370.3000001907349}],"measures":[],"correlationId":"75ed9233142c77","effectiveType":"4g","downlink":9,"rtt":0,"serverDuration":99,"dbReadsTimeInMs":13,"dbConnsTimeInMs":20,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Note, that while I do support fsync (if innodb_buffer_pool_size is not configured), I think you're pushing this 20x a little bit too much. First, why would anyone consider btrfs at all - that's not the default, and its read speed as shown by your benchmark, is questionable. So for a database, maybe I'd stay away from btrfs pretty much.
Second, asking about how much buffer pool size to dedicate to database, might be considered the installer's or if you want, packager's responsibility . MariaDB's Windows MSI by default asks about 12.5% of RAM, and also lets user overwrite this, in both UI and silent mode. MSI has been doing it since 10 years, so maybe zypper also can consider asking user that one question.