introduce a new innodb command line option that will make compressed tables writable. by default they should be read-only. if the write support is enabled, it should print a warning on startup.
I think that it was easier to complete this than to provide an estimate. Most of the time was spent on adding SET GLOBAL innodb_read_only_compressed=OFF around tests that cover the ROW_FORMAT=COMPRESSED format. We must keep those tests until we remove the write support.
Marko Mäkelä
added a comment - I think that it was easier to complete this than to provide an estimate. Most of the time was spent on adding SET GLOBAL innodb_read_only_compressed=OFF around tests that cover the ROW_FORMAT=COMPRESSED format. We must keep those tests until we remove the write support.
To allow ROW_FORMAT=COMPRESSED tables to be created or written, we introduce an option:
SETGLOBAL innodb_read_only_compressed=OFF;
That parameter may be deprecated and ignored in the next major version (along with the code to support writing ROW_FORMAT=COMPRESSED tables in InnoDB.
Marko Mäkelä
added a comment - To allow ROW_FORMAT=COMPRESSED tables to be created or written, we introduce an option:
SET GLOBAL innodb_read_only_compressed= OFF ;
That parameter may be deprecated and ignored in the next major version (along with the code to support writing ROW_FORMAT=COMPRESSED tables in InnoDB.
@Marko - will we at least be provided with a way to easily convert ONLINE these tables to the alternative? (PAGE_COMPRESSED=1)
I have large compressed tables on my 24-hour online write-intensive website.
This change seems to mean to me that I won't be able to upgrade my MariaDB version to 10.7+, once the COMPRESSED feature is completely removed...
Also, as discussed in MDEV-22839, PAGE_COMPRESSED doesn't really seem to work well for me. It even increased the non-sparse size of the data & indexes, when I tested...
I'm really not happy with this change.
> "Nowadays, with fast solid-state storage being a commodity,"
I would be careful in saying this.
Not everyone can afford AWS or similar (may save in storage, but doesn't save in data transfer out),
and many are still on Dedicated bare metal servers, where there's no choice to simply "increase" storage.
Nuno
added a comment - - edited @Marko - will we at least be provided with a way to easily convert ONLINE these tables to the alternative? (PAGE_COMPRESSED=1)
I have large compressed tables on my 24-hour online write-intensive website.
This change seems to mean to me that I won't be able to upgrade my MariaDB version to 10.7+, once the COMPRESSED feature is completely removed...
Also, as discussed in MDEV-22839 , PAGE_COMPRESSED doesn't really seem to work well for me. It even increased the non-sparse size of the data & indexes, when I tested...
I'm really not happy with this change.
> "Nowadays, with fast solid-state storage being a commodity,"
I would be careful in saying this.
Not everyone can afford AWS or similar (may save in storage, but doesn't save in data transfer out),
and many are still on Dedicated bare metal servers, where there's no choice to simply "increase" storage.
@Nuno I agree with you. I've also TB's of table's using ROW_FORMAT=COMPRESSED and run on-prem. i'm a bit concerned indeed this feature will be dropped.
Olaf Buitelaar
added a comment - @Nuno I agree with you. I've also TB's of table's using ROW_FORMAT=COMPRESSED and run on-prem. i'm a bit concerned indeed this feature will be dropped.
I implemented online table rebuild in MySQL 5.6.8, and the feature is present in MariaDB 10.0 already. You should be able to just execute something like the following:
The table will be rebuilt, while even allowing concurrent writes. And concurrent writes will be allowed while the table is being rebuilt, as long as
SETGLOBAL innodb_read_only_compressed=OFF;
is in effect.
Marko Mäkelä
added a comment - - edited I implemented online table rebuild in MySQL 5.6.8, and the feature is present in MariaDB 10.0 already. You should be able to just execute something like the following:
ALTER TABLE compressed_table ROW_FORMAT= DYNAMIC page_compressed=1;
The table will be rebuilt, while even allowing concurrent writes. And concurrent writes will be allowed while the table is being rebuilt, as long as
SET GLOBAL innodb_read_only_compressed= OFF ;
is in effect.
Thanks for the suggestion, and i'm sure it would work. But the tables are extremely large 15+TB (however partitioned), this would mean we would require that space double during the rebuild. Also the tables are quite write intensive, so IO would double during the rebuild time. Last in the past i had some page corruption errors, which i was not able to recover. Probably rebuilding the table would let those errors surface again. Also during testing it seemed the compression ratio with
ROW_FORMAT=COMPRESSED
vs
ROW_FORMAT=DYNAMIC page_compressed=1
was much better
Altogether it seems it's going to be quite a difficult upgrade.
Having said that, i understand your motivation to simplify things in InnoDB. And i'm glad you do so.
Olaf Buitelaar
added a comment - Thanks for the suggestion, and i'm sure it would work. But the tables are extremely large 15+TB (however partitioned), this would mean we would require that space double during the rebuild. Also the tables are quite write intensive, so IO would double during the rebuild time. Last in the past i had some page corruption errors, which i was not able to recover. Probably rebuilding the table would let those errors surface again. Also during testing it seemed the compression ratio with
ROW_FORMAT=COMPRESSED
vs
ROW_FORMAT= DYNAMIC page_compressed=1
was much better
Altogether it seems it's going to be quite a difficult upgrade.
Having said that, i understand your motivation to simplify things in InnoDB. And i'm glad you do so.
olafbuitelaar, thank you for your feedback. ALTER TABLE should only read the clustered index, not secondary indexes. In fact, rebuilding a table would fix corrupted secondary indexes (I hope that MDEV-22373 is fixed now, but it is too early to tell).
I hope that page_compressed together with compressed columns (MDEV-11371) would make an acceptable replacement of ROW_FORMAT=COMPRESSED. One piece that is missing is something that allows us to avoid the problematic use of sparse files in page_compressed. I think that we must provide an option to write page_compressed tables as normal non-sparse files and let the a smart SSD compress away the sequences of zero bytes, as explained in https://mariadb.org/fest2020/ssd/.
Marko Mäkelä
added a comment - olafbuitelaar , thank you for your feedback. ALTER TABLE should only read the clustered index, not secondary indexes. In fact, rebuilding a table would fix corrupted secondary indexes (I hope that MDEV-22373 is fixed now, but it is too early to tell).
I hope that page_compressed together with compressed columns ( MDEV-11371 ) would make an acceptable replacement of ROW_FORMAT=COMPRESSED . One piece that is missing is something that allows us to avoid the problematic use of sparse files in page_compressed . I think that we must provide an option to write page_compressed tables as normal non-sparse files and let the a smart SSD compress away the sequences of zero bytes, as explained in https://mariadb.org/fest2020/ssd/ .
Are there any creditable benchmarks on the CPU, I/O, and disk space for ROW_FORMAT=COMPRESSED? I have long been skeptical about the feature. I will continue to steer people away from using that setting.
Rick James
added a comment - Are there any creditable benchmarks on the CPU, I/O, and disk space for ROW_FORMAT=COMPRESSED? I have long been skeptical about the feature. I will continue to steer people away from using that setting.
Marko Mäkelä
added a comment - Based on https://github.com/nextcloud/server/issues/25436 NextCloud appears to be using ROW_FORMAT=COMPRESSED for InnoDB tables.
rjasdfiii, I remember that back in 2006 or 2007, when I had completed the implementation of ROW_FORMAT=COMPRESSED for the InnoDB Plugin for MySQL 5.0 based on some rough ideas of Heikki Tuuri, some performance tests were run, and the results were not great. I think that we really have to compare all performance aspects to page_compressed, as suggested in MDEV-22839.
Marko Mäkelä
added a comment - rjasdfiii , I remember that back in 2006 or 2007, when I had completed the implementation of ROW_FORMAT=COMPRESSED for the InnoDB Plugin for MySQL 5.0 based on some rough ideas of Heikki Tuuri, some performance tests were run, and the results were not great. I think that we really have to compare all performance aspects to page_compressed , as suggested in MDEV-22839 .
MDEV-27736 in MariaDB Server 10.6.6 essentially reverted this change, by changing the default value of innodb_read_only_compressed to OFF. The plan to remove support for InnoDB ROW_FORMAT=COMPRESSED tables (MDEV-22367) has been cancelled.
Marko Mäkelä
added a comment - MDEV-27736 in MariaDB Server 10.6.6 essentially reverted this change, by changing the default value of innodb_read_only_compressed to OFF . The plan to remove support for InnoDB ROW_FORMAT=COMPRESSED tables ( MDEV-22367 ) has been cancelled.
People
Marko Mäkelä
Sergei Golubchik
Votes:
0Vote for this issue
Watchers:
11Start watching this issue
Dates
Created:
Updated:
Resolved:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":1404.2999997138977,"ttfb":353.59999990463257,"pageVisibility":"visible","entityId":90683,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"3ebc36fa-ccb6-4ef0-86d3-b4a53b6d57ad","navigationType":0,"readyForUser":1480.0999999046326,"redirectCount":0,"resourceLoadedEnd":1533.1999998092651,"resourceLoadedStart":361.19999980926514,"resourceTiming":[{"duration":503.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":361.19999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":361.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":864.7999997138977,"responseStart":0,"secureConnectionStart":0},{"duration":503.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":361.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":361.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":865.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":512.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":361.59999990463257,"connectEnd":361.59999990463257,"connectStart":361.59999990463257,"domainLookupEnd":361.59999990463257,"domainLookupStart":361.59999990463257,"fetchStart":361.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":361.59999990463257,"responseEnd":874,"responseStart":874,"secureConnectionStart":361.59999990463257},{"duration":566.2000002861023,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":361.7999997138977,"connectEnd":361.7999997138977,"connectStart":361.7999997138977,"domainLookupEnd":361.7999997138977,"domainLookupStart":361.7999997138977,"fetchStart":361.7999997138977,"redirectEnd":0,"redirectStart":0,"requestStart":361.7999997138977,"responseEnd":928,"responseStart":928,"secureConnectionStart":361.7999997138977},{"duration":569.8999996185303,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":362,"connectEnd":362,"connectStart":362,"domainLookupEnd":362,"domainLookupStart":362,"fetchStart":362,"redirectEnd":0,"redirectStart":0,"requestStart":362,"responseEnd":931.8999996185303,"responseStart":931.8999996185303,"secureConnectionStart":362},{"duration":570.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":362.19999980926514,"connectEnd":362.19999980926514,"connectStart":362.19999980926514,"domainLookupEnd":362.19999980926514,"domainLookupStart":362.19999980926514,"fetchStart":362.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":362.19999980926514,"responseEnd":932.3999996185303,"responseStart":932.3999996185303,"secureConnectionStart":362.19999980926514},{"duration":570.6000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":362.3999996185303,"connectEnd":362.3999996185303,"connectStart":362.3999996185303,"domainLookupEnd":362.3999996185303,"domainLookupStart":362.3999996185303,"fetchStart":362.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":362.3999996185303,"responseEnd":933,"responseStart":933,"secureConnectionStart":362.3999996185303},{"duration":658.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":362.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":362.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1021,"responseStart":0,"secureConnectionStart":0},{"duration":570.9000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":362.59999990463257,"connectEnd":362.59999990463257,"connectStart":362.59999990463257,"domainLookupEnd":362.59999990463257,"domainLookupStart":362.59999990463257,"fetchStart":362.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":362.59999990463257,"responseEnd":933.5,"responseStart":933.5,"secureConnectionStart":362.59999990463257},{"duration":658.3000001907349,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":362.7999997138977,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":362.7999997138977,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1021.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":571.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":363,"connectEnd":363,"connectStart":363,"domainLookupEnd":363,"domainLookupStart":363,"fetchStart":363,"redirectEnd":0,"redirectStart":0,"requestStart":363,"responseEnd":934.0999999046326,"responseStart":934,"secureConnectionStart":363},{"duration":741.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":364.2999997138977,"connectEnd":364.2999997138977,"connectStart":364.2999997138977,"domainLookupEnd":364.2999997138977,"domainLookupStart":364.2999997138977,"fetchStart":364.2999997138977,"redirectEnd":0,"redirectStart":0,"requestStart":364.2999997138977,"responseEnd":1105.3999996185303,"responseStart":1105.3999996185303,"secureConnectionStart":364.2999997138977},{"duration":1143.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":389.59999990463257,"connectEnd":389.59999990463257,"connectStart":389.59999990463257,"domainLookupEnd":389.59999990463257,"domainLookupStart":389.59999990463257,"fetchStart":389.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":389.59999990463257,"responseEnd":1533.1999998092651,"responseStart":1533.1999998092651,"secureConnectionStart":389.59999990463257},{"duration":76.19999980926514,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1033,"connectEnd":1033,"connectStart":1033,"domainLookupEnd":1033,"domainLookupStart":1033,"fetchStart":1033,"redirectEnd":0,"redirectStart":0,"requestStart":1033,"responseEnd":1109.1999998092651,"responseStart":1109.1999998092651,"secureConnectionStart":1033},{"duration":326.80000019073486,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1315.1999998092651,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1315.1999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1642,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":172,"responseStart":354,"responseEnd":360,"domLoading":359,"domInteractive":1579,"domContentLoadedEventStart":1579,"domContentLoadedEventEnd":1642,"domComplete":2104,"loadEventStart":2104,"loadEventEnd":2105,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1535.0999999046326},{"name":"bigPipe.sidebar-id.end","time":1536.0999999046326},{"name":"bigPipe.activity-panel-pipe-id.start","time":1536.3999996185303},{"name":"bigPipe.activity-panel-pipe-id.end","time":1539.5},{"name":"activityTabFullyLoaded","time":1662}],"measures":[],"correlationId":"4127c7deb64109","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":123,"dbReadsTimeInMs":20,"dbConnsTimeInMs":29,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
I think that it was easier to complete this than to provide an estimate. Most of the time was spent on adding SET GLOBAL innodb_read_only_compressed=OFF around tests that cover the ROW_FORMAT=COMPRESSED format. We must keep those tests until we remove the write support.