Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23497

make ROW_FORMAT=COMPRESSED read-only by default

Details

    Description

      introduce a new innodb command line option that will make compressed tables writable. by default they should be read-only. if the write support is enabled, it should print a warning on startup.

      This will allow users to prepare for MDEV-22367

      Attachments

        Issue Links

          Activity

            serg Sergei Golubchik created issue -
            serg Sergei Golubchik made changes -
            Field Original Value New Value
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            marko Marko Mäkelä made changes -
            Status Open [ 1 ] In Progress [ 3 ]

            I think that it was easier to complete this than to provide an estimate. Most of the time was spent on adding SET GLOBAL innodb_read_only_compressed=OFF around tests that cover the ROW_FORMAT=COMPRESSED format. We must keep those tests until we remove the write support.

            marko Marko Mäkelä added a comment - I think that it was easier to complete this than to provide an estimate. Most of the time was spent on adding SET GLOBAL innodb_read_only_compressed=OFF around tests that cover the ROW_FORMAT=COMPRESSED format. We must keep those tests until we remove the write support.

            To allow ROW_FORMAT=COMPRESSED tables to be created or written, we introduce an option:

            SET GLOBAL innodb_read_only_compressed=OFF;
            

            That parameter may be deprecated and ignored in the next major version (along with the code to support writing ROW_FORMAT=COMPRESSED tables in InnoDB.

            marko Marko Mäkelä added a comment - To allow ROW_FORMAT=COMPRESSED tables to be created or written, we introduce an option: SET GLOBAL innodb_read_only_compressed= OFF ; That parameter may be deprecated and ignored in the next major version (along with the code to support writing ROW_FORMAT=COMPRESSED tables in InnoDB.
            marko Marko Mäkelä made changes -
            Fix Version/s 10.6.0 [ 24431 ]
            Fix Version/s 10.6 [ 24028 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            nunop Nuno added a comment - - edited

            @Marko - will we at least be provided with a way to easily convert ONLINE these tables to the alternative? (PAGE_COMPRESSED=1)

            I have large compressed tables on my 24-hour online write-intensive website.

            This change seems to mean to me that I won't be able to upgrade my MariaDB version to 10.7+, once the COMPRESSED feature is completely removed...

            Also, as discussed in MDEV-22839, PAGE_COMPRESSED doesn't really seem to work well for me. It even increased the non-sparse size of the data & indexes, when I tested...

            I'm really not happy with this change.

            > "Nowadays, with fast solid-state storage being a commodity,"

            I would be careful in saying this.
            Not everyone can afford AWS or similar (may save in storage, but doesn't save in data transfer out),
            and many are still on Dedicated bare metal servers, where there's no choice to simply "increase" storage.

            nunop Nuno added a comment - - edited @Marko - will we at least be provided with a way to easily convert ONLINE these tables to the alternative? (PAGE_COMPRESSED=1) I have large compressed tables on my 24-hour online write-intensive website. This change seems to mean to me that I won't be able to upgrade my MariaDB version to 10.7+, once the COMPRESSED feature is completely removed... Also, as discussed in MDEV-22839 , PAGE_COMPRESSED doesn't really seem to work well for me. It even increased the non-sparse size of the data & indexes, when I tested... I'm really not happy with this change. > "Nowadays, with fast solid-state storage being a commodity," I would be careful in saying this. Not everyone can afford AWS or similar (may save in storage, but doesn't save in data transfer out), and many are still on Dedicated bare metal servers, where there's no choice to simply "increase" storage.
            marko Marko Mäkelä made changes -

            @Nuno I agree with you. I've also TB's of table's using ROW_FORMAT=COMPRESSED and run on-prem. i'm a bit concerned indeed this feature will be dropped.

            olafbuitelaar Olaf Buitelaar added a comment - @Nuno I agree with you. I've also TB's of table's using ROW_FORMAT=COMPRESSED and run on-prem. i'm a bit concerned indeed this feature will be dropped.
            marko Marko Mäkelä added a comment - - edited

            I implemented online table rebuild in MySQL 5.6.8, and the feature is present in MariaDB 10.0 already. You should be able to just execute something like the following:

            ALTER TABLE compressed_table ROW_FORMAT=DYNAMIC page_compressed=1;
            

            The table will be rebuilt, while even allowing concurrent writes. And concurrent writes will be allowed while the table is being rebuilt, as long as

            SET GLOBAL innodb_read_only_compressed=OFF;
            

            is in effect.

            marko Marko Mäkelä added a comment - - edited I implemented online table rebuild in MySQL 5.6.8, and the feature is present in MariaDB 10.0 already. You should be able to just execute something like the following: ALTER TABLE compressed_table ROW_FORMAT= DYNAMIC page_compressed=1; The table will be rebuilt, while even allowing concurrent writes. And concurrent writes will be allowed while the table is being rebuilt, as long as SET GLOBAL innodb_read_only_compressed= OFF ; is in effect.

            Thanks for the suggestion, and i'm sure it would work. But the tables are extremely large 15+TB (however partitioned), this would mean we would require that space double during the rebuild. Also the tables are quite write intensive, so IO would double during the rebuild time. Last in the past i had some page corruption errors, which i was not able to recover. Probably rebuilding the table would let those errors surface again. Also during testing it seemed the compression ratio with

            ROW_FORMAT=COMPRESSED

            vs

            ROW_FORMAT=DYNAMIC page_compressed=1

            was much better
            Altogether it seems it's going to be quite a difficult upgrade.
            Having said that, i understand your motivation to simplify things in InnoDB. And i'm glad you do so.

            olafbuitelaar Olaf Buitelaar added a comment - Thanks for the suggestion, and i'm sure it would work. But the tables are extremely large 15+TB (however partitioned), this would mean we would require that space double during the rebuild. Also the tables are quite write intensive, so IO would double during the rebuild time. Last in the past i had some page corruption errors, which i was not able to recover. Probably rebuilding the table would let those errors surface again. Also during testing it seemed the compression ratio with ROW_FORMAT=COMPRESSED vs ROW_FORMAT= DYNAMIC page_compressed=1 was much better Altogether it seems it's going to be quite a difficult upgrade. Having said that, i understand your motivation to simplify things in InnoDB. And i'm glad you do so.

            olafbuitelaar, thank you for your feedback. ALTER TABLE should only read the clustered index, not secondary indexes. In fact, rebuilding a table would fix corrupted secondary indexes (I hope that MDEV-22373 is fixed now, but it is too early to tell).

            I hope that page_compressed together with compressed columns (MDEV-11371) would make an acceptable replacement of ROW_FORMAT=COMPRESSED. One piece that is missing is something that allows us to avoid the problematic use of sparse files in page_compressed. I think that we must provide an option to write page_compressed tables as normal non-sparse files and let the a smart SSD compress away the sequences of zero bytes, as explained in https://mariadb.org/fest2020/ssd/.

            marko Marko Mäkelä added a comment - olafbuitelaar , thank you for your feedback. ALTER TABLE should only read the clustered index, not secondary indexes. In fact, rebuilding a table would fix corrupted secondary indexes (I hope that MDEV-22373 is fixed now, but it is too early to tell). I hope that page_compressed together with compressed columns ( MDEV-11371 ) would make an acceptable replacement of ROW_FORMAT=COMPRESSED . One piece that is missing is something that allows us to avoid the problematic use of sparse files in page_compressed . I think that we must provide an option to write page_compressed tables as normal non-sparse files and let the a smart SSD compress away the sequences of zero bytes, as explained in https://mariadb.org/fest2020/ssd/ .
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            rjasdfiii Rick James added a comment -

            Are there any creditable benchmarks on the CPU, I/O, and disk space for ROW_FORMAT=COMPRESSED? I have long been skeptical about the feature. I will continue to steer people away from using that setting.

            rjasdfiii Rick James added a comment - Are there any creditable benchmarks on the CPU, I/O, and disk space for ROW_FORMAT=COMPRESSED? I have long been skeptical about the feature. I will continue to steer people away from using that setting.
            GeoffMontee Geoff Montee (Inactive) made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -

            Based on https://github.com/nextcloud/server/issues/25436 NextCloud appears to be using ROW_FORMAT=COMPRESSED for InnoDB tables.

            marko Marko Mäkelä added a comment - Based on https://github.com/nextcloud/server/issues/25436 NextCloud appears to be using ROW_FORMAT=COMPRESSED for InnoDB tables.
            marko Marko Mäkelä made changes -

            rjasdfiii, I remember that back in 2006 or 2007, when I had completed the implementation of ROW_FORMAT=COMPRESSED for the InnoDB Plugin for MySQL 5.0 based on some rough ideas of Heikki Tuuri, some performance tests were run, and the results were not great. I think that we really have to compare all performance aspects to page_compressed, as suggested in MDEV-22839.

            marko Marko Mäkelä added a comment - rjasdfiii , I remember that back in 2006 or 2007, when I had completed the implementation of ROW_FORMAT=COMPRESSED for the InnoDB Plugin for MySQL 5.0 based on some rough ideas of Heikki Tuuri, some performance tests were run, and the results were not great. I think that we really have to compare all performance aspects to page_compressed , as suggested in MDEV-22839 .
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 112484 ] MariaDB v4 [ 134322 ]
            marko Marko Mäkelä made changes -

            MDEV-27736 in MariaDB Server 10.6.6 essentially reverted this change, by changing the default value of innodb_read_only_compressed to OFF. The plan to remove support for InnoDB ROW_FORMAT=COMPRESSED tables (MDEV-22367) has been cancelled.

            marko Marko Mäkelä added a comment - MDEV-27736 in MariaDB Server 10.6.6 essentially reverted this change, by changing the default value of innodb_read_only_compressed to OFF . The plan to remove support for InnoDB ROW_FORMAT=COMPRESSED tables ( MDEV-22367 ) has been cancelled.

            People

              marko Marko Mäkelä
              serg Sergei Golubchik
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.