Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19780

Remove the TokuDB storage engine

Details

    Description

      The TokuDB storage engine has been deprecated by upstream:

      TokuDB is deprecated in the 8.0 series and will be supported through the 8.0 series until further notice. This storage engine will not be included in the next major release of Percona Server for MySQL. We recommend MyRocks as a long-term migration path.

      We have neither knowledge nor resources to take over TokuDB maintenance.
      TokuDB users can consider migrating to InnoDB or RocksDB, imperfect as it is. Or ask TokuDB owner to resume TokuDB maintenance.

      Attachments

        Issue Links

          relates to

          Task - A task that needs to be done. MDEV-22839 ROW_FORMAT=COMPRESSED vs PAGE_COMPRESSION=1 - size comparison

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-8843 Ship TokuBackup as part of MariaDB

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11101 storage/tokudb uses much more space than other engines

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21803 wrong dependency for mariadb-plugin-tokudb

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21944 Remove TokuDB from Debian packaging?

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            marko Marko Mäkelä created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä added a comment -

            bb-10.5-MDEV-19780 looks OK with respect to the TokuDB removal

            marko Marko Mäkelä added a comment - bb-10.5- MDEV-19780 looks OK with respect to the TokuDB removal
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.5 [ 23123 ]
            serg Sergei Golubchik added a comment -

            TokuDB is now disabled in 10.5.0.

            serg Sergei Golubchik added a comment - TokuDB is now disabled in 10.5.0.
            GeoffMontee Geoff Montee (Inactive) made changes -
            elenst Elena Stepanova made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Assignee Sergei Golubchik [ serg ] Marko Mäkelä [ marko ]
            marko Marko Mäkelä made changes -
            Fix Version/s 10.6 [ 24028 ]
            marko Marko Mäkelä made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2020-05-14 10:17:44.0 2020-05-14 10:17:44.67
            marko Marko Mäkelä made changes -
            Fix Version/s N/A [ 14700 ]
            Fix Version/s 10.6 [ 24028 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.6.0 [ 24431 ]
            Fix Version/s N/A [ 14700 ]
            marko Marko Mäkelä made changes -
            Comment [ [~serg], please change fixVersion to 10.6.0 and make sure that 10.6 is registered on our CI. I tested it on the bb-10.5- branch which I have now removed. ]
            Alex/AT Alex/AT added a comment -

            Actually it's a very bad idea.
            I have converted few huge tables to RocksDB and well... it's barely usable.
            First of all, it takes a whole lot of gigabytes of RAM to sustain the tables, MariaDB server memory usage went from 10G on TokuDB to 20G on RocksDB.
            Second, TokuDB has online DDL, not only index creation, but column operations. And it's pretty much replication safe, compared to RocksDB.
            Imagine extending a 100-200GB worth RocksDB table with a new index or column, locking table for the while of operation, stalling the system, and you'll quickly understand what's terribly wrong about it.

            To sum it, please re-enable TokuDB in current versions. Even if it won't get any new fixes, it's production ready and was the only lifesaver many people with big data tables had.

            Alex/AT Alex/AT added a comment - Actually it's a very bad idea. I have converted few huge tables to RocksDB and well... it's barely usable. First of all, it takes a whole lot of gigabytes of RAM to sustain the tables, MariaDB server memory usage went from 10G on TokuDB to 20G on RocksDB. Second, TokuDB has online DDL, not only index creation, but column operations. And it's pretty much replication safe, compared to RocksDB. Imagine extending a 100-200GB worth RocksDB table with a new index or column, locking table for the while of operation, stalling the system, and you'll quickly understand what's terribly wrong about it. To sum it, please re-enable TokuDB in current versions. Even if it won't get any new fixes, it's production ready and was the only lifesaver many people with big data tables had.
            serg Sergei Golubchik made changes -
            Description The TokuDB storage engine has been [deprecated by upstream|https://www.percona.com/doc/percona-server/8.0/tokudb/tokudb_intro.html]:
            {quote}
            TokuDB is deprecated in the 8.0 series and will be supported through the 8.0 series until further notice. This storage engine will not be included in the next major release of Percona Server for MySQL. We recommend MyRocks as a long-term migration path.
            {quote}
            Because MariaDB Server includes MyRocks as well, it makes sense to remove TokuDB in favor of MyRocks.             		The TokuDB storage engine has been [deprecated by upstream|https://www.percona.com/doc/percona-server/8.0/tokudb/tokudb_intro.html]:
            {quote}
            TokuDB is deprecated in the 8.0 series and will be supported through the 8.0 series until further notice. This storage engine will not be included in the next major release of Percona Server for MySQL. We recommend MyRocks as a long-term migration path.
            {quote}
            We have neither knowledge nor resources to take over TokuDB maintenance.
            TokuDB users can consider migrate to InnoDB or RocksDB, imperfect as it is. Or ask TokuDB owner to resume TokuDB maintenance.
            serg Sergei Golubchik made changes -
            Description The TokuDB storage engine has been [deprecated by upstream|https://www.percona.com/doc/percona-server/8.0/tokudb/tokudb_intro.html]:
            {quote}
            TokuDB is deprecated in the 8.0 series and will be supported through the 8.0 series until further notice. This storage engine will not be included in the next major release of Percona Server for MySQL. We recommend MyRocks as a long-term migration path.
            {quote}
            We have neither knowledge nor resources to take over TokuDB maintenance.
            TokuDB users can consider migrate to InnoDB or RocksDB, imperfect as it is. Or ask TokuDB owner to resume TokuDB maintenance.             		The TokuDB storage engine has been [deprecated by upstream|https://www.percona.com/doc/percona-server/8.0/tokudb/tokudb_intro.html]:
            {quote}
            TokuDB is deprecated in the 8.0 series and will be supported through the 8.0 series until further notice. This storage engine will not be included in the next major release of Percona Server for MySQL. We recommend MyRocks as a long-term migration path.
            {quote}
            We have neither knowledge nor resources to take over TokuDB maintenance.
            TokuDB users can consider migrating to InnoDB or RocksDB, imperfect as it is. Or ask TokuDB owner to resume TokuDB maintenance.
            Alex/AT Alex/AT added a comment -

            Sorry for commenting again on a closed issue, but I think it would be good for some users to know.

            A tolerable replacement for TokuDB is InnoDB + page compression (we use zlib, but it would be good to have something like zstd & lzma compiled in for next versions, maybe this can be considered).
            While it's terrible on HDDs, on SSDs in our case of few hundred gigabytes tables it gives around only 50% performance drop compared to TokuDB on large queries, inserts are around 15% to 30% slower. Which is precisely tolerable for our case - of course everyone mileage may vary. Page compression actually gives around 2x to 4x worse compression compared to TokuDB (depending on table structure and data), but this is still better than 10x to 20x growth we have with uncompressed InnoDB. Disk throughput corresponds, and while it's still 1.5x to 4x load on queries, it's tolerable (while uncompressed InnoDB easily saturates 10G iSCSI link on the same query set of ours).

            I saw another issue (don't remember the number) suggesting page compression removal for InnoDB, please don't! If this would be removed, this would be a total disaster. I want to stress Rocks is unusable for our case in terms of replication support, transaction isolation behavior, lack of online DDL, memory consumption and much more.

            Alex/AT Alex/AT added a comment - Sorry for commenting again on a closed issue, but I think it would be good for some users to know. A tolerable replacement for TokuDB is InnoDB + page compression (we use zlib, but it would be good to have something like zstd & lzma compiled in for next versions, maybe this can be considered). While it's terrible on HDDs, on SSDs in our case of few hundred gigabytes tables it gives around only 50% performance drop compared to TokuDB on large queries, inserts are around 15% to 30% slower. Which is precisely tolerable for our case - of course everyone mileage may vary. Page compression actually gives around 2x to 4x worse compression compared to TokuDB (depending on table structure and data), but this is still better than 10x to 20x growth we have with uncompressed InnoDB. Disk throughput corresponds, and while it's still 1.5x to 4x load on queries, it's tolerable (while uncompressed InnoDB easily saturates 10G iSCSI link on the same query set of ours). I saw another issue (don't remember the number) suggesting page compression removal for InnoDB, please don't! If this would be removed, this would be a total disaster. I want to stress Rocks is unusable for our case in terms of replication support, transaction isolation behavior, lack of online DDL, memory consumption and much more.
            marko Marko Mäkelä added a comment -

            Alex/AT, it sounds like you are using page_compressed with InnoDB, and not ROW_FORMAT=COMPRESSED (which I would like to remove; see MDEV-23497 for a first ‘deprecation warning’ step in 10.6)? Did you find a filesystem where fragmentation is not an issue? Note: in 10.5, MDEV-15528 and MDEV-8139 improved the handling of page_compressed tables, by punching holes over completely freed pages.

            Please reply in MDEV-22839.

            marko Marko Mäkelä added a comment - Alex/AT , it sounds like you are using page_compressed with InnoDB, and not ROW_FORMAT=COMPRESSED (which I would like to remove; see MDEV-23497 for a first ‘deprecation warning’ step in 10.6)? Did you find a filesystem where fragmentation is not an issue? Note: in 10.5, MDEV-15528 and MDEV-8139 improved the handling of page_compressed tables, by punching holes over completely freed pages. Please reply in MDEV-22839 .
            marko Marko Mäkelä made changes -
            denji Denis Denisov made changes -
            Comment [ bq. Because MariaDB Server includes MyRocks as well, it makes sense to remove TokuDB in favor of MyRocks.

            You haven't even looked at the numerous shortcomings and limitations of RocksDB for MySQL Engine https://www.percona.com/doc/percona-server/8.0/myrocks/limitations.html

            Racing to victory for efficient managers to the disadvantage of the customer. MyRocks is not ready for production and requires detailed hacks, can cause OOM. ]
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 97543 ] MariaDB v4 [ 133994 ]

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.