[MDEV-19780] Remove the TokuDB storage engine Created: 2019-06-17  Updated: 2021-10-28  Resolved: 2020-05-14

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB
Fix Version/s: 10.6.0

Type: Task Priority: Major
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 1
Labels: None

Issue Links:
Relates
relates to MDEV-22839 ROW_FORMAT=COMPRESSED vs PAGE_COMPRES... Open
relates to MDEV-8843 Ship TokuBackup as part of MariaDB Closed
relates to MDEV-11101 storage/tokudb uses much more space t... Confirmed
relates to MDEV-21803 wrong dependency for mariadb-plugin-t... Closed
relates to MDEV-21944 Remove TokuDB from Debian packaging? Closed

 Description   

The TokuDB storage engine has been deprecated by upstream:

TokuDB is deprecated in the 8.0 series and will be supported through the 8.0 series until further notice. This storage engine will not be included in the next major release of Percona Server for MySQL. We recommend MyRocks as a long-term migration path.

We have neither knowledge nor resources to take over TokuDB maintenance.
TokuDB users can consider migrating to InnoDB or RocksDB, imperfect as it is. Or ask TokuDB owner to resume TokuDB maintenance.



 Comments   
Comment by Marko Mäkelä [ 2019-06-17 ]

bb-10.5-MDEV-19780 looks OK with respect to the TokuDB removal

Comment by Sergei Golubchik [ 2019-08-25 ]

TokuDB is now disabled in 10.5.0.

Comment by Alex/AT [ 2020-09-19 ]

Actually it's a very bad idea.
I have converted few huge tables to RocksDB and well... it's barely usable.
First of all, it takes a whole lot of gigabytes of RAM to sustain the tables, MariaDB server memory usage went from 10G on TokuDB to 20G on RocksDB.
Second, TokuDB has online DDL, not only index creation, but column operations. And it's pretty much replication safe, compared to RocksDB.
Imagine extending a 100-200GB worth RocksDB table with a new index or column, locking table for the while of operation, stalling the system, and you'll quickly understand what's terribly wrong about it.

To sum it, please re-enable TokuDB in current versions. Even if it won't get any new fixes, it's production ready and was the only lifesaver many people with big data tables had.

Comment by Alex/AT [ 2021-01-28 ]

Sorry for commenting again on a closed issue, but I think it would be good for some users to know.

A tolerable replacement for TokuDB is InnoDB + page compression (we use zlib, but it would be good to have something like zstd & lzma compiled in for next versions, maybe this can be considered).
While it's terrible on HDDs, on SSDs in our case of few hundred gigabytes tables it gives around only 50% performance drop compared to TokuDB on large queries, inserts are around 15% to 30% slower. Which is precisely tolerable for our case - of course everyone mileage may vary. Page compression actually gives around 2x to 4x worse compression compared to TokuDB (depending on table structure and data), but this is still better than 10x to 20x growth we have with uncompressed InnoDB. Disk throughput corresponds, and while it's still 1.5x to 4x load on queries, it's tolerable (while uncompressed InnoDB easily saturates 10G iSCSI link on the same query set of ours).

I saw another issue (don't remember the number) suggesting page compression removal for InnoDB, please don't! If this would be removed, this would be a total disaster. I want to stress Rocks is unusable for our case in terms of replication support, transaction isolation behavior, lack of online DDL, memory consumption and much more.

Comment by Marko Mäkelä [ 2021-01-28 ]

Alex/AT, it sounds like you are using page_compressed with InnoDB, and not ROW_FORMAT=COMPRESSED (which I would like to remove; see MDEV-23497 for a first ‘deprecation warning’ step in 10.6)? Did you find a filesystem where fragmentation is not an issue? Note: in 10.5, MDEV-15528 and MDEV-8139 improved the handling of page_compressed tables, by punching holes over completely freed pages.

Please reply in MDEV-22839.

Generated at Thu Feb 08 08:54:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.