I could reproduce the described behavior, at least the part about the first OPTIMIZE. I've never seen further growth on the 2nd OPTIMIZE, but sometimes TokuDB files are updated slowly, maybe TokuDB was still writing the file when you measured the size after the first OPTIMIZE; or, maybe in some cases it does happen that the 2nd OPTIMIZE causes this effect.
Anyway, for the 1st OPTIMIZE, it's also reproducible on TokuDB builds, both with MariaDB and MySQL (I tried 5.5.36 for both), and with all compression algorithms.
I hope Tokutek people will be able to shed some light on it.
Will you forward this report to Tokutek or i should contact them myself?
We already have. The report is assigned to Rich Prohaska who is our Tokutek contact here in JIRA, let's see if he confirms it is a bug.
This is expected behavior for small tables, and it's because OPTIMIZE TABLE has a different meaning in TokuDB.
TokuDB stores data in fractal trees, which are like B-trees but with buffers on the internal nodes that can store messages (inserts, updates, deletes, etc) destined for leaves below them. The OPTIMIZE TABLE command flushes all of those buffers in depth-first left-to-right order, to flatten the tree. This dirties all the nodes in the tree over time, so they'll all get written out again.
Since our block allocator is copy-on-write, checkpoint makes new copies of those nodes that were changed between each checkpoint, and the old versions of those nodes are freed after the checkpoint completes.
There is an awkward interaction between checkpoints, the block allocator, and OPTIMIZE TABLE on small trees: if the OPTIMIZE runs fast enough, it can dirty the entire tree in between checkpoints, and then on the next checkpoint, the block allocator essentially needs to duplicate the entire tree. If the tree was already fairly compact, this will nearly double the file size, and when the checkpoint completes, most of the beginning half of the file will be freed. After this, you'll have to wait until all the nodes get dirtied again and get written out to fill in all those holes early in the file, before the file can be truncated and start to shrink.
So my advice is, don't run OPTIMIZE TABLE on small TokuDB tables, it generally isn't necessary and will just make strange things happen.
The final 'alter table engine=tokudb' operation rebuilds the table by coping all of the rows from the original table to a new table. At the end, mysql renames the new table and deletes the old table. Since the new table is created from scratch, its size is expected to be smaller than the original table since there is only one copy of the fractal tree blocks.
Big thanks for the explanation, but can you clarity what "small" means?
For example table with 100M rows. It's about 10 GB of data. It's "small" for TokuDB, right?
'select * from information_schema.tokudb_fractal_tree_info' returns bytes used and bytes allocated for each fractal tree. 'allocated-used' is the amount of free space in the fractal tree file that will be used by new blocks as data gets added to the table. someday, we will implement algorithms that return the free space to the file system.
Thanks. BTW, i guess it can be a good idea to invent a session variable tokudb_optimizetable (integer 0 or 1). If 0 - "optimize table" action will just do nothing, but returns success. Or 1 - it will act just like now. Defaults to 1. As far as it session variable, in my.cnf it will set default value, but it can be changed in the session.
It can be useful for people, who have many small table (let's say <100M rows), but a few big tables
I don't understand the proposal. Why not just decide to not execute an optimize statement?
Well, there are sometimes no way to control an application or user(s) which use database. Application and users can do Optimize table on tables and increase database size this way. Most of Content Management Systems use MySQL database optimize every day. It's also can be accidently caused by yourself and then it will be a pain to revert the process, it'll cause a lot of downtime. Also the bad thing at the moment, is that there are no Hot optimize table operation at MariaDB. So it can be better to prevent it with MySQL server config.
From what i tested, with 500M rows, i still have 2x increase in database size. It can really break predictions of drive space needed for database 
If you think that it is not needed, nothing wrong here. Thanks for your time.
TokuDB (at least Tokutek's version) supports hot optimize table. hot in the sense that that concurrent reads and writes can occur while optimize is running. Would this feature be useful?
Sure it will. I am using MariaDB native version at the moment.
But Optimize table is still can take hours to be done.
TokuDB 7.1.7 has online optimize table for MariaDB 10.0 which will allow concurrent table optimize + table reads + table writes.
Yeah, i know, thanks.
tokudb 7.17. is merged into mariadb 5.5.39 (and will be in 10.0.13)