[MDEV-23196] Compression opts not enabled on user defined settings Created: 2020-07-16  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Affects Version/s: 10.2.25, 10.5.4
Fix Version/s: 10.4, 10.5

Type: Bug Priority: Major
Reporter: Jonas Krauss Assignee: Sergei Petrunia
Resolution: Unresolved Votes: 1
Labels: None
Environment:

mariadb.org binary distribution
Ubuntu 18.04.3 LTS
Ubuntu 19.10

Attachments: Text File log.txt     Text File log_not_working.txt    

 Description   

When setting compression_opts in rocksdb_default_cf_options, it is supposed to set the rocksdb internal setting Options.compression_opts.enabled to true. Checking the rocksdb LOG in MariaDB the value stays false, unless explicitly set to true.

Not working sample conf: rocksdb_default_cf_options=...;compression_opts:-14:3:0;...

Working sample conf: rocksdb_default_cf_options=...;compression_opts:-14:3:0:0:0:true;...

This behavior seems not desired according to rocksdb's description of this parameter:
// When the compression options are set by the user, it will be set to "true".
// For bottommost_compression_opts, to enable it, user must set enabled=true.
// Otherwise, bottommost compression will use compression_opts as default
// compression options.
//
// For compression_opts, if compression_opts.enabled=false, it is still
// used as compression options for compression process.
//
// Default: false.
bool enabled;
https://github.com/facebook/rocksdb/blob/master/include/rocksdb/advanced_options.h

Not sure if this is supposed to be different in MyRocks than in RocksDB but my best guess is that it is not. My fear is that set compression options are not actually used.

 Comments   
Comment by Jonas Krauss [ 2020-07-17 ]

Debugging further, the parameter "enabled" might not play a role for the compression opts to be used. I have run a couple a imports with and without bulk load on a vanilla MariaDB instance (with wiping the database clean between imports). It turns out the size of the SST files is the same with "enabled" = false and "enabled" = true.

E.g. 

compression_opts=-14:3:0;
 
 
 
ls -lh /data/rocksdb/data
 
total 928M
 
-rw-r----- 1 mysql mysql 131M Jul 17 10:27 000030.sst
 
-rw-r----- 1 mysql mysql 1,8K Jul 17 10:31 000046.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:33 000053.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:33 000054.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:33 000055.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:33 000056.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:33 000057.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:33 000058.sst
 
-rw-r----- 1 mysql mysql  15M Jul 17 10:33 000059.sst
 
-rw-r----- 1 mysql mysql  12M Jul 17 10:34 000061.sst
 
-rw-r----- 1 mysql mysql 1,5K Jul 17 10:34 000062.sst

produces the same size of rocksdb files as

compression_opts=-14:3:0:0:0:true;
 
 
 
ls -lh /data/rocksdb/data/
 
total 928M
 
-rw-r----- 1 mysql mysql 131M Jul 17 10:46 000019.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:51 000041.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:51 000042.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:51 000043.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:52 000044.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:52 000045.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 10:52 000046.sst
 
-rw-r----- 1 mysql mysql  15M Jul 17 10:52 000047.sst
 
-rw-r----- 1 mysql mysql  12M Jul 17 10:52 000049.sst
 
-rw-r----- 1 mysql mysql 1,8K Jul 17 10:52 000051.sst

Changing the compression opts to a higher level, without enable set to true, does bring down the size:

compression_opts=-14:9:0;
 
 
 
ls -lh /data/rocksdb/data/
 
total 873M
 
-rw-r----- 1 mysql mysql 125M Jul 17 11:59 000013.sst
 
-rw-r----- 1 mysql mysql 1,6K Jul 17 11:59 000014.sst
 
-rw-r----- 1 mysql mysql 1,6K Jul 17 12:01 000021.sst
 
-rw-r----- 1 mysql mysql 1,5K Jul 17 12:03 000028.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 12:06 000035.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 12:06 000036.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 12:06 000037.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 12:06 000038.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 12:06 000039.sst
 
-rw-r----- 1 mysql mysql  97M Jul 17 12:07 000040.sst
 
-rw-r----- 1 mysql mysql  11M Jul 17 12:07 000042.sst
 
-rw-r----- 1 mysql mysql 1,5K Jul 17 12:07 000043.sst

I might just have misunderstood the meaning of this parameter. Would be glad if someone with deeper knowledge would shed some light onto this. By now I have probably read every page of the rocksdb and myrocks github wiki twice, but haven't found a clear explanation.

Btw: setting the max_dict_bytes/zstd_max_train_bytes does not seem to have an effect. This might be due to the fact, that in this simple scenario I have tested, not enough levels exits to trigger bottommost_compression or the data I have used is not benefiting from the dictionary. I have also tested with a block size of 64kb, the dictionary effect is likely smaller at this size.

compression_opts=-14:3:0:131072:131072:true;
 
 
 
ls -lh /data/rocksdb/data/
 
total 928M
 
-rw-r----- 1 mysql mysql 131M Jul 17 11:34 000013.sst
 
-rw-r----- 1 mysql mysql 1,6K Jul 17 11:34 000014.sst
 
-rw-r----- 1 mysql mysql 1,6K Jul 17 11:36 000021.sst
 
-rw-r----- 1 mysql mysql 1,5K Jul 17 11:37 000028.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 11:39 000035.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 11:39 000036.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 11:39 000037.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 11:39 000038.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 11:39 000039.sst
 
-rw-r----- 1 mysql mysql 129M Jul 17 11:40 000040.sst
 
-rw-r----- 1 mysql mysql  15M Jul 17 11:40 000041.sst
 
-rw-r----- 1 mysql mysql  12M Jul 17 11:41 000043.sst
 
-rw-r----- 1 mysql mysql 1,5K Jul 17 11:41 000044.sst

Generated at Thu Feb 08 09:20:35 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.