[MDEV-15578] MyRocks: support zstandard compression where the distro allows it Created: 2018-03-15  Updated: 2020-10-03  Resolved: 2020-10-03

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB
Fix Version/s: 10.2.33, 10.3.24, 10.4.14, 10.5.5

Type: Task Priority: Major
Reporter: Sergei Petrunia Assignee: Vicențiu Ciorbaru
Resolution: Fixed Votes: 3
Labels: None

Issue Links:
Duplicate
is duplicated by MDEV-23848 MyRocks/RocksDB add ZSTD compression Closed
Relates
relates to MDEV-13281 Rocksdb doesn't have ZSTD in mariadb ... Open

 Description   

Modern distros include ZStandard compression library.
This task is to have MyRocks compiled with that compression library, and have it as a dependency.
The goal is only to support it when the OS has a package for ZStandard.

Fedora 24 has these packages:

zstd.x86_64 : Zstd compression library
libzstd.i686 : Zstd shared library
libzstd.x86_64 : Zstd shared library
libzstd-devel.i686 : Header files for Zstd library

Ubuntu Xenial 16.04.3 LTS has

libzstd-dev - fast lossless compression algorithm -- development files
libzstd0 - fast lossless compression algorithm
zstd - fast lossless compression algorithm -- CLI tool
libzstd1 - fast lossless compression algorithm
libzstd1-dev - fast lossless compression algorithm -- development files

Debian Stretch has it: https://packages.debian.org/stretch/libzstd1
CentOS 7 doesn't have it.



 Comments   
Comment by Sergei Petrunia [ 2018-03-29 ]

There might be a problem with fixing this in 10.2 as changing package dependencies in a stable version may create problems. In this case, let's compile/link/depend on ZStandard in 10.3.

Comment by Daniel Black [ 2018-05-24 ]

Centos/RHEL 7 - there is an EPEL package for libzstd-devel: https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/l/

Comment by MG [ 2020-08-05 ]

Lack of zstd support in MyRocks for MariaDB was called out in a recent blog post and I have also wished to use this library for InnoDB page compression, so I voted on this issue.

Comment by Charalampos Michael [ 2020-10-03 ]

Please add zstd support!

Comment by MG [ 2020-10-03 ]

I see that in the changelog for 10233 and 10324 this was added, and the linked commit mentions also that 10414 and 1055 corrected the makefile problem.

Quick test seems to show that zstd works in 10.4.14:

MariaDB [test2]> select   @@version, SP.sst_name, SP.compression_algo from   information_schema.rocksdb_sst_props SP,   information_schema.rocksdb_ddl D,   information_schema.rocksdb_index_file_map IFM where   D.table_schema='test2' and D.table_name='myrockst1' and   D.index_number= IFM.index_number and   IFM.sst_name=SP.sst_name;
+---------------------+------------+------------------+
| @@version           | sst_name   | compression_algo |
+---------------------+------------+------------------+
| 10.4.14-MariaDB-log | 000096.sst | ZSTD             |
+---------------------+------------+------------------+
1 row in set (0.001 sec)

Sadly this fix being for storage/rocksdb/build_rocksdb.cmake and the libzstd dependency coming with MariaDB-rocksdb-engine means that innodb_compression_algorithm was unaffected.

I suspect that the folks at MariaDB can now close this bug.

Comment by Daniel Black [ 2020-10-03 ]

Thanks mg for confirming

MDEV-12933 is there to unify the provision of compression across engines

Generated at Thu Feb 08 08:22:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.