[MDEV-22310] Support zstd Compression algorithm for IO Created: 2020-04-20  Updated: 2023-12-21

Status: Open
Project: MariaDB Server
Component/s: Protocol
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: Diego Dupin Assignee: Unassigned
Resolution: Unresolved Votes: 4
Labels: None

Issue Links:
Blocks
blocks MDEV-11068 Review which innodb_compression_algor... Closed

 Description   

Goal

Server support zlib library for compression.
Facebook zstd (open-source dual BSD and GPLv2 license) argue to have better compression and decompression speed for better ratio.

MySQL implement that in 8.0.
see task , task, task and documentation

New variables

[Compression_algorithm|https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html#statvar_Compression_algorithm] : The name of the compression algorithm in use for the current connection to the server
[Compression_level|https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html#statvar_Compression_level]: The compression level in use for the current connection to the server
protocol_compression_algorithms: List the compression algorithms that the server permits for incoming connections.

Protocol change

New capability CLIENT_ZSTD_COMPRESSION_ALGORITHM (1UL << 26) indicate supporting zstd algorithm. (enable when global variable protocol-compression-algorithms contains zstd).

Handshake Response Packet has a new "compress_level" value

* int<4> client capabilities
* int<4> max packet size
* int<1> client character collation
* string<19> reserved
* if not (server_capabilities & CLIENT_MYSQL)
  * int<4> extended client capabilities
* else
  * string<4> reserved
* string<NUL> username
* if (server_capabilities & PLUGIN_AUTH_LENENC_CLIENT_DATA)
  * string<lenenc> authentication data
* else if (server_capabilities & CLIENT_SECURE_CONNECTION)
  * int<1> length of authentication response
  * string<fix> authentication response (length is indicated by previous field)
* else
  * int<1> 0x00
* if (server_capabilities & CLIENT_CONNECT_WITH_DB)
  * string<NUL> default database name
* if (server_capabilities & CLIENT_PLUGIN_AUTH)
  * string<NUL> authentication plugin name
* if (server_capabilities & CLIENT_CONNECT_ATTRS)
  * int<lenenc> size of connection attributes
  * loop
    * string<lenenc> key
    * string<lenenc> value
* if (server_capabilities & CLIENT_ZSTD_COMPRESSION_ALGORITHM && client_capabilities & CLIENT_ZSTD_COMPRESSION_ALGORITHM )
  * string<1> compression level

No dictionnary is used in MySQL implementation, but dictionary with zstd benchmarks generally improve encryption/decryption time drastically for small amount of data. Some verification might be done to see if that this can be applicable to MariaDB exchanges



 Comments   
Comment by Vladislav Vaintroub [ 2020-04-20 ]

If I can be allowed a sceptical remark.

  • this change in MySQL have been pushed by the Facebook. There is no evidence that outside of facebook anyone is interested in this. As far as I can tell, so far , there is no indication that zstd is going to enjoy the same level of popularity and support as zlib. Standard libraries in higher level languages do not support it, at least in JDK and .NET(standard and core) as most prominent examples, I have not looked further
  • zlib allows different levels, and the choosing the correct level (usually, lower than standard, which is 6) , could provide faster compression decompression to supposedly better algorithms, at a comparable speed.
  • At least on the server side, if I remember correctly, compression decompression operations have high overhead, and take a long time because of extra allocations, one per compression/decompression operation. maybe we should look first at improving that first.

As for dictionary , what can be added to it except "\03def"?

Generated at Thu Feb 08 09:13:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.