Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22310

Support zstd Compression algorithm for IO

Details

    • New Feature
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • Protocol
    • None

    Description

      Goal

      Server support zlib library for compression.
      Facebook zstd (open-source dual BSD and GPLv2 license) argue to have better compression and decompression speed for better ratio.

      MySQL implement that in 8.0.
      see task , task, task and documentation

      New variables

      [Compression_algorithm|https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html#statvar_Compression_algorithm] : The name of the compression algorithm in use for the current connection to the server
      [Compression_level|https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html#statvar_Compression_level]: The compression level in use for the current connection to the server
      protocol_compression_algorithms: List the compression algorithms that the server permits for incoming connections.

      Protocol change

      New capability CLIENT_ZSTD_COMPRESSION_ALGORITHM (1UL << 26) indicate supporting zstd algorithm. (enable when global variable protocol-compression-algorithms contains zstd).

      Handshake Response Packet has a new "compress_level" value

      * int<4> client capabilities
      * int<4> max packet size
      * int<1> client character collation
      * string<19> reserved
      * if not (server_capabilities & CLIENT_MYSQL)
        * int<4> extended client capabilities
      * else
        * string<4> reserved
      * string<NUL> username
      * if (server_capabilities & PLUGIN_AUTH_LENENC_CLIENT_DATA)
        * string<lenenc> authentication data
      * else if (server_capabilities & CLIENT_SECURE_CONNECTION)
        * int<1> length of authentication response
        * string<fix> authentication response (length is indicated by previous field)
      * else
        * int<1> 0x00
      * if (server_capabilities & CLIENT_CONNECT_WITH_DB)
        * string<NUL> default database name
      * if (server_capabilities & CLIENT_PLUGIN_AUTH)
        * string<NUL> authentication plugin name
      * if (server_capabilities & CLIENT_CONNECT_ATTRS)
        * int<lenenc> size of connection attributes
        * loop
          * string<lenenc> key
          * string<lenenc> value
      * if (server_capabilities & CLIENT_ZSTD_COMPRESSION_ALGORITHM && client_capabilities & CLIENT_ZSTD_COMPRESSION_ALGORITHM )
        * string<1> compression level

      No dictionnary is used in MySQL implementation, but dictionary with zstd benchmarks generally improve encryption/decryption time drastically for small amount of data. Some verification might be done to see if that this can be applicable to MariaDB exchanges

      Attachments

        Issue Links

          Activity

            wlad Vladislav Vaintroub added a comment - - edited

            If I can be allowed a sceptical remark.

            • this change in MySQL have been pushed by the Facebook. There is no evidence that outside of facebook anyone is interested in this. As far as I can tell, so far , there is no indication that zstd is going to enjoy the same level of popularity and support as zlib. Standard libraries in higher level languages do not support it, at least in JDK and .NET(standard and core) as most prominent examples, I have not looked further
            • zlib allows different levels, and the choosing the correct level (usually, lower than standard, which is 6) , could provide faster compression decompression to supposedly better algorithms, at a comparable speed.
            • At least on the server side, if I remember correctly, compression decompression operations have high overhead, and take a long time because of extra allocations, one per compression/decompression operation. maybe we should look first at improving that first.

            As for dictionary , what can be added to it except "\03def"?

            wlad Vladislav Vaintroub added a comment - - edited If I can be allowed a sceptical remark. this change in MySQL have been pushed by the Facebook. There is no evidence that outside of facebook anyone is interested in this. As far as I can tell, so far , there is no indication that zstd is going to enjoy the same level of popularity and support as zlib. Standard libraries in higher level languages do not support it, at least in JDK and .NET(standard and core) as most prominent examples, I have not looked further zlib allows different levels, and the choosing the correct level (usually, lower than standard, which is 6) , could provide faster compression decompression to supposedly better algorithms, at a comparable speed. At least on the server side, if I remember correctly, compression decompression operations have high overhead, and take a long time because of extra allocations, one per compression/decompression operation. maybe we should look first at improving that first. As for dictionary , what can be added to it except "\03def"?

            People

              Unassigned Unassigned
              diego dupin Diego Dupin
              Votes:
              4 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.