Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12933

sort out the compression library chaos

Details

    Description

      As MariaDB is getting more storage engines and as they're getting more features, MariaDB can optionally use more and more compression libraries for various purposes.

      InnoDB, TokuDB, RocksDB — they all can use different sets of compression libraries. Compiling them all in would result in a lot of run-time/rpm/deb dependencies, most of which will be never used by most of the users. Not compiling them in, would result in requests to compile them in. While most users don't use all these libraries, many users use some of these libraries.

      A solution could be to load these libraries on request, without creating a packaging dependency. There are different ways to do it

      • hide all compression libraries behind a single unified compression API. Either develop our own or use something like Squash. This would require changing all engines to use this API
      • use the same approach as in server services — create a service per compression library, a service implementation will just return an error code for any function invocation if the corresponding library is not installed. this way — may be — we could avoid modifying all affected storage engines

      Attachments

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. MDEV-21877 Enable snappy compression by default for InnoDB page compression

          • Major - Major loss of function.
          • Open
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26762 Compressed table remains accessible without provider after unclean restart

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26773 MariaBackup backup does not work with compression providers

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26774 Compression provider unloading at runtime has no effect but doesn't produce a warning

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26775 Error messages about unavailable compression libraries not returned in diagnostics

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26776 mroonga_libgroonga_support_lz4 is set to ON even when lz4 is not available, CREATE behavior changes

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26791 MariaBackup logs compression provider plugins as encryption plugin

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26792 MariaBackup prepare crashes with invalid pointer with compression providers and extra options

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26794 MariaBackup does not recognize added providers upon prepare of incremental backup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26795 InnoDB writes error messages about compressed tables upon startup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-26888 Fedora 33/34 in the foundation buildbot is missing lzo and bzip2

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Task - A task that needs to be done. MDEV-26924 document compatibility considerations re. provider plugins

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27887 InnoDB may crash when required compression plugins are not loaded

          • Major - Major loss of function.
          • Confirmed
          includes

          Task - A task that needs to be done. MDEV-22895 Implement server support for making compression library dependencies optional

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15048 storage/innobase: Fix cmake compression library detection

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20255 InnoDB LZ4 and LZMA compression algorithms aren't available on Debian 9

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26757 Compression-related tests fail with MSAN errors in buf_page_is_corrupted

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26907 innochecksum fails to validate full_crc32 for compressed or encrypted tables

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-9821 Distribute MariaDB with more compression algorithms in the build

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. MDEV-11068 Review which innodb_compression_algorithm to support in binary packages

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-13281 Rocksdb doesn't have ZSTD in mariadb 10.2.6

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Task - A task that needs to be done. MDEV-22762 Investigate if using Squash is viable for unifying compression APIs

          • Major - Major loss of function.
          • Stalled

          Activity

            Ascending order - Click to sort in descending order
            serg Sergei Golubchik created issue -
            serg Sergei Golubchik made changes -
            Field Original Value New Value
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.4 [ 22408 ]
            serg Sergei Golubchik made changes -
            Labels gsoc19
            serg Sergei Golubchik made changes -
            Fix Version/s 10.4 [ 22408 ]
            cvicentiu Vicențiu Ciorbaru made changes -
            Labels gsoc19 gsoc19 gsoc20
            GeoffMontee Geoff Montee (Inactive) made changes -
            ratzpo Rasmus Johansson (Inactive) made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            robertbindar Robert Bindar made changes -
            robertbindar Robert Bindar made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            julien.fritsch Julien Fritsch made changes -
            Labels gsoc19 gsoc20 DesignNeeded gsoc19 gsoc20
            robertbindar Robert Bindar made changes -
            Assignee Sergei Golubchik [ serg ] Robert Bindar [ robertbindar ]
            robertbindar Robert Bindar made changes -
            Labels DesignNeeded gsoc19 gsoc20 gsoc19 gsoc20
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            GeoffMontee Geoff Montee (Inactive) made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            serg Sergei Golubchik made changes -
            Assignee Robert Bindar [ robertbindar ] Sergei Golubchik [ serg ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.7 [ 24805 ]
            serg Sergei Golubchik made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]
            serg Sergei Golubchik made changes -
            Status Confirmed [ 10101 ] In Review [ 10002 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Due Date 2021-09-14
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Marko Mäkelä [ marko ]
            Status Stalled [ 10000 ] In Review [ 10002 ]
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Sergei Golubchik [ serg ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Elena Stepanova [ elenst ]
            elenst Elena Stepanova made changes -
            serg Sergei Golubchik made changes -
            Assignee Elena Stepanova [ elenst ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Elena Stepanova [ elenst ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Due Date 2021-09-14
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            elenst Elena Stepanova made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 10.7.1 [ 26120 ]
            Fix Version/s 10.7 [ 24805 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Assignee Elena Stepanova [ elenst ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 80978 ] MariaDB v4 [ 133269 ]
            marko Marko Mäkelä made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Labels gsoc19 gsoc20 Preview_10.7 gsoc19 gsoc20
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 181586 193884
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -

            People

              serg Sergei Golubchik
              serg Sergei Golubchik
              Votes:
              5 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.