Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12933

sort out the compression library chaos

Details

    Description

      As MariaDB is getting more storage engines and as they're getting more features, MariaDB can optionally use more and more compression libraries for various purposes.

      InnoDB, TokuDB, RocksDB — they all can use different sets of compression libraries. Compiling them all in would result in a lot of run-time/rpm/deb dependencies, most of which will be never used by most of the users. Not compiling them in, would result in requests to compile them in. While most users don't use all these libraries, many users use some of these libraries.

      A solution could be to load these libraries on request, without creating a packaging dependency. There are different ways to do it

      • hide all compression libraries behind a single unified compression API. Either develop our own or use something like Squash. This would require changing all engines to use this API
      • use the same approach as in server services — create a service per compression library, a service implementation will just return an error code for any function invocation if the corresponding library is not installed. this way — may be — we could avoid modifying all affected storage engines

      Attachments

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. MDEV-21877 Enable snappy compression by default for InnoDB page compression

          • Major - Major loss of function.
          • Open
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26762 Compressed table remains accessible without provider after unclean restart

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26773 MariaBackup backup does not work with compression providers

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26774 Compression provider unloading at runtime has no effect but doesn't produce a warning

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26775 Error messages about unavailable compression libraries not returned in diagnostics

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26776 mroonga_libgroonga_support_lz4 is set to ON even when lz4 is not available, CREATE behavior changes

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26791 MariaBackup logs compression provider plugins as encryption plugin

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26792 MariaBackup prepare crashes with invalid pointer with compression providers and extra options

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26794 MariaBackup does not recognize added providers upon prepare of incremental backup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26795 InnoDB writes error messages about compressed tables upon startup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-26888 Fedora 33/34 in the foundation buildbot is missing lzo and bzip2

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Task - A task that needs to be done. MDEV-26924 document compatibility considerations re. provider plugins

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27887 InnoDB may crash when required compression plugins are not loaded

          • Major - Major loss of function.
          • Confirmed
          includes

          Task - A task that needs to be done. MDEV-22895 Implement server support for making compression library dependencies optional

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15048 storage/innobase: Fix cmake compression library detection

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20255 InnoDB LZ4 and LZMA compression algorithms aren't available on Debian 9

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26757 Compression-related tests fail with MSAN errors in buf_page_is_corrupted

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26907 innochecksum fails to validate full_crc32 for compressed or encrypted tables

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-9821 Distribute MariaDB with more compression algorithms in the build

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. MDEV-11068 Review which innodb_compression_algorithm to support in binary packages

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-13281 Rocksdb doesn't have ZSTD in mariadb 10.2.6

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Task - A task that needs to be done. MDEV-22762 Investigate if using Squash is viable for unifying compression APIs

          • Major - Major loss of function.
          • Stalled

          Activity

            Transition Time In Source Status Execution Times
            Sergei Golubchik made transition -
            Open Confirmed
            		1516d 2h 49m 1
            Sergei Golubchik made transition -
            Confirmed In Review
            		14s 1
            Sergei Golubchik made transition -
            Stalled In Review
            		4d 20h 18m 1
            Marko Mäkelä made transition -
            In Review Stalled
            		36d 2h 49m 2
            Sergei Golubchik made transition -
            Stalled Closed
            		57d 6h 9m 1

            People

              serg Sergei Golubchik
              serg Sergei Golubchik
              Votes:
              5 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.