Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12933

sort out the compression library chaos

    XMLWordPrintable

    Details

      Description

      As MariaDB is getting more storage engines and as they're getting more features, MariaDB can optionally use more and more compression libraries for various purposes.

      InnoDB, TokuDB, RocksDB — they all can use different sets of compression libraries. Compiling them all in would result in a lot of run-time/rpm/deb dependencies, most of which will be never used by most of the users. Not compiling them in, would result in requests to compile them in. While most users don't use all these libraries, many users use some of these libraries.

      A solution could be to load these libraries on request, without creating a packaging dependency. There are different ways to do it

      • hide all compression libraries behind a single unified compression API. Either develop our own or use something like Squash. This would require changing all engines to use this API
      • use the same approach as in server services — create a service per compression library, a service implementation will just return an error code for any function invocation if the corresponding library is not installed. this way — may be — we could avoid modifying all affected storage engines

        Attachments

          Issue Links

          blocks

          Task - A task that needs to be done. MDEV-21877 Enable snappy compression by default for InnoDB page compression

          • Major - Major loss of function.
          • Open
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26762 Compressed table remains accessible without provider after unclean restart

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26773 MariaBackup backup does not work with compression providers

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26774 Compression provider unloading at runtime has no effect but doesn't produce a warning

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26775 Error messages about unavailable compression libraries not returned in diagnostics

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26776 mroonga_libgroonga_support_lz4 is set to ON even when lz4 is not available, CREATE behavior changes

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26791 MariaBackup logs compression provider plugins as encryption plugin

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26792 MariaBackup prepare crashes with invalid pointer with compression providers and extra options

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26794 MariaBackup does not recognize added providers upon prepare of incremental backup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26795 InnoDB writes error messages about compressed tables upon startup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-26888 Fedora 33/34 in the foundation buildbot is missing lzo and bzip2

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Task - A task that needs to be done. MDEV-26924 document compatibility considerations re. provider plugins

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          includes

          Task - A task that needs to be done. MDEV-22895 Implement server support for making compression library dependencies optional

          • Major - Major loss of function.
          • Closed
          is blocked by

          Task - A task that needs to be done. MDEV-22762 Investigate if using Squash is viable for unifying compression APIs

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15048 storage/innobase: Fix cmake compression library detection

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26757 Compression tests fail with MSAN errors in buf_page_is_corrupted

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26907 innochecksum fails to validate full_crc32 for compressed or encrypted tables

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-9821 Distribute MariaDB with more compression algorithms in the build

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Task - A task that needs to be done. MDEV-13281 Rocksdb doesn't have ZSTD in mariadb 10.2.6

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

            Activity

              People

              Assignee:
              serg Sergei Golubchik
              Reporter:
              serg Sergei Golubchik
              Votes:
              5 Vote for this issue
              Watchers:
              20 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration