[MDEV-22895] Implement server support for making compression library dependencies optional Created: 2020-06-15 Updated: 2021-08-11 Resolved: 2021-08-11 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Backup, Compiling, Packaging, Storage Engine - InnoDB |
| Fix Version/s: | N/A |
| Type: | Task | Priority: | Major |
| Reporter: | Robert Bindar | Assignee: | Sergei Golubchik |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | Compatibility | ||
| Issue Links: |
|
||||||||||||||||||||
| Description |
|
Squash is off the table for now, there are two strong arguments against it: 1. Using Squash requires changing code in all the storage engines using compression libraries, including 3rd party storage engines and this is problematic because it is very unlikely we will be able to propagate these changes upstream. Given the above, we should implement this with MariaDB services using an approach like the one below: 1. Add new `--use-compression=snappy,bzip2` option in the server. 2. Choose one compression library and create a new mariadb service that contains dummy functions for all corresponding functions within the compression library (only functions that are used by the storage engines we support). Make sure the service is built correctly, test it by calling one of its functions from a dynamic plugin’s init function and see it printing something. 3. Alter the service so that it works with static plugins as well (we need this for storage engines such as InnoDB). Quickly test that it works with a dummy print as well, from InnoDB for example. 4. We need to have the ability to trick storage engines into calling our custom service API instead of the compression library code. Thus, we need to tweak the include paths so that storage engines using this library will include our own header, with our own functions, and this without actually changing any storage engine source code. 5. The server should try to load the compression libraries based on the values passed within the `use-compression` option. If a library is loaded correctly, then the function pointers within the corresponding compression service should be set to point within that compression library. 6. Tests should be written for each service that takes care of one compression library. |
| Comments |
| Comment by Kartik Soneji [ 2020-07-27 ] | ||||||||||||
|
We should add global status variables to the server so that the user can easily view which libraries are loaded.
will not cause an error even if lzma is not loaded. | ||||||||||||
| Comment by Kartik Soneji [ 2020-07-27 ] | ||||||||||||
|
Maybe it is better to change all to auto for the --use-compression= switch.
| ||||||||||||
| Comment by Marko Mäkelä [ 2021-03-25 ] | ||||||||||||
|
I think that before this task can be meaningfully worked on, we need a resolution on I would strongly advise against enabling support for any compression algorithms, unless we will commit to that support for the indefinite future. I think that we have an implied promise of seamless upgrades. If we enabled support of (say) innodb_compression_algorithm=bzip2 in some package that we distribute, some user could complain when we disabled that support later, after deciding that the memory and CPU overhead during compression and decompression is not worth the possible savings compared to innodb_compression_algorithm=zlib (which we can commit to always supporting, because zlib is a core dependency of the server). A minimum requirement for enabling new compression algorithms should be the development of a tool that can convert data files from one innodb_compression_algorithm to another in-place, to facilitate upgrades. | ||||||||||||
| Comment by Kartik Soneji [ 2021-03-25 ] | ||||||||||||
|
Which compression algorithms are actually useful in practice? Is there any telemetry/user poll data on this? | ||||||||||||
| Comment by Sergei Golubchik [ 2021-08-11 ] | ||||||||||||
|
part of |