Details
-
Sub-Task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
6.1.1
-
None
-
2021-5, 2021-6, 2021-7, 2021-8, 2021-9
Description
LZ4 compression support for `CompressedInetStreamSocket` and `JoinPartition`.
1. `getUncompressedSize`: currently `CompressedInetStreamSocket` and `JoinPartition` require a function `getUncompressedSize`, so we can allocate appropriate size for uncompressed buffer. LZ4 does not have by default this function, reading the `snappy` documentation I found out that the cost of this function is O(1) so it probably stores the input size in the compressed block, so I think we can do the same way for those two specific parts.
2. Currently `CompressedInetStreamSocket` and `JoinPartition` use config to enable/disable compression, for sockets and for temporal files. Probably we should add a one more field in config to specify the compression type.