Details
-
Type:
Sub-Task
-
Status: Closed (View Workflow)
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 6.1.1
-
Fix Version/s: 6.1.1
-
Component/s: writeengine
-
Labels:None
-
Sprint:2021-5, 2021-6, 2021-7, 2021-8, 2021-9
Description
LZ4 compression support for `CompressedInetStreamSocket` and `JoinPartition`.
1. `getUncompressedSize`: currently `CompressedInetStreamSocket` and `JoinPartition` require a function `getUncompressedSize`, so we can allocate appropriate size for uncompressed buffer. LZ4 does not have by default this function, reading the `snappy` documentation I found out that the cost of this function is O(1) so it probably stores the input size in the compressed block, so I think we can do the same way for those two specific parts.
2. Currently `CompressedInetStreamSocket` and `JoinPartition` use config to enable/disable compression, for sockets and for temporal files. Probably we should add a one more field in config to specify the compression type.