[MCOL-4810] Redundant copying and wasting memory in PrimProc Created: 2021-07-12  Updated: 2022-03-29  Resolved: 2021-10-08

Status: Closed
Project: MariaDB ColumnStore
Component/s: ExeMgr, PrimProc
Affects Version/s: 6.1.1
Fix Version/s: 6.2.1, 6.2.2

Type: New Feature Priority: Critical
Reporter: Roman Assignee: Roman
Resolution: Fixed Votes: 0
Labels: performance

Issue Links:
Problem/Incident
causes MCOL-4626 Columnstore cluster becomes non opera... Stalled
causes MCOL-4954 Add support for new DBRM socket bytes... Closed
Epic Link: ColumnStore Performance Improvements
Sprint: 2021-10, 2021-11, 2021-12

 Description   

There is a class called ByteStream that represents ... a byte stream. BS allocates and uses another arbitrary sized chunk of RAM storing RGData with long strings. BS also runs a memcpy to move the data.
This behavior is too wasteful resource-wise. The current implementation of BS can't mix together the stream from its own buffer and data from a smart pointer backed buffer.
The suggested approach is to replace BS'es internal buffer in certain cases(when it serializes RGData to send another portion of records over the network) with a stack of smart pointers that contains actual data buffers. This saves RAM that is shared b/w RGData and ByteStream and time b/c runtime skips memcpy calls.



 Comments   
Comment by Roman [ 2021-08-27 ]

4QA We changed the way MCS serializes long strings sending it over the cluster. The main check should be: there is no errors, RAM consumption for queries with long strings is the same or less in PP and EM, performance is the same or better.

Generated at Thu Feb 08 02:53:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.