[MCOL-1151] mcsapi needs a new read API Design Created: 2018-01-09  Updated: 2023-10-26  Resolved: 2022-11-05

Status: Closed
Project: MariaDB ColumnStore
Component/s: None
Affects Version/s: None
Fix Version/s: Icebox

Type: Task Priority: Major
Reporter: Andrew Hutchings (Inactive) Assignee: Unassigned
Resolution: Won't Do Votes: 0
Labels: gsoc18, gsoc19, gsoc20

Epic Link: Consolidate & Redevelop All Columnstore Tools (SDK, Adapters, Backup, Restore, mcsimport)
Epic/Theme: MCOL-4571

 Description   

We need an ORM-style NoSQL read API to go along with the bulk write API of mcsapi.

This will likely take the form of:

1. A reader in ExeMgr which will convert messages from mcsapi into jobs
2. Code in mcsapi to send/receive the messages

Although ExeMgr can already receive messages with an execution plan the format is very complex and ABI breaks easily (we often send whole C++ objects).

We should look at other ORM frameworks for inspiration as the the API design.

This task to do the design for this API.



 Comments   
Comment by David Thompson (Inactive) [ 2018-01-10 ]

I would actually look at an ORM framework as reference, like for example hibernate criteria: https://docs.jboss.org/hibernate/entitymanager/3.5/reference/en/html/querycriteria.html. However not saying this is the best example

My thinking is to provide a simplistic query ability over what we can do with primitives such as filter and project. The main use case i see is bulk extracts either for backup or replication or pulling out large datasets with some basic filtering and limiting of columns into a machine learning framework in python or spark. Functions / mapping of these values is probably out of scope as it can be done in the client code. For the former use case of replication / filtering it likely would be an advantage to offer some positional metadata / filtering too allowing for incremental extracts.

Obviously the hope is that this can be a lot faster (but less flexible) than going through the sql layer.

Comment by Todd Stoffel (Inactive) [ 2022-11-05 ]

This item is being closed because it was well passed the expiration date with no activity. If you suspect this was done in error please create a new ticket.

Generated at Thu Feb 08 02:26:35 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.