[MCOL-1151] mcsapi needs a new read API Design Created: 2018-01-09 Updated: 2023-10-26 Resolved: 2022-11-05 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | Icebox |
| Type: | Task | Priority: | Major |
| Reporter: | Andrew Hutchings (Inactive) | Assignee: | Unassigned |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | gsoc18, gsoc19, gsoc20 | ||
| Epic Link: | Consolidate & Redevelop All Columnstore Tools (SDK, Adapters, Backup, Restore, mcsimport) |
| Epic/Theme: | MCOL-4571 |
| Description |
|
We need an ORM-style NoSQL read API to go along with the bulk write API of mcsapi. This will likely take the form of: 1. A reader in ExeMgr which will convert messages from mcsapi into jobs Although ExeMgr can already receive messages with an execution plan the format is very complex and ABI breaks easily (we often send whole C++ objects). We should look at other ORM frameworks for inspiration as the the API design. This task to do the design for this API. |
| Comments |
| Comment by David Thompson (Inactive) [ 2018-01-10 ] |
|
I would actually look at an ORM framework as reference, like for example hibernate criteria: https://docs.jboss.org/hibernate/entitymanager/3.5/reference/en/html/querycriteria.html. However not saying this is the best example My thinking is to provide a simplistic query ability over what we can do with primitives such as filter and project. The main use case i see is bulk extracts either for backup or replication or pulling out large datasets with some basic filtering and limiting of columns into a machine learning framework in python or spark. Functions / mapping of these values is probably out of scope as it can be done in the client code. For the former use case of replication / filtering it likely would be an advantage to offer some positional metadata / filtering too allowing for incremental extracts. Obviously the hope is that this can be a lot faster (but less flexible) than going through the sql layer. |
| Comment by Todd Stoffel (Inactive) [ 2022-11-05 ] |
|
This item is being closed because it was well passed the expiration date with no activity. If you suspect this was done in error please create a new ticket. |