native write data api
(MCOL-511)
|
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | writeengine |
| Affects Version/s: | 1.1.0 |
| Fix Version/s: | 1.1.0 |
| Type: | Sub-Task | Priority: | Major |
| Reporter: | Andrew Hutchings (Inactive) | Assignee: | David Thompson (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Sprint: | 2017-8, 2017-9, 2017-10, 2017-11, 2017-12, 2017-13, 2017-14, 2017-15, 2017-16, 2017-17, 2017-18 | ||||||||
| Description |
|
I had originally planned to use WE_SVR_BATCH_INSERT but it is far too verbose for what I need and the performance wouldn't be great. I need to create a similar command which uses binary data and removes the verbose command messaging. |
| Comments |
| Comment by Andrew Hutchings (Inactive) [ 2017-06-13 ] |
|
Branch created for this but mcsapi additions are needed to test it. So putting on hold for now. |
| Comment by Andrew Hutchings (Inactive) [ 2017-06-16 ] |
|
Also added GET_UNCOMMITTED_LBIDS which is a wrapper around a VSS call which doesn't use network. We need this to commit the lbids and update the HWM. |
| Comment by Andrew Hutchings (Inactive) [ 2017-06-16 ] |
|
Two commands added in the tree as indicated. Not testable by QA yet but I've used these in mcsapi to write and commit thousands of rows of data. |
| Comment by Andrew Hutchings (Inactive) [ 2017-06-16 ] |
|
Moving back to me. We are using the version buffer when we really don't want to |
| Comment by Andrew Hutchings (Inactive) [ 2017-06-19 ] |
|
Performance isn't great yet, 4,000,000 rows of two int columns takes a minute in debug build. Profiling shows this mostly down to the liberal use of boost::any which has a performance hit with the copys we do in it. We should consider boost::spirit::hold_any for this function. In addition we rather liberally fsync when it isn't needed. |
| Comment by Andrew Hutchings (Inactive) [ 2017-06-27 ] |
|
Current version has been running pretty well with mcsapi. |
[MCOL-769]