[MCOL-3242] Improve load/save in BRM structs Created: 2019-04-09 Updated: 2019-09-20 Resolved: 2019-09-20 |
|
| Status: | Closed |
| Project: | MariaDB ColumnStore |
| Component/s: | writeengine |
| Affects Version/s: | None |
| Fix Version/s: | 1.4.0 |
| Type: | Task | Priority: | Minor |
| Reporter: | Patrick LeBlanc (Inactive) | Assignee: | Daniel Lee (Inactive) |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Sprint: | 2019-06 |
| Description |
|
Hacking up the BRM structs to make them use IDBDataFile consistently, I noticed that the EM, VBBM, and VSS are still using my v 0.1 prototype code for loading and saving. That code, written like 12 years ago, loads and saves elements one by one. That won't scale, and it'll cause noticeable pauses across the system while it does that. This is another one of those things I had to fix at Tune (saving a 600MB extent map ~50 bytes at a time = pain). This task is for making them save consecutive groups of elements at a time. Much much quicker. One other thing I noticed is that it's taking a full snapshot of all of the structs on commit. That's expensive, esp with the current impl, and it doesn't need to do that. |
| Comments |
| Comment by Patrick LeBlanc (Inactive) [ 2019-06-14 ] |
|
Made the PR: https://github.com/mariadb-corporation/mariadb-columnstore-engine/pull/785 Assign to whoever deserves it most. |
| Comment by Patrick LeBlanc (Inactive) [ 2019-06-14 ] |
|
Just realized I didn't prevent it from writing a snapshot after every commit. Will resubmit in a bit. |
| Comment by Patrick LeBlanc (Inactive) [ 2019-06-14 ] |
|
OK that's in |
| Comment by Andrew Hutchings (Inactive) [ 2019-06-17 ] |
|
dleeyh with Patrick's patch we should see transactional BRM snapshots happening faster, I suspect when there are many columns and many tables in particular. I'm guessing you will see this most in DML. |
| Comment by Patrick LeBlanc (Inactive) [ 2019-07-17 ] |
|
It's been working for about a month it seems. Just merged the final cleanup. |
| Comment by Daniel Lee (Inactive) [ 2019-09-20 ] |
|
Build verified: 1.4.0-1 engine commit: Have been running autopilot on different builds. Verified by regression. |