[MCOL-3242] Improve load/save in BRM structs Created: 2019-04-09  Updated: 2019-09-20  Resolved: 2019-09-20

Status: Closed
Project: MariaDB ColumnStore
Component/s: writeengine
Affects Version/s: None
Fix Version/s: 1.4.0

Type: Task Priority: Minor
Reporter: Patrick LeBlanc (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 1
Labels: None

Sprint: 2019-06

 Description   

Hacking up the BRM structs to make them use IDBDataFile consistently, I noticed that the EM, VBBM, and VSS are still using my v 0.1 prototype code for loading and saving. That code, written like 12 years ago, loads and saves elements one by one. That won't scale, and it'll cause noticeable pauses across the system while it does that. This is another one of those things I had to fix at Tune (saving a 600MB extent map ~50 bytes at a time = pain).

This task is for making them save consecutive groups of elements at a time. Much much quicker.

One other thing I noticed is that it's taking a full snapshot of all of the structs on commit. That's expensive, esp with the current impl, and it doesn't need to do that.



 Comments   
Comment by Patrick LeBlanc (Inactive) [ 2019-06-14 ]

Made the PR: https://github.com/mariadb-corporation/mariadb-columnstore-engine/pull/785

Assign to whoever deserves it most.

Comment by Patrick LeBlanc (Inactive) [ 2019-06-14 ]

Just realized I didn't prevent it from writing a snapshot after every commit. Will resubmit in a bit.

Comment by Patrick LeBlanc (Inactive) [ 2019-06-14 ]

OK that's in

Comment by Andrew Hutchings (Inactive) [ 2019-06-17 ]

dleeyh with Patrick's patch we should see transactional BRM snapshots happening faster, I suspect when there are many columns and many tables in particular. I'm guessing you will see this most in DML.

Comment by Patrick LeBlanc (Inactive) [ 2019-07-17 ]

It's been working for about a month it seems. Just merged the final cleanup.

Comment by Daniel Lee (Inactive) [ 2019-09-20 ]

Build verified: 1.4.0-1

engine commit:
975463c

Have been running autopilot on different builds. Verified by regression.

Generated at Thu Feb 08 02:41:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.