Details

    • Task
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • None
    • 1.4.0
    • writeengine
    • None
    • 2019-06

    Description

      Hacking up the BRM structs to make them use IDBDataFile consistently, I noticed that the EM, VBBM, and VSS are still using my v 0.1 prototype code for loading and saving. That code, written like 12 years ago, loads and saves elements one by one. That won't scale, and it'll cause noticeable pauses across the system while it does that. This is another one of those things I had to fix at Tune (saving a 600MB extent map ~50 bytes at a time = pain).

      This task is for making them save consecutive groups of elements at a time. Much much quicker.

      One other thing I noticed is that it's taking a full snapshot of all of the structs on commit. That's expensive, esp with the current impl, and it doesn't need to do that.

      Attachments

        Activity

          pleblanc Patrick LeBlanc (Inactive) added a comment - Made the PR: https://github.com/mariadb-corporation/mariadb-columnstore-engine/pull/785 Assign to whoever deserves it most.

          Just realized I didn't prevent it from writing a snapshot after every commit. Will resubmit in a bit.

          pleblanc Patrick LeBlanc (Inactive) added a comment - Just realized I didn't prevent it from writing a snapshot after every commit. Will resubmit in a bit.

          OK that's in

          pleblanc Patrick LeBlanc (Inactive) added a comment - OK that's in

          dleeyh with Patrick's patch we should see transactional BRM snapshots happening faster, I suspect when there are many columns and many tables in particular. I'm guessing you will see this most in DML.

          LinuxJedi Andrew Hutchings (Inactive) added a comment - dleeyh with Patrick's patch we should see transactional BRM snapshots happening faster, I suspect when there are many columns and many tables in particular. I'm guessing you will see this most in DML.

          It's been working for about a month it seems. Just merged the final cleanup.

          pleblanc Patrick LeBlanc (Inactive) added a comment - It's been working for about a month it seems. Just merged the final cleanup.

          Build verified: 1.4.0-1

          engine commit:
          975463c

          Have been running autopilot on different builds. Verified by regression.

          dleeyh Daniel Lee (Inactive) added a comment - Build verified: 1.4.0-1 engine commit: 975463c Have been running autopilot on different builds. Verified by regression.

          People

            dleeyh Daniel Lee (Inactive)
            pleblanc Patrick LeBlanc (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.