Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-3270

Improve cpimport ingest speed into Dictionary columns

    XMLWordPrintable

Details

    • New Feature
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 1.2.3
    • 1.2.4
    • cpimport
    • None
    • 2019-04

    Description

      Given 800 000 000 records with a couple Dictionary columns with lots of equal length strings in the data set. It took 4 167 seconds to ingest the data set into CS.
      After the patch it takes only 467 seconds.

      There were two main sources of latency:

      • Dctnry::getTokenFromArray represented de-dup buffer as array and called memcpy for any equal-sized string
      • COND_WAIT_SECONDS was 3 seconds per default

      Attachments

        Issue Links

          Activity

            People

              dleeyh Daniel Lee (Inactive)
              drrtuy Roman
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.