Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6572

Research possible effect of the use of ZDDs on internal data size and processing time

    XMLWordPrintable

Details

    • New Feature
    • Status: In Progress (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      https://en.wikipedia.org/wiki/Zero-suppressed_decision_diagram

      Zero-suppressed decisions diagrams represent sets of sets of elements of some universe. Their notable feature is that they compress representation as much as possible. For example, representing Cartesian product of two sets A and B requires size(A) + size(B) space instead of size(A)*size(B). Our join algorithm does the latter and it would be nice to have space savings of ZDDs with materialization as late as possible.

      There is a join operation over ZDDs: https://www.jstage.jst.go.jp/article/transinf/advpub/0/advpub_2025EDP7062/_pdf

      Finally, if we allow for several ZDDs roots with same compressed representation, we can associate i-th root with the 2^i multiplier. This way we can represent multisets and they will be compressed as well - subsets for 2^i and 2^j will share data between them as much as possible.

      The task here is the following:

      1. Create TPC-DS data and scale 1, 5 and 10 and look whether data inside tables can be compressed with ZDDs, report results for different scales,
      2. Implement joins from TPC-DS queries (without GROUP BY - queries 16 and 96 are two such examples) and report on memory requirements and runtimes for different scales.

      The library to use: https://github.com/kunisura/tdzdd (maybe some other - there are libraries for parallel ZDD operations).

      Attachments

        Activity

          People

            sergey.zefirov Sergey Zefirov
            sergey.zefirov Sergey Zefirov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - 7d
                7d
                Remaining:
                Time Spent - 5h Remaining Estimate - 6d 3h
                6d 3h
                Logged:
                Time Spent - 5h Remaining Estimate - 6d 3h
                5h

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.