Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4218 Remove NFS/EFS/ Filestore prereq for S3 storage
  3. MCOL-5721

Aggregate S3 metadata to reduce number of keys in FoundationDB

Details

    • Sub-Task
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • 2021-3, 2021-4, 2021-5, 2021-6, 2021-7, 2021-8, 2021-9, 2021-10, 2021-11, 2021-12, 2021-16, 2021-17, 2022-22, 2022-23, 2023-4, 2023-5, 2023-6, 2023-7, 2023-8, 2023-10, 2023-11, 2023-12, 2024-2

    Attachments

      Activity

        drrtuy Roman added a comment -

        The granularity for JSON doc that contains a list of S3 object files is MCS data file. This subtask describes the optimization to reduce the granularity of the S3 meta to partition. The suggested value size for FoundationDB is 10KB and this is enough to store meta for a partition. This would reduce the number of keys in FDB down to 8-10X from the original number of metadata files.

        drrtuy Roman added a comment - The granularity for JSON doc that contains a list of S3 object files is MCS data file. This subtask describes the optimization to reduce the granularity of the S3 meta to partition. The suggested value size for FoundationDB is 10KB and this is enough to store meta for a partition. This would reduce the number of keys in FDB down to 8-10X from the original number of metadata files.

        People

          Unassigned Unassigned
          julien.fritsch Julien Fritsch
          Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

            Created:
            Updated:

            Git Integration

              Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.