Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4566

rebuildEM utility must support compressed segment files

    XMLWordPrintable

    Details

    • Sprint:
      2021-4, 2021-5, 2021-6, 2021-8, 2021-9, 2021-10, 2021-11, 2021-12, 2021-13

      Description

      There are certain situations when it is impossible to restore metadata information that either was completely or partially lost. Metadata in this case includes Extent Map and auxilary database OID counter used to allocate OIDs for database object to be created.
      This issue describes the part that manages with Extent Map partial or comple loss. There is a tool under tool/rebuildEM called rebuildEM which algo creates an Extent Map using the existing columnar data files. Its algorithm is very simple it counts a number of blocks and creates corresponding number of extents in the new in-memory Extent Map. In the end it writes an image of the Extent Map to disk.
      The main problem with the tool is that it doesn't support compressed(MCS as of 6.1.1 supports Snappy compression) files. The suggested approach is to change the tool's algorithm so that it uses compressed files to produce Extent Map. A compressed file contains a structure CompressedDBFileHeader as the header. The fBlockCount attribute points to a number of blocks in the file.
      The important assumption for the tool is that all dbroots must be available at the node where rebuildEM is run so the cluster must either have a shared or S3 storage.
      To be noted that we are going to add LZ4 as a compression method.
      One must remove extent map file to test how rebuildEM works with compressed files.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              drrtuy Roman
              Reporter:
              drrtuy Roman
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.