Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-6677

Log progress of disk-based aggregation (spill, generations, finalize merge) with the session ID

    XMLWordPrintable

Details

    • Task
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 23.02.24
    • None
    • PrimProc
    • None

    Description

      Summary: when a query's aggregation spills to disk, PrimProc logs nothing between the query's start and end. On a production 23.02 cluster, a daily aggregation query ran with no end, and PrimProc's journal was silent from its start until the node was restarted. The logs can't tell a disk-aggregation merge that is still progressing (possibly slowly, on a shared/NFS spill directory) from one that is stuck.

      Proposal: log a few lines from disk-based aggregation (RowAggStorage), each with the session ID:

      • spill start: spill directory, memory limit hit;
      • each new generation: generation number, bytes and rows spilled so far;
      • finalize start: generation count, total bytes on disk;
      • finalize progress at most every N seconds (e.g. 60): generations merged / total, rows merged, bytes read back;
      • finalize end: elapsed time.

      Logging only: no behavior change. Throttled so a long merge produces at most one line per interval per aggregation thread group.

      Status: not started.

      Attachments

        Activity

          People

            Unassigned Unassigned
            kyle.hutchinson Kyle Hutchinson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.