Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40215

Add staged Parallel Query prototype for TPC-H Q1/Q6-style analytical scans

    XMLWordPrintable

Details

    Description

      Short Description

      This task proposes a staged Parallel Query prototype for MariaDB. The current
      prototype focuses on selected single-table InnoDB analytical scans, especially
      TPC-H Q6-style scalar aggregation and constrained TPC-H Q1-style grouped
      aggregation.

      The goal is to start technical review and agree on the first upstreamable
      slice. This is not a request to merge a complete general-purpose Parallel
      Query implementation in one step.

      Problem / Motivation

      MariaDB currently executes many analytical single-table scan and aggregate
      queries serially. For large InnoDB tables, TPC-H Q1/Q6-style workloads spend
      substantial time scanning rows, evaluating predicates, and aggregating values.

      The prototype demonstrates an execution path where a leader thread starts
      worker threads, workers scan disjoint ranges and build partial results, and
      the leader merges the final scalar or grouped result.

      This is intended to improve elapsed time for large scan-heavy read-only
      queries while preserving safe fallback for unsupported query shapes.

      Proposed Solution

      The prototype adds an opt-in Parallel Query path with these properties:

      • SQL-layer admission for a bounded set of single-table InnoDB read-only
        query shapes.
      • Worker-side scan, predicate evaluation, and partial aggregation.
      • Leader-side worker join, error/cancellation cleanup, and result merge.
      • Statement snapshot handoff from leader to workers for MVCC consistency.
      • TaurusDB-compatible PQ and NO_PQ hint surface where it maps cleanly to
        MariaDB internals.
      • DOP and worker resource controls.
      • EXPLAIN, optimizer trace, status counter, and slow-log observability.
      • Serial fallback for unsupported or unsafe query shapes.

      Supported demonstration examples:

      SELECT SUM(l_extendedprice * l_discount)
      FROM lineitem
      WHERE l_shipdate >= DATE '1994-01-01'
        AND l_shipdate < DATE '1995-01-01'
        AND l_discount BETWEEN 0.05 AND 0.07
        AND l_quantity < 24;
      

      SELECT l_returnflag, l_linestatus,
             SUM(l_quantity), SUM(l_extendedprice),
             SUM(l_extendedprice * (1 - l_discount)),
             SUM(l_extendedprice * (1 - l_discount) * (1 + l_tax)),
             AVG(l_quantity), AVG(l_extendedprice), AVG(l_discount),
             COUNT(*)
      FROM lineitem
      WHERE l_shipdate <= DATE '1998-09-02'
      GROUP BY l_returnflag, l_linestatus
      ORDER BY l_returnflag, l_linestatus;
      

      Current Non-Goals

      The current prototype does not implement:

      • general joins or hash joins;
      • UNION, derived tables, recursive CTE, or correlated subqueries;
      • write-select, CTAS, or DML parallel execution;
      • full worker JOIN/TABLE/Item plan clone;
      • TaurusDB message-queue transport;
      • complete InnoDB B-tree/subtree split;
      • broad secondary-index, ref, reverse, or ICP parity;
      • MySQL/TaurusDB `EXPLAIN FORMAT=TREE` parity.

      These should be discussed as follow-up slices after the first review boundary
      is agreed.

      Correctness Notes

      • Unsupported query shapes fall back to existing serial execution.
      • Workers open their own table instances and do not share the leader handler.
      • Workers use the leader statement snapshot instead of creating independent
        read views.
      • Focused tests cover Q1/Q6 MVCC behavior under REPEATABLE READ and
        READ COMMITTED scenarios.
      • The prototype keeps the default behavior conservative and opt-in.

      Local Performance Evidence

      These numbers are local engineering evidence only, not official MariaDB
      benchmark results.

      Test context:

      • Dataset: TPC-H SF10 standard model subset with `orders`, `partsupp`, and
        `lineitem`.
      • Row counts:
      • `orders`: 15,000,000
      • `partsupp`: 8,000,000
      • `lineitem`: 59,986,052
      • Server binary: `build_release_pq/sql/mariadbd`
      • Buffer pool: 12G
      • Table charset/collation: default `utf8mb4_uca1400_ai_ci`
      • `lineitem` primary key: `PRIMARY KEY(l_orderkey, l_linenumber)`
      • Two standard LINEITEM foreign keys were added.
      • No secondary index was added on `l_shipdate`; Q6 measured clustered-primary
        full scan plus predicate filtering and aggregation.

      Observed elapsed time:

      Query Serial PQ(2) PQ(4) Notes
      Q1 full grouped aggregate 36.44s / 33.92s not rerun 9.28s / 9.25s `Parallel_query_count=1`; about 3.7x at DOP 4
      Q6 scalar aggregate 4.93s / 5.43s warm repeat 2.45s / 2.52s 1.85s / 1.87s `Parallel_query_count=1`; DOP 4 was best on this host

      Q6 does not scale linearly beyond DOP 4 on this host. Current evidence points
      to a row scan/shared-resource bottleneck rather than hint parsing failure,
      fallback, or split imbalance.

      Validation Evidence

      The contribution branch currently contains focused MTR coverage for:

      • Q1/Q6 positive execution paths;
      • serial/PQ result equality for supported aggregate shapes;
      • worker lifecycle and cleanup;
      • resource controls and queue timeout behavior;
      • EXPLAIN, optimizer trace, status counter, and slow-log observability;
      • MVCC and transaction boundaries;
      • fallback behavior for unsupported grouped, subquery, write-select, routine,
        locking, isolation, descriptor, and secondary-provider shapes.

      No full MTR, ASAN, or official benchmark certification is claimed at this
      stage.

      Implementation Branch

      GitHub branch:

      https://github.com/ZhuQingping/mariadb_server/tree/support_parallel_query

      Current commit series:

      c5819ad2a9c MDEV-PQ: Add single-table parallel query prototype
      cf5c1a5a701 MDEV-PQ: Add resource controls for parallel query workers
      b986f00e71f MDEV-PQ: Test staged parallel query prototype
      abc6e4b28f1 MDEV-PQ: Document staged parallel query contribution
      

      Review documents in the branch:

      Docs/parallel_query_tasks/README.md
      Docs/parallel_query_tasks/parallel-query-high-level-design.md
      Docs/parallel_query_tasks/parallel-query-low-level-design.md
      Docs/parallel_query_tasks/parallel-query-performance.md
      Docs/parallel_query_tasks/parallel-query-roadmap.md
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            qingping Qingping Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.