Details
-
New Feature
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
Description
Short Description
This task proposes a staged Parallel Query prototype for MariaDB. The current
prototype focuses on selected single-table InnoDB analytical scans, especially
TPC-H Q6-style scalar aggregation and constrained TPC-H Q1-style grouped
aggregation.
The goal is to start technical review and agree on the first upstreamable
slice. This is not a request to merge a complete general-purpose Parallel
Query implementation in one step.
Problem / Motivation
MariaDB currently executes many analytical single-table scan and aggregate
queries serially. For large InnoDB tables, TPC-H Q1/Q6-style workloads spend
substantial time scanning rows, evaluating predicates, and aggregating values.
The prototype demonstrates an execution path where a leader thread starts
worker threads, workers scan disjoint ranges and build partial results, and
the leader merges the final scalar or grouped result.
This is intended to improve elapsed time for large scan-heavy read-only
queries while preserving safe fallback for unsupported query shapes.
Proposed Solution
The prototype adds an opt-in Parallel Query path with these properties:
- SQL-layer admission for a bounded set of single-table InnoDB read-only
query shapes. - Worker-side scan, predicate evaluation, and partial aggregation.
- Leader-side worker join, error/cancellation cleanup, and result merge.
- Statement snapshot handoff from leader to workers for MVCC consistency.
- TaurusDB-compatible PQ and NO_PQ hint surface where it maps cleanly to
MariaDB internals. - DOP and worker resource controls.
- EXPLAIN, optimizer trace, status counter, and slow-log observability.
- Serial fallback for unsupported or unsafe query shapes.
Supported demonstration examples:
SELECT SUM(l_extendedprice * l_discount) |
FROM lineitem |
WHERE l_shipdate >= DATE '1994-01-01' |
AND l_shipdate < DATE '1995-01-01' |
AND l_discount BETWEEN 0.05 AND 0.07 |
AND l_quantity < 24; |
SELECT l_returnflag, l_linestatus, |
SUM(l_quantity), SUM(l_extendedprice), |
SUM(l_extendedprice * (1 - l_discount)), |
SUM(l_extendedprice * (1 - l_discount) * (1 + l_tax)), |
AVG(l_quantity), AVG(l_extendedprice), AVG(l_discount), |
COUNT(*) |
FROM lineitem |
WHERE l_shipdate <= DATE '1998-09-02' |
GROUP BY l_returnflag, l_linestatus |
ORDER BY l_returnflag, l_linestatus; |
Current Non-Goals
The current prototype does not implement:
- general joins or hash joins;
- UNION, derived tables, recursive CTE, or correlated subqueries;
- write-select, CTAS, or DML parallel execution;
- full worker JOIN/TABLE/Item plan clone;
- TaurusDB message-queue transport;
- complete InnoDB B-tree/subtree split;
- broad secondary-index, ref, reverse, or ICP parity;
- MySQL/TaurusDB `EXPLAIN FORMAT=TREE` parity.
These should be discussed as follow-up slices after the first review boundary
is agreed.
Correctness Notes
- Unsupported query shapes fall back to existing serial execution.
- Workers open their own table instances and do not share the leader handler.
- Workers use the leader statement snapshot instead of creating independent
read views. - Focused tests cover Q1/Q6 MVCC behavior under REPEATABLE READ and
READ COMMITTED scenarios. - The prototype keeps the default behavior conservative and opt-in.
Local Performance Evidence
These numbers are local engineering evidence only, not official MariaDB
benchmark results.
Test context:
- Dataset: TPC-H SF10 standard model subset with `orders`, `partsupp`, and
`lineitem`. - Row counts:
- `orders`: 15,000,000
- `partsupp`: 8,000,000
- `lineitem`: 59,986,052
- Server binary: `build_release_pq/sql/mariadbd`
- Buffer pool: 12G
- Table charset/collation: default `utf8mb4_uca1400_ai_ci`
- `lineitem` primary key: `PRIMARY KEY(l_orderkey, l_linenumber)`
- Two standard LINEITEM foreign keys were added.
- No secondary index was added on `l_shipdate`; Q6 measured clustered-primary
full scan plus predicate filtering and aggregation.
Observed elapsed time:
| Query | Serial | PQ(2) | PQ(4) | Notes |
| Q1 full grouped aggregate | 36.44s / 33.92s | not rerun | 9.28s / 9.25s | `Parallel_query_count=1`; about 3.7x at DOP 4 |
| Q6 scalar aggregate | 4.93s / 5.43s warm repeat | 2.45s / 2.52s | 1.85s / 1.87s | `Parallel_query_count=1`; DOP 4 was best on this host |
Q6 does not scale linearly beyond DOP 4 on this host. Current evidence points
to a row scan/shared-resource bottleneck rather than hint parsing failure,
fallback, or split imbalance.
Validation Evidence
The contribution branch currently contains focused MTR coverage for:
- Q1/Q6 positive execution paths;
- serial/PQ result equality for supported aggregate shapes;
- worker lifecycle and cleanup;
- resource controls and queue timeout behavior;
- EXPLAIN, optimizer trace, status counter, and slow-log observability;
- MVCC and transaction boundaries;
- fallback behavior for unsupported grouped, subquery, write-select, routine,
locking, isolation, descriptor, and secondary-provider shapes.
No full MTR, ASAN, or official benchmark certification is claimed at this
stage.
Implementation Branch
GitHub branch:
https://github.com/ZhuQingping/mariadb_server/tree/support_parallel_query
Current commit series:
c5819ad2a9c MDEV-PQ: Add single-table parallel query prototype
|
cf5c1a5a701 MDEV-PQ: Add resource controls for parallel query workers
|
b986f00e71f MDEV-PQ: Test staged parallel query prototype
|
abc6e4b28f1 MDEV-PQ: Document staged parallel query contribution
|
Review documents in the branch:
Docs/parallel_query_tasks/README.md
|
Docs/parallel_query_tasks/parallel-query-high-level-design.md
|
Docs/parallel_query_tasks/parallel-query-low-level-design.md
|
Docs/parallel_query_tasks/parallel-query-performance.md
|
Docs/parallel_query_tasks/parallel-query-roadmap.md
|