Details
-
Task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
Description
This is a followup to MDEV-28073.
Problem description
Join optimizer can be very slow when the join has many tables and join prefix pruning feature doesn't manage to prune join prefixes.
An important special case is when a subset of join tables are joined with an equi-join condition on a primary key, and so can be joined using eq_ref in any order. Then, query plans that use different permutations of these tables have the same cost. The optimizer will consider all possible permutations. This can take a lot of CPU but provide little benefit.
The fix for MDEV-28073 addresses these issues to some extent and was pushed into 10.6. This MDEV is about more comprehensive fix.
Solution
TODO description. Copy from:
https://jira.mariadb.org/browse/MDEV-28073?focusedCommentId=225701&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-225701
Join prefix pruning is now controlled by @@optimizer_prune_level variable. The new pruning is done when @@optimizer_prune_level=2, which is the new default. One can disable certain aspects (TODO elaborate) of the new behavior by setting @@optimizer_prune_level=1 (the default before 10.10).
The tree
It is preview-10.10-MDEV-28852-join-pruning
Attachments
Issue Links
- causes
-
MDEV-28928 Plan selection takes 2+ times longer than before MDEV-28852
-
- Closed
-
-
MDEV-29072 Slower query with MDEV-28852 comparing to 10.6 or 10.10 (5 sec vs 0.03 sec)
-
- Closed
-
-
MDEV-29073 Slower query #2 with MDEV-28852 comparing to 10.6 or 10.10 (820 sec vs 0.9 sec)
-
- Closed
-
- relates to
-
MDEV-28882 Assertion `tmp >= 0' failed in best_access_path
-
- Stalled
-
-
MDEV-28073 Slow query performance in MariaDB when using many tables
-
- Closed
-
-
MDEV-28929 Plan selection takes forever with MDEV-28852 comparing to reasonable time without
-
- Closed
-
-
MDEV-29094 Many-table join optimization: re-run Join Order Benchmark
-
- Closed
-
ralf.gebhardt, I've checked.
There were multiple commits pushed that were marked with
MDEV-28073.The fix for
MDEV-28073, pushed into 10.6+:commit b729896d00e022f6205399376c0cc107e1ee0704
Author: Monty <monty@mariadb.org>
Date: Tue May 10 11:47:20 2022 +0300
MDEV-28073 Query performance degradation in newer MariaDB versions when using many tables
The issue was that best_extension_by_limited_search() had to go through
too many plans with the same cost as there where many EQ_REF tables.
Fixed by shortcutting EQ_REF (AND REF) when the result only contains one
row. This got the optimization time down from hours to sub seconds.
Then, there were a series of commits pushed into 10.10.1+:
commit b3c74bdc1f09b2b8babd7f2bd9d52df2749ddcc3
Author: Monty <monty@mariadb.org>
Date: Tue May 31 17:36:32 2022 +0300
Improve pruning in greedy_search by sorting tables during search
MDEV-28073 Slow query performance in MariaDB when using many tables
...
... a few more related commits follow, until this one:
commit 515b9ad05a6de9dac3871ef2769dde1b5834c6e3
Author: Monty <monty@mariadb.org>
Date: Thu Jun 2 19:47:23 2022 +0300
Added EQ_REF chaining to the greedy_optimizer
MDEV-28073 Slow query performance in MariaDB when using many table
The idea is to prefer and chain EQ_REF tables (tables that uses an
unique key to find a row) when searching for the best table combination.
This significantly reduces row combinations that has to be examined.
This is optimization is enabled when setting optimizer_prune_level=2
(which is now default).
Implementation:
- optimizer_prune_level has a new level, 2, which enables EQ_REF
optimization in addition to the pruning done by level 1.
...
Then, there is this commit (also in 10.10.1+) which is a fix for patch for this MDEV and is the only one mentioning
MDEV-28929:commit 8c2faad576d6a77314e92755a389de2c41e21242
Author: Sergei Petrunia <sergey@mariadb.com>
Date: Tue Jul 19 14:13:17 2022 +0300
MDEV-28929: Plan selection takes forever with MDEV-28852 ...
Part #2: Extend heuristic pruning to use multiple tables as the
"Model tables".
The commits that fix this MDEV are b3c74bdc1f09b2b8babd7f2bd9d52df2749ddcc3..515b9ad05a6de9dac3871ef2769dde1b5834c6e3 , endpoints inclusive.