Make it possible for a storage engine to take any query fragment to be executed internally in the engine. In particular, it should be possible to push joins, GROUP BY, ORDER BY, WHERE, HAVING.
Assorted thoughts:
this is step 1, only do simple cases, see below.
only push down joins when all tables in a query are in the same storage engine
interface to the server: replace all that with a pseudo-table? with a handler?
PS/SP reexecution — restore or keep? probably, restore. easier for the engine. and we rerun optimizer anyway
optimizer: no changes, in step 1 there is no cost estimation for pushed down joins.
EXPLAIN:
| 1 | t1 | ... | pushed down
| 2 | t2 | ... | pushed down
| 3 | <PUSHDOWN RESULT> | ... | for 1 and 2
UNION — don't, every UNION part is handled separately
derived tables, views: decide to materialize or merge, as usual. if merged — push together with the upper join, if materialized — push separately
subqueries: if materialized — push separately, if semijoin — push with the upper join, otherwise — don't.
partitioning: not affected
use FederatedX as a PoC
Attachments
Issue Links
is blocked by
MDEV-17096Pushdown of simple derived tables to storage engines
Closed
is part of
MDEV-22366Changes in optimizer to allow better operation of Smart Engine
Open
relates to
MDEV-16502Join pushdown to something like an IN filter for different storage engine
Here's a suggested scheme of the implementation of this feature.
Let Q be an n-way join query of tables such that some of tables belong to the engines that can execute join operations SELECT ... FROM t[0],...,t[n] WHERE P(t[0],...,t[n]).
Consider a possible join order t[i[0]],...,t[i[n]] and let T[0],...T[k] be partitioning of this join order such that any
partition contains only tables from the same engine and no neighbors has tables from the same engine.
Let's also assume that for any two neighbors only one belongs to the engine that can execute join by its own means while the other executes joins by MariaDB SQL processor. Let's assume that T[1] belongs to the engine that can execute join by its own means. Then any odd partition also belongs to such an engine while joins of any even partition are executed by SQL processor.
Let partition T[j] consists of tables t[j][0],...t[j][lj]
It's obvious that query Q is equivalent to the following query
SELECT ... FROM
(SELECT (...
(SELECT ...FROM
(SELECT ... FROM t[0][0],...,t[0][l0]) AS dt[1],
t[1][0],...,t[1][l1]) AS dt[2] ...) AS dtk, t[k][0],...,t[k][lk]
WHERE P(t[0],...,t[n]).
In this query first we define a derived table dt[1] that joins tables from T[0]. Then we define the derived table dt[2] that joins dt[1] and tables from T[1]. The derived table dt[j+1] joins the derived table dt[j] with tables from T[j].
Igor Babaev (Inactive)
added a comment - - edited Here's a suggested scheme of the implementation of this feature.
Let Q be an n-way join query of tables such that some of tables belong to the engines that can execute join operations SELECT ... FROM t [0] ,...,t [n] WHERE P(t [0] ,...,t [n] ).
Consider a possible join order t[i [0] ],...,t[i [n] ] and let T [0] ,...T [k] be partitioning of this join order such that any
partition contains only tables from the same engine and no neighbors has tables from the same engine.
Let's also assume that for any two neighbors only one belongs to the engine that can execute join by its own means while the other executes joins by MariaDB SQL processor. Let's assume that T [1] belongs to the engine that can execute join by its own means. Then any odd partition also belongs to such an engine while joins of any even partition are executed by SQL processor.
Let partition T [j] consists of tables t [j] [0] ,...t [j] [lj]
It's obvious that query Q is equivalent to the following query
SELECT ... FROM
(SELECT (...
(SELECT ...FROM
(SELECT ... FROM t[0][0],...,t[0][l0]) AS dt[1],
t[1][0],...,t[1][l1]) AS dt[2] ...) AS dtk, t[k][0],...,t[k][lk]
WHERE P(t[0],...,t[n]).
In this query first we define a derived table dt [1] that joins tables from T [0] . Then we define the derived table dt [2] that joins dt [1] and tables from T [1] . The derived table dt [j+1] joins the derived table dt [j] with tables from T [j] .
Another scheme just encapsulate tables from odd partitions into derived tables and pushes conditions
into these derived tables
SELECT ... FROM
...
(SELECT ... FROM t[k][0],...,t[k][lk] WHERE P(t[k][0],...,t[k][lk])) as dtk
...
WHERE P(t[0],...,t[n])
Here P(t[k][0],...,t[k][lk])) is the condition over t[k][0],...,t[k][lk] that can be extracted from P(t[0],...,t[n]).
With this scheme we rather push into the engine SELECTs that specify derived tables.
This well fits into the scheme how SELECT queries are handled by ColumnStore.
Igor Babaev (Inactive)
added a comment - Another scheme just encapsulate tables from odd partitions into derived tables and pushes conditions
into these derived tables
SELECT ... FROM
...
(SELECT ... FROM t[k][0],...,t[k][lk] WHERE P(t[k][0],...,t[k][lk])) as dtk
...
WHERE P(t[0],...,t[n])
Here P(t [k] [0] ,...,t [k] [lk] )) is the condition over t [k] [0] ,...,t [k] [lk] that can be extracted from P(t [0] ,...,t [n] ).
With this scheme we rather push into the engine SELECTs that specify derived tables.
This well fits into the scheme how SELECT queries are handled by ColumnStore.
Let's call a derived table External Derived Table (EDT) if it is specified by a SELECT over tables belonging to the same external database engine capable to execute SQL operations.
If an EDT is used in a query we always consider it as a materialized derived table whose materialization is performed by an external engine.
Igor Babaev (Inactive)
added a comment - - edited Let's call a derived table External Derived Table (EDT) if it is specified by a SELECT over tables belonging to the same external database engine capable to execute SQL operations.
If an EDT is used in a query we always consider it as a materialized derived table whose materialization is performed by an external engine.
Some observations
1. Rows of a EDT for ColumnStore engine could be produced in the same way as the result set for a select query.
2. To process EDT the server needs a special function to fill the EDT that would use the iterator function over the rows of EDT
3. The optimizer has to know the cardinality of EDT. This cardinality either calculated by the external engine or is calculated by the optimizer itself employing the statistical data on the tables and columns referenced in the specification of EDT.
Igor Babaev (Inactive)
added a comment - Some observations
1. Rows of a EDT for ColumnStore engine could be produced in the same way as the result set for a select query.
2. To process EDT the server needs a special function to fill the EDT that would use the iterator function over the rows of EDT
3. The optimizer has to know the cardinality of EDT. This cardinality either calculated by the external engine or is calculated by the optimizer itself employing the statistical data on the tables and columns referenced in the specification of EDT.
I spinned off MDEV-17096 "Pushdown of derived tables to storage engines".
This is the first step to resolve of the problem of pushing joins into storage engines.
This is probably the only task that realistically can be done within 10.4.
Igor Babaev (Inactive)
added a comment - - edited I spinned off MDEV-17096 "Pushdown of derived tables to storage engines".
This is the first step to resolve of the problem of pushing joins into storage engines.
This is probably the only task that realistically can be done within 10.4.
People
Unassigned
Sergei Golubchik
Votes:
3Vote for this issue
Watchers:
7Start watching this issue
Dates
Created:
Updated:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":1434.1000001430511,"ttfb":525.9000000953674,"pageVisibility":"visible","entityId":54012,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"925b8a26-79da-49ac-854b-61e742858d09","navigationType":0,"readyForUser":1523,"redirectCount":0,"resourceLoadedEnd":1613.1000001430511,"resourceLoadedStart":552.9000000953674,"resourceTiming":[{"duration":303.2999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bsh/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":552.9000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":552.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":856.2000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":303.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bsh/820016/12ta74/eb142f92e4bd16bd1ef8b08c1b9d5d56/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":553.2000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":553.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":856.6000001430511,"responseStart":0,"secureConnectionStart":0},{"duration":313.69999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/c54b129276d75dc2a3460e1d78f37913-CDN/lu2bsh/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":553.3000001907349,"connectEnd":553.3000001907349,"connectStart":553.3000001907349,"domainLookupEnd":553.3000001907349,"domainLookupStart":553.3000001907349,"fetchStart":553.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":553.3000001907349,"responseEnd":867,"responseStart":867,"secureConnectionStart":553.3000001907349},{"duration":390.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/f867843cd2fdb209d4d1d4f760f86346-CDN/lu2bsh/820016/12ta74/eb142f92e4bd16bd1ef8b08c1b9d5d56/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":553.4000000953674,"connectEnd":553.4000000953674,"connectStart":553.4000000953674,"domainLookupEnd":553.4000000953674,"domainLookupStart":553.4000000953674,"fetchStart":553.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":553.4000000953674,"responseEnd":943.5,"responseStart":943.5,"secureConnectionStart":553.4000000953674},{"duration":434.10000014305115,"initiatorType":"script","name":"https://jira.mariadb.org/s/ffdb17665775c5b4d6f097f3974ee359-CDN/lu2bsh/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":553.5,"connectEnd":553.5,"connectStart":553.5,"domainLookupEnd":553.5,"domainLookupStart":553.5,"fetchStart":553.5,"redirectEnd":0,"redirectStart":0,"requestStart":553.5,"responseEnd":987.6000001430511,"responseStart":987.6000001430511,"secureConnectionStart":553.5},{"duration":521.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":553.6000001430511,"connectEnd":553.6000001430511,"connectStart":553.6000001430511,"domainLookupEnd":553.6000001430511,"domainLookupStart":553.6000001430511,"fetchStart":553.6000001430511,"redirectEnd":0,"redirectStart":0,"requestStart":553.6000001430511,"responseEnd":1075.2000000476837,"responseStart":1075.2000000476837,"secureConnectionStart":553.6000001430511},{"duration":522.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":553.6000001430511,"connectEnd":553.6000001430511,"connectStart":553.6000001430511,"domainLookupEnd":553.6000001430511,"domainLookupStart":553.6000001430511,"fetchStart":553.6000001430511,"redirectEnd":0,"redirectStart":0,"requestStart":553.6000001430511,"responseEnd":1075.8000001907349,"responseStart":1075.8000001907349,"secureConnectionStart":553.6000001430511},{"duration":529.6000001430511,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bsh/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":553.7000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":553.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1083.3000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":522.5,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":553.8000001907349,"connectEnd":553.8000001907349,"connectStart":553.8000001907349,"domainLookupEnd":553.8000001907349,"domainLookupStart":553.8000001907349,"fetchStart":553.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":553.8000001907349,"responseEnd":1076.3000001907349,"responseStart":1076.3000001907349,"secureConnectionStart":553.8000001907349},{"duration":528.8999998569489,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bsh/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":554.6000001430511,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":554.6000001430511,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1083.5,"responseStart":0,"secureConnectionStart":0},{"duration":522.1000001430511,"initiatorType":"script","name":"https://jira.mariadb.org/s/81b5d7c27af3ebc078cc4a36383678ba-CDN/lu2bsh/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":554.7000000476837,"connectEnd":554.7000000476837,"connectStart":554.7000000476837,"domainLookupEnd":554.7000000476837,"domainLookupStart":554.7000000476837,"fetchStart":554.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":554.7000000476837,"responseEnd":1076.8000001907349,"responseStart":1076.8000001907349,"secureConnectionStart":554.7000000476837},{"duration":1045.8000001907349,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":555.5,"connectEnd":555.5,"connectStart":555.5,"domainLookupEnd":555.5,"domainLookupStart":555.5,"fetchStart":555.5,"redirectEnd":0,"redirectStart":0,"requestStart":555.5,"responseEnd":1601.3000001907349,"responseStart":1601.3000001907349,"secureConnectionStart":555.5},{"duration":1046.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":555.6000001430511,"connectEnd":555.6000001430511,"connectStart":555.6000001430511,"domainLookupEnd":555.6000001430511,"domainLookupStart":555.6000001430511,"fetchStart":555.6000001430511,"redirectEnd":0,"redirectStart":0,"requestStart":555.6000001430511,"responseEnd":1601.7000000476837,"responseStart":1601.7000000476837,"secureConnectionStart":555.6000001430511},{"duration":112,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1078.7000000476837,"connectEnd":1078.7000000476837,"connectStart":1078.7000000476837,"domainLookupEnd":1078.7000000476837,"domainLookupStart":1078.7000000476837,"fetchStart":1078.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":1078.7000000476837,"responseEnd":1190.7000000476837,"responseStart":1190.7000000476837,"secureConnectionStart":1078.7000000476837},{"duration":297.60000014305115,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2bsh/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":1315.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1315.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1613.1000001430511,"responseStart":0,"secureConnectionStart":0},{"duration":410.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bsh/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":1316.7000000476837,"connectEnd":1316.7000000476837,"connectStart":1316.7000000476837,"domainLookupEnd":1316.7000000476837,"domainLookupStart":1316.7000000476837,"fetchStart":1316.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":1316.7000000476837,"responseEnd":1726.9000000953674,"responseStart":1726.9000000953674,"secureConnectionStart":1316.7000000476837},{"duration":435.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/4a55b33fb6014f6055f885e193bf4347-CDN/lu2bsh/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":1317,"connectEnd":1317,"connectStart":1317,"domainLookupEnd":1317,"domainLookupStart":1317,"fetchStart":1317,"redirectEnd":0,"redirectStart":0,"requestStart":1317,"responseEnd":1752.2000000476837,"responseStart":1752.2000000476837,"secureConnectionStart":1317},{"duration":331.2999999523163,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1424.9000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1424.9000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1756.2000000476837,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":185,"responseStart":526,"responseEnd":551,"domLoading":533,"domInteractive":1693,"domContentLoadedEventStart":1693,"domContentLoadedEventEnd":1750,"domComplete":2260,"loadEventStart":2260,"loadEventEnd":2262,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1657},{"name":"bigPipe.sidebar-id.end","time":1657.8000001907349},{"name":"bigPipe.activity-panel-pipe-id.start","time":1658.1000001430511},{"name":"bigPipe.activity-panel-pipe-id.end","time":1661.4000000953674},{"name":"activityTabFullyLoaded","time":1775.3000001907349}],"measures":[],"correlationId":"a73e048a0fa10","effectiveType":"4g","downlink":9.8,"rtt":0,"serverDuration":290,"dbReadsTimeInMs":12,"dbConnsTimeInMs":19,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Here's a suggested scheme of the implementation of this feature.
Let Q be an n-way join query of tables such that some of tables belong to the engines that can execute join operations SELECT ... FROM t[0],...,t[n] WHERE P(t[0],...,t[n]).
Consider a possible join order t[i[0]],...,t[i[n]] and let T[0],...T[k] be partitioning of this join order such that any
partition contains only tables from the same engine and no neighbors has tables from the same engine.
Let's also assume that for any two neighbors only one belongs to the engine that can execute join by its own means while the other executes joins by MariaDB SQL processor. Let's assume that T[1] belongs to the engine that can execute join by its own means. Then any odd partition also belongs to such an engine while joins of any even partition are executed by SQL processor.
Let partition T[j] consists of tables t[j][0],...t[j][lj]
It's obvious that query Q is equivalent to the following query
SELECT ... FROM
(SELECT (...
(SELECT ...FROM
(SELECT ... FROM t[0][0],...,t[0][l0]) AS dt[1],
t[1][0],...,t[1][l1]) AS dt[2] ...) AS dtk, t[k][0],...,t[k][lk]
WHERE P(t[0],...,t[n]).
In this query first we define a derived table dt[1] that joins tables from T[0]. Then we define the derived table dt[2] that joins dt[1] and tables from T[1]. The derived table dt[j+1] joins the derived table dt[j] with tables from T[j].