Details
-
New Feature
-
Status: Open (View Workflow)
-
Critical
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
-
MXS-CAPACITY-Q4-2026
Description
Problem
For MariaDB Exa, the newest slice of a table can live in MariaDB while the full history lives in Exasol. The smart router has no way to know which tables work this way or what time range MariaDB still holds, so it cannot send a query to MariaDB when it is safe and to Exasol otherwise.
Goal
Enhance the smart router to route a query to MariaDB only when its time predicate provably falls inside the data MariaDB holds (maybe still racing both queries incase its analytic and still resolves faster in exa), and to Exasol in every other case because it holds all the data. It's expected to cost extra latency as we parse the query.
Design Proposal (not final)
- Eligible tables: the filter links to a binlogrouter CDC service (cdc_service) and treats the tables in its odbc_insert_only_tables as historical-store candidates.
- Time column: by default the first DATE/DATETIME/TIMESTAMP column of the table (by ordinal position). It must be indexed. A schema comment overrides the default, key set by column_comment_key.
- Boundary: the oldest value of the time column per table, read at startup and refreshed every boundary_refresh_interval.
- Routing rules (outermost query only):
- Route to MariaDB if the outermost FROM table is an eligible table and the outermost WHERE filters its time column with =, BETWEEN, > or < and the range lies fully inside the boundary.
- Route to Exasol if the range overlaps or goes beyond the boundary, if the predicate is missing, uses a function on the column, or is inside an OR, or if the table is not eligible.
- With joins or subqueries, only the outermost query's main FROM table and its WHERE clause are evaluated.
- An explicit routing hint always overrides the filter.
- If a table has no indexed time column or the comment is invalid, behaviour follows on_unresolved, with a warning in the log.
- a process that on some interval, deletes, prunes data from mariadb keep that slice small within the defined data retention policy
New fields (all proposed)
[Historical]
|
type=filter
|
module=historicalfilter # new filter module |
cdc_service=CDC-Service # binlogrouter service whose odbc_insert_only_tables define eligible tables
|
column_comment_key=maxscale.historical_column # schema comment key that overrides the default time column |
boundary_refresh_interval=60s # how often to re-read each table's oldest value
|
on_unresolved=exasol # table with no usable column: exasol | error
|
Acceptance criteria
- A query with a time range inside MariaDB's data goes to MariaDB; a range that overlaps or precedes it goes to Exasol.
- A query without a time predicate on the column goes to Exasol.
- The comment override changes which column is used, without restart after a refresh.
- A non-indexed or missing time column produces a warning and the configured fallback.
- A hint overrides the routing decision.
- For joins, only the outermost FROM table and WHERE clause determine routing.
- Table and boundary state can be inspected (maxctrl), showing the column used, the boundary, and the last refresh.
Open questions
- Late-arriving rows with old timestamps make the oldest value unreliable as a boundary. Add a configured watermark or partition ranges as alternatives?
- Writes to rows outside the window: reject, or leave unrouted?
- Depends on the insert-only limitations of
MXS-6772(no REPLACE or ON DUPLICATE KEY UPDATE, no silent key reuse); confirm acceptable for candidate tables.