Research and Estimation for Adapting VIDEX to MariaDB

VIDEX (Virtual Index for MySQL) is an open-source virtual index storage engine developed by ByteDance, designed to simulate the impact of indexes on query plans without requiring real data. It supports what-if analysis for index strategies, integrates AI-based cardinality and NDV (number of distinct values) estimation algorithms, and has been validated with benchmarks like TPC-H. VIDEX is currently implemented for MySQL 8.0 and deployed at scale, optimizing thousands of MySQL instances daily.

ByteDance has expressed interest in adapting VIDEX for MariaDB compatibility, seeking collaboration with the MariaDB community due to differences between MySQL 8.0 and the MariaDB codebase. This feature story requests research and estimation to identify the technical requirements, challenges, and effort needed to integrate VIDEX in MariaDB Server, enabling MariaDB users to benefit from advanced query optimization capabilities.

Objectives

Understand the architectural differences between MySQL 8.0 and MariaDB that impact VIDEX compatibility.

Identify modifications needed in the MariaDB Server codebase to support VIDEX as a storage engine plugin.

Estimate the effort required for development.

Acceptance Criteria

A Jira story report documenting:

Key differences between MySQL 8.0 and MariaDB affecting VIDEX integration.

Specific changes required in MariaDB Server to support VIDEX (e.g., plugin interfaces, statistical metadata handling).

Estimated effort for implementation.

Recommendations for collaboration with ByteDance to streamline adaptation.

Validation that the proposed changes align with MariaDB’s architecture and performance goals.

A plan for testing VIDEX integration using benchmarks like TPC-H and JOB to ensure simulation accuracy.

Background

VIDEX Overview: VIDEX simulates query plans by providing statistical information (e.g., NDV, cardinality) via a RESTful algorithm service, supporting AI-driven estimation models. It skips index query interfaces (e.g., index_first, index_read) and focuses on optimizer interactions (ha_engine::info, ha_engine::records_in_range).

VIDEX Architecture: VIDEX uses a three-layer disaggregated design (production database, VIDEX-Optimizer, VIDEX-Statistic-Server), allowing lightweight statistical replication and AI model integration (VIDEX paper, arXiv:2503.23776).