Details
-
Task
-
Status: In Progress (View Workflow)
-
Critical
-
Resolution: Unresolved
-
None
-
Q2/2025 Development
Description
The goal of this task is to create a prototype to port the clone feature from MySQL 8.0. The At the end of this prototype we should have ...
1. Able to clone data locally using SQL syntax with Innodb and ARIA SE.
MariaDB> INSTALL PLUGIN clone SONAME 'mariadb_clone.so';
|
MariaDB> CLONE LOCAL DATA DIRECTORY [=] 'clone_dir';
|
MariaDB> UNINSTALL PLUGIN clone;
|
2. Divide the entire work of MDEV-14992 into multiple individual MDEVs with estimation and sequencing.
Sub work Items
Local Clone:
[Done] 1. Port Clone Plugin and create SE (handler) Interfaces: Disabled dynamic PFS table related code.
[Done] 2. Convert component services required by Clone to Plugin Services. Disabled SSL for now required by remote Cloning.
[Done] 3. Port Server changes: Syntax, Clone Statement, Clone handler to interface with plugin.
[Done] 4. Port Innodb SE changes for snapshot.
5. Synchronize for Binary log and GTID, XA transaction
Design: Block commits (MDL_BACKUP_WAIT_COMMIT) to synchronize clone across all SEs and note the replication co-ordinates.
6. Clone other non-SE data like FRM files
Design: The idea is to follow the similar design to Maria Backup for synchronization and block DDLs before copying frm files.
7. Clone other SEs: ARIA and other non-transactional SEs
8. At the end of this MDEV-14992 will be divided the entire work of into multiple individual MDEVs with estimation and sequencing.
Attachments
Issue Links
- blocks
-
MDEV-14992 BACKUP: in-server backup
-
- Open
-
-
-
- Open
-
-
-
- Open
-
- duplicates
-
-
- Closed
-
-
-
- Closed
-
- split from
-
-
- Open
-
Activity
[Done] 3. Port Server changes: Syntax, Clone Statement, Clone handler to interface with plugin.
INSTALL PLUGIN clone SONAME 'mariadb_clone.so';
|
UNINSTALL PLUGIN clone;
|
Clone syntax works the flow reaches clone plugin and returns from SE interfaces [NO OP] right now.
CLONE LOCAL DATA DIRECTORY [=] 'clone_dir';
|
CLONE INSTANCE FROM 'user'@'host':port IDENTIFIED BY 'password'
|
[DATA DIRECTORY [=] 'clone_dir'] [REQUIRE [NO] SSL];
|
Started working on Innodb changes for clone.
[In Progress] 4. Port Innodb SE changes for snapshot.
I am adding another task to this prototype work after the initial part is complete. Adding one more week for that.
8. Divide the entire work of MDEV-14992 into multiple individual MDEVs with estimation and sequencing.
The scope of the work is increased to include to divide the backup work into multiple MDEVs with estimates. +1 week added to "Remaining Estimate".
In server backup can improve the backup design by allowing easier synchronization with other server activities going on. A BACKUP command to stream data to client has one drawback that it has to use a single socket to stream the data. A powerful alternative is CLONE feature that I had implemented for MySQL8.0.
https://dev.mysql.com/doc/refman/8.4/en/clone-plugin.html
Clone architecture uses a plugin that orchestrate the operation across multiple SEs and handles the network transfer for remote cloning. Clone uses server to server communication with the "Recipient" server establishing parallel connections to the "Donor" server for the data transfer. The dynamic snapshot creation for a specific engine is abstracted within the engine and currently implemented only for Innodb in MySQL.
Clone can be conveniently used for replica provisioning in replication setup. It would be easy to integrate with Galera later as it already uses clone as the SST method for MySQL.
https://dev.mysql.com/blog-archive/clone-create-mysql-instance-replica/
https://galeracluster.com/library/documentation/sst-physical.html
Clone is also known to provide the best performance from internal and external testing mainly due to the advantage of in server synchronization and 3 phase snapshot strategy with in-memory page tracking.
https://www.mydbops.com/blog/mysql-clone-plugin-speed-test
Clone page tracking feature can be extended to provide persistent tracking logs and improve incremental backup.
https://dev.mysql.com/blog-archive/innodb-clone-and-page-tracking/
A practical approach to implement clone like solution for Mariadb is to first port the clone feature from MySQL. This would require some effort to integrate with Mariadb We can then extend it to provide cloning for all supported storage engines.
We can then incrementally add features like incremental clone and partial clone. At this point clone solution in Mariadb would be much more advanced than what is there in MySQL.