[MCOL-5403] Design a consistent snapshot backup procedure Created: 2023-01-27  Updated: 2023-12-15

Status: Open
Project: MariaDB ColumnStore
Component/s: ?
Affects Version/s: 22.08.7
Fix Version/s: Icebox

Type: New Feature Priority: Major
Reporter: Roman Assignee: Max Mether
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
PartOf
includes MCOL-5402 save_brm must save oidbitmap as a par... Open
includes MCOL-5404 Flush S3 object files from SM cache i... Open

 Description   

There are number of elements that must be saved to take a snapshot of S3-based MCS installation.
1) MDB tables meta in a form of .frm files that lives in MDB data dir(/var/lib/mariadb/).
2) Consistent meta data snapshot that consists of: extent_map, EM journal, vbbm, vss, oidbitmap files.
3) Actual data that is mostly in S3 storage but parts might be found in StorageManager journal:
/var/lib/mariadb/columnstore/storagemanager/cache.
4) S3 meta data files that are the names of the actual S3 object files that constitutes data or MCS metadata files. These files resides in /var/lib/mariadb/columnstore/storagemanager/meta.

The suggested procedure is to
1) Disable writes on the cluster. set cluster to readonly is one way of doing this but its effect should be tested in a middle of TXN.
2) Flush S3 object files from SM cache into S3.
3) Call save_brm to take MCS meta + SM meta snapshot.
4) Re-enable writes

The suggested procedure must be tested to measure its delays. Especially the 2d step of the procedure.


Generated at Thu Feb 08 02:57:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.