[MDEV-18363] Use external table to store history instead of partitions Created: 2019-01-23  Updated: 2023-04-13

Status: Stalled
Project: MariaDB Server
Component/s: Partitioning, Versioned Tables
Fix Version/s: None

Type: Task Priority: Minor
Reporter: Sam Tagney Assignee: Aleksey Midenkov
Resolution: Unresolved Votes: 4
Labels: Compatibility

Issue Links:
Relates
relates to MDEV-19191 FK support for CURRENT partition of v... Stalled

 Description   

We have been using IBM DB2 system versioning since 2015. It is a great feature.

We are trying to switch to MariaDB and have run into a roadblock with system versioning.

DB2 stores historic data in a separate table.
Table ADDRESS has associated history table ADDRESS_HIST

This makes it very easy to backup and restore history, it doesn't effect main table query performance, and allows for dropping versioning on the main table without wiping out history; in order to do ALTERing of columns, indexes, constraints, etc.

Utilizing the main table or partitioning presents a whole host of problems. We have almost all of our tables versioned, so if we partition all of them, we lose referential integrity on all of them due to disallowed foreign keys on partitions. Without partitions, and utilizing the default behavior of just storing the history in the main table, we are unable to insert our existing history into the main table, so we would be starting from scratch. Removing versioning on the main table wipes out the history; making it difficult to do ALTERing maintenance. We currently don't use replication, but will in the future. I can imagine replication is effected by the current versioning design.

Was system versioning designed this way due to the MariaDB architecture? I've read about it to some degree and it seems very advanced. Is there any hope for MariaDB storing history in separate tables? If we can assist in making this happen sooner than later, please contact me, as we have to make a decision to switch very soon.



 Comments   
Comment by Aleksey Midenkov [ 2019-02-18 ]

Partitioning in fact is MariaDB way of using multiple tables. It'd be not very good to develop multiple mechanisms that solves the same thing. What other problems apart from prohibited FK partitioning rises? I see future development in allowing FK for working partition of versioned tables. Maybe it will be more easy with shorter syntax for CREATE TABLE that will automatically create working partition and history partition.

Comment by Sam Tagney [ 2019-02-21 ]

The foreign key problem is only part of the problem. We cannot lose our history. If we need to remove versioning on a table, under MariaDB, the history is wiped out. That's unacceptable. In DB2, the separate history table does not have foreign key relationships or referential integrity; nor is it needed. Just the fact that it exists and retains the history when versioning in disabled, is what is important. Also, when versioning is enabled, it resumes where it left off, utilizing all the history in the separate history table.

As for replication. I can't even begin to imagine the amount of difficulty involved in replicating a partitioned table, much less a table partitioned exclusively for versioning.

We would never use versioning as it is currently defaulted to in MariaDB, where the history is mixed in with the table data. It's a performance hit and unmanageable.

Comment by Aleksey Midenkov [ 2019-02-21 ]

So you basically saying about 3 problems:

1. FK don't work with partitions;
2. Unable to turn off system versioning and retain history;
3. Doubtful replication usefulness on partitioned table.

If all 3 points are arranged does that satisfy your needs? There will be 2 partitions: CURRENT and HISTORY. Foreign keys will work only against CURRENT. Two partitions are actually two InnoDB tables. Expect not much more trouble with replication as it is now with any kind of tables.

> Also, when versioning is enabled, it resumes where it left off, utilizing all the history in the separate history table.

What will happen if the schema is altered on working table while system versioning is off?

Comment by Sam Tagney [ 2019-02-21 ]

In DB2, when versioning is disabled, the separate history table remains as just a stand alone table. The linkage to it's parent is lost. When versioning is enabled, any changes to the parent's schema must be applied to the history table, otherwise it errors.

My colleagues asked me to point out that, in MariaDB, when the parent table is dropped or truncated, we lose the history too. This is not the case for DB2.

They also said they had difficulties adding a timestamp column to a partitioned table

Comment by Aleksey Midenkov [ 2019-02-22 ]

Sam, thank you for outlining drawbacks! I must admit that separate history table has its own advantages over partitioning. We can consider this as a complementary feature after partitioning tasks are finished.

Comment by Sam Tagney [ 2019-02-22 ]

Your very welcome Aleksey. We can do testing or try to help get answers to your questions. If there's anything else you think we can help with, please let me know

Comment by Federico Razzoli [ 2019-08-07 ]

May we have more details on how this will be implemented (if details are already decided)?

For example, will the 2 tables need to use exactly the same structure (indexes, etc) and the same storage engine? What will happen in case of an ALTER TABLE? Will it be possible to exclude the history table from replication? Will the same triggers work for both the tables?

Comment by Sergei Golubchik [ 2019-08-08 ]

No, details are not decided yet.

I thought that if we'll be looking into this, we'll start from checking how it works in DB2.

So, thanks for the questions, now we will not forget to consider them!

Generated at Thu Feb 08 08:43:32 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.