[MDEV-17554] Auto-create history partitions for system-versioned tables Created: 2018-10-26  Updated: 2023-08-07  Resolved: 2022-05-07

Status: Closed
Project: MariaDB Server
Component/s: Partitioning, Versioned Tables
Fix Version/s: 10.9.1

Type: Task Priority: Critical
Reporter: Ulrich Moser (Inactive) Assignee: Aleksey Midenkov
Resolution: Fixed Votes: 0
Labels: Preview_10.8

Issue Links:
Blocks
blocks MDEV-23639 Auto-create does not work under LOCK ... Closed
blocks MDEV-25495 Auto-drop old history partition for s... In Review
is blocked by MDEV-25347 DML events for auto-partitioned table... Closed
Duplicate
is duplicated by MDEV-23639 Auto-create does not work under LOCK ... Closed
PartOf
includes MDEV-20068 History partition rotation is not don... Closed
Problem/Incident
causes MDEV-25477 Auto-create breaks replication when t... Closed
causes MDEV-27456 Assertion `!thd->is_error()' fails in... Closed
causes MDEV-28567 Assertion `0' in open_tables upon fun... Closed
causes MDEV-29872 MSAN/Valgrind uninitialised value err... Stalled
causes MDEV-29873 MSAN uninitialized value errors in bc... Closed
Relates
relates to MDEV-19903 Setup default partitions for system v... Closed
relates to MDEV-21003 Per-partition INTERVAL for history pa... Open
relates to MDEV-21747 Refactor partition auto-creation via ... Closed
relates to MDEV-25475 Auto-create: LIMIT partitions are not... Closed
relates to MDEV-25478 Auto-create: Auto partitioning is uns... Closed
relates to MDEV-25479 Auto-create: 2nd and further executio... Closed
relates to MDEV-25482 Auto-create: Server hangs after a fai... Closed
relates to MDEV-25529 Auto-create: Pre-existing historical ... Stalled
relates to MDEV-25531 Auto-create: Bad error message upon f... Closed
relates to MDEV-25540 Auto-create: There is no clear way to... Closed
relates to MDEV-25545 Auto-create: ODKU does not cause part... Closed
relates to MDEV-25547 Auto-create: Undetected deadlock last... Closed
relates to MDEV-25559 Auto-create: infinite loop after inte... Closed
relates to MDEV-25567 Auto-create: Partition creation happe... Open
relates to MDEV-28207 Auto-create: Many partitions generate... Open
relates to MDEV-28288 System versioning doesn't support cor... Confirmed
relates to MDEV-28337 Update documentation for SYSTEM_TIME ... Stalled
relates to MDEV-28338 Auto-create: doesn't work for engine=... Open
relates to MDEV-28413 Incorrect statistics and data for his... Open
relates to MDEV-28414 Auto-create: If XA transaction is ro... Open
relates to MDEV-17553 Enable setting start datetime for int... Closed
relates to MDEV-19938 Reorganize multiple history partitions Open
relates to MDEV-20336 Assertion bitmap_is_set(read_partitio... Closed
relates to MDEV-22247 History partition overflow leads to w... Closed
relates to MDEV-23640 Versioned InnoDB table doesn't auto-c... Closed
relates to MDEV-23641 Default settings for partitioning and... Closed
relates to MDEV-23642 Locking timeout caused by auto-creati... Closed
relates to MDEV-25390 Pruning boundary for history partitio... Closed

 Description   

Syntax change

Keyword AUTO enables partition auto-creation.

create table t1 (x int) with system versioning
partition by system_time interval 1 hour auto;
 
create table t1 (x int) with system versioning
partition by system_time interval 1 month
starts '2021-01-01 00:00:00' auto partitions 12;
 
create table t1 (x int) with system versioning
partition by system_time limit 1000 auto;

Or with explicit partitions:

create table t1 (x int) with system versioning
partition by system_time interval 1 hour auto
(partition p0 history, partition pn current);

Description

Before executing history-generating DML command add N history
partitions, so that N would be sufficient for potentially generated
history. N > 1 may be required when history is rotated by INTERVAL and
timestamp was jumped to future further than interval value.

If last history partition exceeds LIMIT records before DML command
then new history partition is created and the history filling is
switched to the new partition. Thus LIMIT does not carry strict
limitation and the history partition size must be planned as LIMIT
records plus average number of history one DML generates.

Auto-creation is implemented by synchronous
fast_alter_partition_table() call from the thread of the executed DML
command before the command itself (by the fallback and retry mechanism
similar to Discovery feature, see Open_table_context).

The name for newly added partitions are generated like default
partition names with extension of MDEV-22155 (which avoids name
clashes by extending assignment counter to next free-enough gap).

These DML commands trigger auto-creation:

  • DELETE (including multi-delete, excluding DELETE HISTORY)
  • UPDATE (including multi-update)
  • REPLACE (including REPLACE .. SELECT)
  • INSERT .. ON DUPLICATE KEY UPDATE
  • LOAD DATA .. REPLACE

MDEV-23642 Locking timeout caused by auto-creation affects original DML

The reasons for this are:

  • Do not disrupt main business process (the history is auxiliary
    service);
  • Consequences are non-fatal (history is not lost, but comes into wrong
    partition; fixed by partitioning rebuild);
  • There is more freedom for application to fail in this case or
    not: it may read warning info and find corresponding error number.
  • While non-failing command is easy to handle by an application and
    fail it, the opposite is hard to handle: there is no automatic
    actions to fix failed command and retry, DBA intervention is
    required and until then application is non-functioning.


 Comments   
Comment by Aleksey Midenkov [ 2019-07-17 ]

This first implementation is done in standalone thread. The drawback of this is that ALTER may be theoretically executed on unwanted table:

1. Table t1 is locked with LOCK TABLES;
2. ALTER TABLE t1 ADD PARTITION is initiated in parallel and blocked on lock;
3. t1 is unlocked, dropped and new t1 is created.
4. ALTER continues and adds partition to "unwanted" table.

Current implementation does not have proper solution for this. But the ALTER will succeed only if new t1 is partitioned by SYSTEM_TIME and there is no partition by the name it tries to add. Though this behavior is generally unwanted, it has minor impact of one extra history partition in new table. In most cases ALTER will fail after old t1 is dropped, so this is low probability.

Comment by Aleksey Midenkov [ 2020-02-17 ]

This task is continued by MDEV-21747.

Comment by Aleksey Midenkov [ 2021-12-04 ]

Please review bb-10.7-midenok-MDEV-17554

Comment by Sergei Golubchik [ 2021-12-11 ]

commit e9f1bd41a29 looks ok!
please, push into preview-10.8-MDEV-17554-auto-create-partition (or any other few-word description that you prefer) and set the status to "In Testing".

Comment by Sergei Golubchik [ 2022-03-18 ]

the branch name is now bb-10.8-MDEV-17554-auto-create-partition

Comment by Elena Stepanova [ 2022-03-19 ]

Please rebase it on the current 10.9 or at least 10.8 before moving to testing.

Comment by Aleksey Midenkov [ 2022-03-22 ]

Rebased. Please test bb-10.9-MDEV-17554-auto-create-partition

Comment by Lena Startseva [ 2022-04-28 ]

Testing done.

Generated at Thu Feb 08 08:37:21 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.