Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17567

Atomic DDL

    XMLWordPrintable

    Details

      Description

      The purpose of this task is to ensure that the following internal operations
      are atomic (either all or nothing) for all storage engines:

      Manage .frm
      Storage engine data dictionary
      Binary log

      This will solve the following things:

      • .frm context and storage engine dictionary is always in sync
      • no #sql-xxxx files if server crashes during alter table
      • binary log will always contain the DDL statement if the DDL successfully and committed to the storage engine. If the DDL
        was not successful, all traces of the DDL (like temporary files) will be deleted/rolled back.
        For example, if an alter table was successfully done and committed and we get a crash just before the write to the binary
        log, crash recovery will write the ALTER TABLE command to the binary log. In the case of a crash in the middle of a multi-table drop, crash recovery will write to the binary log those tables that was actually dropped.

      The cost of doing the above should not be more than 1-4 sync per DDL.

      High level architecture

      When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
      some new method):

      • Operation
      • Number of tables
      • Table name
      • Table id for original table
      • Table id for resulting table (in case of rename)
      • sql command (for binary log)

      If there is no table id (for example for CSV) we would use the
      timestamp of the files.

      With the above information we would be able to continue from the place
      where the operation failed.

      Low level architecture is done for each sub project

      Some requirements for a storage engine to be 'Atomic compliment':

      • drop table and rename_table needs to be either atomic or can be retried if there was a crash in middle of the
        operation.
      • Engines supporting inplace alter table, must also support the handlerton->check_version() call to allow the
        ddl recovery code to check if the inline alter table succeed. This is only needed if there was a crash between
        the inplace alter table commit and the rename of the .frm file.

      Supported engines in 10.6 are (among others)

      • InnoDB
      • MyRocks
      • Aria (transactional and non transactional tables)
      • MyISAM
      • Any engine that has only one table file (as then drop and rename will be atomic)

        Attachments

          Issue Links

          blocks

          Task - A task that needs to be done. MDEV-11633 Make the InnoDB system tablespace optional

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-11655 Transactional data dictionary

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-15140 Implement Partial / Filtered Indexes

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-16417 Store Foreign Key metadata outside of InnoDB

          • Critical - Crashes, loss of data, severe memory leak.
          • In Review

          Technical task - A technical task. MDEV-21652 FK migration from old version

          • Critical - Crashes, loss of data, severe memory leak.
          • Stalled

          Task - A task that needs to be done. MDEV-23570 deprecate keep_files_on_create

          • Major - Major loss of function.
          • Open
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25683 Atomic DDL: With innodb_force_recovery=3 InnoDB: Trying to load index but the index tree has been freed

          • Major - Major loss of function.
          • Confirmed
          includes

          Task - A task that needs to be done. MDEV-23842 Atomic RENAME TABLE

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-23844 Atomic DROP TABLE (single table)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-24395 Atomic DROP TRIGGER

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-24408 Crash-safe DROP DATABASE

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-24576 Atomic CREATE TABLE

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-24607 ATOMIC CREATE VIEW

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-24746 Atomic CREATE TRIGGER

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-25180 Atomic ALTER TABLE

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24589 DROP TABLE is not crash-safe

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          is part of

          Task - A task that needs to be done. MDEV-4259 transactional DDL

          • Major - Major loss of function.
          • Open
          relates to

          Task - A task that needs to be done. MDEV-18518 Implement atomic multi-table (or multi-partition) CREATE TABLE for InnoDB

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22928 SUMMARY: AddressSanitizer: SEGV storage/innobase/include/dict0mem.h:1178 in dict_index_t::is_spatial() const

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23741 InnoDB: Table ... contains <n> indexes inside InnoDB, which is different from the number of indexes <n> defined in the MariaDB

          • Major - Major loss of function.
          • Closed

          Technical task - A technical task. MDEV-24755 Rebase FK tasks on top of Atomic DDL

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25588 Atomic DDL: Binlog query event written upon recovery is corrupt

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25606 Concurrent CREATE TRIGGER statements mix up in binlog and break replication

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25655 Atomic DDL: InnoDB: Cannot replay rename of tablespace, error Data structure corruption

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25656 Atomic DDL: Assertion `!full_crc32(flags)' failed in fil_space_t::logical_size(ulint)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25657 Atomic DDL: InnoDB: Cannot apply log, double free or corruption (fasttop)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25662 Atomic DDL: InnoDB: Expected tablespace id ... but found 0 upon crash recovery

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25683 Atomic DDL: With innodb_force_recovery=3 InnoDB: Trying to load index but the index tree has been freed

          • Major - Major loss of function.
          • Confirmed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25705 Atomic DDL: atomic.rename_case and atomic.drop_db_long_names fail on Windows

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-25920 Atomic DROP DATABASE

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-6642 Server crashed with assertion failure in file ha_innodb.cc line 8619

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. MDEV-11675 Lag Free Alter On Slave

          • Critical - Crashes, loss of data, severe memory leak.
          • Confirmed

          Task - A task that needs to be done. MDEV-16290 ALTER TABLE ... RENAME COLUMN syntax

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Technical task - A technical task. MDEV-21053 Crash safety of foreign key DDL

          • Critical - Crashes, loss of data, severe memory leak.
          • In Review

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22056 Rocks db corrupts data when disk is out of space

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25604 Atomic DDL: Binlog event written upon recovery does not have default database

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25646 Atomic DDL: InnoDB: Datafile './test/<tablename>.ibd' is corrupted. Cannot determine the space ID from the first 64 pages

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25666 Atomic DDL: InnoDB: Operating system error number 2, Could not find a valid tablespace file

          • Major - Major loss of function.
          • Closed

            Activity

              People

              Assignee:
              monty Michael Widenius
              Reporter:
              monty Michael Widenius
              Votes:
              8 Vote for this issue
              Watchers:
              20 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: