Details

    Description

      The purpose of this task is to ensure that the following internal operations
      are atomic (either all or nothing) for all storage engines:

      Manage .frm
      Storage engine data dictionary
      Binary log

      This will solve the following things:

      • .frm context and storage engine dictionary is always in sync
      • no #sql-xxxx files if server crashes during alter table
      • binary log will always contain the DDL statement if the DDL successfully and committed to the storage engine. If the DDL
        was not successful, all traces of the DDL (like temporary files) will be deleted/rolled back.
        For example, if an alter table was successfully done and committed and we get a crash just before the write to the binary
        log, crash recovery will write the ALTER TABLE command to the binary log. In the case of a crash in the middle of a multi-table drop, crash recovery will write to the binary log those tables that was actually dropped.

      The cost of doing the above should not be more than 1-4 sync per DDL.

      High level architecture

      When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
      some new method):

      • Operation
      • Number of tables
      • Table name
      • Table id for original table
      • Table id for resulting table (in case of rename)
      • sql command (for binary log)

      If there is no table id (for example for CSV) we would use the
      timestamp of the files.

      With the above information we would be able to continue from the place
      where the operation failed.

      Low level architecture is done for each sub project

      Some requirements for a storage engine to be 'Atomic compliment':

      • drop table and rename_table needs to be either atomic or can be retried if there was a crash in middle of the
        operation.
      • Engines supporting inplace alter table, must also support the handlerton->check_version() call to allow the
        ddl recovery code to check if the inline alter table succeed. This is only needed if there was a crash between
        the inplace alter table commit and the rename of the .frm file.

      Supported engines in 10.6 are (among others)

      • InnoDB
      • MyRocks
      • Aria (transactional and non transactional tables)
      • MyISAM
      • Any engine that has only one table file (as then drop and rename will be atomic)

      Attachments

        Issue Links

          Activity

            monty Michael Widenius created issue -
            monty Michael Widenius made changes -
            Field Original Value New Value
            monty Michael Widenius made changes -
            marko Marko Mäkelä made changes -
            Description The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if it succeeded

            The cost of doing the above should not be more than 1-2 sync per DDL.

            High level architecture

            When doing a dll, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CVS) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture (to be done before implementation starts)

            Describe in detail the source changes needed for:

            CREATE TABLE
            CREATE TABLE ... SELECT
            DROP TABLE t1,t2,t3,t4;
            RENAME TABLE t1 as t2, t2 as t1;
            ALTER TABLE
            The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if it succeeded

            The cost of doing the above should not be more than 1-2 sync per DDL.

            High level architecture

            When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CSV) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture (to be done before implementation starts)

            Describe in detail the source changes needed for:

            CREATE TABLE
            CREATE TABLE ... SELECT
            DROP TABLE t1,t2,t3,t4;
            RENAME TABLE t1 as t2, t2 as t1;
            ALTER TABLE
            marko Marko Mäkelä made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Assignee Michael Widenius [ monty ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Epic Link PT-76 [ 68557 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Epic Link PT-76 [ 68557 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.4 [ 22408 ]
            marko Marko Mäkelä made changes -
            midenok Aleksey Midenkov made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 10.5 [ 23123 ]
            serg Sergei Golubchik made changes -
            marko Marko Mäkelä made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            midenok Aleksey Midenkov made changes -
            midenok Aleksey Midenkov made changes -
            marko Marko Mäkelä made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.5 [ 23123 ]
            marko Marko Mäkelä made changes -
            mpflaum Maria M Pflaum (Inactive) made changes -
            Assignee Michael Widenius [ monty ] Ralf Gebhardt [ ralf.gebhardt@mariadb.com ]
            mpflaum Maria M Pflaum (Inactive) made changes -
            Assignee Ralf Gebhardt [ ralf.gebhardt@mariadb.com ] Michael Widenius [ monty ]
            marko Marko Mäkelä made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            midenok Aleksey Midenkov made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            monty Michael Widenius made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            monty Michael Widenius made changes -
            monty Michael Widenius made changes -
            monty Michael Widenius made changes -
            monty Michael Widenius made changes -
            monty Michael Widenius made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            monty Michael Widenius made changes -
            monty Michael Widenius made changes -
            midenok Aleksey Midenkov made changes -
            monty Michael Widenius made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            monty Michael Widenius made changes -
            Description The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if it succeeded

            The cost of doing the above should not be more than 1-2 sync per DDL.

            High level architecture

            When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CSV) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture (to be done before implementation starts)

            Describe in detail the source changes needed for:

            CREATE TABLE
            CREATE TABLE ... SELECT
            DROP TABLE t1,t2,t3,t4;
            RENAME TABLE t1 as t2, t2 as t1;
            ALTER TABLE
            The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if it succeeded

            The cost of doing the above should not be more than 1-2 sync per DDL.

            High level architecture

            When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CSV) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture is done for each sub project

            Some requirements for a storage engine to be 'Atomic compliment':
            - drop table and rename_table needs to be either atomic or can be retried if there was a crash in middle of the
              operation.
            - Engines supporting inplace alter table, must also support the handlerton->check_version() call to allow the
              ddl recovery code to check if the inline alter table succeed. This is only needed if there was a crash between
              the inplace alter table commit and the rename of the .frm file.

            Supported engines in 10.6 are (among others)
            - InnoDB
            - Aria (transactional and non transactional tables)
            - MyISAM
            - Any engine that has only one table file (as then drop and rename will be atomic)
            monty Michael Widenius made changes -
            Description The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if it succeeded

            The cost of doing the above should not be more than 1-2 sync per DDL.

            High level architecture

            When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CSV) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture is done for each sub project

            Some requirements for a storage engine to be 'Atomic compliment':
            - drop table and rename_table needs to be either atomic or can be retried if there was a crash in middle of the
              operation.
            - Engines supporting inplace alter table, must also support the handlerton->check_version() call to allow the
              ddl recovery code to check if the inline alter table succeed. This is only needed if there was a crash between
              the inplace alter table commit and the rename of the .frm file.

            Supported engines in 10.6 are (among others)
            - InnoDB
            - Aria (transactional and non transactional tables)
            - MyISAM
            - Any engine that has only one table file (as then drop and rename will be atomic)
            The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if it succeeded

            The cost of doing the above should not be more than 1-2 sync per DDL.

            High level architecture

            When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CSV) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture is done for each sub project

            Some requirements for a storage engine to be 'Atomic compliment':
            - drop table and rename_table needs to be either atomic or can be retried if there was a crash in middle of the
              operation.
            - Engines supporting inplace alter table, must also support the handlerton->check_version() call to allow the
              ddl recovery code to check if the inline alter table succeed. This is only needed if there was a crash between
              the inplace alter table commit and the rename of the .frm file.

            Supported engines in 10.6 are (among others)
            - InnoDB
            - MyRocks
            - Aria (transactional and non transactional tables)
            - MyISAM
            - Any engine that has only one table file (as then drop and rename will be atomic)
            midenok Aleksey Midenkov made changes -
            monty Michael Widenius made changes -
            Description The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if it succeeded

            The cost of doing the above should not be more than 1-2 sync per DDL.

            High level architecture

            When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CSV) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture is done for each sub project

            Some requirements for a storage engine to be 'Atomic compliment':
            - drop table and rename_table needs to be either atomic or can be retried if there was a crash in middle of the
              operation.
            - Engines supporting inplace alter table, must also support the handlerton->check_version() call to allow the
              ddl recovery code to check if the inline alter table succeed. This is only needed if there was a crash between
              the inplace alter table commit and the rename of the .frm file.

            Supported engines in 10.6 are (among others)
            - InnoDB
            - MyRocks
            - Aria (transactional and non transactional tables)
            - MyISAM
            - Any engine that has only one table file (as then drop and rename will be atomic)
            The purpose of this task is to ensure that the following internal operations
            are atomic (either all or nothing) for all storage engines:

            Manage .frm
            Storage engine data dictionary
            Binary log

            This will solve the following things:
            - .frm context and storage engine dictionary is always in sync
            - no #sql-xxxx files if server crashes during alter table
            - binary log will always contain the DDL statement if the DDL successfully and committed to the storage engine. If the DDL
              was not successful, all traces of the DDL (like temporary files) will be deleted/rolled back.
              For example, if an alter table was successfully done and committed and we get a crash just before the write to the binary
              log, crash recovery will write the ALTER TABLE command to the binary log. In the case of a crash in the middle of a multi-table drop, crash recovery will write to the binary log those tables that was actually dropped.

            The cost of doing the above should not be more than 1-4 sync per DDL.

            High level architecture

            When doing a DDL, store somewhere (either trough write_ddl_log_entry() or
            some new method):

            - Operation
            - Number of tables
            - Table name
            - Table id for original table
            - Table id for resulting table (in case of rename)
            - sql command (for binary log)

            If there is no table id (for example for CSV) we would use the
            timestamp of the files.

            With the above information we would be able to continue from the place
            where the operation failed.

            Low level architecture is done for each sub project

            Some requirements for a storage engine to be 'Atomic compliment':
            - drop table and rename_table needs to be either atomic or can be retried if there was a crash in middle of the
              operation.
            - Engines supporting inplace alter table, must also support the handlerton->check_version() call to allow the
              ddl recovery code to check if the inline alter table succeed. This is only needed if there was a crash between
              the inplace alter table commit and the rename of the .frm file.

            Supported engines in 10.6 are (among others)
            - InnoDB
            - MyRocks
            - Aria (transactional and non transactional tables)
            - MyISAM
            - Any engine that has only one table file (as then drop and rename will be atomic)
            marko Marko Mäkelä made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            marko Marko Mäkelä made changes -
            elenst Elena Stepanova made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 10.6.1 [ 24437 ]
            Fix Version/s 10.6 [ 24028 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            Labels ServiceNow
            serg Sergei Golubchik made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            Labels ServiceNow 76qDvLB8Gju6Hs7nk3VY3EX42G795W5z
            midenok Aleksey Midenkov made changes -
            midenok Aleksey Midenkov made changes -
            serg Sergei Golubchik made changes -
            Labels 76qDvLB8Gju6Hs7nk3VY3EX42G795W5z
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            marko Marko Mäkelä made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 90302 ] MariaDB v4 [ 133731 ]
            marko Marko Mäkelä made changes -
            midenok Aleksey Midenkov made changes -
            midenok Aleksey Midenkov made changes -
            serg Sergei Golubchik made changes -
            serg Sergei Golubchik made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 201658 191541
            Zendesk active tickets 201658

            People

              monty Michael Widenius
              monty Michael Widenius
              Votes:
              8 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.