Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20590

Introduce a file format constraint to ALTER TABLE

Details

    Description

      If a table is altered using the new INSTANT algorithm, it can force the table to use a non-canonical format:

      Some operations cause a table's tablespace file to use a non-canonical storage format when the INSTANT algorithm is used. The affected operations include:

      • Adding a column.
      • Dropping a column.
      • Reordering columns.

      These operations require the following non-canonical changes to the storage format:

      • A hidden metadata record at the start of the clustered index is used to store each column's DEFAULT value. This makes it possible to add new columns that have default values without rebuilding the table.
      • A BLOB in the hidden metadata record is used to store column mappings. This makes it possible to drop or reorder columns without rebuilding the table. This also makes it possible to add columns to any position or drop columns from any position in the table without rebuilding the table.
      • If a column is dropped, old records will contain garbage in that column's former position, and new records will be written with NULL values, empty strings, or dummy values.

      https://mariadb.com/kb/en/library/innodb-online-ddl-operations-with-algorithminstant/#non-canonical-storage-format-caused-by-some-operations

      This is generally not a problem. However, there may be cases where users may want to avoid putting a table into this format. For example, users may want to avoid bugs similar to MDEV-19783 and MDEV-20066:

      There are some known bugs that could lead to issues when an InnoDB DDL operation is performed using the INSTANT algorithm. This algorithm will usually be chosen by default if the operation supports the algorithm.

      The effect of many of these bugs is that the table seems to forget that its tablespace file is in the non-canonical storage format.

      https://mariadb.com/kb/en/library/innodb-online-ddl-operations-with-algorithminstant/#known-bugs

      We may want to add a way for users to avoid putting tables into the non-canonical INSTANT format, if they would like.

      How would it be implemented?:

      • A new value for the ALGORITHM clause and the alter_algorithm system variable that would lead to data files using the non-canonical data format? i.e. CANON_DATAFILES, NOBLOBMAP, STANDARD_DATAFILES, etc.
      • A new InnoDB system variable that could enable/disable operations that would lead to data files using the non-canonical data format? i.e. innodb_noncanonical_alter=0

      Attachments

        Issue Links

          Activity

            I am considering to use append and mapped instead of ‘add’ or ‘drop’. Here is the 10.3 version. The commit message includes some reasoning for these names.

            marko Marko Mäkelä added a comment - I am considering to use append and mapped instead of ‘add’ or ‘drop’. Here is the 10.3 version . The commit message includes some reasoning for these names.
            marko Marko Mäkelä added a comment - - edited

            After some more discussions, I am suggesting the following names:

            --echo #do not allow MDEV-11369 instant ADD / MDEV-15562 instant DROP/reorder
            set global innodb_file_format=barracuda;
            --echo # ditto, and if instant ADD/DROP had been used in the past, ALTER TABLE would rebuild the table
            set global innodb_file_format=force_barracuda_compatible;
            --echo # do not allow MDEV-15562 instant DROP/redorder
            set global innodb_file_format=10_3; 
            --echo # ditto, and any subsequent ALTER TABLE would rebuild to the requested format if needed
            set global innodb_file_format=force_10_3_compatible;
            --echo # allow instant ADD/DROP/reorder, but not possible future format changes
            set global innodb_file_format=10_4; 
            --echo # same as 10_4, at least for now
            set global innodb_file_format=force_10_4_compatible;
            

            I would use _ instead of . so that quotes around the parameter values can be avoided.

            We might want to omit the force_…_compatible variants and always enforce the requested format in ALTER TABLE or OPTIMIZE TABLE. That could cause a negative surprise for users who might have a leftover innodb_file_format=barracuda in their configuration files and could have executed instant ADD/DROP in an earlier 10.3 or 10.4 version. However, they should have noticed the deprecation warning for that parameter.

            For future file format changes after MariaDB Server 10.5, we might introduce another parameter to control whether instant ADD/DROP is allowed, or we could introduce further values for innodb_file_format. In my vision, we could extend the 10.4 metadata record format with further things (persistent COUNT in MDEV-18188, or secondary index metadata for simpler IMPORT in MDEV-11658), and users could want to use that format while preventing instant ADD/DROP/reorder column.

            marko Marko Mäkelä added a comment - - edited After some more discussions, I am suggesting the following names: --echo #do not allow MDEV-11369 instant ADD / MDEV-15562 instant DROP/reorder set global innodb_file_format=barracuda; --echo # ditto, and if instant ADD/DROP had been used in the past, ALTER TABLE would rebuild the table set global innodb_file_format=force_barracuda_compatible; --echo # do not allow MDEV-15562 instant DROP/redorder set global innodb_file_format=10_3; --echo # ditto, and any subsequent ALTER TABLE would rebuild to the requested format if needed set global innodb_file_format=force_10_3_compatible; --echo # allow instant ADD/DROP/reorder, but not possible future format changes set global innodb_file_format=10_4; --echo # same as 10_4, at least for now set global innodb_file_format=force_10_4_compatible; I would use _ instead of . so that quotes around the parameter values can be avoided. We might want to omit the force_…_compatible variants and always enforce the requested format in ALTER TABLE or OPTIMIZE TABLE . That could cause a negative surprise for users who might have a leftover innodb_file_format=barracuda in their configuration files and could have executed instant ADD/DROP in an earlier 10.3 or 10.4 version. However, they should have noticed the deprecation warning for that parameter. For future file format changes after MariaDB Server 10.5, we might introduce another parameter to control whether instant ADD/DROP is allowed, or we could introduce further values for innodb_file_format . In my vision, we could extend the 10.4 metadata record format with further things (persistent COUNT in MDEV-18188 , or secondary index metadata for simpler IMPORT in MDEV-11658 ), and users could want to use that format while preventing instant ADD/DROP/reorder column.

            From the user point of view, the file format might be an uninteresting implementation detail. What the user wants is a guarantee that there will be no overhead for future DML operations. Instant ADD COLUMN or DROP COLUMN always incurs some of that. So, reviving the deprecated parameter innodb_file_format might be a bad idea, especially given the future plans to improve the .ibd file format in ways that have nothing to do with instant ALTER TABLE operations.

            I revise my suggestion to introduce a new parameter innodb_instant_alter_column_allowed:

            --echo #Do not allow MDEV-11369 instant ADD / MDEV-15562 instant DROP/reorder
            --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE.
            --echo #Coincidentally, IMPORT TABLESPACE should work to any MariaDB 10.x version.
            SET GLOBAL innodb_instant_alter_column_allowed=no;
            --echo #Do not allow MDEV-15562 instant DROP/reorder but do allow MDEV-11369
            --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE.
            SET GLOBAL innodb_instant_alter_column_allowed=add_last;
            --echo #Allow any instant ALTER TABLE.
            SET GLOBAL innodb_instant_alter_column_allowed=add_drop_reorder;
            

            The maximum and default values would be as follows:

            • 10.3: innodb_instant_alter_column_allowed=add_last (other allowed values: no)
            • 10.4 and 10.5: innodb_instant_alter_column_allowed=add_drop_reorder (other allowed values: no, add_last)
            marko Marko Mäkelä added a comment - From the user point of view, the file format might be an uninteresting implementation detail. What the user wants is a guarantee that there will be no overhead for future DML operations. Instant ADD COLUMN or DROP COLUMN always incurs some of that. So, reviving the deprecated parameter innodb_file_format might be a bad idea, especially given the future plans to improve the .ibd file format in ways that have nothing to do with instant ALTER TABLE operations. I revise my suggestion to introduce a new parameter innodb_instant_alter_column_allowed : --echo #Do not allow MDEV-11369 instant ADD / MDEV-15562 instant DROP/reorder --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE. --echo #Coincidentally, IMPORT TABLESPACE should work to any MariaDB 10.x version. SET GLOBAL innodb_instant_alter_column_allowed= no ; --echo #Do not allow MDEV-15562 instant DROP/reorder but do allow MDEV-11369 --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE. SET GLOBAL innodb_instant_alter_column_allowed=add_last; --echo #Allow any instant ALTER TABLE. SET GLOBAL innodb_instant_alter_column_allowed=add_drop_reorder; The maximum and default values would be as follows: 10.3: innodb_instant_alter_column_allowed=add_last (other allowed values: no ) 10.4 and 10.5: innodb_instant_alter_column_allowed=add_drop_reorder (other allowed values: no , add_last )

            Result of RQG testing with a derivate (additional SET GLOBAL innodb_file_format = <number>) of the grammar conf/mariadb/table_stress_innodb.yy
            on bb-10.4-MDEV-20590 commit 163dc5f919d3a8bf12288a8653d4c90e75809acc
            No new asserts or SEGV's
            

            mleich Matthias Leich added a comment - Result of RQG testing with a derivate (additional SET GLOBAL innodb_file_format = <number>) of the grammar conf/mariadb/table_stress_innodb.yy on bb-10.4-MDEV-20590 commit 163dc5f919d3a8bf12288a8653d4c90e75809acc No new asserts or SEGV's
            marko Marko Mäkelä added a comment - - edited

            I pushed this to 10.3 and merged up to 10.5. Starting with 10.3.24, we introduce the parameter with the default value add_last, and the non-default value never would introduce a constraint. Starting with MariaDB 10.4.14, 10.5.3, and later, we introduce the parameter with the default value will be add_drop_reorder.

            --echo #Do not allow MDEV-11369 instant ADD / MDEV-15562 instant DROP/reorder
            --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE.
            --echo #Coincidentally, IMPORT TABLESPACE should work to any MariaDB 10.x version.
            SET GLOBAL innodb_instant_alter_column_allowed=never;
            --echo #Do not allow MDEV-15562 instant DROP/reorder but do allow MDEV-11369
            --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE.
            SET GLOBAL innodb_instant_alter_column_allowed=add_last;
            --echo #Allow any instant ALTER TABLE.
            SET GLOBAL innodb_instant_alter_column_allowed=add_drop_reorder;
            

            marko Marko Mäkelä added a comment - - edited I pushed this to 10.3 and merged up to 10.5. Starting with 10.3.24, we introduce the parameter with the default value add_last , and the non-default value never would introduce a constraint. Starting with MariaDB 10.4.14, 10.5.3, and later, we introduce the parameter with the default value will be add_drop_reorder . --echo #Do not allow MDEV-11369 instant ADD / MDEV-15562 instant DROP/reorder --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE. --echo #Coincidentally, IMPORT TABLESPACE should work to any MariaDB 10.x version. SET GLOBAL innodb_instant_alter_column_allowed=never; --echo #Do not allow MDEV-15562 instant DROP/reorder but do allow MDEV-11369 --echo #If a table already was in conflicting format, force a rebuild on any subsequent ALTER TABLE. SET GLOBAL innodb_instant_alter_column_allowed=add_last; --echo #Allow any instant ALTER TABLE. SET GLOBAL innodb_instant_alter_column_allowed=add_drop_reorder;

            People

              marko Marko Mäkelä
              GeoffMontee Geoff Montee (Inactive)
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.