Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14046

Allow ALGORITHM=INPLACE for 10.1 tables that contain virtual columns

Details

    • 10.2.11

    Description

      With the dataset of MDEV-14023, ALTER TABLE will fail in InnoDB with a misleading error message:

      ERROR 1901 (HY000): Function or expression 'AUTO_INCREMENT' cannot be used in the GENERATED ALWAYS AS clause of `AttendDet_ID`
      

      All of the following statements will lead to the above response:

      ALTER TABLE AttendDet ADD INDEX(counter), LOCK=NONE;
      ALTER TABLE AttendDet DROP INDEX IX_AttendDet_Shift, LOCK=NONE;
      ALTER TABLE AttendDet DROP INDEX IX_AttendDet_Shift, ALGORITHM=INPLACE;
      

      We must fix this (insert the missing records to SYS_VIRTUAL), so that early users of virtual columns in 10.1 will be able to quickly create indexes on the virtual columns in 10.2 (MDEV-5800), and to instantly add columns in 10.3 (MDEV-11369), among other things.

      Attachments

        Issue Links

          Activity

            Actually, the same error message will be issued even for ALGORITHM=COPY. It makes sense: the table definition includes a virtual column:

            CREATE TABLE AttendDet (
              AttendDet_ID INT AUTO_INCREMENT PRIMARY KEY,
              Counter INT GENERATED ALWAYS AS (AttendDet_ID) VIRTUAL,
            ) ENGINE=InnoDB;
            

            This would work in 10.1, where virtual columns cannot be indexed, but in 10.2 after MDEV-5800, it makes sense to block this.

            So, the only problem with the mentioned dataset is that the following is refused:

            alter table AttendDet drop column Counter, algorithm=inplace;
            

            marko Marko Mäkelä added a comment - Actually, the same error message will be issued even for ALGORITHM=COPY . It makes sense: the table definition includes a virtual column: CREATE TABLE AttendDet ( AttendDet_ID INT AUTO_INCREMENT PRIMARY KEY , Counter INT GENERATED ALWAYS AS (AttendDet_ID) VIRTUAL, ) ENGINE=InnoDB; This would work in 10.1, where virtual columns cannot be indexed, but in 10.2 after MDEV-5800 , it makes sense to block this. So, the only problem with the mentioned dataset is that the following is refused: alter table AttendDet drop column Counter, algorithm=inplace;

            I am testing the following kind of a fix:

            1. Remove the crude check from ha_innobase::check_if_supported_inplace_alter().
            2. Do not invoke prepare_inplace_drop_virtual() if the table was created before 10.2.2.
            3. In prepare_inplace_add_virtual(), if the table was created before 10.2.2, always add all virtual columns inside InnoDB.

            The last part is needed so that ADD COLUMN…VIRTUAL will actually add the metadata to the InnoDB internal dictionary table SYS_VIRTUAL if there was no intermediate ALTER TABLE that would have dropped all virtual columns from the old table.

            marko Marko Mäkelä added a comment - I am testing the following kind of a fix: Remove the crude check from ha_innobase::check_if_supported_inplace_alter() . Do not invoke prepare_inplace_drop_virtual() if the table was created before 10.2.2. In prepare_inplace_add_virtual() , if the table was created before 10.2.2, always add all virtual columns inside InnoDB. The last part is needed so that ADD COLUMN…VIRTUAL will actually add the metadata to the InnoDB internal dictionary table SYS_VIRTUAL if there was no intermediate ALTER TABLE that would have dropped all virtual columns from the old table.

            Failure reported in MDEV-17199 makes most of relevant upgrade tests fail.

            elenst Elena Stepanova added a comment - Failure reported in MDEV-17199 makes most of relevant upgrade tests fail.

            Note: The following scenario should be covered as well:

            create table t1(a int, b int as (a) virtual, c int as(a+a) virtual) engine=innodb;
            --echo # upgrade from 10.0 or 10.1 to 10.2 or later
            alter table t1 drop b,algorithm=inplace;
            alter table t1 add d int as (-a) virtual,algorithm=inplace;
            --echo # up to this point, no changes to the InnoDB dictionary are necessary
            alter table t1 add index(c), algorithm=inplace;
            

            Without any fix, we would refuse anything else than ALGORITHM=COPY. With the fix, InnoDB would not be aware of any virtual column metadata. As long as we are dropping virtual columns, nothing would need to be done inside InnoDB, because the virtual columns never existed in the first place, as far as InnoDB is concerned. We could also allow adding non-indexed virtual columns while not updating the InnoDB data dictionary. But, as soon as we add an index on a virtual column, we must make InnoDB aware of all virtual columns.

            marko Marko Mäkelä added a comment - Note: The following scenario should be covered as well: create table t1(a int , b int as (a) virtual, c int as (a+a) virtual) engine=innodb; --echo # upgrade from 10.0 or 10.1 to 10.2 or later alter table t1 drop b,algorithm=inplace; alter table t1 add d int as (-a) virtual,algorithm=inplace; --echo # up to this point, no changes to the InnoDB dictionary are necessary alter table t1 add index (c), algorithm=inplace; Without any fix, we would refuse anything else than ALGORITHM=COPY . With the fix, InnoDB would not be aware of any virtual column metadata. As long as we are dropping virtual columns, nothing would need to be done inside InnoDB, because the virtual columns never existed in the first place, as far as InnoDB is concerned. We could also allow adding non-indexed virtual columns while not updating the InnoDB data dictionary. But, as soon as we add an index on a virtual column, we must make InnoDB aware of all virtual columns.

            While this could have been useful to implement, I think that the ship has already sailed, because everyone should already have upgraded from MariaDB Server 10.1, which reached its end of life in October 2020.

            marko Marko Mäkelä added a comment - While this could have been useful to implement, I think that the ship has already sailed, because everyone should already have upgraded from MariaDB Server 10.1, which reached its end of life in October 2020.
            nunop Nuno added a comment -

            Hi marko
            With your last comment, are you saying that since MariaDB 10.2 we can ALTER TABLE with ALGORITHM=INPLACE ?

            I'm on MariaDB 10.5, and I cannot use ALGORITHM=INPLACE if the table has a virtual generated column (indexed).

            Thanks much.

            nunop Nuno added a comment - Hi marko With your last comment, are you saying that since MariaDB 10.2 we can ALTER TABLE with ALGORITHM=INPLACE ? I'm on MariaDB 10.5, and I cannot use ALGORITHM=INPLACE if the table has a virtual generated column (indexed). Thanks much.

            nunop, before MDEV-5800 MariaDB Server did not support indexed virtual columns, but it did support virtual non-indexed columns. This ticket would have been about facilitating native ALTER TABLE on such tables that had been created before MariaDB 10.2.

            There are several restrictions around native ALTER TABLE and virtual columns. Most of them were inherited from the implementation in MySQL 5.7, and the work-around of MDEV-13795 and MDEV-14332 added one more: online table rebuild is disallowed if indexed virtual columns exist.

            marko Marko Mäkelä added a comment - nunop , before MDEV-5800 MariaDB Server did not support indexed virtual columns, but it did support virtual non-indexed columns. This ticket would have been about facilitating native ALTER TABLE on such tables that had been created before MariaDB 10.2. There are several restrictions around native ALTER TABLE and virtual columns. Most of them were inherited from the implementation in MySQL 5.7, and the work-around of MDEV-13795 and MDEV-14332 added one more: online table rebuild is disallowed if indexed virtual columns exist.
            nunop Nuno added a comment - - edited

            marko Thank you very much for the clarification!

            Do you know if there are plans to make Online ALTER TABLE in these scenarios possible?
            Or are there limitations that totally make that impossible?

            I might need to stop using indexed VIRTUAL columns completely, due to this limitation, unfortunately
            (unless in the future it will be possible to do Online ALTER TABLE)

            Thank you very much again. Good day to you.

            nunop Nuno added a comment - - edited marko Thank you very much for the clarification! Do you know if there are plans to make Online ALTER TABLE in these scenarios possible? Or are there limitations that totally make that impossible? I might need to stop using indexed VIRTUAL columns completely, due to this limitation, unfortunately (unless in the future it will be possible to do Online ALTER TABLE) Thank you very much again. Good day to you.

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.