[MDEV-15563] Instant failure-free data type conversions Created: 2018-03-14  Updated: 2023-12-19  Resolved: 2019-02-13

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Fix Version/s: 10.4.3

Type: Task Priority: Critical
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 1
Labels: instant, online-ddl

Attachments: File UNSIGNED-bigger-signed.patch    
Issue Links:
Blocks
blocks MDEV-11424 Instant ALTER TABLE of failure-free r... Closed
blocks MDEV-15564 Avoid table rebuild in ALTER TABLE on... Closed
Problem/Incident
causes MDEV-18124 PK on inplace-enlarged type fails Closed
causes MDEV-18596 ASAN unknown-crash in row_mysql_store... Closed
causes MDEV-18598 Assertion `is_string == dtype_is_stri... Closed
causes MDEV-18623 Assertion `ha_alter_info->key_count +... Closed
causes MDEV-18627 Wrong result after instant size chang... Closed
causes MDEV-18719 Assertion `!((c.prtype ^ o->prtype) &... Closed
causes MDEV-21539 Assertion `!((new_col->prtype ^ col->... Closed
causes MDEV-22465 DROP COLUMN, DROP INDEX is wrongly cl... Closed
causes MDEV-25236 Online log apply fails for redundant ... Closed
Relates
relates to MDEV-16849 Extending indexed VARCHAR column shou... Closed
relates to MDEV-18035 InnoDB: Failing assertion: field->col... Closed
relates to MDEV-18649 Assertion `supremum[7] == index.n_cor... Closed
relates to MDEV-18780 Assertion `col->prtype == prtype' fai... Confirmed
relates to MDEV-22771 Assertion `fields[i].same(instant.fie... Closed
relates to MDEV-28060 Online DDL fails while checking for i... Closed
relates to MDEV-33069 Instant Modify Column Open

 Description   

The original InnoDB record format (retroactively named ROW_FORMAT=REDUNDANT in MySQL 5.0.3) basically treats every column as variable-length and possibly NULL. That is why we can freely change those columns, by changing the metadata only.

MariaDB 10.2 supports VARCHAR column extension, but only if the maximum length does not change from less than 256 to at least 256 bytes. For ROW_FORMAT=REDUNDANT we should support this unconditionally.

Likewise, for ROW_FORMAT=REDUNDANT we can support the extension of any CHAR column, or converting VARCHAR to a CHAR that is at least as wide.

We can also support converting INT to BIGINT and similar. Maybe also conversion from an unsigned integer to a signed wider integer (such as INT UNSIGNED to BIGINT), but this would require a change to the 'metadata row' format to indicate that the column originally was in unsigned format.

To support instant conversion of CHAR to VARCHAR, we should store a flag in the MDEV-15562 metadata record to indicate that trailing space should be trimmed, just like it would if the table was rebuilt. In this case, any secondary indexes on the column would have to be rebuilt and this is, obviously, not and INSTANT operation:

CREATE TABLE t(a CHAR(2)) ENGINE=InnoDB;
INSERT INTO t VALUES('a ');
SELECT HEX(a) FROM t;
ALTER TABLE t CHANGE a a VARCHAR(2);
SELECT HEX(a) FROM t;
DROP TABLE t;

The above example would read 'a' from the column and not 'a '.
Similarly, for changing INT UNSIGNED to BIGINT we should add a flag that the shorter column was unsigned.

TO BE DETERMINED: On secondary indexes of full columns (not column prefixes), do we have to rebuild the index when the length of a fixed-length column (CHAR, BINARY) is changed?



 Comments   
Comment by Marko Mäkelä [ 2018-09-20 ]

When it comes to avoiding rebuilding secondary indexes, MDEV-16849 might be useful.

Comment by Eugene Kosov (Inactive) [ 2018-10-24 ]

Please, review a first part of an issue.

Comment by Marko Mäkelä [ 2018-10-31 ]

The first part (NOT NULL to NULL) conversion looks mostly OK, except for one condition in the code. We must check the ROW_FORMAT from InnoDB dict_table_t, because it could be wrong in the .frm file.

Comment by Aleksey Midenkov [ 2018-11-04 ]

Analysis is in progress for part 2: https://github.com/tempesta-tech/mariadb/issues/555

Comment by Marko Mäkelä [ 2018-11-05 ]

The first part (NOT NULL removal) was moved to MDEV-17520.
The second part (instant failure-free changes of column type) will depend on MDEV-17520. Until MDEV-17520 is pushed to 10.4, the initial version of the first part can be used for developing the column type changes for ROW_FORMAT=REDUNDANT.

Comment by Marko Mäkelä [ 2018-12-11 ]

I ported all generic changes from bb-10.4-MDEV-17520 to a new branch bb-10.4-MDEV-15563.
It implements an instantaneous removal of NOT NULL constraints for ROW_FORMAT=REDUNDANT tables, and it includes some minor refactoring.

Comment by Matthias Leich [ 2018-12-13 ]

Results of RQG testing (several hours with rqg_batch.pl on box with 8 CPU's) on
bb-10.4-MDEV-15563  commit b710e66b4912c0a0806609330c78679d809ecb36
- three new bugs found but all were not related to bb-10.4-MDEV-15563
  - https://jira.mariadb.org/browse/MDEV-17989    InnoDB problem which exists already in 10.2
  - to https://jira.mariadb.org/browse/MDEV-15850  added as activity    Two RQG problems+now fixed.
- the RQG grammar 'table_stress' was extended to include
   - certain InnoDB Row formats (especially redundant)
   - ALTERING columns of different type between NULL (allowed) and NOT NULL and back
- especially the RQG test "table_stress" revealed a crowd of asserts but they were all
  already covered by bug reports for 10.2 till 10.4 

Comment by Marko Mäkelä [ 2018-12-13 ]

I pushed bb-10.4-MDEV-15563 (which only implemented the original first part of this task, instant NOT NULL removal for ROW_FORMAT=REDUNDANT tables) to 10.4, and it should be part of the 10.4.1 release.

This task should no longer depend on MDEV-17520, if we change the course so that the most generic instant length and type changes are limited to ROW_FORMAT=REDUNDANT.

For ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPACT, we should be able to do the following instantly, without changing the storage format:

  1. change CHAR(0) to any VARCHAR (I believe that it is encoded as variable-length)
  2. change a CHAR that is longer than 255 bytes to VARCHAR or wider CHAR
  3. change a VARCHAR that is shorter than 128 bytes into any longer VARCHAR
  4. (10.2 already supports changing VARCHAR to wider one, except if the maximum length grows from less than 256 bytes to at least 256 bytes)
Comment by Marko Mäkelä [ 2019-02-13 ]

I split this into 3 more commits:
MDEV-15563: Allow instant VARCHAR extension from <128 bytes
MDEV-15563: Instant VARCHAR extension for ROW_FORMAT=REDUNDANT
MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
The last two are based on the work of midenok.

I decided to omit UNSIGNED-bigger-signed.patch, because something like that could be better done later as part of MDEV-17520, in a way that does not add much overhead to the current ROW_FORMAT=DYNAMIC.

Comment by Marko Mäkelä [ 2019-02-18 ]

The third change (instant fixed-size column extension for ROW_FORMAT=REDUNDANT) had to be reverted due to incorrect results for indexed columns, in MDEV-18627. We might revive it in a later version related to the task MDEV-17520.

Generated at Thu Feb 08 08:22:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.