Not only SYS_COLUMNS.PRTYPE changes on charset/collation change but also SYS_COLUMNS.LEN could increase f.ex. on ascii -> utf8mb3 or utf8mb3 -> utf8mb4.

Right. 10.2 already supports instant changes of the maximum length of VARCHAR (in bytes) except when the old length was less than 256 bytes and the new length is at least 256 bytes. Other changes to wider type would be covered by MDEV-15563 (ROW_FORMAT=REDUNDANT) and MDEV-11424 (ROW_FORMAT=DYNAMIC, ROW_FORMAT=COMPACT).

As part of this task, we can keep the existing limitation regarding table rebuilds, but still support ALGORITHM=INPLACE, LOCK=NONE instead of the currently only choice ALGORITHM=COPY.

Current open questions:
1) Can CHAR(200) be converted from utf8mb3 to utf8mb4? It's size should increase from 600 to 800 in that case. If InnoDB really allocates 600 bytes it can't increase it to 800 bytes instantly. This should be examined in source code.
2) There are a lot of charsets in MariaDB and are much more charset/collation pairs. I do not know the all charset/collate -> charset/collate pair which could be altered instantly. E.g. I don't know whether it's possible to instantly convert ascii_general_nopad_ci to latin2_general_nopad_ci.
3) I don't know how to test every combination of charset/collation convert or not convert to other charset/collation instantly. Actually I don't think it's ever needed. I'm trying to write the most generic check code and test every corner case, instead of every possible case.
4) SYS_COLUMNS.PRTYPE and SYS_COLUMNS.LEN are updated on my instant queries. And thus InnoDB cache data structures should be checked somehow. And I don't know a simple way to do it. A complex way could look like ALTER ... INSERT ... SELECT ... ORDER BY.

First of all, ROW_FORMAT=REDUNDANT basically treats every field as if it is variable-length and possibly NULL. I think that we can allow any size extension with that format. For other formats (COMPACT, DYNAMIC, COMPRESSED) the following holds:

1) Internally, long CHAR (maybe the limit is 255 bytes) is treated as VARCHAR, and I think that it should be possible to instantly extend this. As an optimization (which is not available in ROW_FORMAT=REDUNDANT) for variable-length character sets, CHAR will be stored as n*mbminlen‥n*mbmaxlen bytes, and trailing space will be trimmed. Only if the ‘payload’ (not counting trailing space) exceeds n*mbminlen bytes, more storage will be allocated. This ought to imply that such CHAR columns are stored as variable-length (explicit length is encoded). I implemented this in MySQL 5.0.3, but cannot remember the details right now.
2), 3) I think that it is OK to start with some safe subset. bar should be able to provide a list.
4) Yes, we should update the dict_table_t. Maybe you could take a look at the code that implements the instant VARCHAR extension, which was introduced in MySQL 5.7.

VARCHAR(200) + ROW_FORMAT=REDUNDANT + UTF8MB3:

Russian 'фыва' occupied 8 bytes, not 4*3=12 bytes.

VARCHAR(200) + ROW_FORMAT=COMPACT + UTF8MB3:

Russian 'фыва' occupied 8 bytes.

VARCHAR(200) + ROW_FORMAT=COMPRESSED + UTF8MB3:

Russian 'фыва' occupied 8 bytes.

VARCHAR(200) + ROW_FORMAT=DYNAMIC + UTF8MB3:

Russian 'фыва' occupied 8 bytes.

CHAR(200) + ROW_FORMAT=REDUNDANT + UTF8MB3:

Russian 'фыва' occupied 600 bytes.

CHAR(200) + ROW_FORMAT=COMPACT + UTF8MB3:

Russian 'фыва' occupied 200 bytes.

CHAR(200) + ROW_FORMAT=COMPRESSED + UTF8MB3:

Russian 'фыва' occupied 200 bytes.

CHAR(200) + ROW_FORMAT=DYNAMIC + UTF8MB3:

Russian 'фыва' occupied 200 bytes.

So, CHAR + ROW_FORMAT=REDUNDANT may need size increase on charset change. Disabling it for simplicity.

Correct: ROW_FORMAT=REDUNDANT will reserve n*mbmaxlen bytes for {{CHAR(n)}}. It will do that even when the column is NULL. The idea is to allow update-in-place. The later ROW_FORMAT optimize the space and allocate n*mbminlen to n*mbmaxlen bytes. Trailing space is ‘compressed away’. For example, 'фыва' is 4×2 bytes. If the column were CHAR(8), we would allocate only 8 bytes for it, because the trailing space 'фыва ' does not need to be stored. If there was one more letter before the trailing space, we would have to allocate more space.

It could be that {{CHAR(n)}} with a variable-length character encoding is actually internally stored as VARCHAR, that is, an explicit length is stored. I cannot remember it.

For ROW_FORMAT=REDUNDANT, I think that it should be OK to bend the existing rules and allow an instant extension of the maximum length of the column, that is, allow CHAR(200) to be converted to a bigger mbmaxlen without changing the existing data. This would be a step towards MDEV-15563, which would allow CHAR or VARCHAR to be instantaneously extended arbitrarily in ROW_FORMAT=REDUNDANT.

Thanks, this looks generally good. I am mainly requesting some code refactoring.

Please document (with examples) in which way the file format is being effectively changed, as witnessed by the change to the CHECK TABLE function btr_index_rec_validate(). We must prevent older MariaDB versions from accessing tables where the file format has been changed.

Some necessary code changes are common between this and MDEV-15563. I think that MDEV-15563 is closer to completion at this point, and should be submitted first.

I've taken a look into a number of commits:

@kevgs pushed 6 commits.

20cb37e add GBK
053e48e add UJIS
4e3f6ef add SJIS
54e4cc0 add BIG5
90b4b4f add latin2
3237b5e added latin7

There is no need to list all ASCII compatible character set names and check them using strcmp().
Please check:

(cs->state & MY_CS_NONASCII)

to know if a character set is not ASCII compatible.

Note, SJIS is not ASCII compatible.
This change is probably not correct:
https://github.com/MariaDB/server/commit/54e4cc0b6571a6a929538eca46f940c465e42e68

Due to conflicts with MDEV-15563, I refactored the InnoDB part of this and also the functions Field_str::is_equal() and Field_varstring::is_equal().

The following bugs were filed for limitations that I think we should try to fix soon:
MDEV-18583 Avoid copying when changing the type of an indexed column
MDEV-18584 Avoid copying when altering CHAR column in InnoDB table