Some character encodings are proper subsets of others. ASCII is a proper subset of latin1 or latin2 or UTF-8. The 3-byte UTF-8 is a proper subset of the 4-byte UTF-8.
When the character encoding of a column changes such that the storage encoding does not change, we should avoid rebuilding the table.
Likewise, when the collation of a column changes such that the encoding stays the same, we should avoid rebuilding the table.
Collations do not matter if the column is not indexed.
If the collation of an indexed column changes, then the affected indexes may have to be rebuilt. For certain collations we might avoid that as well. For example, changing the encoding and collation from binary 3-byte to binary 4-byte UTF-8 might not require any change.
When the collation change involves a column that is part of the PRIMARY KEY and we have determined that a collation change is necessary, then the whole table will have to be rebuilt.