An I/O error on the final .frm rename of an in-place DROP INDEX / ADD INDEX leaves the .frm and the InnoDB dictionary disagreeing: queries on the table fail with error 1030 (or the new index is unusable), CHECK TABLE says OK, a restart does not repair it

Description:

An in-place ALTER TABLE commits the change in InnoDB first and then replaces
the .frm by renaming "#sql-alter-....frm" over "<table>.frm". If that rename
fails (EIO here; any rename error), the statement returns

ERROR 1025 (HY000): Error on rename of './test/#sql-alter-1-6-0' to './test/t4' (errno: 5 "Input/output error")

and leaves the old .frm in place, but the InnoDB change is not undone.
The error is honest about the statement, but the table is left inconsistent,
and the inconsistency is permanent (it survives a clean restart):

• DROP INDEX ix2_t4 ON t4: the .frm still lists ix2_t4, InnoDB no longer has
  it. SELECT count, any query whose plan uses ix2_t4, and FORCE INDEX
  (ix2_t4) fail with ERROR 1030 "Got error 1 "Operation not permitted" from
  storage engine InnoDB"; the error log says "InnoDB could not find key no 1
  with name ix2_t4 from dict cache for table test/t4". The data is intact.
• CREATE INDEX ix_k ON t5 (k): InnoDB has the new index, the .frm does not.
  The index is maintained by InnoDB but cannot be used ("Key 'ix_k' doesn't
  exist in table 't5'"), and the error log reports "Table test/t5 contains 2
  indexes inside InnoDB, which is different from the number of indexes 1
  defined in the .frm file".

CHECK TABLE reports status OK for both tables. Nothing repairs the table
automatically; for t4, running "ALTER TABLE t4 DROP INDEX ix2_t4" again
fixes it.

How to repeat:

docker run -d --name m004 -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -e MARIADB_DATABASE=test \
quay.io/mariadb-foundation/mariadb-devel:verylatest

1. (wait until the server is ready)
   q() { docker exec m004 mariadb -uroot test -e "$*"; }

   q "CREATE TABLE t4 (id INT PRIMARY KEY, k INT, n INT, v TEXT, KEY ix2_t4 (k, n));
   INSERT INTO t4 SELECT seq, seq % 7, seq % 11, REPEAT('v', 20) FROM seq_1_to_55;
   CREATE TABLE t5 (id INT PRIMARY KEY, k INT); INSERT INTO t5 SELECT seq, seq FROM seq_1_to_20;"
   fail_rename() { # the first rename() of the statement fails with EIO
   sudo timeout 8 strace -f -qq -y -p $(docker inspect -f '.State.Pid' m004) -e trace=rename \
   e inject=rename:error=EIO:when=1 -o trace$RANDOM.txt & sleep 2
   q "$1"; wait
   }
   fail_rename "DROP INDEX ix2_t4 ON t4"
   fail_rename "CREATE INDEX ix_k ON t5 (k)"
   docker restart m004 # optional; same result before and after

2. (wait until the server is ready)
   q "SHOW INDEX FROM t4"
   q "SELECT i.NAME FROM information_schema.INNODB_SYS_INDEXES i JOIN information_schema.INNODB_SYS_TABLES t
   ON i.TABLE_ID = t.TABLE_ID WHERE t.NAME IN ('test/t4','test/t5')"
   q "SELECT count FROM t4"
   q "SELECT id FROM t4 WHERE k=3 AND n=3"
   q "CHECK TABLE t4"
   q "SELECT count FROM t5 FORCE INDEX (ix_k)"
   q "CHECK TABLE t5"

Result (identical on 13.2 main and 13.0.2):

DROP INDEX ix2_t4 ON t4 -> ERROR 1025 (HY000): Error on rename of './test/#sql-alter-1-6-0' to './test/t4' (errno: 5 "Input/output error")
CREATE INDEX ix_k ON t5 (k) -> ERROR 1025 (HY000): Error on rename of './test/#sql-alter-1-7-1' to './test/t5' (errno: 5 "Input/output error")
trace: rename("./test/#sql-alter-1-6-0.frm", "./test/t4.frm") = -1 EIO (Input/output error) (INJECTED)
rename("./test/#sql-alter-1-7-1.frm", "./test/t5.frm") = -1 EIO (Input/output error) (INJECTED)

SHOW INDEX FROM t4 -> PRIMARY, ix2_t4 (k, n)
InnoDB indexes -> t4: PRIMARY t5: PRIMARY, ix_k
SELECT count FROM t4 -> ERROR 1030 (HY000): Got error 1 "Operation not permitted" from storage engine InnoDB
SELECT id FROM t4 WHERE k=3 AND n=3 -> ERROR 1030 (same)
CHECK TABLE t4 -> status OK
SELECT count FROM t5 FORCE INDEX (ix_k) -> ERROR 1176 (42000): Key 'ix_k' doesn't exist in table 't5'
CHECK TABLE t5 -> status OK
error log:
[ERROR] InnoDB could not find key no 1 with name ix2_t4 from dict cache for table test/t4
[Warning] InnoDB: change_active_index(1) failed
[ERROR] Got error 1 when reading table './test/t4'
[ERROR] InnoDB: Table test/t5 contains 2 indexes inside InnoDB, which is different from the number of indexes 1 defined in the .frm file.

Expected: after a failed DDL statement the table is in a consistent state,
either the old one or the new one. The atomic-DDL log already records the
ALTER (DDL_ALTER_TABLE_PHASE_INPLACE_COPIED means "the new table is ready ...
in case of crash it should use the new one"), so after the engine commit the
.frm swap could be retried or completed through the DDL log instead of
abandoning it; at minimum CHECK TABLE should detect that the .frm and the
engine dictionary disagree.

Cause, as far as I can tell:

In mysql_inplace_alter_table(), after the InnoDB commit and the DDL log phase
update, a failure of mysql_rename_table() for the .frm goes to err, with the
comment "Since changes were done in-place, we can't revert them"; nothing
reconciles the two dictionaries afterwards (the DDL log is only replayed at
crash recovery):
https://github.com/MariaDB/server/blob/327437ade1d6dc6e4e57a6f1723b4abe4ac6c4b4/sql/sql_table.cc#L8778
https://github.com/MariaDB/server/blob/327437ade1d6dc6e4e57a6f1723b4abe4ac6c4b4/sql/sql_table.cc#L8812-L8820

Related: MDEV-17567 (atomic DDL). No report found for a failed .frm rename of
an in-place ALTER.