Aria: one failed fdatasync() of aria_log silently makes the log read-only; later writes to Aria tables (incl. mysql.global_priv) fail with misleading errors yet persist after restart, tables show corrupt, and a new table loops in auto-repair

If one fdatasync() of aria_log.00000001 fails, the statement that commits
gets error 27 "Can't sync file ... aria_log.00000001". That is fine. But the
failure also switches the Aria log to read-only for the rest of the server's
life (translog_stop_writing()), and the error log does not say so. From then
on, until a restart:

• every write to any transactional Aria table fails with error 1030 "Got
  error 122 "Internal (unspecified) error in handler" from storage engine
  Aria". This includes the system tables: CREATE USER fails with error 1032
  "Can't find record in 'global_priv'";
• the failed statement leaves the table in a state that CHECK TABLE reports
  as Corrupt ("Wrong data in bitmap"), also for mysql.global_priv;
• the changes of these failed statements are not visible while the server
  runs, but after a restart (kill -9 or clean) they are there: the row of
  the failed INSERT and the user of the failed CREATE USER exist;
• an INSERT into a newly created Aria table does not return. It stays in
  "Opening tables" / "Repair with keycache", writes "Got error 145 ... marked
  as crashed" to the error log and creates a new t3.MAD-<timestamp>.BAK file
  about once per second (30 files after 30 s);
• a failed ALTER TABLE on an Aria table leaves its #sql-alter-* table on disk,
  marked as crashed. The next ALTER of an Aria table from the same connection
  fails on it with error 145, and RENAME TABLE of the original table fails
  with errno 1 "Operation not permitted". The #sql-alter files stay after
  restarts.

How to repeat:

docker run -d --name aria012 -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -e MARIADB_DATABASE=test \
quay.io/mariadb-foundation/mariadb-devel:verylatest

1. (wait until the server is ready)
   q() { docker exec aria012 mariadb -uroot test -e "$1"; }

   q "CREATE TABLE t1 (id INT PRIMARY KEY, n INT) ENGINE=Aria TRANSACTIONAL=1; CREATE TABLE t2 LIKE t1;
   INSERT INTO t1 SELECT seq, seq FROM seq_1_to_1000; INSERT INTO t2 SELECT seq, seq FROM seq_1_to_1000;"

2. fail the next fdatasync() of the Aria log once:
   sudo timeout 10 strace -f -qq -y -p $(docker inspect -f '.State.Pid' aria012) -e trace=fdatasync \
   -e inject=fdatasync:error=EIO:when=1 -P /var/lib/mysql/aria_log.00000001 -o trace.txt &
   sleep 2
   q "INSERT INTO t1 VALUES (1001, 1001)"
   wait # strace detaches after 10 s
   q "INSERT INTO t2 VALUES (1001, 1001)"
   q "SELECT count FROM t2 WHERE id=1001"
   q "CHECK TABLE t2"
   q "CREATE USER u1@localhost"
   q "CHECK TABLE mysql.global_priv"
   q "CREATE TABLE t3 (a INT) ENGINE=Aria TRANSACTIONAL=1"
   timeout 15 docker exec aria012 mariadb -uroot test -e "INSERT INTO t3 VALUES (1)"
   docker exec aria012 ls /var/lib/mysql/test | grep -c BAK
   docker kill -s KILL aria012; docker start aria012
   q "SELECT id FROM t2 WHERE id > 1000; SELECT user FROM mysql.global_priv WHERE user='u1'"

Result (13.2 main; 13.0.2 gives the same for the statements tried on it):

trace.txt: fdatasync(6</var/lib/mysql/aria_log.00000001>) = -1 EIO (Input/output error) (INJECTED)

INSERT INTO t1 VALUES (1001, 1001)
-> ERROR 27 (HY000): Can't sync file '/var/lib/mysql/aria_log.00000001' to disk (Errcode: 5 "Input/output error")
INSERT INTO t2 VALUES (1001, 1001)
-> ERROR 1030 (HY000): Got error 122 "Internal (unspecified) error in handler" from storage engine Aria
SELECT count FROM t2 WHERE id=1001 -> 0
CHECK TABLE t2
-> error Page 6: Wrong data in bitmap. Page_type: 1 full: 0 empty_space: 5992 Bitmap-bits: 4 'full'
error Corrupt
CREATE USER u1@localhost
-> ERROR 1032 (HY000): Can't find record in 'global_priv'
CHECK TABLE mysql.global_priv
-> error Page 1: Wrong data in bitmap. ... error Corrupt
CREATE TABLE t3 ... -> OK
INSERT INTO t3 VALUES (1) -> does not return (timeout); SHOW PROCESSLIST:
"Opening tables" / "Repair with keycache"; error log repeats
[ERROR] mariadbd: Got error '145 "Table was marked as crashed and should be repaired"' for './test/t3'
[Warning] Recovering table: './test/t3'
ls ... | grep -c BAK -> 13, then 29 a few seconds later
after kill -9 and restart ("Aria engine: starting recovery ... recovery done"):
SELECT id FROM t2 WHERE id > 1000 -> 1001 (the INSERT that failed with error 122)
SELECT user FROM mysql.global_priv ... -> u1 (the CREATE USER that failed)
CHECK TABLE t1, t2, mysql.global_priv -> OK

The error log has no message about the Aria log having become read-only.
Besides the error-145 lines of the repair loop (hundreds per second) and the
"Can't find record in 'global_priv'" line, the only hint is, up to 30 s
later, "[ERROR] mariadbd: Aria engine: checkpoint failed at
translog_write_record with error 0". After the restart the error log shows
that t2 and mysql.global_priv are opened as crashed and repaired
automatically from their data files:
[ERROR] mariadbd: Got error '145 "Table was marked as crashed and should be repaired"' for './mysql/global_priv'
[Note] Found 7 of 6 rows when repairing './mysql/global_priv'
[Note] Found 1001 of 1000 rows when repairing './test/t2'

With ALTER TABLE t1 ADD INDEX ix1_t1 as the statement whose log sync
fails (found with a fault-injection harness, reproduced 3 of 3 times on 13.0.2
and on 13.2 main):
ALTER TABLE t1 ADD INDEX ix1_t1 -> ERROR 27 Can't sync file '/var/lib/mysql/aria_log.00000001'
ALTER TABLE t2 ADD INDEX ix3_t2 (v(200))
-> ERROR 145 Got error '145 "Table was marked as crashed and should be repaired"' for './ddh/#sql-alter-1-5-2'
RENAME TABLE t1 TO t4
-> ERROR 1025 Error on rename of './ddh/t1' to './ddh/t4' (errno: 1 "Operation not permitted")
after restart: #sql-alter-1-5-2.MAD and #sql-alter-1-5-2.MAI are still in the database directory.

Expected: if Aria cannot continue after a log sync error, it should say so in
the error log and fail later statements cleanly before they change any page
(a clear "Aria log is read-only" error, no table left corrupt, no change
that shows up after a restart although the statement failed). A statement
must not loop in automatic repair. The alternative is to stop the server, as
InnoDB does after a redo log write error. The "Internal (unspecified) error"
and "Can't find record in 'global_priv'" messages do not point to the cause.

Cause, as far as I can tell:

translog_sync_files() calls translog_stop_writing() when mysql_file_sync()
fails:
https://github.com/MariaDB/server/blob/327437ade1d6dc6e4e57a6f1723b4abe4ac6c4b4/storage/maria/ma_loghandler.c#L7918-L7923
translog_stop_writing() sets translog_status to TRANSLOG_READONLY and
contains DBUG_ASSERT(0), so this path is treated as unexpected, and nothing
is written to the error log:
https://github.com/MariaDB/server/blob/327437ade1d6dc6e4e57a6f1723b4abe4ac6c4b4/storage/maria/ma_loghandler.c#L929-L940
The failed statements seem to change row and bitmap pages before their log
record is refused, and these changes are not undone (CHECK TABLE then finds
the bitmap out of step). They come back after the restart because the
tables are marked crashed and the automatic repair (aria_recover_options=
BACKUP,QUICK) rebuilds them from the data file, which holds the rows of the
failed statements ("Found 1001 of 1000 rows when repairing './test/t2'");
the same result was seen after kill -9 and after a clean shutdown.