Aria: a failed data-page write during FLUSH TABLES drops the dirty page; committed UPDATEs on a TRANSACTIONAL=1 table are silently lost, FLUSH TABLES returns OK

Description:

When one pwrite() of an Aria data page fails while the page cache flushes a
table (FLUSH TABLES, RENAME TABLE, any close of the table), the dirty page is
removed from the page cache anyway. Committed changes on that page are lost:
the next read gets the older page from disk. FLUSH TABLES returns OK to the
client. On the next open the table is found "marked as crashed" and the
automatic repair rebuilds it from the data file that holds the old page. A
restart, clean or after kill -9, does not bring the changes back, although
they were committed to the Aria log.

In the example below, one EIO on one 8 KB write loses an acknowledged UPDATE
of 44 of 2000 rows. Afterwards CHECK TABLE says OK.

How to repeat:

docker run -d --name aria007 -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -e MARIADB_DATABASE=test \
quay.io/mariadb-foundation/mariadb-devel:verylatest

1. (wait until the server is ready)
   q() { docker exec aria007 mariadb -uroot test -e "$1"; }

   q "CREATE TABLE t1 (id INT PRIMARY KEY, v VARCHAR(200)) ENGINE=Aria TRANSACTIONAL=1;
   INSERT INTO t1 SELECT seq, CONCAT('old-', seq, '-', REPEAT('x',150)) FROM seq_1_to_2000;
   FLUSH TABLES;"
   q "UPDATE t1 SET v=CONCAT('new-', id, '-', REPEAT('y',150))" # OK, 2000 rows changed

2. fail the first pwrite64() to t1.MAD with EIO (the file is not opened
3. again, so -P on the path inside the container works on the host):
   sudo timeout 10 strace -f -qq -y -p $(docker inspect -f '.State.Pid' aria007) -e trace=pwrite64 \
   -e inject=pwrite64:error=EIO:when=1 -P /var/lib/mysql/test/t1.MAD -o trace.txt &
   sleep 2
   q "FLUSH TABLES" # returns OK
   wait # strace detaches after 10 s
   q "SELECT count, SUM(v LIKE 'old-%') AS old_rows FROM t1"
   docker kill -s KILL aria007; docker start aria007 # or a clean restart
   q "SELECT SUM(v LIKE 'new-%') AS new_rows, SUM(v LIKE 'old-%') AS old_rows FROM t1"
   q "CHECK TABLE t1 EXTENDED"

Result (identical on 13.2 main and 13.0.2):

trace.txt:
pwrite64(25</var/lib/mysql/test/t1.MAD>, "...new-1-yyyy"..., 8192, ...) = -1 EIO (Input/output error) (INJECTED)
(46 more writes to t1.MAD succeed)

FLUSH TABLES -> OK, no warning
SELECT ... before the restart > count 2000, old_rows 44 (rows 1, 2, 3, ... show 'old...')
error log: [ERROR] mariadbd: Got error '145 "Table was marked as crashed and should be repaired"' for './test/t1'
[Warning] Recovering table: './test/t1' (t1.MAD-<timestamp>.BAK is created)
after the restart -> new_rows 1956, old_rows 44
"Aria engine: starting recovery ... recovery done"
CHECK TABLE t1 EXTENDED -> OK

Expected: the committed UPDATE stays visible. If the page cannot be written,
the page should stay dirty (and the write be retried, or the table be marked
crashed while the page is kept and redone from the log), and the statement
that triggered the flush should report the error. The error log never says
that a write failed; the only message is the "marked as crashed" line, and
the automatic repair makes the loss permanent.

The same fault during other statements gives other wrong results. Found with
a fault-injection harness, with one EIO on one write of the data file:

• RENAME TABLE t1 TO t7 (which flushes t1): acknowledged; 2 rows silently show
  older versions, live and after restart.
• During a 2000-row INSERT (the page cache evicts a dirty page of the table
  being written): the INSERT fails with error 126 "Index is corrupted" for
  t3.MAI; every later statement on the table fails (error 1034 "Index for
  table 't3' is corrupt", error 176 "Read page with wrong checksum"); after
  the restart, recovery applies 1994 of the 2000 rows of the INSERT that was
  reported as failed.

Cause, as far as I can tell:

flush_cached_blocks() in storage/maria/ma_pagecache.c handles a failed
pagecache_fwrite() by setting PCBLOCK_ERROR and remembering the errno, and
then, for FLUSH_RELEASE (used when a table is closed), frees the block like a
successfully written one; for the FLUSH_KEEP types it calls
link_to_file_list(), which clears PCBLOCK_CHANGED and sets rec_lsn to LSN_MAX,
so the page is no longer dirty and no longer needs redo:
https://github.com/MariaDB/server/blob/327437ade1d6dc6e4e57a6f1723b4abe4ac6c4b4/storage/maria/ma_pagecache.c#L4849-L4884
https://github.com/MariaDB/server/blob/327437ade1d6dc6e4e57a6f1723b4abe4ac6c4b4/storage/maria/ma_pagecache.c#L1275-L1289
The error comes back to maria_close()/ha_maria::close(), but the close error
of FLUSH TABLES does not reach the client (closefrm()'s return value is not
used by the table cache). MDEV-33877 (disk full on a transactional Aria table
can hang) suggests, for a failed flush of an old page, to "mark the current
table as corrupted and keep the old page in the page cache"; for EIO the page
is not kept.

Related: MDEV-33877 (disk full / hang, open), MDEV-33813 (closed).