When I merged MDEV-13328 from 10.1 to 10.2, the innodb.innodb test suddenly started to fail.

The culprit seemed to be a correct-looking conflict resolution in FlushObserver::flush(). As part of the test, innodb.innodb is executing the following:

let $default=`select @@storage_engine`;

set storage_engine=INNODB;

source include/varchar.inc;

This includes the following:

--error ER_DUP_ENTRY

alter table t1 add unique(v);

The duplicate key error triggers the following code at the end of row_merge_build_indexes():

		if (error != DB_SUCCESS) {

			flush_observer->interrupted();

		}

		flush_observer->flush();

This in turn causes a choice of a dangerous parameter:

/** Flush dirty pages and wait. */

void

FlushObserver::flush()

{

/** Flush dirty pages and wait. */

void

FlushObserver::flush()

{

	buf_remove_t	buf_remove;

	if (m_interrupted) {

		buf_remove = BUF_REMOVE_FLUSH_NO_WRITE;

	} else {

		buf_remove = BUF_REMOVE_FLUSH_WRITE;

…

	}

	/* Flush or remove dirty pages. */

	buf_LRU_flush_or_remove_pages(m_space_id, buf_remove, m_trx);

The danger here is that m_space_id=0, the system tablespace. This is potentially discarding other writes to the InnoDB system tablespace, potentially corrupting the whole instance.

We must use the equivalent of BUF_REMOVE_FLUSH_WRITE for the system tablespace (and in MySQL 5.7, for any table that resides in a persistent shared tablespace). Failure to do so caused all sorts of trouble when running innodb.innodb after the merge, especially when using --innodb-buffer-pool-size=5m (the minimum).
Apparently the corruption was caused by the following sequence of events:

1. Some pages in the system tablespace are modified.
2. The ADD UNIQUE INDEX operation runs and fails, and finally removes system tablespace pages from the buf_pool->flush_list without writing to the file (see above).
3. Some affected not-written-back pages are evicted from the buffer pool.
4. A page is eventually needed and read back to the buffer pool, with too old contents, or with all-zero contents.

Various assertions failed due to a supposedly-initialized page being all-zero.

Note: For a failed ALTER TABLE in a tablename.ibd file, it is perfectly OK to discard the entries from the flush_list, and to subsequently mark the pages as freed, or to delete the file (if it was a table-rebuilding ALTER).