Change in MTR made in scope of MDEV-12042 allow it to rebootstrap the server between consequent tests, if a test's options are suspected to be not compatible with the original bootstrap. The list of such options is hardcoded:

     if ( $extra_opts and

           "@$extra_opts" =~ /--innodb[-_](?:page[-_]size|checksum[-_]algorithm|undo[-_]tablespaces|log[-_]group[-_]home[-_]dir|data[-_]home[-_]dir)/ )

      {

        mysql_install_db($mysqld, undef, $extra_opts);

      }

There are some problems to fix and things to improve.

1. Re-bootstrap with inconsistent set of options

The above logic does not work well enough when InnoDB options depend on other server settings. For example, if we run

$ perl ./mtr encryption.innodb_encryption --mysqld=--innodb-undo-tablespaces=0 --mem 

we'll get

encryption.innodb_encryption 'innodb_plugin' [ fail ]  Found warnings/errors in server log file!

        Test ended at 2017-08-18 19:04:19

line

2017-08-18 19:03:42 140593044430976 [Warning] InnoDB: New log files created, LSN=45887

2017-08-18 19:03:42 140593044430976 [Warning] InnoDB: Creating foreign key constraint system tables.

^ Found warnings in /data/src/bb-10.1-elenst/mysql-test/var/log/mysqld.1.err

ok

This is because rebootstrap here doesn't actually work. It attempts to start with innodb options which require encryption, but without the encryption plugin itself, because it only chooses InnoDB options from $extra_opts, while plugin-load-add is ignored; so InnoDB doesn't start at all, and hence the tables and log files don't get created upon bootstrap.
From the bootstrap log:

2017-08-18 19:12:20 139995187245184 [ERROR] InnoDB: cannot enable encryption, encryption plugin is not available

2017-08-18 19:12:20 139995187245184 [ERROR] Plugin 'InnoDB' init function returned error.

2017-08-18 19:12:20 139995187245184 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

2. Re-creating bootstrap.sql upon re-bootstrap

Every time re-bootstrap is issued, it re-creates bootstrap.sql file from scratch. It shouldn't take that much resources, but it is unnecessary; besides, we are having troubles on Windows, where tests sometimes fail upon re-bootsrap apparently because the bootstrap.sql file is locked when they want to re-create it. While we don't know root cause of the problem, it might well go away if we just stop re-creating it when it already exists. One catch is that currently it's located in var/tmp, which is removed between the tests; but we can store it in var/log instead, it might even be beneficial to keep it after test run when we want to analyze some obscure bootstrap problems.

3. Bootstrap log on Windows

Not directly related to MDEV-12042, but rather to investigating problems upon re-bootstrap – Wlad requested adding --console to bootstrap, to get more useful log.