This has been a recurrent discussion for long time. Here are some points that have been raised before, and that we briefly revisited with serg yesterday in Berlin.

1) "not disabling" plugins and "enabling" plugins are different matters – we cannot, even hypothetically, simply enable all plugins in MTR, because some plugins cannot live together, e.g. InnoDB and XtraDB, Federated and FederatedX; besides, enabling all plugins is as unrealistic setup as disabling all plugins which we have now. So, I suggest that in the scope of this task we think about not disabling plugins which are enabled by default. Enabling non-default plugins can only be done selectively, but then MTR isn't the best place to do it, it can be done more efficiently in individual test runs or suites;
2) hypothetically, we can stop disabling plugins, but it will make many tests unstable – not because of real failures, but because of limitations of the result-based MTR: different plugins will show up on different builds and test runs, and it will be next to impossible to make existing tests deal with it;
3) allowing InnoDB to start in all tests will be insanely time-consuming, especially on debug builds. While I raise this question every now and then, realistically we cannot afford it for all test runs.

As the entry point, I suggest the following: I'll add an experimental option "keep-default-configuration" or something like that in my development tree and let buildbot have a run with it. This way we'll know how many problems we deal with in (2) above. If it appears to be viable, we can add a build run with this option at least for suites where it makes sense.

Everyone, feel free to add your thoughts, ideas or objections.

mtr disables plugins to make sure they don't affect unrelated tests. The test is supposed to test only one specific thing, one code path — if the plugin is part of that, it should be enabled, otherwise it should be disabled. That was the logic.

keep-default-configuration looks like a useful idea. But as it'll cause a lot of changes in the result files, it will be tricky to run it in buildbot on a regular basis. (may be with a set of rdiffs?..)

I think the problem we've run into with the cracklib plugin is that it's not always easy to figure which tests are related. Besides, although on one hand we could say that everything that cracklib plugin could have affected must have been covered by its own tests, on the other hand it would be an obvious duplication of test cases. It's certainly better to re-use existing test cases whenever possible, we just need to come up with a strategy for doing that, i don't have a ready solution.

For the changes in result files – yes, I expect it will be the case, but we need to see which exactly result files, which differences and how many of them we would be dealing with, then we can make a decision on how to deal with them.

So, here are results of a full buildbot run with plugins not being globally disabled – that is, whatever plugins are enabled by default and not disabled by individual tests/suites stay enabled. Please note that it does not make non-default plugins enabled, e.g. the tests were still running without the cracklib plugin.

Additional notes:

• example_key_management was still disabled (because of MDEV-9948);
• loose-file-key-management-filename was added to options;
• dan builders did not run, but fulltest ones did, so it should be more or less covered;
• P8 builders did not run;
• valgrind and sol10-64 builders were ignored.

In total, there were ~50 failed tests (by name, not by failure count, combinations are grouped under the same name).

      2 binlog.binlog_mysqlbinlog2

      8 connect.infoschema-9739

     10 encryption.debug_key_management

     13 encryption.filekeys_nofile

     16 funcs_1.is_columns_is

      8 funcs_1.is_columns_is_embedded

     16 funcs_1.is_columns_mysql

      4 funcs_1.is_columns_mysql_embedded

     16 funcs_1.is_statistics

     16 funcs_1.is_tables_is

      4 funcs_1.is_tables_is_embedded

     19 main.bad_frm_crash_5029

     19 main.bootstrap

     18 main.features

     15 main.information_schema

     15 main.information_schema_all_engines

      4 main.information_schema-big

      1 main.mysqlbinlog_row_minimal

     15 main.mysql_upgrade-6984

     15 main.mysql_upgrade_no_innodb

     15 main.mysql_upgrade_view

     19 main.view

     18 maria.small_blocksize

      1 mroonga/storage.variable_match_escalation_threshold_global

      1 perfschema.csv_table_io

      1 perfschema.func_file_io

      1 perfschema.func_mutex

      1 perfschema.indexed_table_io

      2 perfschema.innodb_table_io

      1 perfschema.memory_table_io

      1 perfschema.merge_table_io

      1 perfschema.multi_table_io

      1 perfschema.myisam_file_io

      1 perfschema.myisam_table_io

      1 perfschema.part_table_io

      1 perfschema.privilege_table_io

      2 perfschema.rollback_table_io

      2 perfschema.start_server_innodb

      1 perfschema.start_server_no_file_inst

      1 perfschema.start_server_on

      1 perfschema.temp_table_io

      1 perfschema.trigger_table_io

      1 perfschema.view_table_io

      1 plugins.audit_null

      1 spider/bg.ha

     18 sql_discovery.simple

     10 sys_vars.innodb_encrypt_tables_basic

      4 sys_vars.sysvars_server_embedded

     14 sys_vars.sysvars_server_notembedded

The failures are mainly mismatches due to missing warnings about disabled InnoDB or partitioning, P_S differences (which in fact might even be an actual problem, but it would be minor and out of scope of this task); however, there were also actual failures which cause test abort (as opposed to mismatches), e.g. caused by wrong error codes and such. It is important to know for choosing further approach (see below).

Regarding my earlier concern about timing with enabled InnoDB, in fact it wasn't nearly as awful as I expected. Very roughly, test runs (individual test* entries in builds) take ~20-50% more time than on the regular 10.1 tree, so it's not times more.

Technically, 50 tests is a treatable amount, so we could add a mode to MTR to run such tests. It would require some hacks, so that from the command point of view it worked similar to, lets say, --ps-protocol option – that is, it would tell MTR not to disable optional plugins; but further, it should work more like a combination, so that we could add rdiffs to mismatches. Tests which abort would have to be disabled in this mode, so we would also need include/no_optional_plugins.inc or something like that.

However, I don't know how valuable this mode will be in practice. It will be rather fragile – whenever a plugin gets enabled/disabled by default, it could cause false positives. And as the last experiment shows, even when it was run first time ever, it did not reveal that many real problems – MDEV-9948, a vague issue with P_S, and a test case for an old-ish bug MDEV-6368, that's about it.

So, I have doubts it's really worth trouble. Opinions are welcome.

As already said before, it was about not disabling default plugins. Enabling all plugins is a totally different story, there will be another comment about it.

Now, regarding actually enabling plugins, which the original issue was about.

First, as it was also said before, we cannot just blindly enable all plugins at once. Even if we find the way to tell server plugins from client plugins and load only libraries for server ones, still, some plugins just can't live together, like InnoDB and XtraDB, Federated and FederatedX; and some don't make sense together even if they actually can survive, like file_key_management and example_key_management. So, we would have to maintain an explicit list of plugins to enable.

Secondly, if we just enable a plugin and run MTR tests with it, one of two things will happen: either the plugin is irrelevant to the test flow, then the tests won't be affected (apart from possible mismatches in result files due to different output of SHOW commands and such; or, the plugin is relevant to the test flow, and then the tests will start failing because the flow is affected. It does not mean the plugin works wrong, it might be just doing its job.

For just one example, if we run the main suite with the cracklib_password_check enabled – yes, we'll get the crashes reported earlier, and we should have caught them. But we'll also get totally legitimate failures whenever a test attempts to set a password which does not meet requirements enforced by the cracklib_password_check. And these failures aren't easy to work around, even with all rdiff and combination tricks, as they are not just mismatches.

Given all that, I don't think it's doable to regular MTR tests with "enabled plugins".

But the description of this issue says "more proactively identify these types of problems." If "these types" refer to crashes, that's something we might be able to do, and which might be useful.
We can add an option --crashtest to MTR. It will presume --force --force --nowarnings --nocheck-testcases, and will also disable result mismatch check. So, theoretically it should only fail when the server crashes or the test times out. That, plus the explicit list of plugins to enable (maybe via a cnf template), should cover "these types of problems".

Does anyone consider it useful? If so, I can try to do that and see where it gets us.
Alternative ideas are also welcome.

If we agree that can't use MTR powers of functional result-based testing for arbitrary combinations of plugins and have to limit this task to crash testing, as the last comment describes (and as the initial description can be interpreted), it may fit into the scope of pquery testing which Roel performs. If I understand correctly, [one of the] sources of the inflow which pquery uses are randomly mixed up queries from existing MTR tests. The same tests could be run with different plugins enabled.

if we can make sure that mtr tests (most of them, anyway) do not depend on the loaded plugin set, then sure, those tests can be run with any plugins loaded. Like, ./mtr --mysqld=--plugin-load-add=someplugin.so etc.

What do do with the tests are would fail because of extra plugins being loaded?

Yes, that's what my previous lengthy comments are about. At that time, I attempted to run MTR tests with default plugins not disabled (which is a smaller step than non-default plugins enabled), and already that caused a number of test failures – not due to bugs, but test issues to deal with, sometimes in a non-trivial way. Enabling arbitrary non-default plugins would be a more difficult problem. I expect it to complicate the MTR test maintenance significantly and am far from sure it is worth the effort.

Crash testing, on the other hand, is a fairly cheap exercise. It can be done even with MTR itself, by simply running the tests with extra options and then ignoring all failures except for crashes; but since Roel's tests already do pretty much the same "natively", they seem to be a good fit.