There are lots of MTR tests which fail sporadically. They create a problem for distribution builds, where a single test failure means a failed build.
Many of these failures have been identified in scope of the effort to make buildbot green (MDEV-7069), but this activity is still far from succeeding. For most of failures separate JIRA issues exist.
We don't want to discard these tests, eventually they need to be fixed (or the product needs to be fixed, if the failure is real); but meanwhile we don't want these tests to disrupt distribution builds.
Distribution builds already run with non-default options, e.g. --skip-test-list, but so far Debian has been maintaining these lists on their own, which is inefficient.
We need to create such a list in the main tree. Apart from distribution builds, it can be beneficial for anyone who wants to run a smaller but more reliable set of tests.
So the current plan is to take a pessimistic approach and create a rather generous list. It shouldn't matter much for distribution builds – default MTR test suite is excessive for testing a build, so skipping 10-15% of tests shouldn't hurt much. For now, the list (mysql-test/unstable-tests) will have the contents as below; it can be re-considered and tuned later.
- all tests which have been identified as unstable by Debian, either through bug reports they filed, or through unstable lists they are currently maintaining;
- all tests which have failed in buildbot on 10.0 main tree since the beginning of the year, unless those were clearly environmental failures, e.g. disk space problems and such, and have not been fixed after the last failure (in general, the observation period is roughly 6 months, based on lowest frequency for known sporadic failures been 2-3 times a year);
- tests which have been modified since the <upcoming release>-2 (that is, for 10.0.27 it will contain tests which have been modified since 10.0.25). The idea here is is that if a test has been modified, we cannot guarantee its behavior to be stable until we have enough statistics for that;
Note: An exception here is TokuDB tests – since they come through a merge, technically they all get marked as modified, and since there are lots of them, going through them one by one is too time-consuming. So, their modifications will not be taken into account (currently it's not important because they are disabled anyway due to
- create a separate JIRA item for each disabled test, if such item does not exist.
Additionally, there will be a small change in mtr_cases.pm to allow wildcards in the skip list. It's just a convenience measure, because we need to disable big chunks of tests, such as TokuDB or Spider.