Repeatability of test scenario

Definition: Here is with a maintenance release meant a release that fixes a number (may be 1) specific issues in a previous release.

In order to lower the threshold to make such a release, we need to be able to reproduce as exactly as possible the test-suite and test environment that was used when the immediately preceding previous release was made. This in order to ensure that we do not find issues that have been revealed due to, for instance, new and improved test-cases.

When making a maintenance release we are only interested in it being better than the previous release, not that it fixes all known problems at the point the maintenance release is made, as that would make the threshold much higher and effectively turn the maintenance release into a regular release.

The test suite used for testing the new maintenance release should be the very same test suite used for testing the previous release, plus test cases for the specific issues the maintenance release is supposed to address.

One approach would be to tag the test-suite using the same tags that are used for tagging the actual Maxscale release.