Things to fix:

• All missing configurations (need list)
• The special handling of galera, as shown in https://buildbot.mariadb.org/#/builders/67/builds/36
• Centos installation is using fedora packages, not Centos.

To summarize once again which of installation/upgrade tests from the old buildbot are expected to be in the new buildbot from the beginning.

Notes:

Special tricks may be needed to work around existing problems. See old buildbot tests for the hints.

Some tests aren't run by the old buildbot on early development releases.

"deb-based platforms" are all versions of Ubuntu and Debian for which MariaDB provides packages to the public. The list is MariaDB-version-specific and dynamic (new Ubuntu releases happen often).
"rpm-based platforms" are all versions of CentOS, RHEL, and Fedora for which MariaDB provides packages to the public. The list is MariaDB-version-specific and dynamic (new Fedora releases happen often).
"zyp-based platforms" are all versions of openSUSE and SLES for which MariaDB provides packages to the public. They are in fact also RPM-based, but with zypper as a package manager.

Installation tests on all platforms ("install" step in old buildbot)

The test installs all packages, and makes sure the server is functioning. The test is performed on a clean image, without any specific non-default packages pre-installed. Only standard distro repos, Galera repository and the repository pointing at the packages under test (for debs) are configured. Galera repo of the corresponding version is needed in this test because while MariaDB public repositories have the galera library, package sets under test don't. Alternatively the test can download the galera package and install it from the local file.

Major upgrade tests on rpm-based platforms ("major-upgrade-server" step in kvm-rpm* builders in the old buildbot)

The test is performed on a clean image, without any specific non-default packages pre-installed.
Based on the version of the packages under test, the test determines the previous release line, configures the package manager to point at the official MariaDB repo of this release line, and installs selected packages from there. The test starts the server, makes sure it works, creates some data. Then it removes the installed packages and the repo configuration, configures it to point at a new Galera if needed, and installs new packages instead. It checks that there are no old packages left, starts the server, runs mysql_upgrade, makes sure the previously created data is accessible.

The test shouldn't fail if it can't find any previously released version in the repos, as it can happen naturally.

On historical reasons, the test isn't run for zyp-based systems. While adding it to new buildbot, there is no point to make this distinction, the test should run there as well.

Major upgrade tests on deb-based platforms ("upgrade1" and "upgrade2" steps in kvm-deb* builders in the old buildbot)

In the old buildbot, major upgrade tests on deb-based platforms are performed from fixed pre-defined versions of MySQL and MariaDB. The test starts a VM with a pre-installed server, creates some data, configures Galera library, runs new package installation and checks that the server is upgraded and the data is accessible. The tests can be replicated in the new buildbot if it's simpler for the start, but in future they should better be converted to the same approach as other upgrade tests take, installing a previous version dynamically. The pre-defined upgrade can still have a value if on some reason we want to test upgrade from a specific server version rather than the latest one, but it's an extension.

Minor upgrade tests on deb-based, rpm-based and zyp-based platforms ("minor-upgrade-*" steps in the old buildbot)

The test is performed on a clean image, without any specific non-default packages pre-installed.
Based on the version of the packages under test, the test determines the release line, configures the package manager to point at the official MariaDB repo of this release line, and installs selected packages from there. The test starts the server (for RPMs), makes sure it works, creates some data. Then it removes/modifies the repo configuration and installs/upgrades to new packages. It checks that there are no old packages left, starts the server, makes sure the previously created data is accessible.

The test shouldn't fail if it can't find any previously released version in the repos, as it can happen naturally.

There are several types of minor upgrade tests in the old buildbot:

• minor-upgrade-server: Installs and upgrades the minimal possible set of packages to get the server, like suggested in MariaDB KB, e.g. sudo apt-get install mariadb-server.
• minor-upgrade-deps: Installs and upgrades a limited set of packages. The point of the test is to make sure that the new minor version doesn't bring any new dependencies, so before running the upgrade, the test disables all repositories in the package manager configuration, even the default ones. Thus, if a new dependency appears, the upgrade will fail, which is the idea.
• minor-upgrade-all: installs all packages available in MariaDB repository (except for Columnstore, see below) and then upgrades them. The test performs some extra checks, particularly that the list of plugins and engines hasn't changed between the minor versions, and that all binaries are linked with the same libraries as before.
• minor-upgrade-columnstore: installs and upgrades only the server and Columnstore. That's a new hopefully temporary test, added because Columnstore isn't yet mature enough to be a part of "minor-upgrade-all". It should be removed in the future, but then extra logic will need to be added to minor-upgrade-all, with additional Columnstore-specific checks.

An additional category of tests which is currently performed as an extra step after installation on deb-based platforms is SST tests. I don't insist on adding them as they are to the new buildbot. It would be better to communicate with the Galera team and maybe re-implement the tests the way they want.