As a quick workaround, I've changed dist-upgrade to minor upgrade (install current packages over the last release of the same major version). Since it does not involve system upgrade, there is no need to disable default repos.

Still, it would be good to have dist-upgrade among tests, because sometimes it behaves differently from install. I'll keep it open until we have a good solution for it.

I can install libnuma1 on whatever VMs it is needed on.

I understand, but I also want to find a general solution for the problem. Running dist-upgrade without default repos is not the right thing to do, it was done this way because I didn't know any better (and still don't).

Thinking about it, maybe it's actually a good thing that the test is run without repos. We just shouldn't run it for pre-RC versions.
This way we can timely detect if our GA packages suddenly get a new dependency (which they shouldn't normally have).

So, the plan is, to have both

• the minor upgrade test (apt-get install mariadb-server X.Y.Z+1 over X.Y.Z, which would run for all versions;
• the dist-upgrade test apt-get dist-upgrade over X.Y.Z with the repo pointing at X.Y.Z+1, which would run for GA+ or RC+ versions, and only for main trees.

Unfortunately, buildbot does not know which are GA and which are not, so it will have to be configured manually, but new major releases don't happen so often, so it's not a big problem.

Thus, there should be no need to pre-install libnuma1. The minor upgrade test runs with default repos enabled, so it should be able to pull the library as a dependency. When the release approaches GA, the previously released version should already have all dependencies, so they will be pulled when we install X.Y.Z (it happens while default repos are still enabled). We disable the repos after that, but it shouldn't matter, because there must be no new dependencies for X.Y.Z+1.

There is now a variant of minor-upgrade test "deps" (stands for dependencies) which run the upgrade with default repos disabled, to ensure that no dependencies are introduced in a post-RC release.
It could be changed to run dist-upgrade on Debians.
However, given that we are planning to switch to the new buildbot, I'm not sure it is worth doing now. Decreasing priority (and will probably close it as "won't do" when/if we really switch to the new buildbot).