Eliminate (or reduce) multi-master setup

Scope

This task covers the refactoring effort to drop (or significantly reduce) the number of buildbot masters. This task will simplify future development and allow an easier way to ensure proper buildbot uptime.

History

During buildbot.mariadb.org's history there have been issues with scalability based on number of log-lines inserted / second. After a certain point, buildbot masters would freeze up and not be able to issue any new worker commands (start build, run tests, etc).

The only identified fix at the time was to scale to a multi-master setup, that introduced significant complexity to the system.

Root cause identified

Following performance testing, we have identified that in fact the culprit is MTRLogObserver capturing the event loop CPU cycles (which are limited to a single thread in python, by doing line-by-line regex matching and other SQL generation logic) and also waiting for IO (SQL queries, insert statements) in a "non-interruptible" callback.

By removing MTRLogObserver from the test runs, we are able to scale to >1 million lines processed per second on a single master (as opposed to less than 1000/s during MTR runs).

Technical details

In order to reduce risk during buildbot version upgrade, we should remove the technical complexity before upgrading. Hence this task blocks the upgrade.
Main changes

• Builder factories should not be defined in any master.cfg, but rather a separate module.
• Builder definitions should import factories within master.cfg
• Finally, move builder definitions into fewer masters.
• There will likely be 4 masters left, as opposed to the current 13:
  • master-protected-branches (to ensure no impact on developers ability to push)
  • master-web (to ensure a stable UI interface)
  • master-other-server-builds (for all other server tests)
  • master-third-party-projects (for other projects not testing MariaDB Server directly, eg. galera, Connector C, ecosystem builds etc.)