Type:
Task
Priority:
Major
Resolution:
Fixed
Affects Version/s:
N/A
Component/s:
None
For install and upgrade tests we use LibVirt managed KVMs.
We currently have one LibVirt worker that handles these types of workers (bg-bbw5-x64). We need an automatic way of creating VMs for all major operating systems and platforms that we are supporting (basically, for all the OSs and platforms for which we create Docker images - see https://github.com/MariaDB/mariadb.org-tools/blob/master/.github/workflows/bb_containers.yml )
VM creation
Create the image and configure LibVirt to use it. The current naming scheme consists in "buildbot-osname". Probably we need to adjust that.
Setup the base image (see VM installed packages below)
Configure LibVirt hook to always start from a clean VM
VM installed packages
The VMs need to be clean and have nothing installed except:
import fnmatch
import os
import sys
from twisted.application import service
from twisted.python.log import FileLogObserver
from twisted.python.log import ILogObserver
from buildbot_worker.bot import Worker
# setup worker
basedir = os.path.abspath(os.path.dirname(__file__))
application = service.Application('buildbot-worker')
application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit)
# and worker on the same process!
buildmaster_host = os.environ.get("BUILDMASTER", 'buildbot.mariadb.org')
port = int(os.environ.get("BUILDMASTER_PORT", 9990))
workername = os.environ.get("WORKERNAME", 'buildbot-ubuntu1804')
passwd = os.environ.get("WORKERPASS", 'PASSWORD')
# delete the password from the environ so that it is not leaked in the log
blacklist = os.environ.get("WORKER_ENVIRONMENT_BLACKLIST", "WORKERPASS").split()
for name in list(os.environ.keys()):
for toremove in blacklist:
if fnmatch.fnmatch(name, toremove):
del os.environ[name]
keepalive = 600
umask = None
maxdelay = 300
allow_shutdown = None
maxretries = 10
s = Worker(buildmaster_host, port, workername, passwd, basedir,
keepalive, umask=umask, maxdelay=maxdelay,
allow_shutdown=allow_shutdown, maxRetries=maxretries)
s.setServiceParent(application)
Attention the WORKERNAME must match the name of the LibVirt VM. The password also needs to be set.
Ideally, include a possibility to easily add packages if needed.
is part of
MDBF-121
Add remaining installation and upgrade tests
Closed
{"report":{"fcp":870.5,"ttfb":269.39999997615814,"pageVisibility":"visible","entityId":91374,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"ffb9b863-6910-4b70-999f-e509bdd22e20","navigationType":0,"readyForUser":946.8999999761581,"redirectCount":0,"resourceLoadedEnd":616.3000000119209,"resourceLoadedStart":282.30000001192093,"resourceTiming":[{"duration":50.39999997615814,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":282.30000001192093,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":282.30000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":332.69999998807907,"responseStart":0,"secureConnectionStart":0},{"duration":50.40000003576279,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":282.69999998807907,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":282.69999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":333.10000002384186,"responseStart":0,"secureConnectionStart":0},{"duration":142,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":282.89999997615814,"connectEnd":282.89999997615814,"connectStart":282.89999997615814,"domainLookupEnd":282.89999997615814,"domainLookupStart":282.89999997615814,"fetchStart":282.89999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":334.30000001192093,"responseEnd":424.89999997615814,"responseStart":347.5,"secureConnectionStart":282.89999997615814},{"duration":292.4000000357628,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":283.89999997615814,"connectEnd":283.89999997615814,"connectStart":283.89999997615814,"domainLookupEnd":283.89999997615814,"domainLookupStart":283.89999997615814,"fetchStart":283.89999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":334.69999998807907,"responseEnd":576.3000000119209,"responseStart":348.39999997615814,"secureConnectionStart":283.89999997615814},{"duration":70.39999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":284.10000002384186,"connectEnd":284.10000002384186,"connectStart":284.10000002384186,"domainLookupEnd":284.10000002384186,"domainLookupStart":284.10000002384186,"fetchStart":284.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":339.80000001192093,"responseEnd":354.5,"responseStart":353,"secureConnectionStart":284.10000002384186},{"duration":77.90000003576279,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":284.19999998807907,"connectEnd":284.19999998807907,"connectStart":284.19999998807907,"domainLookupEnd":284.19999998807907,"domainLookupStart":284.19999998807907,"fetchStart":284.19999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":349.69999998807907,"responseEnd":362.10000002384186,"responseStart":360.60000002384186,"secureConnectionStart":284.19999998807907},{"duration":88.30000001192093,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":284.39999997615814,"connectEnd":284.39999997615814,"connectStart":284.39999997615814,"domainLookupEnd":284.39999997615814,"domainLookupStart":284.39999997615814,"fetchStart":284.39999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":352.10000002384186,"responseEnd":372.69999998807907,"responseStart":371.5,"secureConnectionStart":284.39999997615814},{"duration":66.79999995231628,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":284.60000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":284.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":351.39999997615814,"responseStart":0,"secureConnectionStart":0},{"duration":91,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":284.69999998807907,"connectEnd":284.69999998807907,"connectStart":284.69999998807907,"domainLookupEnd":284.69999998807907,"domainLookupStart":284.69999998807907,"fetchStart":284.69999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":354.30000001192093,"responseEnd":375.69999998807907,"responseStart":373.89999997615814,"secureConnectionStart":284.69999998807907},{"duration":69,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":284.80000001192093,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":284.80000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":353.80000001192093,"responseStart":0,"secureConnectionStart":0},{"duration":91.10000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":285,"connectEnd":285,"connectStart":285,"domainLookupEnd":285,"domainLookupStart":285,"fetchStart":285,"redirectEnd":0,"redirectStart":0,"requestStart":356.30000001192093,"responseEnd":376.10000002384186,"responseStart":374.5,"secureConnectionStart":285},{"duration":329.60000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":286.69999998807907,"connectEnd":286.69999998807907,"connectStart":286.69999998807907,"domainLookupEnd":286.69999998807907,"domainLookupStart":286.69999998807907,"fetchStart":286.69999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":508.10000002384186,"responseEnd":616.3000000119209,"responseStart":611.6999999880791,"secureConnectionStart":286.69999998807907},{"duration":328.80000001192093,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":286.80000001192093,"connectEnd":286.80000001192093,"connectStart":286.80000001192093,"domainLookupEnd":286.80000001192093,"domainLookupStart":286.80000001192093,"fetchStart":286.80000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":507.89999997615814,"responseEnd":615.6000000238419,"responseStart":610.3999999761581,"secureConnectionStart":286.80000001192093},{"duration":102.69999998807907,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":547,"connectEnd":547,"connectStart":547,"domainLookupEnd":547,"domainLookupStart":547,"fetchStart":547,"redirectEnd":0,"redirectStart":0,"requestStart":616,"responseEnd":649.6999999880791,"responseStart":648,"secureConnectionStart":547}],"fetchStart":0,"domainLookupStart":42,"domainLookupEnd":94,"connectStart":94,"connectEnd":113,"secureConnectionStart":102,"requestStart":114,"responseStart":269,"responseEnd":285,"domLoading":272,"domInteractive":1003,"domContentLoadedEventStart":1003,"domContentLoadedEventEnd":1050,"domComplete":1459,"loadEventStart":1459,"loadEventEnd":1459,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":983.3000000119209},{"name":"bigPipe.sidebar-id.end","time":984},{"name":"bigPipe.activity-panel-pipe-id.start","time":984.1000000238419},{"name":"bigPipe.activity-panel-pipe-id.end","time":985.1999999880791},{"name":"activityTabFullyLoaded","time":1067.5}],"measures":[],"correlationId":"aeaf44e7c4a9bb","effectiveType":"4g","downlink":9.2,"rtt":0,"serverDuration":102,"dbReadsTimeInMs":18,"dbConnsTimeInMs":25,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}