Type:
Task
Priority:
Major
Resolution:
Fixed
Affects Version/s:
N/A
Component/s:
None
For install and upgrade tests we use LibVirt managed KVMs.
We currently have one LibVirt worker that handles these types of workers (bg-bbw5-x64). We need an automatic way of creating VMs for all major operating systems and platforms that we are supporting (basically, for all the OSs and platforms for which we create Docker images - see https://github.com/MariaDB/mariadb.org-tools/blob/master/.github/workflows/bb_containers.yml )
VM creation
Create the image and configure LibVirt to use it. The current naming scheme consists in "buildbot-osname". Probably we need to adjust that.
Setup the base image (see VM installed packages below)
Configure LibVirt hook to always start from a clean VM
VM installed packages
The VMs need to be clean and have nothing installed except:
import fnmatch
import os
import sys
from twisted.application import service
from twisted.python.log import FileLogObserver
from twisted.python.log import ILogObserver
from buildbot_worker.bot import Worker
# setup worker
basedir = os.path.abspath(os.path.dirname(__file__))
application = service.Application('buildbot-worker')
application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit)
# and worker on the same process!
buildmaster_host = os.environ.get("BUILDMASTER", 'buildbot.mariadb.org')
port = int(os.environ.get("BUILDMASTER_PORT", 9990))
workername = os.environ.get("WORKERNAME", 'buildbot-ubuntu1804')
passwd = os.environ.get("WORKERPASS", 'PASSWORD')
# delete the password from the environ so that it is not leaked in the log
blacklist = os.environ.get("WORKER_ENVIRONMENT_BLACKLIST", "WORKERPASS").split()
for name in list(os.environ.keys()):
for toremove in blacklist:
if fnmatch.fnmatch(name, toremove):
del os.environ[name]
keepalive = 600
umask = None
maxdelay = 300
allow_shutdown = None
maxretries = 10
s = Worker(buildmaster_host, port, workername, passwd, basedir,
keepalive, umask=umask, maxdelay=maxdelay,
allow_shutdown=allow_shutdown, maxRetries=maxretries)
s.setServiceParent(application)
Attention the WORKERNAME must match the name of the LibVirt VM. The password also needs to be set.
Ideally, include a possibility to easily add packages if needed.
is part of
MDBF-121
Add remaining installation and upgrade tests
Closed
{"report":{"fcp":738.1000000238419,"ttfb":172.10000002384186,"pageVisibility":"visible","entityId":91374,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"66c6ffa8-f3f8-43bb-96af-0984a8ba44fd","navigationType":0,"readyForUser":819.8000000119209,"redirectCount":0,"resourceLoadedEnd":670.5,"resourceLoadedStart":177.4000000357628,"resourceTiming":[{"duration":13,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":177.4000000357628,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":177.4000000357628,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":190.4000000357628,"responseStart":0,"secureConnectionStart":0},{"duration":13.399999976158142,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":177.60000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":177.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":191,"responseStart":0,"secureConnectionStart":0},{"duration":67.39999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/e9b27a47da5fb0f74a35acd57e9847fb-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":177.80000001192093,"connectEnd":177.80000001192093,"connectStart":177.80000001192093,"domainLookupEnd":177.80000001192093,"domainLookupStart":177.80000001192093,"fetchStart":177.80000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":177.80000001192093,"responseEnd":245.19999998807907,"responseStart":245.19999998807907,"secureConnectionStart":177.80000001192093},{"duration":176.60000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/s/c32eb0da7ad9831253f8397e6cc26afd-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":178,"connectEnd":178,"connectStart":178,"domainLookupEnd":178,"domainLookupStart":178,"fetchStart":178,"redirectEnd":0,"redirectStart":0,"requestStart":178,"responseEnd":354.60000002384186,"responseStart":354.60000002384186,"secureConnectionStart":178},{"duration":180,"initiatorType":"script","name":"https://jira.mariadb.org/s/bc0bcb146314416123c992714ee00ff7-CDN/lu2bv2/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":178.19999998807907,"connectEnd":178.19999998807907,"connectStart":178.19999998807907,"domainLookupEnd":178.19999998807907,"domainLookupStart":178.19999998807907,"fetchStart":178.19999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":178.19999998807907,"responseEnd":358.19999998807907,"responseStart":358.19999998807907,"secureConnectionStart":178.19999998807907},{"duration":180,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":178.5,"connectEnd":178.5,"connectStart":178.5,"domainLookupEnd":178.5,"domainLookupStart":178.5,"fetchStart":178.5,"redirectEnd":0,"redirectStart":0,"requestStart":178.5,"responseEnd":358.5,"responseStart":358.5,"secureConnectionStart":178.5},{"duration":180.4000000357628,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":178.5,"connectEnd":178.5,"connectStart":178.5,"domainLookupEnd":178.5,"domainLookupStart":178.5,"fetchStart":178.5,"redirectEnd":0,"redirectStart":0,"requestStart":178.5,"responseEnd":358.9000000357628,"responseStart":358.9000000357628,"secureConnectionStart":178.5},{"duration":209.10000002384186,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bv2/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":178.69999998807907,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":178.69999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":387.80000001192093,"responseStart":0,"secureConnectionStart":0},{"duration":180.5,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":178.9000000357628,"connectEnd":178.9000000357628,"connectStart":178.9000000357628,"domainLookupEnd":178.9000000357628,"domainLookupStart":178.9000000357628,"fetchStart":178.9000000357628,"redirectEnd":0,"redirectStart":0,"requestStart":178.9000000357628,"responseEnd":359.4000000357628,"responseStart":359.4000000357628,"secureConnectionStart":178.9000000357628},{"duration":208.89999997615814,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":179.10000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":179.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":388,"responseStart":0,"secureConnectionStart":0},{"duration":180.80000001192093,"initiatorType":"script","name":"https://jira.mariadb.org/s/719848dd97ebe0663199f49a3936487a-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":179.19999998807907,"connectEnd":179.19999998807907,"connectStart":179.19999998807907,"domainLookupEnd":179.19999998807907,"domainLookupStart":179.19999998807907,"fetchStart":179.19999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":179.19999998807907,"responseEnd":360,"responseStart":360,"secureConnectionStart":179.19999998807907},{"duration":260.9000000357628,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":180.19999998807907,"connectEnd":180.19999998807907,"connectStart":180.19999998807907,"domainLookupEnd":180.19999998807907,"domainLookupStart":180.19999998807907,"fetchStart":180.19999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":180.19999998807907,"responseEnd":441.10000002384186,"responseStart":441.10000002384186,"secureConnectionStart":180.19999998807907},{"duration":261,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":180.4000000357628,"connectEnd":180.4000000357628,"connectStart":180.4000000357628,"domainLookupEnd":180.4000000357628,"domainLookupStart":180.4000000357628,"fetchStart":180.4000000357628,"redirectEnd":0,"redirectStart":0,"requestStart":180.4000000357628,"responseEnd":441.4000000357628,"responseStart":441.4000000357628,"secureConnectionStart":180.4000000357628},{"duration":9.400000035762787,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":400,"connectEnd":400,"connectStart":400,"domainLookupEnd":400,"domainLookupStart":400,"fetchStart":400,"redirectEnd":0,"redirectStart":0,"requestStart":400,"responseEnd":409.4000000357628,"responseStart":409.4000000357628,"secureConnectionStart":400},{"duration":22.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2bv2/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":419.30000001192093,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":419.30000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":441.80000001192093,"responseStart":0,"secureConnectionStart":0},{"duration":22.30000001192093,"initiatorType":"link","name":"https://jira.mariadb.org/s/50bc9be5bfead1a25e72c1a9338c94f6-CDN/lu2bv2/820016/12ta74/e108c7645258ccb43280ed3404e3e949/_/download/contextbatch/css/com.atlassian.jira.plugins.jira-development-integration-plugin:0,-_super,-jira.view.issue,-jira.global,-jira.general,-jira.browse.project,-project.issue.navigator,-atl.general/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":419.80000001192093,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":419.80000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":442.10000002384186,"responseStart":0,"secureConnectionStart":0},{"duration":239.69999998807907,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":420.60000002384186,"connectEnd":420.60000002384186,"connectStart":420.60000002384186,"domainLookupEnd":420.60000002384186,"domainLookupStart":420.60000002384186,"fetchStart":420.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":420.60000002384186,"responseEnd":660.3000000119209,"responseStart":660.1999999880791,"secureConnectionStart":420.60000002384186},{"duration":249.39999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/53a43b6764f587426c7bb9a150184c00-CDN/lu2bv2/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":421.10000002384186,"connectEnd":421.10000002384186,"connectStart":421.10000002384186,"domainLookupEnd":421.10000002384186,"domainLookupStart":421.10000002384186,"fetchStart":421.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":421.10000002384186,"responseEnd":670.5,"responseStart":670.5,"secureConnectionStart":421.10000002384186},{"duration":122,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":517.6999999880791,"connectEnd":517.6999999880791,"connectStart":517.6999999880791,"domainLookupEnd":517.6999999880791,"domainLookupStart":517.6999999880791,"fetchStart":517.6999999880791,"redirectEnd":0,"redirectStart":0,"requestStart":517.6999999880791,"responseEnd":639.6999999880791,"responseStart":639.6999999880791,"secureConnectionStart":517.6999999880791}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":16,"responseStart":172,"responseEnd":176,"domLoading":175,"domInteractive":893,"domContentLoadedEventStart":893,"domContentLoadedEventEnd":943,"domComplete":1451,"loadEventStart":1451,"loadEventEnd":1453,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":858.1999999880791},{"name":"bigPipe.sidebar-id.end","time":859.1000000238419},{"name":"bigPipe.activity-panel-pipe-id.start","time":859.3000000119209},{"name":"bigPipe.activity-panel-pipe-id.end","time":861.6999999880791},{"name":"activityTabFullyLoaded","time":954}],"measures":[],"correlationId":"3b60d51f0ef55b","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":110,"dbReadsTimeInMs":11,"dbConnsTimeInMs":18,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}