[MDBF-122] Automate VM creation process for LibVirt workers Created: 2020-09-02  Updated: 2022-08-09  Resolved: 2022-08-09

Status: Closed
Project: MariaDB Foundation Development
Component/s: None
Affects Version/s: N/A
Fix Version/s: N/A

Type: Task Priority: Major
Reporter: Vlad Bogolin Assignee: Faustin Lammler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: 0d
Time Spent: 2.5d
Original Estimate: Not Specified

Issue Links:
PartOf
is part of MDBF-121 Add remaining installation and upgrad... Closed

 Description   

For install and upgrade tests we use LibVirt managed KVMs.

We currently have one LibVirt worker that handles these types of workers (bg-bbw5-x64). We need an automatic way of creating VMs for all major operating systems and platforms that we are supporting (basically, for all the OSs and platforms for which we create Docker images - see https://github.com/MariaDB/mariadb.org-tools/blob/master/.github/workflows/bb_containers.yml)

VM creation

Create the image and configure LibVirt to use it. The current naming scheme consists in "buildbot-osname". Probably we need to adjust that.
Setup the base image (see VM installed packages below)
Configure LibVirt hook to always start from a clean VM

VM installed packages

The VMs need to be clean and have nothing installed except:

import fnmatch
 
import os
 
import sys
 
 
 
from twisted.application import service
 
from twisted.python.log import FileLogObserver
 
from twisted.python.log import ILogObserver
 
 
 
from buildbot_worker.bot import Worker
 
 
 
# setup worker
 
basedir = os.path.abspath(os.path.dirname(__file__))
 
application = service.Application('buildbot-worker')
 
 
 
 
 
application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit)
 
# and worker on the same process!
 
buildmaster_host = os.environ.get("BUILDMASTER", 'buildbot.mariadb.org')
 
port = int(os.environ.get("BUILDMASTER_PORT", 9990))
 
workername = os.environ.get("WORKERNAME", 'buildbot-ubuntu1804')
 
passwd = os.environ.get("WORKERPASS", 'PASSWORD')
 
 
 
# delete the password from the environ so that it is not leaked in the log
 
blacklist = os.environ.get("WORKER_ENVIRONMENT_BLACKLIST", "WORKERPASS").split()
 
for name in list(os.environ.keys()):
 
    for toremove in blacklist:
 
        if fnmatch.fnmatch(name, toremove):
 
            del os.environ[name]
 
 
 
keepalive = 600
 
umask = None
 
maxdelay = 300
 
allow_shutdown = None
 
maxretries = 10
 
 
 
s = Worker(buildmaster_host, port, workername, passwd, basedir,
 
           keepalive, umask=umask, maxdelay=maxdelay,
 
           allow_shutdown=allow_shutdown, maxRetries=maxretries)
 
s.setServiceParent(application)

Attention the WORKERNAME must match the name of the LibVirt VM. The password also needs to be set.

  • Ideally, include a possibility to easily add packages if needed.

 Comments   
Comment by Vlad Bogolin [ 2022-04-18 ]

First deployment created by faust https://gitlab.com/mariadb/sysadmin/-/merge_requests/28

Comment by Faustin Lammler [ 2022-04-20 ]
  • POC for generating BB VM, next:
  • use ansible to deploy and configure VM;
  • improve the bb_worker_vm role;
  • test full destroy;
  • test deploy from scratch (ok with debian/ubuntu).
  • Finish cloud-init POC (test other OS):
  • test centos;
  • test fedora;
  • handle case where VM are created manually (do not destroy them, and keep qemu hook in place);
  • create PR (https://gitlab.com/mariadb/sysadmin/-/merge_requests/28);
  • activate VM's with Vlad;
  • improve installation mechanism (pb with unbuntu18.04 and fedora35);
  • still pb with centos-8-stream.
Comment by Faustin Lammler [ 2022-04-20 ]

TODO:

  • other arch (ARM, PPC)
  • rhel7/8
  • centos 8
Comment by Faustin Lammler [ 2022-04-21 ]

ARM is now correctly handled by the ansible role deployment system.
https://gitlab.com/mariadb/sysadmin/-/commit/c0667b4def7cb5947fb0e9cbab2a11a7e750c6ce

Following VMs are now available to test installation and upgrade on ARM in BB:

  • bb-centos-7
  • bb-debian-10
  • bb-debian-11
  • bb-debian-sid
  • bb-fedora-34
  • bb-fedora-35
  • bb-ubuntu-1804
  • bb-ubuntu-2004
  • bb-ubuntu-2204
Comment by Faustin Lammler [ 2022-05-12 ]

Following VM are now also available for activation (ARM and x64)

  • fedora-36
  • centos-8-stream
  • centos-9-stream

TODO:

  • rhel-7
  • rhel-8
Comment by Faustin Lammler [ 2022-05-16 ]

Rhel is now also available:

  • rhel 7 (x86 - i can't find rhel7 images for ARM so far.)
  • rhel 8 (x86/ARM)
Comment by Faustin Lammler [ 2022-05-18 ]

TODO: SELinux enforcing

Comment by Faustin Lammler [ 2022-05-23 ]

Rhel9 (x64/ARM) have been deployed.

Comment by Faustin Lammler [ 2022-06-10 ]

SELinux is now in Enforcing mode for libvirt VM:
https://gitlab.com/mariadb/sysadmin/-/commit/ccb90fccadbaa11bf45ea3f629a034f689b1cffb

Comment by Faustin Lammler [ 2022-06-10 ]

First attempt to deploy PPC libvirt workers.

  • same problem as for containers, the installation of cryptography module needs rust/libssl-dev and takes ages!;
  • try to generate wheel for PPC using manylinux containers;
  • this is not as simple as I though (there is some python magic that I do not understand);
  • I will try to do as they are in https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml
  • this can potentially simplify and drastically speed up quay.io image generations and libvirt deployments.
Comment by Faustin Lammler [ 2022-06-14 ]

ppc64le vm have been deployed, here is the list:

  • bb-db-p9-bbw1-centos-7
  • bb-db-p9-bbw1-centos-8-stream
  • bb-db-p9-bbw1-centos-9-stream
  • bb-db-p9-bbw1-debian-10
  • bb-db-p9-bbw1-debian-11
  • bb-db-p9-bbw1-debian-sid
  • bb-db-p9-bbw1-rhel-8
  • bb-db-p9-bbw1-rhel-9
  • bb-db-p9-bbw1-ubuntu-1804
  • bb-db-p9-bbw1-ubuntu-2004
  • bb-db-p9-bbw1-ubuntu-2204
Comment by Faustin Lammler [ 2022-08-09 ]

The automated creation process is working for ppc/amd/arm.
Remaining: deploy s390x machines.

Generated at Thu Feb 08 03:35:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.