[MDBF-122] Automate VM creation process for LibVirt workers - Jira

Details

Type: Task
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: N/A
Fix Version/s: N/A
Component/s: None
Labels:
None

Description

For install and upgrade tests we use LibVirt managed KVMs.

We currently have one LibVirt worker that handles these types of workers (bg-bbw5-x64). We need an automatic way of creating VMs for all major operating systems and platforms that we are supporting (basically, for all the OSs and platforms for which we create Docker images - see https://github.com/MariaDB/mariadb.org-tools/blob/master/.github/workflows/bb_containers.yml)

VM creation

Create the image and configure LibVirt to use it. The current naming scheme consists in "buildbot-osname". Probably we need to adjust that.
Setup the base image (see VM installed packages below)
Configure LibVirt hook to always start from a clean VM

VM installed packages

The VMs need to be clean and have nothing installed except:

buildbot-worker - Python package that needs to be installed similar to the steps in the Dockerfile creation (https://github.com/MariaDB/mariadb.org-tools/blob/bd5c4a6a072e8c9232e203ec985efcf25843fd1d/buildbot.mariadb.org/ci_build_images/common.Dockerfile#L29)
configure a buildbot.tac that is unique for each worker similar to:

import fnmatch

import os

import sys

from twisted.application import service

from twisted.python.log import FileLogObserver

from twisted.python.log import ILogObserver

from buildbot_worker.bot import Worker

# setup worker

basedir = os.path.abspath(os.path.dirname(__file__))

application = service.Application('buildbot-worker')

application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit)

# and worker on the same process!

buildmaster_host = os.environ.get("BUILDMASTER", 'buildbot.mariadb.org')

port = int(os.environ.get("BUILDMASTER_PORT", 9990))

workername = os.environ.get("WORKERNAME", 'buildbot-ubuntu1804')

passwd = os.environ.get("WORKERPASS", 'PASSWORD')

# delete the password from the environ so that it is not leaked in the log

blacklist = os.environ.get("WORKER_ENVIRONMENT_BLACKLIST", "WORKERPASS").split()

for name in list(os.environ.keys()):

    for toremove in blacklist:

        if fnmatch.fnmatch(name, toremove):

            del os.environ[name]

keepalive = 600

umask = None

maxdelay = 300

allow_shutdown = None

maxretries = 10

s = Worker(buildmaster_host, port, workername, passwd, basedir,

           keepalive, umask=umask, maxdelay=maxdelay,

           allow_shutdown=allow_shutdown, maxRetries=maxretries)

s.setServiceParent(application)

Attention the WORKERNAME must match the name of the LibVirt VM. The password also needs to be set.

Ideally, include a possibility to easily add packages if needed.

Attachments

Issue Links

is part of

MDBF-121 Add remaining installation and upgrade tests

Closed

Activity

Ascending order - Click to sort in descending order

Vlad Bogolin created issue - 2020-09-02 12:49

Vlad Bogolin made changes - 2020-09-02 12:49

Field	Original Value	New Value
Link		This issue is part of ~~MDBF-121~~ [ ~~MDBF-121~~ ]

Sergei Golubchik made changes - 2022-02-01 08:33

Workflow	MFWA: Task Management Workflow [ 113176 ]	MariaDB v4 [ 162284 ]
Status	To Do [ 10003 ]	Open [ 1 ]

Vlad Bogolin made changes - 2022-02-04 14:53

Component/s		None [ 18105 ]
Fix Version/s		N/A [ 27305 ]

Vlad Bogolin made changes - 2022-02-04 15:03

Description

Have an automatic way of creating VMs for install and upgrade tests

For install and upgrade tests we use LibVirt managed KVMs.

We currently have one LibVirt worker that handles these types of workers (bg-bbw5-x64). We need an automatic way of creating VMs for all major operating systems and platforms that we are supporting (basically, for all the OSs and platforms for which we create Docker images - see https://github.com/MariaDB/mariadb.org-tools/blob/master/.github/workflows/bb_containers.yml)

VM creation

The VMs need to be clean and have nothing installed except:

* buildbot-worker - Python package that needs to be installed similar to the steps in the Dockerfile creation (https://github.com/MariaDB/mariadb.org-tools/blob/bd5c4a6a072e8c9232e203ec985efcf25843fd1d/buildbot.mariadb.org/ci_build_images/common.Dockerfile#L29)
* configure a buildbot.tac that is unique for each worker similar to:

{noformat}
import fnmatch
import os
import sys

from twisted.application import service
from twisted.python.log import FileLogObserver
from twisted.python.log import ILogObserver

from buildbot_worker.bot import Worker

# setup worker
basedir = os.path.abspath(os.path.dirname(__file__))
application = service.Application('buildbot-worker')

application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit)
# and worker on the same process!
buildmaster_host = os.environ.get("BUILDMASTER", 'buildbot.mariadb.org')
port = int(os.environ.get("BUILDMASTER_PORT", 9990))
workername = os.environ.get("WORKERNAME", 'buildbot-ubuntu1804')
passwd = os.environ.get("WORKERPASS", 'PASSWORD')

# delete the password from the environ so that it is not leaked in the log
blacklist = os.environ.get("WORKER_ENVIRONMENT_BLACKLIST", "WORKERPASS").split()
for name in list(os.environ.keys()):
    for toremove in blacklist:
        if fnmatch.fnmatch(name, toremove):
            del os.environ[name]

keepalive = 600
umask = None
maxdelay = 300
allow_shutdown = None
maxretries = 10

s = Worker(buildmaster_host, port, workername, passwd, basedir,
           keepalive, umask=umask, maxdelay=maxdelay,
           allow_shutdown=allow_shutdown, maxRetries=maxretries)
s.setServiceParent(application)
{noformat}

*Attention* the WORKERNAME must match the name of the LibVirt VM. The password also needs to be set.

Vlad Bogolin made changes - 2022-02-04 15:06

Description

For install and upgrade tests we use LibVirt managed KVMs.

We currently have one LibVirt worker that handles these types of workers (bg-bbw5-x64). We need an automatic way of creating VMs for all major operating systems and platforms that we are supporting (basically, for all the OSs and platforms for which we create Docker images - see https://github.com/MariaDB/mariadb.org-tools/blob/master/.github/workflows/bb_containers.yml)

h3. VM creation
Create the image and configure LibVirt to use it. The current naming scheme consists in "buildbot-osname". Probably we need to adjust that.
Setup the base image (see VM installed packages below)
Configure LibVirt hook to always start from a clean VM

h3. VM installed packages
The VMs need to be clean and have nothing installed except:

* buildbot-worker - Python package that needs to be installed similar to the steps in the Dockerfile creation (https://github.com/MariaDB/mariadb.org-tools/blob/bd5c4a6a072e8c9232e203ec985efcf25843fd1d/buildbot.mariadb.org/ci_build_images/common.Dockerfile#L29)
* configure a buildbot.tac that is unique for each worker similar to:

{noformat}
import fnmatch
import os
import sys

from twisted.application import service
from twisted.python.log import FileLogObserver
from twisted.python.log import ILogObserver

from buildbot_worker.bot import Worker

# setup worker
basedir = os.path.abspath(os.path.dirname(__file__))
application = service.Application('buildbot-worker')

application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit)
# and worker on the same process!
buildmaster_host = os.environ.get("BUILDMASTER", 'buildbot.mariadb.org')
port = int(os.environ.get("BUILDMASTER_PORT", 9990))
workername = os.environ.get("WORKERNAME", 'buildbot-ubuntu1804')
passwd = os.environ.get("WORKERPASS", 'PASSWORD')

# delete the password from the environ so that it is not leaked in the log
blacklist = os.environ.get("WORKER_ENVIRONMENT_BLACKLIST", "WORKERPASS").split()
for name in list(os.environ.keys()):
    for toremove in blacklist:
        if fnmatch.fnmatch(name, toremove):
            del os.environ[name]

keepalive = 600
umask = None
maxdelay = 300
allow_shutdown = None
maxretries = 10

s = Worker(buildmaster_host, port, workername, passwd, basedir,
           keepalive, umask=umask, maxdelay=maxdelay,
           allow_shutdown=allow_shutdown, maxRetries=maxretries)
s.setServiceParent(application)
{noformat}

*Attention* the WORKERNAME must match the name of the LibVirt VM. The password also needs to be set.

* Ideally, include a possibility to easily add packages if needed.

Vlad Bogolin made changes - 2022-02-04 15:07

Summary

Automate VM creation process

Automate VM creation process for LibVirt workers

Permalink

Faustin Lammler logged work - 2022-04-13 06:49

Time Spent:

1.25d

- POC for generating BB VM, next:
  - use ansible to deploy and configure VM;
  - improve the bb_worker_vm role;
  - test full destroy;
  - test deploy from scratch (ok with debian/ubuntu).

- Finish cloud-init POC (test other OS):
  - test centos;
  - test fedora;
  - handle case where VM are created manually (do not destroy them, and keep qemu hook in place);
  - create PR (https://gitlab.com/mariadb/sysadmin/-/merge_requests/28);
  - activate VM's with Vlad;
  - improve installation mechanism (pb with unbuntu18.04 and fedora35);
  - still pb with centos-8-stream.

Vlad Bogolin made changes - 2022-04-18 13:58

Assignee

Faustin Lammler [ faust ]

Vlad Bogolin added a comment - 2022-04-18 13:59

First deployment created by faust https://gitlab.com/mariadb/sysadmin/-/merge_requests/28

Vlad Bogolin added a comment - 2022-04-18 13:59 First deployment created by faust https://gitlab.com/mariadb/sysadmin/-/merge_requests/28

Vlad Bogolin made changes - 2022-04-18 13:59

Affects Version/s

N/A [ 27305 ]

Faustin Lammler made changes - 2022-04-20 06:50

Worklog Id	97375 [ 97375 ]
Remaining Estimate		0d [ 0 ]
Time Spent		1.25d [ 36000 ]

Faustin Lammler added a comment - 2022-04-20 06:51

POC for generating BB VM, next:
use ansible to deploy and configure VM;
improve the bb_worker_vm role;
test full destroy;
test deploy from scratch (ok with debian/ubuntu).

Finish cloud-init POC (test other OS):
test centos;
test fedora;
handle case where VM are created manually (do not destroy them, and keep qemu hook in place);
create PR (https://gitlab.com/mariadb/sysadmin/-/merge_requests/28);
activate VM's with Vlad;
improve installation mechanism (pb with unbuntu18.04 and fedora35);
still pb with centos-8-stream.

Faustin Lammler added a comment - 2022-04-20 06:51 POC for generating BB VM, next: use ansible to deploy and configure VM; improve the bb_worker_vm role; test full destroy; test deploy from scratch (ok with debian/ubuntu). Finish cloud-init POC (test other OS): test centos; test fedora; handle case where VM are created manually (do not destroy them, and keep qemu hook in place); create PR ( https://gitlab.com/mariadb/sysadmin/-/merge_requests/28 ); activate VM's with Vlad; improve installation mechanism (pb with unbuntu18.04 and fedora35); still pb with centos-8-stream.

Faustin Lammler added a comment - 2022-04-20 06:51 - edited

TODO:

other arch (ARM, PPC)
rhel7/8
centos 8

Faustin Lammler added a comment - 2022-04-20 06:51 - edited TODO: other arch (ARM, PPC) rhel7/8 centos 8

Faustin Lammler made changes - 2022-04-20 06:52

Status

Open [ 1 ]

In Progress [ 3 ]

Permalink

Faustin Lammler logged work - 2022-04-21 09:51

Time Spent:

3h

Deploy VMs on ARM.

Faustin Lammler added a comment - 2022-04-21 09:51

ARM is now correctly handled by the ansible role deployment system.
https://gitlab.com/mariadb/sysadmin/-/commit/c0667b4def7cb5947fb0e9cbab2a11a7e750c6ce

Following VMs are now available to test installation and upgrade on ARM in BB:

bb-centos-7
bb-debian-10
bb-debian-11
bb-debian-sid
bb-fedora-34
bb-fedora-35
bb-ubuntu-1804
bb-ubuntu-2004
bb-ubuntu-2204

Faustin Lammler added a comment - 2022-04-21 09:51 ARM is now correctly handled by the ansible role deployment system. https://gitlab.com/mariadb/sysadmin/-/commit/c0667b4def7cb5947fb0e9cbab2a11a7e750c6ce Following VMs are now available to test installation and upgrade on ARM in BB: bb-centos-7 bb-debian-10 bb-debian-11 bb-debian-sid bb-fedora-34 bb-fedora-35 bb-ubuntu-1804 bb-ubuntu-2004 bb-ubuntu-2204

Faustin Lammler made changes - 2022-04-21 09:52

Worklog Id	97438 [ 97438 ]
Time Spent	1.25d [ 36000 ]	1d 5h [ 46800 ]

Permalink

Faustin Lammler logged work - 2022-05-11 08:32

Time Spent:

1h

<No comment>

Faustin Lammler added a comment - 2022-05-12 08:31 - edited

Following VM are now also available for activation (ARM and x64)

fedora-36
centos-8-stream
centos-9-stream

TODO:

rhel-7
rhel-8

Faustin Lammler added a comment - 2022-05-12 08:31 - edited Following VM are now also available for activation (ARM and x64) fedora-36 centos-8-stream centos-9-stream TODO: rhel-7 rhel-8

Faustin Lammler made changes - 2022-05-12 08:32

Worklog Id	98391 [ 98391 ]
Time Spent	1d 5h [ 46800 ]	1.75d [ 50400 ]

Permalink

Faustin Lammler logged work - 2022-05-16 14:02

Time Spent:

3h

make deployment of rhel7/8 via ansible working.

Faustin Lammler added a comment - 2022-05-16 14:02 - edited

Rhel is now also available:

rhel 7 (x86 - i can't find rhel7 images for ARM so far.)
rhel 8 (x86/ARM)

Faustin Lammler added a comment - 2022-05-16 14:02 - edited Rhel is now also available: rhel 7 (x86 - i can't find rhel7 images for ARM so far.) rhel 8 (x86/ARM)

Faustin Lammler made changes - 2022-05-16 14:03

Worklog Id	98499 [ 98499 ]
Time Spent	1.75d [ 50400 ]	2d 1h [ 61200 ]

Faustin Lammler added a comment - 2022-05-18 08:14 - edited

~~TODO: SELinux enforcing~~

Faustin Lammler added a comment - 2022-05-18 08:14 - edited TODO: SELinux enforcing

Faustin Lammler added a comment - 2022-05-23 16:08

Rhel9 (x64/ARM) have been deployed.

Faustin Lammler added a comment - 2022-05-23 16:08 Rhel9 (x64/ARM) have been deployed.

Faustin Lammler added a comment - 2022-06-10 10:57

SELinux is now in Enforcing mode for libvirt VM:
https://gitlab.com/mariadb/sysadmin/-/commit/ccb90fccadbaa11bf45ea3f629a034f689b1cffb

Faustin Lammler added a comment - 2022-06-10 10:57 SELinux is now in Enforcing mode for libvirt VM: https://gitlab.com/mariadb/sysadmin/-/commit/ccb90fccadbaa11bf45ea3f629a034f689b1cffb

Permalink

Faustin Lammler logged work - 2022-06-10 16:40

Time Spent:

3h

See last today comment

Faustin Lammler made changes - 2022-06-10 16:40

Worklog Id	99606 [ 99606 ]
Time Spent	2d 1h [ 61200 ]	2.5d [ 72000 ]

Faustin Lammler added a comment - 2022-06-10 16:41

First attempt to deploy PPC libvirt workers.

same problem as for containers, the installation of cryptography module needs rust/libssl-dev and takes ages!;
try to generate wheel for PPC using manylinux containers;
this is not as simple as I though (there is some python magic that I do not understand);
I will try to do as they are in https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml
this can potentially simplify and drastically speed up quay.io image generations and libvirt deployments.

Faustin Lammler added a comment - 2022-06-10 16:41 First attempt to deploy PPC libvirt workers. same problem as for containers, the installation of cryptography module needs rust/libssl-dev and takes ages!; try to generate wheel for PPC using manylinux containers; this is not as simple as I though (there is some python magic that I do not understand); I will try to do as they are in https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml this can potentially simplify and drastically speed up quay.io image generations and libvirt deployments.

Faustin Lammler added a comment - 2022-06-14 17:16

ppc64le vm have been deployed, here is the list:

bb-db-p9-bbw1-centos-7
bb-db-p9-bbw1-centos-8-stream
bb-db-p9-bbw1-centos-9-stream
bb-db-p9-bbw1-debian-10
bb-db-p9-bbw1-debian-11
bb-db-p9-bbw1-debian-sid
bb-db-p9-bbw1-rhel-8
bb-db-p9-bbw1-rhel-9
bb-db-p9-bbw1-ubuntu-1804
bb-db-p9-bbw1-ubuntu-2004
bb-db-p9-bbw1-ubuntu-2204

Faustin Lammler added a comment - 2022-06-14 17:16 ppc64le vm have been deployed, here is the list: bb-db-p9-bbw1-centos-7 bb-db-p9-bbw1-centos-8-stream bb-db-p9-bbw1-centos-9-stream bb-db-p9-bbw1-debian-10 bb-db-p9-bbw1-debian-11 bb-db-p9-bbw1-debian-sid bb-db-p9-bbw1-rhel-8 bb-db-p9-bbw1-rhel-9 bb-db-p9-bbw1-ubuntu-1804 bb-db-p9-bbw1-ubuntu-2004 bb-db-p9-bbw1-ubuntu-2204

Faustin Lammler added a comment - 2022-08-09 07:05

The automated creation process is working for ppc/amd/arm.
Remaining: deploy s390x machines.

Faustin Lammler added a comment - 2022-08-09 07:05 The automated creation process is working for ppc/amd/arm. Remaining: deploy s390x machines.

Faustin Lammler made changes - 2022-08-09 07:05

Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Julien Fritsch made changes - 2025-01-21 15:10

Workflow

MariaDB v4 [ 162284 ]

MariaDB Foundation v1 [ 188409 ]

Vicențiu Ciorbaru made changes - 2025-01-24 08:05

Component/s

None [ 18105 ]

People

Assignee:: Faustin Lammler

Reporter:: Vlad Bogolin

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2020-09-02 12:49

Updated:: 2025-01-24 08:05

Resolved:: 2022-08-09 07:05

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

2.5d

MariaDB Foundation Development

Details

Description

VM creation

VM installed packages

Attachments

Issue Links

Activity

People

Dates

Time Tracking