Uploaded image for project: 'MariaDB Foundation Development'
  1. MariaDB Foundation Development
  2. MDBF-122

Automate VM creation process for LibVirt workers

Details

    • Task
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • N/A
    • N/A
    • None
    • None

    Description

      For install and upgrade tests we use LibVirt managed KVMs.

      We currently have one LibVirt worker that handles these types of workers (bg-bbw5-x64). We need an automatic way of creating VMs for all major operating systems and platforms that we are supporting (basically, for all the OSs and platforms for which we create Docker images - see https://github.com/MariaDB/mariadb.org-tools/blob/master/.github/workflows/bb_containers.yml)

      VM creation

      Create the image and configure LibVirt to use it. The current naming scheme consists in "buildbot-osname". Probably we need to adjust that.
      Setup the base image (see VM installed packages below)
      Configure LibVirt hook to always start from a clean VM

      VM installed packages

      The VMs need to be clean and have nothing installed except:

      import fnmatch
      import os
      import sys
       
      from twisted.application import service
      from twisted.python.log import FileLogObserver
      from twisted.python.log import ILogObserver
       
      from buildbot_worker.bot import Worker
       
      # setup worker
      basedir = os.path.abspath(os.path.dirname(__file__))
      application = service.Application('buildbot-worker')
       
       
      application.setComponent(ILogObserver, FileLogObserver(sys.stdout).emit)
      # and worker on the same process!
      buildmaster_host = os.environ.get("BUILDMASTER", 'buildbot.mariadb.org')
      port = int(os.environ.get("BUILDMASTER_PORT", 9990))
      workername = os.environ.get("WORKERNAME", 'buildbot-ubuntu1804')
      passwd = os.environ.get("WORKERPASS", 'PASSWORD')
       
      # delete the password from the environ so that it is not leaked in the log
      blacklist = os.environ.get("WORKER_ENVIRONMENT_BLACKLIST", "WORKERPASS").split()
      for name in list(os.environ.keys()):
          for toremove in blacklist:
              if fnmatch.fnmatch(name, toremove):
                  del os.environ[name]
       
      keepalive = 600
      umask = None
      maxdelay = 300
      allow_shutdown = None
      maxretries = 10
       
      s = Worker(buildmaster_host, port, workername, passwd, basedir,
                 keepalive, umask=umask, maxdelay=maxdelay,
                 allow_shutdown=allow_shutdown, maxRetries=maxretries)
      s.setServiceParent(application)
      

      Attention the WORKERNAME must match the name of the LibVirt VM. The password also needs to be set.

      • Ideally, include a possibility to easily add packages if needed.

      Attachments

        Issue Links

          Activity

            Rhel9 (x64/ARM) have been deployed.

            faust Faustin Lammler added a comment - Rhel9 (x64/ARM) have been deployed.
            faust Faustin Lammler added a comment - SELinux is now in Enforcing mode for libvirt VM: https://gitlab.com/mariadb/sysadmin/-/commit/ccb90fccadbaa11bf45ea3f629a034f689b1cffb

            First attempt to deploy PPC libvirt workers.

            • same problem as for containers, the installation of cryptography module needs rust/libssl-dev and takes ages!;
            • try to generate wheel for PPC using manylinux containers;
            • this is not as simple as I though (there is some python magic that I do not understand);
            • I will try to do as they are in https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml
            • this can potentially simplify and drastically speed up quay.io image generations and libvirt deployments.
            faust Faustin Lammler added a comment - First attempt to deploy PPC libvirt workers. same problem as for containers, the installation of cryptography module needs rust/libssl-dev and takes ages!; try to generate wheel for PPC using manylinux containers; this is not as simple as I though (there is some python magic that I do not understand); I will try to do as they are in https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml this can potentially simplify and drastically speed up quay.io image generations and libvirt deployments.

            ppc64le vm have been deployed, here is the list:

            • bb-db-p9-bbw1-centos-7
            • bb-db-p9-bbw1-centos-8-stream
            • bb-db-p9-bbw1-centos-9-stream
            • bb-db-p9-bbw1-debian-10
            • bb-db-p9-bbw1-debian-11
            • bb-db-p9-bbw1-debian-sid
            • bb-db-p9-bbw1-rhel-8
            • bb-db-p9-bbw1-rhel-9
            • bb-db-p9-bbw1-ubuntu-1804
            • bb-db-p9-bbw1-ubuntu-2004
            • bb-db-p9-bbw1-ubuntu-2204
            faust Faustin Lammler added a comment - ppc64le vm have been deployed, here is the list: bb-db-p9-bbw1-centos-7 bb-db-p9-bbw1-centos-8-stream bb-db-p9-bbw1-centos-9-stream bb-db-p9-bbw1-debian-10 bb-db-p9-bbw1-debian-11 bb-db-p9-bbw1-debian-sid bb-db-p9-bbw1-rhel-8 bb-db-p9-bbw1-rhel-9 bb-db-p9-bbw1-ubuntu-1804 bb-db-p9-bbw1-ubuntu-2004 bb-db-p9-bbw1-ubuntu-2204

            The automated creation process is working for ppc/amd/arm.
            Remaining: deploy s390x machines.

            faust Faustin Lammler added a comment - The automated creation process is working for ppc/amd/arm. Remaining: deploy s390x machines.

            People

              faust Faustin Lammler
              vladbogo Vlad Bogolin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 2.5d
                  2.5d