I think we should first create a prototype and measure performance to make sure we are not losing anything.

I've got a reasonable docker build/test environment. I'll attach some scripts in the next few days. Its easy enough to add a few at a time. First tip is to use overlayfs for the docker backing filesystem.

Host

Sysctls

Running a shared kernel between contains will hit on these limits in test:

/etc/sysctl.d/80-mysql-limits.conf

fs.aio-max-nr=16777216

/etc/sysctl.d/30-mysql-vm-huge-pages.conf

vm.nr_hugepages=2048

Docker

overlayfs is better performing that the default aufs

On an ubuntu-16.04 base the following was sufficent to switch

/etc/systemd/system/docker.service.d/overlay.conf

[Service]

ExecStart=

ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=overlay

The following was useful to inject common environment variables into the docker executable (like proxy servers)

/etc/systemd/system/docker.service.d/env.conf

[Service]

EnvironmentFile=/etc/environment

Docker build containers

Overview

The attached scripts are based on a Jenkins build environment. They where more aimed at building cross MySQL variants and across many compilers rather than cross OS variant. source maps to a checked out source and hopefully the environment variables used are obvious to map to buildbot.

Important bits are that docker containers use VOLUMES maps to the host (like artefacts) for preserving build logs (and ccache) between builds.

docker_run_build.sh creates a image, and then runs it as a container. Dockerfile is an ubuntu-16.04 base image and only one of these should be needed per (arch,distro). Bits in here are needed for non-mariadb MySQL variants

build_mysql.sh runs from within the container and calls func_test.sh to run functional tests.

Architectures

The Docker support for multiple architectures in pulling images via docker hub is currently a bit limited though being worked on.

Workaround like running the following to establish a consistent name on Power machines to those on intel so the same Dockerfile can be used helps.

Pulling from dockerhub ppc64le images https://hub.docker.com/r/ppc64le/

    docker pull ppc64le/ubuntu:16.04

    docker tag ppc64le/ubuntu:16.04 ubuntu:16.04

The same applies for Debian, Fedora, Centos, Opensuse

To build from scratch would invole using something like https://github.com/moby/moby/blob/master/contrib/mkimage-debootstrap.sh and

 chmod 755 debootstrap.sh

sudo ./debootstrap.sh ubuntu –components=main,universe trusty

sudo tar -C ubuntu -c . | docker import - ubuntu:14.04.4

docker tag ubuntu:14.04.4-new ubuntu:14.04

With Fedora something like spins/kickstarts could be used to generate images.

PPC64 (Big endian) - docker for BE doesn't exist

edit: no BE docker.

Any question let me know. Happy to provide guidance.

Note MDEV-658 - use of debhelper from jessie-backports and manual edits of dh_strip.

Note: MDEV-4646 / MDEV-11258 - CMake-3.7.0+ (pref 3.8.0+) on RPM builders to generate debug packages however keep one builder at distro version.

I have a functional implementation of latest Buildbot using Docker for targeting Linux builds:
https://buildbot.mariadb.org

Code/config available here:
https://github.com/MariaDB/mariadb.org-tools/tree/master/buildbot.mariadb.org

Dockerfiles for current builders:
https://github.com/MariaDB/mariadb.org-tools/tree/master/buildbot.mariadb.org/dockerfiles

Also see associated README.md in the repo for details.

So far no major drawbacks from using Docker have been encountered. The only downside is that we can't really use it to target anything else besides Linux x86 builds. For all other architectures, we need to revert back to KVM (any other options suggested worth a try?).