[MDBF-518] Diffrences between Debian Salsa and Buildbot testing Created: 2022-12-19  Updated: 2023-01-10

Status: Open
Project: MariaDB Foundation Development
Component/s: Buildbot, debian
Affects Version/s: N/A
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Tuukka Pasanen Assignee: Tuukka Pasanen
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: 0d
Time Spent: 5h
Original Estimate: Not Specified


 Description   

Debian Salsa-CI and Buildbot has lot's of differences and they are documented here. These are only from Debian version 10.6 and if there is some major like 10.11 then it's mentioned.

Source pre-tests

These test

gnitpick

This test uses Seravo Gnitpick which nitpick commits language.

Command which is run is:

gnitpick --target-repository https://salsa.debian.org/mariadb-team/mariadb-server.git --target-branch debian/latest

This test can fail

packaging-fix-checks

This test depends on Debian package git devscripts python3-debian
Checks if there is no unaddressed patches in debian/patches directory.

if [ "$(find debian/patches/ -type f -not -name series | wc -l)" -eq "$(cat debian/patches/series | wc -l)" ]

and after that it runs wrap-and-sort-command which does (taken from man page):

wrap-and-sort wraps the package lists in Debian control files. By default the lists will only split into multiple lines if the entries are longer than the maximum line length limit of 79 characters. wrap-and-sort sorts the package lists in Debian

command which does this is:

wrap-and-sort -a

What it mostly does it makes sure that files are in alphabetical order under debian-directory and takes care or debian/control that it does not contain rows over 80-bytes long

Building

build bullseye-backports

This is not the test but builds Debian 11 version of MariaDB and it's used on later tests after build is done

build buster-backports

This is not the test but builds Debian 10 version of MariaDB and it's used on later tests after build is done. This one is available on MariaDB 10.6 and after that there can't be Buster version because of change of dependencies.

Upgrading and install

autopkgtest

Autopkg is different from other tests as it is run inside of LXC virtualization (inside of docker container) and not straightly inside docker container as rest of the tests. Good thing about this is that there is possibility to use systemd and goodies. Autopkg test runs tests that are located under debian/tests/

Git repository for LXC image is https://salsa.debian.org/salsa-ci-team/autopkgtest-lxc

LXC image used is (for Sid): https://salsa.debian.org/salsa-ci-team/autopkgtest-lxc/-/jobs/artifacts/master/raw/artifacts/lxc.tar?job=sid

Command which is executed is (for debci you can read here:

su -P debci -c "debci localtest $WORKING_DIR/*.changes --suite ${RELEASE} --logs-dir ${DEBCI_LOG_PATH}" || ( ret=$?; [ $ret -eq 8 ] || [ $ret -eq 2 ] )

Where RELEASE is something like buster and DEBCI_LOG_PATH comes from Salsa-CI env variables. Under debian/tests there is currently two tests and then control file where dependencies are collected for these tests.

smoke test

Smoke test creates use and database and test weather compression plugins are available then it checks if RocksDB is available

upstream test

It runs MariaDB test suite and it has unstable-tests for tests that does not work

piuparts

Piuparts is QA tool which installs and uninstalls all the deb packages created in building process

Piuparts needs some of tooling before running inside docker. CHROOT is not chroot as we are inside container is set up as CHROOT_PATH="/tmp/debian-chroot" 

mknod -m 666 ${CHROOT_PATH}/dev/urandom c 1 9
 
mkdir -p /srv/local-apt-repository/ && cp -a ${WORKING_DIR}/*.deb /srv/local-apt-repository/ && /usr/lib/local-apt-repository/rebuild
 
mkdir -p ${CHROOT_PATH}/etc-target/apt/sources.list.d ${CHROOT_PATH}/etc-target/apt/preferences.d
 
cp -Hv /etc/apt/sources.list.d/local-apt-repository.list ${CHROOT_PATH}/etc-target/apt/sources.list.d/
 
cp -aTLv /etc/apt/preferences.d  ${CHROOT_PATH}/etc-target/apt/preferences.d
 
cp -aTLv /srv/local-apt-repository ${CHROOT_PATH}/srv/local-apt-repository
 
cp -aTLv /var/lib/local-apt-repository/ ${CHROOT_PATH}/var/lib/local-apt-repository/
 
add_extra_repository.sh -v -e "${SALSA_CI_EXTRA_REPOSITORY}" -k "${SALSA_CI_EXTRA_REPOSITORY_KEY}" -t "${CHROOT_PATH}/etc-target"
 
sed  '/127.0.0.1/s/localhost/pipeline.salsa.debian.org localhost/' /etc/hosts > ${CHROOT_PATH}/etc/hosts
 
PIUPARTS_DISTRIBUTION_ARG="--distribution $RELEASE"

After these it's run against build deb packages:

(for PACKAGE in $(ls ${WORKING_DIR}/*.deb); do
 
        piuparts --mirror "${SALSA_CI_MIRROR} ${SALSA_CI_COMPONENTS}" ${SALSA_CI_PIUPARTS_ARGS} --scriptsdir /etc/piuparts/scripts --allow-database --warn-on-leftovers-after-purge --hard-link -e ${CHROOT_PATH} ${PIUPARTS_DISTRIBUTION_ARG} ${PACKAGE}
 
      done) | filter-output

blhc

BLHC stands for build log hardening check, checks build logs for missing hardening flags which blhc is a small tool which checks build logs for missing hardening flags.

Command behind it is:

 blhc --debian --line-numbers --color ${SALSA_CI_BLHC_ARGS} ${WORKING_DIR}/*.build

Lintian

Lintian is tool to spot problems in Deb-packages. It's little bit conversational as it's tries to keep up with Debian packaging documentation but it's also mixed with good practices.

Command to run lintian

lintian --suppress-tags --display-info --pedantic --fail-on error --allow-root ${WORKING_DIR}/*.changes | tee lintian.output || ECODE=$?

missing-breaks

Missing breaks tool is Python code which lives outside of Debian package and can be found here. I checks packages to have correct Breaks/Conflicts that they can be installed and uninstalled

Command to run

check_for_missing_breaks_replaces.py -o ${WORKING_DIR}/missing_breaks.xml --changes-file ${WORKING_DIR}/*.changes

rc-bugs

RC bugs tries to find if there is open RC bugs in Debian bug database for the package. Script can be found here

Command to run

check_rc_bugs.py -v -o ${WORKING_DIR}/rc_bugs.xml --changes-file ${WORKING_DIR}/*.changes

test basic features

Test basic feature test some MariaDB basic features. It tries to see if MariaDB is functioning ok after fresh install.

print info about server

 mariadb --skip-column-names -e "select @@version, @@version_comment"
 
 mariadb --skip-column-names -e "select engine, support, transactions, savepoints from information_schema.engines order by engine" | sort
 
 mariadb --skip-column-names -e "select plugin_name, plugin_status, plugin_type, plugin_library, plugin_license from information_schema.all_plugins order by plugin_name, plugin_library"

Test various features

mariadb -e "CREATE DATABASE db"
 
mariadb -e "CREATE TABLE db.t_innodb(a1 SERIAL, c1 CHAR(8)) ENGINE=InnoDB; INSERT INTO db.t_innodb VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
 
mariadb -e "CREATE TABLE db.t_myisam(a2 SERIAL, c2 CHAR(8)) ENGINE=MyISAM; INSERT INTO db.t_myisam VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
 
mariadb -e "CREATE TABLE db.t_aria(a3 SERIAL, c3 CHAR(8)) ENGINE=Aria; INSERT INTO db.t_aria VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
 
mariadb -e "CREATE TABLE db.t_memory(a4 SERIAL, c4 CHAR(8)) ENGINE=MEMORY; INSERT INTO db.t_memory VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
 
mariadb -e "CREATE ALGORITHM=MERGE VIEW db.v_merge AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria"
 
mariadb -e "CREATE ALGORITHM=TEMPTABLE VIEW db.v_temptable AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria"
 
mariadb -e "CREATE PROCEDURE db.p() SELECT * FROM db.v_merge"
 
mariadb -e "CREATE FUNCTION db.f() RETURNS INT DETERMINISTIC RETURN 1"

Test that the features still work (this step can be done e.g. after an upgrade)

mariadb -e "SHOW TABLES IN db"
 
mariadb -e "SELECT * FROM db.t_innodb; INSERT INTO db.t_innodb VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
 
mariadb -e "SELECT * FROM db.t_myisam; INSERT INTO db.t_myisam VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
 
mariadb -e "SELECT * FROM db.t_aria; INSERT INTO db.t_aria VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
 
mariadb -e "SELECT * FROM db.t_memory; INSERT INTO db.t_memory VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
 
mariadb -e "SELECT COUNT(*) FROM db.v_merge"
 
mariadb -e "SELECT COUNT(*) FROM db.v_temptable"
 
mariadb -e "CALL db.p()"
 
mariadb -e "SELECT db.f()"

Test TLS connections

dpkg -l | grep -i -e tls -e ssl
 
# Create user for TCP connection, must have password
 
 mariadb -e "SET PASSWORD FOR 'mysql'@'localhost' = PASSWORD('asdf234');"
 
cat <<EOF > /root/.my.cnf
 
[client]
 
user=mysql
 
password=asdf234
 
protocol=tcp
 
EOF
 
export CERT_PATH=/usr/share/mysql/mysql-test/std_data
 
openssl verify -CAfile $CERT_PATH/cacert.pem $CERT_PATH/server-cert.pem
 
openssl x509 -subject -issuer -noout -in $CERT_PATH/cacert.pem
 
openssl x509 -subject -issuer -noout -in $CERT_PATH/server-cert.pem
 
cat <<EOF > /etc/mysql/mariadb.conf.d/tls.cnf
 
[client-server]
 
ssl = on
 
ssl-ca = $CERT_PATH/cacert.pem
 
ssl-cert = $CERT_PATH/server-cert.pem
 
ssl-key = $CERT_PATH/server-key.pem
 
[server]
 
require-secure-transport = on
 
[client]
 
ssl-verify-server-cert = on
 
EOF
 
service mariadb restart
 
mariadb -Bse 'STATUS' | tee result
 
# Ensure important values present, otherwise fail job
 
grep --quiet "localhost via TCP/IP" result
 
mariadb -Bse 'SHOW VARIABLES' | grep -e tls -e ssl | tee result
 
grep --quiet "have_ssl	YES" result
 
grep --quiet TLSv1.3 result
 
mariadb -Bse 'SHOW SESSION STATUS' | grep -i -e tls -e ssl | tee result
 
grep --quiet TLSv1.3 result

fresh install

Installs build packages to container and tests if server is up and running with service mariadb status and

mkdir -p debug # Ensure dir exists before using it
 
find /var/lib/mysql -ls > debug/var-lib-mysql.list || true # Ignore errors about "no such file or directory"
 
cp -ra /etc/mysql debug/etc-mysql
 
cp -ra /var/log/mysql debug/var-log-mysql
 
mariadb --skip-column-names -e "select @@version, @@version_comment" # Show version
 
mariadb --table -e "SHOW DATABASES;" # List databases
 
mariadb --table -e "SELECT host,user,plugin,authentication_string FROM user;" mysql
 
mariadb --table -e "SELECT * FROM plugin;" mysql
 
mariadb --table -e "SHOW PLUGINS;" mysql
 
# Test that InnoDB works and that command 'mysql' is also still usable
 
mysql -e "CREATE DATABASE test; USE test; CREATE TABLE t(a INT PRIMARY KEY) ENGINE=INNODB; INSERT INTO t VALUEs (1); SELECT * FROM t; DROP TABLE t; DROP DATABASE test;"

build mariadbclient consumer Python-MySQLdb

Builds MariaDB Python 3.x client library against build debs

pkg-config --cflags --libs mysqlclient # See what MySQLdb builds with
 
apt-get install -y python3-pip
 
pip3 install mysqlclient # Compiles module against libmysqlclient
 
apt-get purge -y libmariadb-dev # Not needed for run-time
 
python3 -c "import MySQLdb; print(MySQLdb.get_client_info())"

Upgrade MariaDB

default-libmysqlclient-dev Sid upgrade

Install Debian default-libmysqlclient-dev package and then libmariadb-packages over them

Prepare

apt-get install -y pkg-config default-libmysqlclient-dev default-libmysqld-dev
 
pkg-config --list-all

Install all libraries over them. This is also function that is re-used in several tests

apt-get install -y ./libmariadb3_*.deb ./libmariadb-dev_*.deb ./libmariadb-dev-compat_*.deb ./libmariadbd19_*.deb ./libmariadbd-dev_*.deb ./mariadb-common_*.deb

upgrade default-libmysqlclient-dev and Bullseye (also Buster)

Same as above but uses Buster or Bullseye packages in test

libmysql* to libmariadb* upgrade

Can libmariadb* be installed over libmysql* packages

apt-get install -y pkg-config libmysqlclient-dev
 
pkg-config --list-all
 
pkg-config --cflags mysqlclient # mysqlclient.pc from original package
 
apt-get install -y ./libmariadb3_*.deb ./mariadb-common_*.deb
 
pkg-config --list-all
 
apt-get install -y ./libmariadb-dev_*.deb
 
pkg-config --list-all
 
apt-get install -y ./libmariadb-dev-compat_*.deb
 
pkg-config --cflags mysqlclient # mysqlclient.pc from compat package
 
pkg-config --list-all
 
apt-get install -y ./libmariadbd19_*.deb
 
pkg-config --list-all
 
apt-get install -y ./libmariadbd-dev_*.deb
 
pkg-config --list-all
 
apt-get install -y default-libmysqlclient-dev default-libmysqld-dev

Upgrading in distro

Upgrading distro packages are done in Bullseye and Buster images. First they install distro version on MariaDB. In Buster version is 10.3 and in Bullseye version is 10.5. They goes nearly the same as next chapter but they are just not done in Sid.

This command is used to install distro packages

apt-get install -y 'default-mysql*' 'mariadb-*' 'libmariadb*'

Upgrading from MariaDB versions

Salsa-CI has upgrading tests from 10.2-10.6 (currently Debian Sid contains 10.6 version). Most of them goes the same.

For example 10.3 goes like this

apt-get -qq install --no-install-recommends --yes ca-certificates curl
 
curl -sS https://mariadb.org/mariadb_release_signing_key.asc -o /etc/apt/trusted.gpg.d/mariadb.asc
 
echo "deb https://archive.mariadb.org/mariadb-10.3/repo/debian ${RELEASE} main" > /etc/apt/sources.list.d/mariadb.list
 
apt-get update -qq
 
*test-install-readline-in-sid-for-backwards-compat
 
apt-get install -y mariadb-server-10.3
 
*test-verify-initial
 
*test-install
 
service mysql status
 
# Give the mariadb-upgrade plenty of time to complete, otherwise next commands
 
# fail on non-existing mariadb.sys user
 
sleep 15



 Comments   
Comment by Faustin Lammler [ 2022-12-20 ]

Thanks illuusio! This is very helpfull.

I think that we should classify those in 2 categories:

  • need to be run on VM (systemd), or something else (LXC);
  • can be run in docker.

Also, I see some tests that are very specific to Debian packaging (inside debian/*) and that we probably don't want to implement:

  • packaging-fix-checks (only needed when packaging diverge from upstream, so not going to happen on BB).

Maybe you could summarize those in an array so that it's easier to decide which we should implement and which not?

Some tests that we already decide we probably should implement:

Those are tests that I think could be good to have:

  • piuparts --> docker I guess;
  • upgrade tests (see what's missing) --> VM;
  • TLS (not sure if not already in MTR) --> VM/Docker?;
  • autopkgtest --> VM.

I don't know well MTR so there might be things that we already test or the contrary...

WRT to what is already done, you can take a look at BB (search for upgrade/install builders). Also as said, all the scripts are in https://github.com/MariaDB/buildbot/tree/main/scripts).

I spend quite some time to make those scripts executable on a local VM (outside of BB execution). So for instance, it's easy to test things if you have a build URL with something like:

./deb-upgrade.sh https://buildbot.mariadb.org/#/builders/404/builds/1133

Same goes for RPM you can execute those script on any clean VM to test new things outside of BB.

Comment by Faustin Lammler [ 2022-12-20 ]

ping vladbogo and danblack also.

Comment by Tuukka Pasanen [ 2022-12-21 ]

If we have KVM then we don't need LXC. I think it's kind of tradeof to run Docker and then LXC top of it (or there is something that I don't understand). If there is some build KVM image I guess everything can be done there.

I'll take a look at those scripts and start testing how to run these new test.

Comment by Faustin Lammler [ 2022-12-21 ]

True, KVM is already what we use and also what old BB use (and we are quite expert with the technology).
I was not excluding LXC because there might be reason to consider other solutions (vagrant is one of them) for Galera cluster testing.

Comment by Tuukka Pasanen [ 2022-12-21 ]

Ok let's keep all the options open and see what fits in. LXC is not bad and it's battle tested tech.

Comment by Daniel Black [ 2023-01-10 ]

faust I think your breakdown is good.
On TLS, I would have though this was already covered in MTR. If it isn't, it probably doesn't need a VM, but its so light it couldn't hurt and saves managing background processes in containers.

What bothered me about the autopkgtest in LXC on salsa is its use of init scripts to mimic what systemd does, and only in very rough terms - https://salsa.debian.org/grooverdan/mariadb-server/-/blob/debian/latest/debian/salsa-ci.yml#L652.

Generated at Thu Feb 08 03:38:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.