Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
N/A
-
None
-
None
Description
Debian Salsa-CI and Buildbot has lot's of differences and they are documented here. These are only from Debian version 10.6 and if there is some major like 10.11 then it's mentioned.
Source pre-tests
These test
gnitpick
This test uses Seravo Gnitpick which nitpick commits language.
Command which is run is:
gnitpick --target-repository https://salsa.debian.org/mariadb-team/mariadb-server.git --target-branch debian/latest
|
This test can fail
packaging-fix-checks
This test depends on Debian package git devscripts python3-debian
Checks if there is no unaddressed patches in debian/patches directory.
if [ "$(find debian/patches/ -type f -not -name series | wc -l)" -eq "$(cat debian/patches/series | wc -l)" ]
|
and after that it runs wrap-and-sort-command which does (taken from man page):
wrap-and-sort wraps the package lists in Debian control files. By default the lists will only split into multiple lines if the entries are longer than the maximum line length limit of 79 characters. wrap-and-sort sorts the package lists in Debian
|
command which does this is:
wrap-and-sort -a
|
What it mostly does it makes sure that files are in alphabetical order under debian-directory and takes care or debian/control that it does not contain rows over 80-bytes long
Building
build bullseye-backports
This is not the test but builds Debian 11 version of MariaDB and it's used on later tests after build is done
build buster-backports
This is not the test but builds Debian 10 version of MariaDB and it's used on later tests after build is done. This one is available on MariaDB 10.6 and after that there can't be Buster version because of change of dependencies.
Upgrading and install
autopkgtest
Autopkg is different from other tests as it is run inside of LXC virtualization (inside of docker container) and not straightly inside docker container as rest of the tests. Good thing about this is that there is possibility to use systemd and goodies. Autopkg test runs tests that are located under debian/tests/
Git repository for LXC image is https://salsa.debian.org/salsa-ci-team/autopkgtest-lxc
LXC image used is (for Sid): https://salsa.debian.org/salsa-ci-team/autopkgtest-lxc/-/jobs/artifacts/master/raw/artifacts/lxc.tar?job=sid
Command which is executed is (for debci you can read here:
su -P debci -c "debci localtest $WORKING_DIR/*.changes --suite ${RELEASE} --logs-dir ${DEBCI_LOG_PATH}" || ( ret=$?; [ $ret -eq 8 ] || [ $ret -eq 2 ] )
|
Where RELEASE is something like buster and DEBCI_LOG_PATH comes from Salsa-CI env variables. Under debian/tests there is currently two tests and then control file where dependencies are collected for these tests.
smoke test
Smoke test creates use and database and test weather compression plugins are available then it checks if RocksDB is available
upstream test
It runs MariaDB test suite and it has unstable-tests for tests that does not work
piuparts
Piuparts is QA tool which installs and uninstalls all the deb packages created in building process
Piuparts needs some of tooling before running inside docker. CHROOT is not chroot as we are inside container is set up as CHROOT_PATH="/tmp/debian-chroot"
mknod -m 666 ${CHROOT_PATH}/dev/urandom c 1 9
|
mkdir -p /srv/local-apt-repository/ && cp -a ${WORKING_DIR}/*.deb /srv/local-apt-repository/ && /usr/lib/local-apt-repository/rebuild
|
mkdir -p ${CHROOT_PATH}/etc-target/apt/sources.list.d ${CHROOT_PATH}/etc-target/apt/preferences.d
|
cp -Hv /etc/apt/sources.list.d/local-apt-repository.list ${CHROOT_PATH}/etc-target/apt/sources.list.d/
|
cp -aTLv /etc/apt/preferences.d ${CHROOT_PATH}/etc-target/apt/preferences.d
|
cp -aTLv /srv/local-apt-repository ${CHROOT_PATH}/srv/local-apt-repository
|
cp -aTLv /var/lib/local-apt-repository/ ${CHROOT_PATH}/var/lib/local-apt-repository/
|
add_extra_repository.sh -v -e "${SALSA_CI_EXTRA_REPOSITORY}" -k "${SALSA_CI_EXTRA_REPOSITORY_KEY}" -t "${CHROOT_PATH}/etc-target"
|
sed '/127.0.0.1/s/localhost/pipeline.salsa.debian.org localhost/' /etc/hosts > ${CHROOT_PATH}/etc/hosts
|
PIUPARTS_DISTRIBUTION_ARG="--distribution $RELEASE"
|
After these it's run against build deb packages:
(for PACKAGE in $(ls ${WORKING_DIR}/*.deb); do
|
piuparts --mirror "${SALSA_CI_MIRROR} ${SALSA_CI_COMPONENTS}" ${SALSA_CI_PIUPARTS_ARGS} --scriptsdir /etc/piuparts/scripts --allow-database --warn-on-leftovers-after-purge --hard-link -e ${CHROOT_PATH} ${PIUPARTS_DISTRIBUTION_ARG} ${PACKAGE}
|
done) | filter-output
|
blhc
BLHC stands for build log hardening check, checks build logs for missing hardening flags which blhc is a small tool which checks build logs for missing hardening flags.
Command behind it is:
blhc --debian --line-numbers --color ${SALSA_CI_BLHC_ARGS} ${WORKING_DIR}/*.build
|
Lintian
Lintian is tool to spot problems in Deb-packages. It's little bit conversational as it's tries to keep up with Debian packaging documentation but it's also mixed with good practices.
Command to run lintian
lintian --suppress-tags --display-info --pedantic --fail-on error --allow-root ${WORKING_DIR}/*.changes | tee lintian.output || ECODE=$?
|
missing-breaks
Missing breaks tool is Python code which lives outside of Debian package and can be found here. I checks packages to have correct Breaks/Conflicts that they can be installed and uninstalled
Command to run
check_for_missing_breaks_replaces.py -o ${WORKING_DIR}/missing_breaks.xml --changes-file ${WORKING_DIR}/*.changes
|
rc-bugs
RC bugs tries to find if there is open RC bugs in Debian bug database for the package. Script can be found here
Command to run
check_rc_bugs.py -v -o ${WORKING_DIR}/rc_bugs.xml --changes-file ${WORKING_DIR}/*.changes
|
test basic features
Test basic feature test some MariaDB basic features. It tries to see if MariaDB is functioning ok after fresh install.
print info about server
mariadb --skip-column-names -e "select @@version, @@version_comment"
|
mariadb --skip-column-names -e "select engine, support, transactions, savepoints from information_schema.engines order by engine" | sort
|
mariadb --skip-column-names -e "select plugin_name, plugin_status, plugin_type, plugin_library, plugin_license from information_schema.all_plugins order by plugin_name, plugin_library"
|
Test various features
mariadb -e "CREATE DATABASE db"
|
mariadb -e "CREATE TABLE db.t_innodb(a1 SERIAL, c1 CHAR(8)) ENGINE=InnoDB; INSERT INTO db.t_innodb VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
|
mariadb -e "CREATE TABLE db.t_myisam(a2 SERIAL, c2 CHAR(8)) ENGINE=MyISAM; INSERT INTO db.t_myisam VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
|
mariadb -e "CREATE TABLE db.t_aria(a3 SERIAL, c3 CHAR(8)) ENGINE=Aria; INSERT INTO db.t_aria VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
|
mariadb -e "CREATE TABLE db.t_memory(a4 SERIAL, c4 CHAR(8)) ENGINE=MEMORY; INSERT INTO db.t_memory VALUES (1,'"'"'foo'"'"'),(2,'"'"'bar'"'"')"
|
mariadb -e "CREATE ALGORITHM=MERGE VIEW db.v_merge AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria"
|
mariadb -e "CREATE ALGORITHM=TEMPTABLE VIEW db.v_temptable AS SELECT * FROM db.t_innodb, db.t_myisam, db.t_aria"
|
mariadb -e "CREATE PROCEDURE db.p() SELECT * FROM db.v_merge"
|
mariadb -e "CREATE FUNCTION db.f() RETURNS INT DETERMINISTIC RETURN 1"
|
Test that the features still work (this step can be done e.g. after an upgrade)
mariadb -e "SHOW TABLES IN db"
|
mariadb -e "SELECT * FROM db.t_innodb; INSERT INTO db.t_innodb VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
|
mariadb -e "SELECT * FROM db.t_myisam; INSERT INTO db.t_myisam VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
|
mariadb -e "SELECT * FROM db.t_aria; INSERT INTO db.t_aria VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
|
mariadb -e "SELECT * FROM db.t_memory; INSERT INTO db.t_memory VALUES (3,'"'"'foo'"'"'),(4,'"'"'bar'"'"')"
|
mariadb -e "SELECT COUNT(*) FROM db.v_merge"
|
mariadb -e "SELECT COUNT(*) FROM db.v_temptable"
|
mariadb -e "CALL db.p()"
|
mariadb -e "SELECT db.f()"
|
Test TLS connections
dpkg -l | grep -i -e tls -e ssl
|
# Create user for TCP connection, must have password
|
mariadb -e "SET PASSWORD FOR 'mysql'@'localhost' = PASSWORD('asdf234');"
|
cat <<EOF > /root/.my.cnf
|
[client]
|
user=mysql
|
password=asdf234
|
protocol=tcp
|
EOF
|
export CERT_PATH=/usr/share/mysql/mysql-test/std_data
|
openssl verify -CAfile $CERT_PATH/cacert.pem $CERT_PATH/server-cert.pem
|
openssl x509 -subject -issuer -noout -in $CERT_PATH/cacert.pem
|
openssl x509 -subject -issuer -noout -in $CERT_PATH/server-cert.pem
|
cat <<EOF > /etc/mysql/mariadb.conf.d/tls.cnf
|
[client-server]
|
ssl = on
|
ssl-ca = $CERT_PATH/cacert.pem
|
ssl-cert = $CERT_PATH/server-cert.pem
|
ssl-key = $CERT_PATH/server-key.pem
|
[server]
|
require-secure-transport = on
|
[client]
|
ssl-verify-server-cert = on
|
EOF
|
service mariadb restart
|
mariadb -Bse 'STATUS' | tee result
|
# Ensure important values present, otherwise fail job
|
grep --quiet "localhost via TCP/IP" result
|
mariadb -Bse 'SHOW VARIABLES' | grep -e tls -e ssl | tee result
|
grep --quiet "have_ssl YES" result
|
grep --quiet TLSv1.3 result
|
mariadb -Bse 'SHOW SESSION STATUS' | grep -i -e tls -e ssl | tee result
|
grep --quiet TLSv1.3 result
|
fresh install
Installs build packages to container and tests if server is up and running with service mariadb status and
mkdir -p debug # Ensure dir exists before using it
|
find /var/lib/mysql -ls > debug/var-lib-mysql.list || true # Ignore errors about "no such file or directory"
|
cp -ra /etc/mysql debug/etc-mysql
|
cp -ra /var/log/mysql debug/var-log-mysql
|
mariadb --skip-column-names -e "select @@version, @@version_comment" # Show version
|
mariadb --table -e "SHOW DATABASES;" # List databases
|
mariadb --table -e "SELECT host,user,plugin,authentication_string FROM user;" mysql
|
mariadb --table -e "SELECT * FROM plugin;" mysql
|
mariadb --table -e "SHOW PLUGINS;" mysql
|
# Test that InnoDB works and that command 'mysql' is also still usable
|
mysql -e "CREATE DATABASE test; USE test; CREATE TABLE t(a INT PRIMARY KEY) ENGINE=INNODB; INSERT INTO t VALUEs (1); SELECT * FROM t; DROP TABLE t; DROP DATABASE test;"
|
build mariadbclient consumer Python-MySQLdb
Builds MariaDB Python 3.x client library against build debs
pkg-config --cflags --libs mysqlclient # See what MySQLdb builds with
|
apt-get install -y python3-pip
|
pip3 install mysqlclient # Compiles module against libmysqlclient
|
apt-get purge -y libmariadb-dev # Not needed for run-time
|
python3 -c "import MySQLdb; print(MySQLdb.get_client_info())"
|
Upgrade MariaDB
default-libmysqlclient-dev Sid upgrade
Install Debian default-libmysqlclient-dev package and then libmariadb-packages over them
Prepare
apt-get install -y pkg-config default-libmysqlclient-dev default-libmysqld-dev
|
pkg-config --list-all
|
Install all libraries over them. This is also function that is re-used in several tests
apt-get install -y ./libmariadb3_*.deb ./libmariadb-dev_*.deb ./libmariadb-dev-compat_*.deb ./libmariadbd19_*.deb ./libmariadbd-dev_*.deb ./mariadb-common_*.deb
|
upgrade default-libmysqlclient-dev and Bullseye (also Buster)
Same as above but uses Buster or Bullseye packages in test
libmysql* to libmariadb* upgrade
Can libmariadb* be installed over libmysql* packages
apt-get install -y pkg-config libmysqlclient-dev
|
pkg-config --list-all
|
pkg-config --cflags mysqlclient # mysqlclient.pc from original package
|
apt-get install -y ./libmariadb3_*.deb ./mariadb-common_*.deb
|
pkg-config --list-all
|
apt-get install -y ./libmariadb-dev_*.deb
|
pkg-config --list-all
|
apt-get install -y ./libmariadb-dev-compat_*.deb
|
pkg-config --cflags mysqlclient # mysqlclient.pc from compat package
|
pkg-config --list-all
|
apt-get install -y ./libmariadbd19_*.deb
|
pkg-config --list-all
|
apt-get install -y ./libmariadbd-dev_*.deb
|
pkg-config --list-all
|
apt-get install -y default-libmysqlclient-dev default-libmysqld-dev
|
Upgrading in distro
Upgrading distro packages are done in Bullseye and Buster images. First they install distro version on MariaDB. In Buster version is 10.3 and in Bullseye version is 10.5. They goes nearly the same as next chapter but they are just not done in Sid.
This command is used to install distro packages
apt-get install -y 'default-mysql*' 'mariadb-*' 'libmariadb*'
|
Upgrading from MariaDB versions
Salsa-CI has upgrading tests from 10.2-10.6 (currently Debian Sid contains 10.6 version). Most of them goes the same.
For example 10.3 goes like this
apt-get -qq install --no-install-recommends --yes ca-certificates curl
|
curl -sS https://mariadb.org/mariadb_release_signing_key.asc -o /etc/apt/trusted.gpg.d/mariadb.asc
|
echo "deb https://archive.mariadb.org/mariadb-10.3/repo/debian ${RELEASE} main" > /etc/apt/sources.list.d/mariadb.list
|
apt-get update -qq
|
*test-install-readline-in-sid-for-backwards-compat
|
apt-get install -y mariadb-server-10.3
|
*test-verify-initial
|
*test-install
|
service mysql status
|
# Give the mariadb-upgrade plenty of time to complete, otherwise next commands
|
# fail on non-existing mariadb.sys user
|
sleep 15
|
