I have everything else pretty much set for uploading mariadb-10.4 to Debian, but this mtr start failure is the only major one remaining to be solved. Could maybe serg check it out and suggest what I could try to debug next?

https://salsa.debian.org/mariadb-team/mariadb-10.4/pipelines/108149

Just a reminder that this is about Debian packaging in their official repos, source at https://salsa.debian.org/mariadb-team/mariadb-10.4/commits/master

If you cvicentiu don't know what is the solution, even giving some hints of what related you've found out could help me debug it...

Hi otto!

I do now know yet what is causing the packages in salsa CI to not-work, however I am now investigating this commit, which is the likely cause of change in behaviour:

https://github.com/MariaDB/server/commit/4fef644303ecdf6ea88ea40d3733dea6bd2cf41d#diff-2231e5936098e13c6b6ee8660f401382L2646

I will keep you updated.

Ok, this was a tricky one...

The roots of this problem start from 2016, with this commit in the debian packaging repository:

commit 5aecef81f774e245f60e8c3f70f44190f7473537

Author: Otto Kekäläinen <otto@debian.org>

Date:   Thu Nov 10 14:23:28 2016 +0200

    Move plugins to $ARCH/*/mariadb18 to meet multiarch needs (Closes: #739452)

    Previously all plugins were installed to usr/lib/mysql/plugin, including

    the plugins in the client library package, which inhibited installing

    other architecture versions of the same package on the same system,

    which is expected to work in multiarch situations.

    As client plugins alone cannot change directory with current build options,

    and moving the server plugins too does not have any negative effects,

    move all plugins to a multiarch directory.

    Using the ABI version in the path also makes it possible to install

    multiple versions of the shared library side-by-side as is normally

    facilitated by the soname .so.18 naming scheme, which the client

    plugins did not follow.

    See also discussion at https://bugs.debian.org/842454

This commit introduces a custom INSTALL_PLUGINDIR (different to what is default in the base MariaDB repository for deb packages) to the cmake invocation that debian packaging uses with debian/autobake-deb.sh. mysql-test-run.pl never properly worked with a custom plugin directory. The tests that require plugins are always skipped. This is also visible even in 10.4.6 packaging on salsa https://salsa.debian.org/mariadb-team/mariadb-10.4/-/jobs/241143, search for the test:

main.blackhole_plugin                    [ skipped ]  Need blackhole plugin

This worked because internally mysql-test-run.pl defaulted to the "current directory" aka dot (.) when it was run from an installed location.

Now, what caused this problem to finally be apparent is when serg committed the following patch https://github.com/MariaDB/server/commit/4fef644303ecdf6ea88ea40d3733dea6bd2cf41d#diff-2231e5936098e13c6b6ee8660f401382L2646

The subtle change is that we no longer have a dot as default plugin-dir value. This is only apparent when we've chosen to modify the install plugin directory and we are not running mtr from the source directory.

Additionally, because the debian/autobake-deb.sh in the upstream MariaDB repository is not the same as the one Salsa, buildbot never caught the problem either.

In essence, the implementation of mtr should "configure" the script to account for INSTALL_PLUGINDIR, but I do not feel comfortable doing these edits right now, not with some input from serg that I did not miss anything in my analysis.

I am however quite comfortable suggesting the following patch to apply to mysql-test-run.pl in debian packaging patches series until we implement a proper fix to account for a custom INSTALL_PLUGINDIR.

diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl

index 6ee8e926ad4..a05607a9418 100755

--- a/mysql-test/mysql-test-run.pl

+++ b/mysql-test/mysql-test-run.pl

@@ -2824,6 +2824,7 @@ sub setup_vardir() {

         set_plugin_var($pname);

         $plugindir=dirname($_) unless $plugindir;

       }

+      $plugindir= ($mysqld_variables{'plugin-dir'} || '.') unless $plugindir;

     }

   }

main.blackhole_plugin                    [ skipped ]  Need blackhole plugin

Good catch! Indeed the plugins in the mariadb-test are in upstream in wrong directory (not arch compatible) and some recent changes even ship plugins in the mariadb-test-data package that should be data-only (the same package is used on all architectures), see MDEV-21654. Directories are not versioned as they should be (thus I did https://github.com/MariaDB/server/commit/b9dea911bf8e3d4d8fc57ce3ef15ab0e2ab4d076).

I have known about these problems since 10.3 was released but because of timing issues I havn't been able to submit or champion proper changes upstream. I plan to do that now in the 10.5 cycle before it gets GA so that the delta between downstream Debian packaging and upstream 10.5 can be finally closed.

MTR in autopkgtest passed now, thanks!
https://salsa.debian.org/mariadb-team/mariadb-10.4/-/jobs/566977

By the way, the Debian builds us the debbuild directly (and thus assumes proper debian/ contents). The MariaDB autobake-deb.sh is not used in official Debian at all.

This is now solved downstream, let's keep it open until also fixed upstream.

https://salsa.debian.org/mariadb-team/mariadb-10.4/pipelines/110060

Should this be applied on 10.5 now?

This is still open, and we still carry the patch https://salsa.debian.org/mariadb-team/mariadb-10.5/-/blob/master/debian/patches/mtr-plugin-path.patch downstream in Debian. Any updates on this cvicentiu (or maybe danblack)?

I removed this patch in https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/bde2cf481fa48a0dd85b9ad40e27ad5005ad1122 and at least mariadb-10.5 builds and test suite seems to run fine without it. See e.g. https://buildd.debian.org/status/package.php?p=mariadb-10.5

Tested also locally and works just fine now:

# cd /usr/share/mysql/mysql-test

# export WORKDIR=$(mktemp -d)

# perl -I. ./mysql-test-run.pl --suite=main --vardir=$WORKDIR/var --tmpdir=$WORKDIR/tmp  --parallel=auto --skip-rpl --force

Logging: ./mysql-test-run.pl  --suite=main --vardir=/var --tmpdir=/tmp --parallel=auto --skip-rpl --force

vardir: /var

Checking leftover processes...

mysql-test-run: WARNING: Found non pid file 'utmp' in '/var/run'

Removing old var directory...

Creating var directory '/var'...

Checking supported features...

Can't exec "patch": No such file or directory at ./mysql-test-run.pl line 2029.

MariaDB Version 10.5.5-MariaDB-4

 - SSL connections supported

Using suites: main

Collecting tests...

Installing system database...

Using parallel: 8

==============================================================================

TEST                                  WORKER RESULT   TIME (ms) or COMMENT

--------------------------------------------------------------------------

...