[MDBF-347] bb 10.2 packages generated in CI ubuntu repository unusable Created: 2022-02-18  Updated: 2022-05-09  Resolved: 2022-02-23

Status: Closed
Project: MariaDB Foundation Development
Component/s: None
Affects Version/s: None
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Daniel Black Assignee: Faustin Lammler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: 0d
Time Spent: 1h
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by MDEV-28258 mariadb-plugin-cracklib-password-chec... Closed
is duplicated by MDEV-28513 mariadb-client-10.7 and mariadb-serve... Closed

 Description   

The docker library builder cannot install mariadb packages from the generated repo.

The result is the continuous delivery of 10.2 container images from quay.io is no-longer updating.

All 10.3+ repositories are generated ok and are building on the docker-library buildbot.

from https://buildbot.mariadb.org/#/builders/311/builds/6783/steps/2/logs/stdio

Get:1 http://ports.ubuntu.com/ubuntu-ports bionic InRelease [242 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease [88.7 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease [74.6 kB]
Ign:4 https://ci.mariadb.org/22519/aarch64-ubuntu-1804-deb-autobake/debs ./ InRelease
Get:5 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease [88.7 kB]
Ign:6 https://ci.mariadb.org/22519/aarch64-ubuntu-1804-deb-autobake/debs ./ Release
Get:7 https://ci.mariadb.org/22519/aarch64-ubuntu-1804-deb-autobake/debs ./ Packages [10.2 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 Packages [11.0 MB]
Get:9 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 Packages [1285 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports bionic/restricted arm64 Packages [572 B]
Get:11 http://ports.ubuntu.com/ubuntu-ports bionic/multiverse arm64 Packages [153 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports bionic-updates/restricted arm64 Packages [4824 B]
Get:13 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 Packages [1798 kB]
Get:14 http://ports.ubuntu.com/ubuntu-ports bionic-updates/multiverse arm64 Packages [5351 B]
Get:15 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 Packages [2005 kB]
Get:16 http://ports.ubuntu.com/ubuntu-ports bionic-backports/universe arm64 Packages [12.2 kB]
Get:17 http://ports.ubuntu.com/ubuntu-ports bionic-backports/main arm64 Packages [11.6 kB]
Get:18 http://ports.ubuntu.com/ubuntu-ports bionic-security/universe arm64 Packages [1307 kB]
Get:19 http://ports.ubuntu.com/ubuntu-ports bionic-security/main arm64 Packages [1412 kB]
Get:20 http://ports.ubuntu.com/ubuntu-ports bionic-security/restricted arm64 Packages [4146 B]
Get:21 http://ports.ubuntu.com/ubuntu-ports bionic-security/multiverse arm64 Packages [2759 B]
Fetched 19.5 MB in 10s (1894 kB/s)
Reading package lists...
+ apt-get install -y mariadb-server=1:10.2.44+maria~bionic mariadb-backup-10.2 socat
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
 mariadb-server : Depends: mariadb-server-10.2 (>= 1:10.2.44+maria~bionic) but it is not going to be installed

This is used for the 10.2 based releases.

Changing the build to:

diff from https://github.com/MariaDB/mariadb-docker

diff --git a/10.2/Dockerfile b/10.2/Dockerfile
index 6ef285d..d298bed 100644
--- a/10.2/Dockerfile
+++ b/10.2/Dockerfile
@@ -1,5 +1,5 @@
 # vim:set ft=dockerfile:
-FROM ubuntu:bionic
+FROM ubuntu:focal
 
 # add our user and group first to make sure their IDs get assigned consistently, regardless of whatever dependencies get added
 RUN groupadd -r mysql && useradd -r -g mysql mysql
@@ -51,7 +51,7 @@ RUN mkdir /docker-entrypoint-initdb.d
 RUN set -ex; \
        apt-get update; \
        DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
-               libjemalloc1 \
+               libjemalloc2 \
                pwgen \
                tzdata \
                xz-utils \

Rebuilding with this in the same buildah command line as the bb generates the same error.

This would eliminate the bionic vs focal apt versions being a cause of difference.

This is generated with buildbot.mariadb.org/master.cfg -> dpkgDeb the same for all repos.

I checked the Packages.gz paths and they seem fine.

Supple differences in 10.2 packaging maybe?



 Comments   
Comment by Faustin Lammler [ 2022-02-22 ]

Daniel,
trying to reproduce this, here seems to be the problem:

root@113a7d5eb63a:~# apt install mariadb-server-10.2
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
 
The following packages have unmet dependencies:
 mariadb-server-10.2 : Depends: iproute but it is not installable or
                                iproute but it is not installable
                       Recommends: libhtml-template-perl but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

This is really wired as iproute2 is available on bionic and if I am correct, the dependency is on either iproute or iproute2 https://github.com/MariaDB/server/commit/9f848da640dd6c3f44d56eae18204370ae7f835c

Digging a bit further, https://ci.mariadb.org/22519/aarch64-ubuntu-1804-deb-autobake/debs/mariadb-server-10.2_10.2.44%2Bmaria~bionic_arm64.deb seems to have removed the iproute2 dependency:

❮ wget https://ci.mariadb.org/22519/aarch64-ubuntu-1804-deb-autobake/debs/mariadb-server-10.2_10.2.44%2Bmaria~bionic_arm64.deb
--2022-02-22 18:06:57--  https://ci.mariadb.org/22519/aarch64-ubuntu-1804-deb-autobake/debs/mariadb-server-10.2_10.2.44%2Bmaria~bionic_arm64.deb
Resolving ci.mariadb.org (ci.mariadb.org)... 135.181.143.118
Connecting to ci.mariadb.org (ci.mariadb.org)|135.181.143.118|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3956192 (3.8M) [application/octet-stream]
Saving to: ‘mariadb-server-10.2_10.2.44+maria~bionic_arm64.deb’
 
mariadb-server-10.2_10.2.44+maria~b 100%[================================================================>]   3.77M  4.60MB/s    in 0.8s
 
2022-02-22 18:06:58 (4.60 MB/s) - ‘mariadb-server-10.2_10.2.44+maria~bionic_arm64.deb’ saved [3956192/3956192]
 
❯ ar x mariadb-server-10.2_10.2.44+maria\~bionic_arm64.deb
 
❯ tar -xvf control.tar.xz
./
./conffiles
./config
./control
./md5sums
./postinst
./postrm
./preinst
./prerm
./templates
./triggers
 
❯ grep iproute control
Depends: bsdutils, coreutils, findutils, galera-3 (>= 25.3), gawk, iproute | iproute, libdbi-perl, lsb-base (>= 3.0-10), lsof, mariadb-client-10.2 (>= 1:10.2.44+maria~bionic), mariadb-server-core-10.2 (>= 1:10.2.44+maria~bionic), passwd, perl (>= 5.6), psmisc, rsync, socat, debconf (>= 0.5) | debconf-2.0, libc6 (>= 2.17), libpam0g (>= 0.99.7.1), libssl1.1 (>= 1.1.1), libstdc++6 (>= 5.2), zlib1g (>= 1:1.2.0)

Dependency is on:

iproute | iproute

instead of:

iproute | iproute2

Not sure why but this is the pb I guess.

Comment by Daniel Black [ 2022-02-23 ]

Because the cache on the container image doesn't contain iproute2, its removed

https://github.com/MariaDB/server/blob/10.2/debian/autobake-deb.sh#L46-L49

Did you fix something? https://buildbot.mariadb.org/#/grid?branch=10.2 appears to have come good

https://quay.io/repository/mariadb-foundation/mariadb-devel?tab=history 10.2 releases there

Comment by Faustin Lammler [ 2022-02-23 ]

> Because the cache on the container image doesn't contain iproute2, its removed
> https://github.com/MariaDB/server/blob/10.2/debian/autobake-deb.sh#L46-L49

But it should not for bionic:

❯ podman run -it ubuntu:bionic bash -c "apt-get update && apt-cache madison iproute2 | grep 'iproute2 *|'"
...
Reading package lists... Done
  iproute2 | 4.18.0-1ubuntu2~ubuntu18.04.1 | http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages
  iproute2 | 4.15.0-2ubuntu1.3 | http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
  iproute2 | 4.15.0-2ubuntu1.1 | http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
  iproute2 | 4.15.0-2ubuntu1 | http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

> Did you fix something? https://buildbot.mariadb.org/#/grid?branch=10.2 appears to have come good
> https://quay.io/repository/mariadb-foundation/mariadb-devel?tab=history 10.2 releases there

Nope

Comment by Daniel Black [ 2022-02-23 ]

The key point is the apt-get update which isn't part of the autobake.sh. If it wasn't in the cache locally it wouldn't be there.

$ podman run --rm quay.io/mariadb-foundation/bb-worker:ubuntu18.04 apt-cache madison iproute2 
Trying to pull quay.io/mariadb-foundation/bb-worker:ubuntu18.04...
Getting image source signatures
Copying blob 68e7bb398b9f skipped: already exists  
Copying blob 2b932ef7fe97 done  
Copying blob 083f14a62f1b done  
Copying blob 270d3ca089f6 done  
Copying blob 016ff0767696 done  
Copying blob 57d07ba92564 done  
Copying config e3de297c33 done  
Writing manifest to image destination
Storing signatures
  iproute2 | 4.18.0-1ubuntu2~ubuntu18.04.1 | http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages
  iproute2 | 4.15.0-2ubuntu1.3 | http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
  iproute2 | 4.15.0-2ubuntu1.1 | http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
  iproute2 | 4.15.0-2ubuntu1 | http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
  iproute2 | 4.15.0-2ubuntu1 | http://archive.ubuntu.com/ubuntu bionic/main Sources
  iproute2 | 4.15.0-2ubuntu1.3 | http://archive.ubuntu.com/ubuntu bionic-updates/main Sources
  iproute2 | 4.18.0-1ubuntu2~ubuntu18.04.1 | http://archive.ubuntu.com/ubuntu bionic-backports/main Sources
  iproute2 | 4.15.0-2ubuntu1.1 | http://security.ubuntu.com/ubuntu bionic-security/main Sources

So there now.

previous image manifest removed

$ podman run --rm quay.io/mariadb-foundation/bb-worker@sha256:1e9c765bd358c8f4e0b1ce1f7727dd670d679f80882b10d3741d0ef0eeaecd33 apt-cache madison iproute2
Trying to pull quay.io/mariadb-foundation/bb-worker@sha256:1e9c765bd358c8f4e0b1ce1f7727dd670d679f80882b10d3741d0ef0eeaecd33...
Error: initializing source docker://quay.io/mariadb-foundation/bb-worker@sha256:1e9c765bd358c8f4e0b1ce1f7727dd670d679f80882b10d3741d0ef0eeaecd33: reading manifest sha256:1e9c765bd358c8f4e0b1ce1f7727dd670d679f80882b10d3741d0ef0eeaecd33 in quay.io/mariadb-foundation/bb-worker: manifest unknown: manifest unknown

... but maybe not on < Feb 19 when the last bb-worker:18.04 change was made, same date quay.io/mariadb-foundation/mariadb-devel:10.2 updated.

Comment by Faustin Lammler [ 2022-02-24 ]

So, this might be the reason:
https://github.com/MariaDB/mariadb.org-tools/commit/cbc21ab61d51a16a0f6d4668218140d20999d964#diff-cf2e76e32cae86bc87bbdc519d110a6ab361fce8911ebe2996100fc573346bc0L76-L78

And if this is a requirement to not clean the apt cache, then it was a smart move. There is still a problem with that approach: I am not sure how long apt will trust it's cache (/var/lib/apt/lists) but if the image is too old, an apt-get update may be needed for this to work as expected.

Comment by Daniel Black [ 2022-02-25 ]

Eventually I think I'd like the debian/autobake-deb.sh to build based on actual distro rather than cache contents.

The Debian base image is updated about monthly https://github.com/docker-library/repo-info/commits/master/repos/debian/remote

package updates but the time you include all the build dependencies, probably monthly, just enough to get updates, and enough to ensure the ccache can be stable based on dependencies pulled in. github actions on a schedule maybe?

Comment by Faustin Lammler [ 2022-02-25 ]

> github actions on a schedule maybe?

Definitively! But not before we have a better solution for MDBF-336.
Also, ideally we should have an automatic rollback mechanism on BB if new images breaks the builds.

Comment by Daniel Black [ 2022-03-04 ]

Seems the aarch64 version from bionic may have been used last release looking at https://doi-janky.infosiftr.net/job/multiarch/job/arm64v8/job/mariadb/

I don't suppose there's an easy work around on released packages?

https://github.com/MariaDB/server/pull/2040 as a cleanup item in testing.

Generated at Thu Feb 08 03:37:13 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.