|
https://buildbot.mariadb.org/#/builders/356/builds/147 too ppc64le-ubuntu-2204-deb-autobake/
This exists on the image:
$ podman run --rm -ti --arch ppc64le quay.io/mariadb-foundation/bb-worker:ubuntu22.04 bash
|
Trying to pull quay.io/mariadb-foundation/bb-worker:ubuntu22.04...
|
Getting image source signatures
|
Copying blob 9942120b68ea skipped: already exists
|
Copying blob 5734b7f56916 skipped: already exists
|
Copying blob a75981f42eff skipped: already exists
|
Copying blob f077f909b969 skipped: already exists
|
Copying blob bbb9b37fdc1d skipped: already exists
|
Copying blob 6142acf6cc92 skipped: already exists
|
Copying config 8e164a001d done
|
Writing manifest to image destination
|
Storing signatures
|
To run a command as administrator (user "root"), use "sudo <command>".
|
See "man sudo_root" for details.
|
|
buildbot@75dbf860dab2:~$ find / -name libfakeroot-sysv.so -ls
|
2389662 132 -rw-r--r-- 1 root root 134952 Jan 19 12:22 /usr/lib/powerpc64le-linux-gnu/libfakeroot/libfakeroot-sysv.so
|
buildbot@75dbf860dab2:~$ fakeroot
|
#
|
$ podman run --rm -ti --arch ppc64le quay.io/mariadb-foundation/bb-worker:debiansid bash
|
Trying to pull quay.io/mariadb-foundation/bb-worker:debiansid...
|
Getting image source signatures
|
Copying blob 801c5fa639e2 skipped: already exists
|
Copying blob 4f4fb700ef54 skipped: already exists
|
Copying blob 071265fa048a skipped: already exists
|
Copying blob 5dde5a0d31b6 done
|
Copying blob 075c0362d19f done
|
Copying blob 27b0c91d3ffb done
|
Copying blob 40a249bbe97c done
|
Copying config c4db41c0fa done
|
Writing manifest to image destination
|
Storing signatures
|
buildbot@06e7201171ab:~$ fakeroot
|
# find / -name libfakeroot-sysv.so -ls
|
2829382 136 -rw-r--r-- 1 root root 135192 Mar 4 14:22 /usr/lib/powerpc64le-linux-gnu/libfakeroot/libfakeroot-sysv.so
|
Is this a worker issue that doesn't have up to date workers? p9-rhel7-bbw1-docker-ubuntu-2204
|
|
Suggestions from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846270#12 adapted
Local tests under qemu podman run --rm -ti --arch ppc64le quay.io/mariadb-foundation/bb-worker:debiansid bash:
|
bash -x fakeroot true
|
buildbot@428b537b848b:~$ bash -x fakeroot true
|
+ FAKEROOT_PREFIX=/usr
|
+ FAKEROOT_BINDIR=/usr/bin
|
+ USEABSLIBPATH=0
|
+ FAKEROOT_LIB=libfakeroot-sysv.so
|
+ PATHS=/usr/lib/powerpc64le-linux-gnu/libfakeroot:/usr/lib64/libfakeroot:/usr/lib32/libfakeroot
|
+ FAKED=/usr/bin/faked-sysv
|
+ FAKED_MODE=unknown-is-root
|
+ export FAKED_MODE
|
+ libfound=no
|
++ getopt -T
|
+ GETOPTTEST=
|
+ test 4 -eq 4
|
++ getopt -l lib: -l faked: -l unknown-is-real -l fd-base: -l version -l help -- +l:f:i:s:ub:vh true
|
+ FAKE_TEMP=' -- '\''true'\'''
|
+ test 0 -ne 0
|
+ eval set -- ' -- '\''true'\'''
|
++ set -- -- true
|
+ FAKEDOPTS=
|
+ PIPEIN=
|
+ WAITINTRAP=0
|
+ test X-- '!=' X--
|
+ shift
|
+ ABSLIB=
|
+ '[' -n /usr/lib/powerpc64le-linux-gnu/libfakeroot:/usr/lib64/libfakeroot:/usr/lib32/libfakeroot ']'
|
++ sed 's/:/ /g'
|
++ echo /usr/lib/powerpc64le-linux-gnu/libfakeroot:/usr/lib64/libfakeroot:/usr/lib32/libfakeroot
|
+ for dir in `echo $PATHS | sed 's/:/ /g'`
|
+ test -r /usr/lib/powerpc64le-linux-gnu/libfakeroot/libfakeroot-sysv.so
|
+ libfound=yes
|
+ ABSLIB=/usr/lib/powerpc64le-linux-gnu/libfakeroot/libfakeroot-sysv.so
|
+ for dir in `echo $PATHS | sed 's/:/ /g'`
|
+ test -r /usr/lib64/libfakeroot/libfakeroot-sysv.so
|
+ for dir in `echo $PATHS | sed 's/:/ /g'`
|
+ test -r /usr/lib32/libfakeroot/libfakeroot-sysv.so
|
+ test yes = no
|
+ test -n ''
|
+ unset FAKEROOTKEY
|
++ eval /usr/bin/faked-sysv
|
+++ /usr/bin/faked-sysv
|
+ KEY_PID=1448255377:27
|
++ echo 1448255377:27
|
++ cut -d: -f1
|
+ FAKEROOTKEY=1448255377
|
++ echo 1448255377:27
|
++ cut -d: -f2
|
+ PID=27
|
+ '[' 0 -eq 0 ']'
|
+ trap 'kill -s TERM 27' EXIT INT
|
+ test -z 1448255377
|
+ test -z 27
|
+ test 0 -ne 0
|
+ test -n ''
|
+ test -n ''
|
+ export FAKEROOT_FD_BASE
|
+ test -z true
|
+ FAKEROOTKEY=1448255377
|
+ LD_LIBRARY_PATH=/usr/lib/powerpc64le-linux-gnu/libfakeroot:/usr/lib64/libfakeroot:/usr/lib32/libfakeroot
|
+ LD_PRELOAD=libfakeroot-sysv.so
|
+ true
|
+ RESULT=0
|
+ exit 0
|
+ kill -s TERM 27
|
buildbot@428b537b848b:~$ LD_LIBRARY_PATH= LD_PRELOAD=libfakeroot-sysv.so ldd /bin/true
|
ERROR: ld.so: object 'libfakeroot-sysv.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
|
ERROR: ld.so: object 'libfakeroot-sysv.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
|
libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x0000004001889000)
|
/lib64/ld64.so.2 (0x0000004000000000)
|
|
buildbot@428b537b848b:~$ LD_LIBRARY_PATH= LD_PRELOAD=/usr/lib/powerpc64le-linux-gnu/libfakeroot/libfakeroot-sysv.so ldd /bin/true
|
/usr/lib/powerpc64le-linux-gnu/libfakeroot/libfakeroot-sysv.so (0x0000004001884000)
|
libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00000040018bc000)
|
libdl.so.2 => /lib/powerpc64le-linux-gnu/libdl.so.2 (0x0000004001ae7000)
|
/lib64/ld64.so.2 (0x0000004000000000)
|
|
|
ldd has a test -r to determine readability. Its obviously not the unix permissions that prevents this.
Test uses access to determine access
|
strace test -r /bin/true
|
...
|
newfstatat(AT_FDCWD, "/bin/true", {st_mode=S_IFREG|0755, st_size=28504, ...}, 0) = 0
|
geteuid() = 1000
|
getegid() = 1000
|
getuid() = 1000
|
getgid() = 1000
|
access("/bin/true", R_OK) = 0
|
Looking how bash -x fakeroot true was resolved for me, test -r /usr/lib64/libfakeroot/libfakeroot-sysv.so is part of the tests, so if test -r /bin/true fails I expect this does too.
Was it only the RHEL 8 host that was the problem? Some selinux restriction in containers reading executables maybe. ls -laZ /bin/true, but most likely investigating access denied errors on the host audit log.
|
|
The RHEL 8 host is not affected by this. So, I disabled the other two (RHEL 7 and Debian) hosts for now. I will continue to look into the permission issue
|
|
Failing builder s390x-ubuntu-2204-deb-autobake, worker s390x-bbw2-docker-ubuntu-2204 (and not s390x-bbw[13]-docker-ubuntu-2204)
|
|
I have digged a bit. I believe that there is a bug in the bash (or sh) test builtin function.
See bellow on the db-p9-bbw1 server:
❯ docker run --user root -it quay.io/mariadb-foundation/bb-worker:debiansid bash
|
root@ebe6a30e7e40:/home/buildbot# test -r /bin/cat ; echo $?
|
1
|
root@ebe6a30e7e40:/home/buildbot# command -v test
|
test
|
root@ebe6a30e7e40:/home/buildbot# enable -n test
|
root@ebe6a30e7e40:/home/buildbot# command -v test
|
/usr/bin/test
|
root@ebe6a30e7e40:/home/buildbot# test -r /bin/cat ; echo $?
|
0
|
But I am not able to reproduce this on another ppc machine:
[faust@proj-p9maria-rhel8 ~]$ docker run --user root -it quay.io/mariadb-foundation/bb-worker:debiansid bash
|
root@de9cfd8ceef8:/home/buildbot# test -r /bin/cat ; echo $?
|
0
|
root@de9cfd8ceef8:/home/buildbot# command -v test
|
test
|
|
|
Well the bash and test are the same in the container.
Suggest maybe docker or secom filters version.
docker run --user root -it --cap-add=SYS_PTRACE quay.io/mariadb-foundation/bb-worker:debiansid bash
|
root@796af1450ff5:/# apt-get update && apt-get install -y strace
|
root@796af1450ff5:/# strace -s 99 test -r /usr/lib/powerpc64le-linux-gnu/libfakeroot/libfakeroot-sysv.so
|
See what access and newfstatat return
|
|
> Well the bash and test are the same in the container.
I am not sure that I understand this...can you explain?
Container has either the bash builtin function or /usr/bin/test, no?
|
|
FYI, I have just verified that this bug is not present if running in a VM (I used the debian-sid qcow2 cloud-init image on the db-p9-bbw1 machine). So, this is a confirmation (if needed) that this is related to docker.
|
|
So this is clearly a permission problem and a bug in docker (or a mix of kernel, cgroups etc.)
❯ docker run -ti quay.io/mariadb-foundation/bb-worker:debiansid fakeroot
|
fakeroot: preload library `libfakeroot-sysv.so' not found, aborting.
|
❯ docker run -ti --privileged quay.io/mariadb-foundation/bb-worker:debiansid fakeroot
|
#
|
I have tried to add specific capabilities (see: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) but without success. And using `--privileged` on BB docker builders is not an option IMO.
vladbogo we have spend lots of time on this (too much?) and I propose we upgrade dbart's machine to bullseye (still on buster). Regarding the fi-rhel7 machine it could be upgraded too IMO. And then we will see what happens.
Also, I am waiting for Otto's comment on removing totally the use of fakeroot.
|
|
faust how about changing to rhel-8 since that machine using rhel-8 doesn't have issues?
|
|
Yep, that's a temporary solution and if it's not too difficult do use this machine specifically for those builds, go on.
|
|
No, I was meaning to use RHEL 8 as the OS for the other machines. As a temporary solution of course we can only use the machine that is not problematic, but that it's just temporary.
|
|
I have just upgraded dbart's machine and it seems that fakeroot is now OK.
|
faust@db-p9-bbw1 ~
|
❮ docker run -ti quay.io/mariadb-foundation/bb-worker:debiansid fakeroot
|
#
|
|
|
MDBF-465 solves this. As in this case, the problem for MDBF-465 translated into having permission denied for some files. The problem was caused by the faccessat2 system call (which is invoked by gdb-add-index) which is not properly handled in an older Docker version. Sample log can be found here https://buildbot.mariadb.org/#/builders/483/builds/345/steps/7/logs/stdio
Easy way to test the problem:
touch ceva && bash -c 'if test ! -r ceva; then echo NOT OK; else echo OK; fi'
As a follow up, upgrading Docker on the problematic machines solves both this issue and MDBF-465. However, this behaviour might appear in the future when adding new OSs.
|