I observed that the docker image build CI for s390x has been timing out since March 2023 therefore some images are missing from the docker registry. I was able to build the image on my s390x machine with no problem. Looks like it is an issue with the build pipeline. Could anyone from the MariaDB community take a look at this issue. I can provide any help if required. Thanks.
pipeline link: https://doi-janky.infosiftr.net/job/multiarch/job/s390x/job/mariadb/
screenshot:
Daniel Black
added a comment - Thanks for the bug report.
It appears to be something in the Docker CI (infosiftr was acquired by Docker Inc).
I've forwarded the bug to them to understand: https://github.com/docker-library/official-images/issues/14923
I'm a bit worried that 10.5, 10.6 succeed and the others failed. This could distro kernel related, 10.9+ are jammy based and would be using iouring, 10.5/10.6 wouldn't). So this could be a s390x kernel uring problem.
I removed the initialization in https://github.com/MariaDB/mariadb-docker/pull/518 that should make it quicker, and avoid the build time exercising of database functions. If there is really a problem in s390x initialization of a database in 10.9+ this needs to be found. I'll look at pushing a release early next week that passes hopefully and there should at least be a release there.
Can you test quay.io/mariadb-foundation/mariadb-devel:$version images on real hardware as they haven't had trouble building at all.
Pushing a hammerdb/sysbench oltp (RW) at the container would be useful too.
Daniel Black
added a comment - - edited I'm a bit worried that 10.5, 10.6 succeed and the others failed. This could distro kernel related, 10.9+ are jammy based and would be using iouring, 10.5/10.6 wouldn't). So this could be a s390x kernel uring problem.
I removed the initialization in https://github.com/MariaDB/mariadb-docker/pull/518 that should make it quicker, and avoid the build time exercising of database functions. If there is really a problem in s390x initialization of a database in 10.9+ this needs to be found. I'll look at pushing a release early next week that passes hopefully and there should at least be a release there.
Can you test quay.io/mariadb-foundation/mariadb-devel:$version images on real hardware as they haven't had trouble building at all.
As a test clone https://github.com/MariaDB/mariadb-docker
and:
.test/run quay.io/mariadb-foundation/mariadb-devel:10.9
Will start/stop exercise the functionality.
Pushing a hammerdb/sysbench oltp (RW) at the container would be useful too.
The s390x released on Docker Hub on the official releases have been updated so are available in all 10.5+ containers like they should have been.
There is still most likely an iouring problem with at least some kernels on s390x for mariadb containers 10.9+. To confirm, after exercising a stalling test case (like previous), disable the liburing with --innodb_use_native_aio=0 and see the same behaviour exists.
Daniel Black
added a comment - The s390x released on Docker Hub on the official releases have been updated so are available in all 10.5+ containers like they should have been.
There is still most likely an iouring problem with at least some kernels on s390x for mariadb containers 10.9+. To confirm, after exercising a stalling test case (like previous), disable the liburing with --innodb_use_native_aio=0 and see the same behaviour exists.
Major 
Thanks for the bug report.
It appears to be something in the Docker CI (infosiftr was acquired by Docker Inc).
I've forwarded the bug to them to understand: https://github.com/docker-library/official-images/issues/14923