[MDEV-31529] MariaDB docker pipeline is failing Created: 2023-06-22  Updated: 2023-06-27  Resolved: 2023-06-27

Status: Closed
Project: MariaDB Server
Component/s: Docker
Affects Version/s: None
Fix Version/s: 11.0.2, 10.5.21, 10.6.14, 10.9.7, 10.10.5, 10.11.4

Type: Bug Priority: Major
Reporter: Ehsan Kiani Far Assignee: Daniel Black
Resolution: Fixed Votes: 0
Labels: None
Environment:

s390x


Attachments: PNG File Screenshot 2023-06-22 at 9.57.43 AM.png    

 Description   

I observed that the docker image build CI for s390x has been timing out since March 2023 therefore some images are missing from the docker registry. I was able to build the image on my s390x machine with no problem. Looks like it is an issue with the build pipeline. Could anyone from the MariaDB community take a look at this issue. I can provide any help if required. Thanks.
pipeline link: https://doi-janky.infosiftr.net/job/multiarch/job/s390x/job/mariadb/
screenshot:



 Comments   
Comment by Daniel Black [ 2023-06-22 ]

Thanks for the bug report.

It appears to be something in the Docker CI (infosiftr was acquired by Docker Inc).

I've forwarded the bug to them to understand: https://github.com/docker-library/official-images/issues/14923

Comment by Daniel Black [ 2023-06-23 ]

I'm a bit worried that 10.5, 10.6 succeed and the others failed. This could distro kernel related, 10.9+ are jammy based and would be using iouring, 10.5/10.6 wouldn't). So this could be a s390x kernel uring problem.

I removed the initialization in https://github.com/MariaDB/mariadb-docker/pull/518 that should make it quicker, and avoid the build time exercising of database functions. If there is really a problem in s390x initialization of a database in 10.9+ this needs to be found. I'll look at pushing a release early next week that passes hopefully and there should at least be a release there.

Can you test quay.io/mariadb-foundation/mariadb-devel:$version images on real hardware as they haven't had trouble building at all.

As a test clone https://github.com/MariaDB/mariadb-docker

and:

.test/run quay.io/mariadb-foundation/mariadb-devel:10.9

Will start/stop exercise the functionality.

Pushing a hammerdb/sysbench oltp (RW) at the container would be useful too.

Comment by Daniel Black [ 2023-06-27 ]

The s390x released on Docker Hub on the official releases have been updated so are available in all 10.5+ containers like they should have been.

There is still most likely an iouring problem with at least some kernels on s390x for mariadb containers 10.9+. To confirm, after exercising a stalling test case (like previous), disable the liburing with --innodb_use_native_aio=0 and see the same behaviour exists.

Comment by Ehsan Kiani Far [ 2023-06-27 ]

Thanks @danblack for your help and support. I am deeply thankful.

Generated at Thu Feb 08 10:24:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.