[MDBF-516] Intermittent archive.mariadb.org issues Created: 2022-12-11  Updated: 2022-12-18  Resolved: 2022-12-12

Status: Closed
Project: MariaDB Foundation Development
Component/s: Downloads
Affects Version/s: N/A
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: cPanel Senior Tech's Assignee: Faustin Lammler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: 0d
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File screenshot-1.png    

 Description   

We received a few reports of MariaDB upgrades failing, for example:

Show all

 
# /usr/bin/yum update
 
Loaded plugins: fastestmirror, universal-hooks
 
Loading mirror speeds from cached hostfile
 
epel/x86_64/metalink                                                                                                      |  22 kB  00:00:00
 
 * EA4: 63.247.64.58
 
 * cpanel-addons-production-feed: 63.247.64.58
 
 * cpanel-plugins: 63.247.64.58
 
 * base: centos.mirror.vexxhost.com
 
 * epel: mirror.dst.ca
 
 * extras: centos.mirror.vexxhost.com
 
 * updates: centos.mirror.vexxhost.com
 
EA4                                                                                                                       | 2.9 kB  00:00:00
 
cpanel-addons-production-feed                                                                                             | 2.9 kB  00:00:00
 
cpanel-plugins                                                                                                            | 2.9 kB  00:00:00
 
https://archive.mariadb.org/mariadb-10.3/yum/centos/7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
 
Trying other mirror.
 
To address this issue please refer to the below wiki article
 
 
 
https://wiki.centos.org/yum-errors
 
 
 
If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
 
 
 
 
 
 
 One of the configured repositories failed (MariaDB103),
 
 and yum doesn't have enough cached data to continue. At this point the only
 
 safe thing yum can do is fail. There are a few ways to work "fix" this:
 
 
 
     1. Contact the upstream for the repository and get them to fix the problem.
 
 
 
     2. Reconfigure the baseurl/etc. for the repository, to point to a working
 
        upstream. This is most often useful if you are using a newer
 
        distribution release than is supported by the repository (and the
 
        packages for the previous distribution release still work).
 
 
 
     3. Run the command with the repository temporarily disabled
 
            yum --disablerepo=MariaDB103 ...
 
 
 
     4. Disable the repository permanently, so yum won't use it by default. Yum
 
        will then just ignore the repository until you permanently enable it
 
        again or use --enablerepo for temporary usage:
 
 
 
            yum-config-manager --disable MariaDB103
 
        or
 
            subscription-manager repos --disable=MariaDB103
 
 
 
     5. Configure the failing repository to be skipped, if it is unavailable.
 
        Note that yum will try to contact the repo. when it runs most commands,
 
        so will have to try and fail each time (and thus. yum will be be much
 
        slower). If it is a very temporary problem though, this is often a nice
 
        compromise:
 
 
 
            yum-config-manager --save --setopt=MariaDB103.skip_if_unavailable=true
 
 
 
failure: repodata/repomd.xml from MariaDB103: [Errno 256] No more mirrors to try.
 
https://archive.mariadb.org/mariadb-10.3/yum/centos/7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

We also saw the issue was with archive.mariadb.org specifically as well (it was showing as empty):

# curl https://archive.mariadb.org/
 
<html>
 
<head><title>404 Not Found</title></head>
 
<body>
 
<center><h1>404 Not Found</h1></center>
 
<hr><center>nginx</center>
 
</body>
 
</html>
 
/</h1>
 
<table id="list"><thead><tr><th style="width:55%"><a href="?C=N&amp;O=A">File Name</a>&nbsp;<a href="?C=N&amp;O=D">&nbsp;&darr;&nbsp;</a></th><th style="width:20%"><a href="?C=S&amp;O=A">File Size</a>&nbsp;<a href="?C=S&amp;O=D">&nbsp;&darr;&nbsp;</a></th><th style="width:25%"><a href="?C=M&amp;O=A">Date</a>&nbsp;<a href="?C=M&amp;O=D">&nbsp;&darr;&nbsp;</a></th></tr></thead>
 
<tbody></tbody></table></body></html>%
 
\

This can cause unintended database outages and we received about 4 reports in a small time frame. Is there any way to know what may have occurred with the archive.mariadb.org repo (around 12 PM CST Dec 11 2022)?

 Comments   
Comment by Daniel Black [ 2022-12-11 ]

As far as I can tell so far it dropped a mount of /mnt/archive and at some point it came back.

I'll keep an eye on it today and ask our sysadmin to look closer at the cause.

Comment by Daniel Black [ 2022-12-12 ]

Note the mirrorbits based rpm.mariadb.org actually has some redundancy on the packages on it and is recommended as a repo url.

Below uses the _v2 key file which includes the new sha2 gpg key beginning operation in 2023.

$ podman run --rm -ti centos:7
 
[root@b345257d54c3 /]# cat > /etc/yum.repos.d/mariadb.repo
 
[mariadb]
 
name = MariaDB
 
baseurl = https://rpm.mariadb.org/10.3/centos7-amd64
 
gpgkey=https://rpm.mariadb.org/RPM-GPG-KEY-MariaDB_v2
 
gpgcheck=1

Comment by Faustin Lammler [ 2022-12-12 ]

Hi,
thank you for warning us about this problem.

We had a downtime yesterday on the archive.mariadb.org service (dates are in GMT):
Start: 2022-12-11 12:00:58
End: 2022-12-11 20:33:52
Duration: 8h 32m 54s

We are still investigating the root cause but it's probably related to a faulty core router (https://status.hetzner.com/incident/75b49f8f-e4df-4bbd-b140-bc0257ccdedf).

As mentioned by Daniel, the archive.mariadb.org service should probably not be used as a default repository for actual releases of MariaDB, it is there to make available old release/binary but is not as reliable as mirror.mariadb.org or any official public mirror.

Comment by Faustin Lammler [ 2022-12-15 ]

cPanelSTA can you make sure that cpanel does not use (hard coded) archive.mariadb.org by default since it's probably the worse candidate (the best being mirror.mariadb.org since it's geographic independent).
Thanks!

Comment by cPanel Senior Tech's [ 2022-12-15 ]

MariaDB 10.2 seems to be missing from mirror.mariadb.org - the absence of versions can cause numerous issues during upgrade attempts. I believe this is why we switched to archive.mariadb.org

Can you ensure 10.2 is re-added to mirror.mariadb.org?
Thanks!

Comment by Daniel Black [ 2022-12-15 ]

Good point cPanelSTA, we restored the last version of the out of date versions on mirrors previously and they are gone again. I'll chase it up.

Comment by Faustin Lammler [ 2022-12-17 ]

cPanelSTA we re-added 10.2.44 (latest 10.2 version that contains also some older versions, see for instance https://mirror.mariadb.org/mariadb-10.2.44/repo/debian/pool/main/m/mariadb-10.2/) on our root mirrors yesterday. It should be synchronized across all mirrors during the week-end. So feel free to switch to mirror.mariadb.org or any mirror from https://mirmon.mariadb.org. The mirror from Netcologne is also a good candidate, it's our EU primary mirror and it's reliable.

Remember that 10.2 is EOL (you probably already know), so I still encourage you to move to the next LTS version 10.6 or 10.11 are probably good candidates. We can help you in that task, do not hesitate to ping us.

Comment by cPanel Senior Tech's [ 2022-12-18 ]

Thank you!

The reason for needing the EOL versions is due to how we upgrade (ie - user on 10.1 updates to 10.6, update process will go incrementally from the versions we support [10.2->10.3->10.4->10.6]). Seems to encounter the least amount of issues and helps ensure schema changes are applied.

Generated at Thu Feb 08 03:38:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.