dbart as discussed last week and to not forget this, I have created this jira task.

I've started on the automation to do this going forward.

For the archive server I have created https://archive.mariadb.org/yum and https://archive.mariadb.org/repo directories with symlinks for all releases of 10.2 through 10.8

Can you take a look to verify they look OK?

Am I able to directly ssh to the server that powers mirror.mariadb.org so that I can add/remove symlinks as necessary?

Thanks.

> Can you take a look to verify they look OK?

LGTM, I used the following to check that there is no broken symlink. Warning, it takes ages on archive (it's faster on hasky).

cd archive_mount_point && find . -xtype l

> Am I able to directly ssh to the server that powers mirror.mariadb.org so that I can add/remove symlinks as necessary?

You were not but you should now be able to:

host hz-mirror

hostname 162.55.42.214

port 65001

dbart I have just tested the new repository by forcing yum.mariadb.org to 162.55.42.214 (this is the IP of mirror.mariadb.org).

The vhost is configured as follow:

server {

  listen 80 ;

  listen [::]:80 ;

  listen 443 ssl;

  listen [::]:443 ssl;

  server_name yum.mariadb.org;

  return 301 $scheme://rpm.mariadb.org/yum$request_uri;

}

This way I was able to use the new service with the following repo configuration:

[root@9c7971e80113 /]# cat /etc/yum.repos.d/Mariadb.repo

[mariadb]

name = MariaDB

baseurl = https://yum.mariadb.org/10.5/fedora35-amd64

gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB

gpgcheck=1

So far everything looks good! So, we could add in the DNS zone a third IP for yum.mariadb.org. Before proceeding I'd like if you could test and double check that it works on your side?

For instance, by modifying your `/etc/hosts` and clicking on https://yum.mariadb.org/10.5/centos73-ppc64le/sha512sums.txt?mirrorlist, it should propose you routing decision based on your Geolocation.

Hmm there is still a problem. The actual https://yum.mariadb.org repo contains more stuff than https://mirror.mariadb.org/yum... Not sure what to do about it.

yum.mariadb.org also contains specific version links, for those that want them. I've added the symlinks for the current versions and can make sure they're there moving forward.

Other than that, we also have galera test repositories that we use for testing new galera versions. I can also manage putting those in place.

Let's quickly discuss this tomorrow because I think it would be better if the mirror.mariadb.org don't diverge from what we have on all our mirrors.

yum.mariadb.org is now compatible with rpm.mariadb.org and redirects to mirror.mariadb.org/yum.

dbart do you think that I can start adding a third IPv4 address to the yum.mariadb.org record (we should see 1/3 of the actual yum.mariadb.org traffic on the new service).

Also, I can create an IPv6 record (AAAA) for this subdomain as the new service is compatible with IPv6.

yes, go ahead and add a third IPv4 address, might as well add IPv6 too

Done:

❯ grep yum domains/mariadb.org.txt

yum.mariadb.org.	1	IN	A	162.55.42.214

yum.mariadb.org.	1	IN	A	142.4.219.197

yum.mariadb.org.	1	IN	A	142.4.217.28

yum.mariadb.org.	1	IN	AAAA	2a01:4f8:1c17:e53d::1

I now see much more traffic on the mirror.mariadb.org machine coming from yum.mariadb.org, see screenshot:

Here are some stuff that we may have to handle:

• treeinfo (or .treeinfo) not found. From what I have read (https://docs.centos.org/en-US/8-docs/advanced-install/assembly_preparing-for-a-network-install/ and https://release-engineering.github.io/productmd/treeinfo-1.0.html) we do not need this file and I should disable logging for this.
• the following versions are not found:
• 5.5
• 10.0
• 10.1

Not sure about what needs to be done as those are EOL...

We ran into one issue with a build config in our Jenkins setup, but elenst was able to work around it. Other than that I don't think there's anything internal that cares about the 5.5/10.0/10.1 repositories. We were mainly keeping them around for people who still may be running those versions. But it would probably be best if we remove repositories of EOL versions after a set amount of time, maybe six months? I'm not sure where we would document this, does the Foundation have a page similar to the Corporation's Engineering Policies page?

dbart I will raise this tomorrow at our Tuesday meeting.
In the mean time, I have removed yum1 (bb01: 142.4.219.197) from the DNS zone.

So only yum2 (bb02: 142.4.217.28) should still receive traffic from yum.mariadb.org.

dbart, beyond Maintenance Policy, no. We didn't discuss at the meeting, but removing EOL versions after six months sounds good to me.

There are a few problems I'm getting which appear to be related to this activity.

1. On ppc64 VMs in buildbot we are getting this:

+ wget http://yum.mariadb.org/10.1/centos73-ppc64/repodata -O repodata.list

--2022-04-15 20:58:29--  http://yum.mariadb.org/10.1/centos73-ppc64/repodata

Resolving yum.mariadb.org (yum.mariadb.org)... 142.4.217.28, 162.55.42.214, 2a01:4f8:1c17:e53d::1

Connecting to yum.mariadb.org (yum.mariadb.org)|142.4.217.28|:80... connected.

HTTP request sent, awaiting response... 301 Moved Permanently

Location: http://yum.mariadb.org/10.1/centos73-ppc64/repodata/ [following]

--2022-04-15 20:58:29--  http://yum.mariadb.org/10.1/centos73-ppc64/repodata/

Reusing existing connection to yum.mariadb.org:80.

HTTP request sent, awaiting response... 200 OK

In fact, http://yum.mariadb.org/10.1/centos73-ppc64/repodata/ is no longer there. It's not a big problem, as 10.1 is long EOL-ed and it's used for 10.2 upgrade tests, which is going to EOL too, so we can tolerate losing it. The actual problem is that it returns 200 OK. Then instead of detecting the absence and giving up, the test proceeds and later fails awkwardly when it attempts to install packages which aren't available.

When I ran the same wget command manually from within the same VM, I got 404 as expected.

2. Certificate problem

+ wget http://yum.mariadb.org/10.3/centos74-amd64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz

--2022-04-14 12:08:52--  http://yum.mariadb.org/10.3/centos74-amd64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz

Resolving yum.mariadb.org (yum.mariadb.org)... 162.55.42.214, 142.4.217.28, 2a01:4f8:1c17:e53d::1

Connecting to yum.mariadb.org (yum.mariadb.org)|162.55.42.214|:80... connected.

HTTP request sent, awaiting response... 301 Moved Permanently

Location: http://mirror.mariadb.org/yum/10.3/centos74-amd64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz [following]

--2022-04-14 12:08:52--  http://mirror.mariadb.org/yum/10.3/centos74-amd64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz

Resolving mirror.mariadb.org (mirror.mariadb.org)... 162.55.42.214, 2a01:4f8:1c17:e53d::1

Reusing existing connection to yum.mariadb.org:80.

HTTP request sent, awaiting response... 302 Found

Location: https://mirror.fi.mariadb.org/mariadb-10.3.34/yum/rhel/7/x86_64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz [following]

--2022-04-14 12:08:53--  https://mirror.fi.mariadb.org/mariadb-10.3.34/yum/rhel/7/x86_64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz

Resolving mirror.fi.mariadb.org (mirror.fi.mariadb.org)... 135.181.6.219, 2a01:4f9:4b:53a4::2

Connecting to mirror.fi.mariadb.org (mirror.fi.mariadb.org)|135.181.6.219|:443... connected.

ERROR: cannot verify mirror.fi.mariadb.org's certificate, issued by ‘/C=US/O=Let's Encrypt/CN=R3’:

  Issued certificate has expired.

To connect to mirror.fi.mariadb.org insecurely, use `--no-check-certificate'.

I can add --no-check-certificate of course, but it doesn't seem right.

This went all right:

+ wget http://yum.mariadb.org/10.3/centos74-amd64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz

--2022-04-14 10:25:46--  http://yum.mariadb.org/10.3/centos74-amd64/repodata/a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz

Resolving yum.mariadb.org (yum.mariadb.org)... 142.4.217.28, 162.55.42.214, 2a01:4f8:1c17:e53d::1

Connecting to yum.mariadb.org (yum.mariadb.org)|142.4.217.28|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 21211 (21K) [application/gzip]

Saving to: ‘a11828bb5dc0c2ed9702e899af8ec0d5db711871ead9bee67c5ec55ff58dd18b-primary.xml.gz’

3. 302 errors during RPM package installation

(17/31): MariaDB-test-debu 40% [======          ] 7.4 MB/s | 151 MB   00:29 ETA 

http://yum.mariadb.org/10.4/centos73-ppc64/rpms/MariaDB-test-debuginfo-10.4.24-1.el7.centos.ppc64.rpm: [Errno 14] HTTPS Error 302 - Found

Trying other mirror.

I've put in a workaround for our build tests.

So for the 3 issues:

1. Doesn't apply to end users (it's 10.1), and my workaround will sort it for our internal uses.

2. I did visit https://mirror.fi.mariadb.org/ directly and its cert is valid, so the fact that the centos74-amd64 builder can't verify it means that the VM needs to have the ca-certificates package updated. In theory any up-to-date centos7 will not have this issue. But it is something to keep an eye on if we see end user reports of the same issue.
(I will also update our centos7 VMs to update the ca-certificates package)

3. I don't know what to make of this one. The 302 error indicates a temp redirect, but then why is it failing after that? Will need some investigation. When trying the url in question locally, the redirection works just fine and I download the file. So like n.2 it may be that end users won't see this issue, but the fact that we saw it means it could happen. My best guess is that it tried to redirect to a mirror where the package didn't exist? I've solved it for our internal tests (removed the possibility for redirection to occur), but we'll need to keep an eye out for user reports.

With the updates to the buildbot config, builds are not failing on server upgrade tests like they were:

• https://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos73-ppc64
• https://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos73-ppc64le

Hi!
I owe you some clarification here, sorry for the delay, I was off for Easter and my previous comments were not clear enough.

So, first of all the actual setup is:

• 1/3 of the IPv4 requests are reaching old yum.mariadb.org server (142.4.219.197);
• 2/3 of the IPv4 requests are reaching the new mirror redirector system (https://mirror.mariadb.org/mirrorstats);
• 100% of IPv6 requests are reaching the new mirror redirector system.

This is accomplished by round robin DNS and can be understood with the following command:

$ watch dig A yum.mariadb.org

You will see a rotation of IPs. If you want to be sure to reach the old yum server, you can modify you '/etc/hosts' file.

Now regarding the problems raised by elenst:

1/ Even if 10.2 is becoming EOL soon, this is still something that we will need to address because we will have it for 10.3 and so on. I suggest we use https://archive.mariadb.org that should be quite reliable for our CI. If it's not, I am happy to find a more robust solution also depending on where (geographically) our CI workers are running.

2/ Yes, dbart is correct. But this is also something that can be handled by not forcing https on the new mirror.mariadb.org system. Not sure that this is desirable though...

3/ That looks like a problem with the old yum server, it's strange because this machine is generally less loaded (than the other yum server that was disabled) and should handle request quite good. It can also be a network problem and that's exactly what should the new mirror.mariadb.org system resolve (or handle better) by picking a nearer mirror.

FYI,
I have just updated the mariadb.org DNS zone and removed the last old yum mirror (142.4.217.28).

In case people still need stuff from those mirrors, they can edit their `/etc/hosts` files with the following:

142.4.217.28 yum.mariadb.org

or

142.4.219.197 yum.mariadb.org

I will continue to renew TLS certificates on those server for some time, just in case we should re-activate them.

Also, we publicly communicated on the new mirror manager system:
https://mariadb.org/mirrorbits