Debian/Ubuntu

Is https://archive.mariadb.org/ usable for you?

archive.mariadb.org is certainly an archive, but it has nothing to do with properly packaged debian/ubuntu repositories. This issue is about the sudden disappearance of the Debian/Ubuntu apt repositories. As I know you know, the apt repositories allow a user to install a particular version with confidence on a Debian/Ubuntu system. archive.mariadb.org, on the other hand, is a hodgepodge of directories of tarballs that might be installable by a knowledgeable person... who knew how to choose the correct one... and who knew how to figure out whether it was fully functional or not. An entirely different thing, and for an entirely different purpose.

I agree there are certainly messy aspects to it. Its repo directories however can be used as repositories.

https://mariadb.com/kb/en/installing-mariadb-deb-files/#pinning-the-mariadb-repository-to-a-specific-minor-release

If there's aspects of this that don't work correctly we'll rectify. Its only useful if its usable.

dbart will know why these where removed from osuosl mirror.

Thanks for you ddev project. I did at some point want to run CI against websites with MariaDB as a backend. This will probably help.

That's a slick trick pointing to the archive as a repo, it might work out for ddev.

Right now, though, if you have MariaDB 10.0, for example, installed, and you have the mariadb specified like this (as it is in the official docker mariadb:10.0):

```
deb http://ftp.osuosl.org/pub/mariadb/repo/10.0/ubuntu xenial main
```
as soon as you do an `apt-get update` you have a fatal error, because that directory is no longer there.

I try to listen on mailing lists and such and didn't hear any announcement that all these debian/ubuntu repos would be removed. Was it announced and I just missed it? Is there a discussion somewhere?

(I checked more than one mirror, it's not just osuosl, pretty sure it's all of them)

Thanks for the nice words about ddev. Lots of people use it for CI as well as for local development. There's even a github action for it, https://github.com/marketplace/actions/setup-ddev

I understand the frustration at its sudden disappearance. I don't recall seeing a discussion/announcement either. cvicentiu do you know?

osuosl is the primary mirror for all the others.

I've found https://github.com/drud/ddev/tree/master/containers/ddev-dbserver as your base. I'm missing how repositories fit into your build picture. I'm always keen to find new projects using mariadb in interesting ways. I do want to make it good for you again.

I'm assuming you know all the mariadb tags got back to late 5.5 - http://soaphub.org/imagehub/?q=mariadb&n=library&p=1

The reason I knew about the removal right away is the ddev nightly build failed (there's a nightly image build as part of that). Each of the mariadb and mysql images derives from the official Docker image of the same version. Since each of those has the appropriate mariadb repo description in /etc/apt/sources.list.d/mariadb.list, the disappearance of these immediately broke the build, because you can't even apt-get update with those disappeared, https://app.circleci.com/pipelines/github/drud/ddev/4568/workflows/f661e289-99fd-47cd-aca5-0578379a1324/jobs/33627

I did find a workaround to get the builds going though, remove the mariadb.list. Since the upstreams already have everything installed, that seems to do no harm. https://github.com/drud/ddev/pull/3096/commits/349225d1cd54aaec5231cc5d398c57f27b034656

Thanks for caring about this issue!

So it looks like your workaround is actually a good ruggard fix.

The only reason I haven't done the same thing in upstream docker library is that from 10.0 onwards there are extra storage engine plugins in separate package here that people may want to install. For future installs I could take my own advice here and use an archive.mariadb.org repo URL as the persistent location in the image for those that depend on it beyond EOL and what might be on the OSLOSU mirror.

As an alternate for your rm -f /etc/apt/sources.list.d/mariadb.list if you wanted to consider plugins:

[ -f /etc/apt/sources.list.d/mariadb.list ] && sed -i -e 's:ftp\.osuosl\.org/pub:archive.mariadb.org:' -e 's:10\.0/:10\.0\.38/:' -e 's:10\.1/:10.1.48/:' /etc/apt/sources.list.d/mariadb.list

With that there seem to be no need to restore the full archives onto OSUOSL mirrors and take the following actions:

• do some announcement about not having EOL versions onto the osuosl mirror, pointing to the archives
• adjust the docker library images to use archive.mariadb.org as a repo url as the preserved URL in some way (dbart I'll chat with you before this to ensure release processes are predictable).

Is that acceptable? Anything else?

I think an announcement and making the docker library images work as is would be good. Normally these things are announced at least months in advance though, so that would be the thing to do.

I'm also wondering about how the EOL of the packages for 5.5 (based on Ubuntu 14.04). They're on archive.ubuntu.com

$ podman run -ti   mariadb:5.5 bash

root@a4e0b208f4f5:/# more /etc/apt/sources.list.d/mariadb.list 

deb http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu trusty main

The archive.ubuntu.com packages aren't the same was the MariaDB released packages. I don't know the exact difference or what compatibility issues need to be considered there. I left of 5.5 in my oneliner as there where no plugins however a replacement to 5.5.68 is ok.

And we'll need to fix the archive repos

root@e320ae9db34f:/# apt-get update

Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease                                                       

Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease

Hit:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease

Hit:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease

Ign:1 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial InRelease

Hit:6 https://repo.percona.com/apt xenial InRelease

Ign:7 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial Release

Ign:8 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main amd64 Packages

Ign:9 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main all Packages

Ign:8 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main amd64 Packages

Ign:9 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main all Packages

Ign:8 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main amd64 Packages

Ign:9 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main all Packages

Ign:8 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main amd64 Packages

Ign:9 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main all Packages

Ign:8 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main amd64 Packages

Ign:9 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main all Packages

Err:8 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main amd64 Packages

  404  Not Found

Ign:9 https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial/main all Packages

Reading package lists... Done

W: The repository 'http://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu xenial Release' does not have a Release file.

N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.

N: See apt-secure(8) manpage for repository creation and user configuration details.

E: Failed to fetch https://archive.mariadb.org/mariadb/repo/10.0.38/ubuntu/dists/xenial/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.

So, fixing one problem led to creating several others. Very sorry about that.

The issue is that we were taking up far too much space relative to other projects on the OSUOSL mirror. To "solve" it I removed old, unmaintained versions of MariaDB and did some de-duplication using symlinks to point to the correct version instead of making full copies of the information under the repo directory. The thinking in my head was that no one would still be using the old unmaintained repositories so it would be ok to just do the removal, but I was obviously wrong about that. I'll discuss with danblack later today on what to do to fix/restore/make-things-right. Thanks.

Thanks so much!

Seem my assessment the archive not being suitable as a mirror was just using an unsupported disto for that version.

The KB page has been updated a little.

https://mariadb.org/?p=22182&preview=1&_ppp=0ba2c1f978 is the draft blog if you think I've missed anything please let me know and I'll take it into account before publishing.

Thanks for this, and for the restoration.

I see that the official images (mariadb:5.5 for example) work again with an apt-get update, thanks!

Blog published showing a few of the workarounds that can be done for older versions.

New docker library images as of about 5 hours ago are using archive.mariadb.org as the repo url on the latest current supported images.

Thanks rfay for bringing this to our attention.

Thanks for all the effort on this!

I don't see any new pushes on 5.5 official docker images for example, https://hub.docker.com/_/mariadb?tab=tags&page=1&ordering=last_updated&name=5.5

Is there a way to get notified about new blog posts? I'm on the mariadb mailing list... but don't think that does it. (I just added it to my rss/feedly feed, but would love it to come by email also)

I wasn't going to update the EOL releases. Though the docker library release process they would show up as current again and I didn't want that to happen. By putting the last EOL releases back onto the mirror those now work for your cases where you apt-update on a MariaDB base image. I'm hoping for now that those using a pre-last of version EOL images aren't adding packages and rebuilding this image from source. If they are, the work around of removing the apt file (like you did) or changing the repo url is in the blog.

On blog email notices, I tried something manual. I don't know if it will work. I'll ask around our other staff more familiar with it.

I understand why you wouldn't update the EOL releases; was just responding to what you said above, but I guess you were saying "New (supported) docker library images as of about 5 hours ago are using archive.mariadb.org as the repo url on the latest current supported images" rather than just "New docker library images as of about 5 hours ago are using archive.mariadb.org as the repo url on the latest current supported images"