[MDEV-29765] Packages signed with weak cipher causing issues with FIPS Created: 2022-10-11  Updated: 2023-02-07  Resolved: 2023-02-07

Status: Closed
Project: MariaDB Server
Component/s: Packaging, Repositories
Affects Version/s: 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10
Fix Version/s: 10.3.38, 10.4.28, 10.5.19, 10.6.12, 10.7.8, 10.8.7, 10.9.5, 10.10.3

Type: Bug Priority: Major
Reporter: Amit Narkhede Assignee: Daniel Bartholomew
Resolution: Fixed Votes: 0
Labels: None
Environment:

SLES 15 SP4


 Description   

We recently started testing deployments of SLES 15 SP4 and we found out that MariaDB repo doesn't work when FIPS is enabled on the machine.

The following steps can be taken to reproduce the issue.
1. Deploy a machine with SLES 15 SP4
2. Add MariaDB package repository script from https://mariadb.com/kb/en/installing-mariadb-with-zypper/#using-the-mariadb-package-repository-setup-script
3. Enable FIPS mode on SLES using steps documented at https://www.suse.com/support/kb/doc/?id=000019432
4. Run command zypper ref on terminal.

Error message we get from doing the above steps looks something like below.

testsles15sp4vm:/tmp # zypper ref
 
 
 
Note: The rpm database seems to contain old V3 version gpg keys which are meanwhile obsolete and
 
considered insecure:
 
 
 
        gpg-pubkey-1bb943db-511147a9
 
 
 
    To see details about a key call 'rpm -qi GPG-PUBKEY-VERSION'.
 
 
 
    Unless you believe the key in question is still in use, you can remove it from the rpm database
 
    calling 'rpm -e GPG-PUBKEY-VERSION'.
 
 
 
Repository 'SLE-Module-Basesystem15-SP4-Pool' is up to date.                                                                                                                                                         
Repository 'SLE-Module-Basesystem15-SP4-Updates' is up to date.                                                                                                                                                      
Repository 'SLE-Module-Containers15-SP4-Pool' is up to date.                                                                                                                                                         
Repository 'SLE-Module-Containers15-SP4-Updates' is up to date.                                                                                                                                                      
Repository 'SLE-Module-HPC15-SP4-Pool' is up to date.                                                                                                                                                                
Repository 'SLE-Module-HPC15-SP4-Updates' is up to date.                                                                                                                                                             
Repository 'SLE-Module-Legacy15-SP4-Pool' is up to date.                                                                                                                                                             
Repository 'SLE-Module-Legacy15-SP4-Updates' is up to date.                                                                                                                                                          
Repository 'SLE-Module-Packagehub-Subpackages15-SP4-Updates' is up to date.                                                                                                                                          
Repository 'SLE-Module-Packagehub-Subpackages15-SP4-Pool' is up to date.                                                                                                                                             
Repository 'SLE-Module-Public-Cloud15-SP4-Pool' is up to date.                                                                                                                                                       
Repository 'SLE-Module-Public-Cloud15-SP4-Updates' is up to date.                                                                                                                                                    
Repository 'SLE-Module-Server-Applications15-SP4-Pool' is up to date.                                                                                                                                                
Repository 'SLE-Module-Server-Applications15-SP4-Updates' is up to date.                                                                                                                                             
Repository 'SLE-Module-Web-Scripting15-SP4-Pool' is up to date.                                                                                                                                                      
Repository 'SLE-Module-Web-Scripting15-SP4-Updates' is up to date.                                                                                                                                                   
Repository 'SLE-Product-SLES15-SP4-Pool' is up to date.                                                                                                                                                              
Repository 'SLE-Product-SLES15-SP4-Updates' is up to date.                                                                                                                                                           
Repository 'SUSE-PackageHub-15-SP4-Backports-Pool' is up to date.                                                                                                                                                    
Repository 'SUSE-PackageHub-15-SP4-Pool' is up to date.                                                                                                                                                                                                                                                                                                          
Looking for gpg key ID 1BB943DB in cache /var/cache/zypp/pubkeys.
 
Repository mariadb_repository_sles15 does not define additional 'gpgkey=' URLs.
 
Warning: File 'repomd.xml' from repository 'mariadb_repository_sles15' is signed with an unknown key 'CBCB082A1BB943DB'.
 
 
 
    Note: Signing data enables the recipient to verify that no modifications occurred after the data
 
    were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
 
    and in extreme cases even to a system compromise.
 
 
 
    Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
 
    whole repo.
 
 
 
    Warning: We can't verify that no one meddled with this file, so it might not be trustworthy
 
    anymore! You should not continue unless you know it's safe.
 
 
 
File 'repomd.xml' from repository 'mariadb_repository_sles15' is signed with an unknown key 'CBCB082A1BB943DB'. Continue? [yes/no] (no)

Looking further why this is happening only with MariaDB and not other third party repositories we have, we noticed this must be weak signature that is probably being blocked with FIPS now.
MariaDB is still using DSA/SHA1 for signing the packges while all other repositories we have on the system are using SHA256 at minimum.

rpm -qip ./MariaDB-server-10.5.17-1.x86_64.rpm
 
Name        : MariaDB-server
 
Version     : 10.5.17
 
Release     : 1
 
Architecture: x86_64
 
Install Date: (not installed)
 
Group       : Applications/Databases
 
Size        : 132794741
 
License     : GPLv2
 
Signature   : DSA/SHA1, Fri Aug 12 00:08:48 2022, Key ID cbcb082a1bb943db
 



rpm -qip ./emacs-nox-27.2-150400.1.49.x86_64.rpm
 
Name        : emacs-nox
 
Version     : 27.2
 
Release     : 150400.1.49
 
Architecture: x86_64
 
Install Date: (not installed)
 
Group       : Productivity/Text/Editors
 
Size        : 15789584
 
License     : GPL-3.0-or-later
 
Signature   : RSA/SHA256, Thu May 12 03:40:45 2022, Key ID 70af9e8139db7c82

Are there existing tickets or plans on fixing this issue? We need FIPS for compliance reasons and can't deploy using SLES15 SP4 for this reason.

 Comments   
Comment by Daniel Bartholomew [ 2022-10-18 ]

For a while now we've used a stronger key for our Debian and Ubuntu repositories and so I'll run some tests with it to see if that key will work on SLES 15 w/FIPS enabled. If it does then we can definitely migrate our SLES repositories to using it. And maybe we should also migrate our RHEL and Fedora repositories too. I don't know if those have a FIPS mode or not, but getting everything onto the same (stronger) key would be beneficial.

Comment by Daniel Bartholomew [ 2022-10-20 ]

I haven't been able to reproduce the error. On a sles15 VM with fips enabled I did the following:

sudo sysctl -a | grep fips
 
# output showed fips was enabled
 
curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup \
 
| sudo bash -s -- --skip-maxscale --skip-tools --mariadb-server-version=10.5
 
sudo zypper refresh
 
sudo zypper install -y MariaDB-server
 
sudo systemctl start mariadb
 
sudo mariadb -e "SELECT @@version;"

The mariadb.repo file looks like this:

buildbot@sles15:~> cat /etc/zypp/repos.d/mariadb.repo
 
 
 
[mariadb-server]
 
name = MariaDB Server
 
baseurl = https://dlm.mariadb.com/repo/mariadb-server/10.5/yum/sles/15/x86_64
 
gpgkey = file:///etc/pki/trust/MariaDB-Server-GPG-KEY
 
gpgcheck = 1
 
type=rpm-md
 
enabled = 1
 
autorefresh=1
 
priority=10

And the gpgkey file, downloaded and saved to the /etc/pki/trust/ folder by the mariadb_repo_setup script from it's default source at https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY has a sha256sum of:

buildbot@sles15:~> sha256sum /etc/pki/trust/MariaDB-Server-GPG-KEY
 
4e0424bd1d917400afeffe6e014ae2133247c3cce687b038a408a571977a3440  /etc/pki/trust/MariaDB-Server-GPG-KEY

Can you verify the above are in line with your machine? If so, and you're still seeing the error, then which SLES 15 SP are you on? Let me know. Thanks.

Comment by Amit Narkhede [ 2022-10-21 ]

sha256 on the machine I have is also the same 

4e0424bd1d917400afeffe6e014ae2133247c3cce687b038a408a571977a3440

What service pack are you using for the test above? So the issue is not reproducible on SLES 15 SP3, because we did use that for some testing earlier this year. It is reproducible on SLES15 SP4. Can you verify that you are indeed using SLES15 SP4 with output of 

cat /etc/os-release

Comment by Daniel Bartholomew [ 2022-10-21 ]

Yes, it turns out my VM was on SP3. So that's why I wasn't able to reproduce. I'll spin up an SP4 VM. Thanks.

Comment by Daniel Bartholomew [ 2022-10-21 ]

Confirmed it on SLES 15 SP4 with fips enabled.

I've created a temporary test repository signed using the same (stronger) key we use for our Debian/Ubuntu repositories (f1656f24c74cd1d8).

I've temporarily put this repository in a location you can access to test:

baseurl = https://downloads.mariadb.com/custom/mariadb-10.5.17/yum/sles/15/x86_64

To try it out, here are some instructions that should work for a stock SLES 15 (fips-enabled or not) system that doesn't have MariaDB installed. If a MariaDB repository is already configured, just change the baseurl line in your MariaDB repo file to the above and then skip the first two steps and jump to step 3 about importing the updated GPG-KEY file and continue on from there.

#1 run mariadb_repo_setup
 
curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup | sudo bash -s -- --skip-maxscale --skip-tools --mariadb-server-version=10.5
 
 
 
#2 Change the base url to the temp test repo
 
sudo sed -i s"/dlm.mariadb.com\/repo\/mariadb-server\/10.5/downloads.mariadb.com\/custom\/mariadb-10.5.17/" /etc/zypp/repos.d/mariadb.repo
 
 
 
#3 Import an updated GPG-KEY file which has the f1656f24c74cd1d8 pub-key
 
pushd /etc/pki/trust/ ; sudo curl -LsSO https://downloads.mariadb.com/custom/mariadb-10.5.17/MariaDB-Server-GPG-KEY ; popd
 
sudo rpm --import /etc/pki/trust/MariaDB-Server-GPG-KEY
 
 
 
#4 Remove the old keys, so zypper stops complaining about them
 
sudo rpm -e gpg-pubkey-1bb943db-511147a9
 
sudo rpm -e gpg-pubkey-307e3d54-5aaa90a5
 
 
 
#5 Refresh repositories
 
sudo zypper refresh
 
 
 
#6 Install and Start MariaDB (to test that the repo is working properly with zypper)
 
sudo zypper install -y MariaDB-server
 
 
 
sudo systemctl start mariadb
 
 
 
sudo mariadb -e "SELECT @@version;"

Assuming things look good for using the f1656f24c74cd1d8 key, I'll start the planning process for an orderly transition to using this key, probably for all of our zypper and dnf repositories (it doesn't hurt for all of them to be using a stronger key).

Comment by Amit Narkhede [ 2022-10-25 ]

Thanks for the detailed steps above. I created a VM on SLES15 SP4. Tried to reproduce the issue on the custom repo and was not able to. So it looks like this indeed fixes the issue here.
Do we have any timeline to when the transition to the new key will be planned? It may be good idea to give organizations time to migration things over in all environments.

Comment by Amit Narkhede [ 2022-10-25 ]

Just to make sure, as things are working fine now can I assume here that the new key will be the one above? I can use that to prepare our migration as this affects VMs in our live environment even if they are not using SLES15 yet.

Comment by Daniel Bartholomew [ 2022-10-25 ]

Yes, the plan is to migrate to using that key for our yum/dnf/zypper repositories. I don't have a definite timeline yet on this, but I'm starting discussions with the relevant people so the roll-out can be as organized and smooth as possible.

To kickstart things I've today updated the official location for the yum/dnf/zypper repository pub-key - https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY - to include both the legacy and 'new' pub keys, but there are other places that need to be updated that I'm working on. This is a transparent change for end users. When rpm imports a pub-key it doesn't care if the file has one or multiple keys in it. So adding the key to that file now has the effect of pre-staging it for those that set up repositories using our repository setup script from today.

We'll also provide documentation in relevant release notes, the GPG page in the Knowledgebase (and other pages), and probably blog posts for updating existing repositories, which I think will be along the lines of:

sudo rpm --import https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY
 
# package-manager-specific commands to clean and refresh the package cache

(At least, that's what I assume will be needed. I haven't tested migrations yet.)

Comment by Faustin Lammler [ 2022-11-08 ]

FYI: https://github.com/MariaDB/buildbot/pull/68 has just been merged and I have triggered rebuild of knowing green install test on RPM platform (new BB: https://buildbot.mariadb.org/#/builders?tags=%2Binstall&tags=kvm&tags=%2Brpm) to test the v2 version of the key.

Comment by Faustin Lammler [ 2022-12-12 ]

The old key has now been replaced on the root mirror with the new one.

I have just roll-backed since I was not aware of:
https://mariadb.com/kb/en/gpg/#rpm-source-key-2023

Comment by Amit Narkhede [ 2023-01-09 ]

Faustin, I see you did get the migration back in December and rolled it back as it was planned for 2023. Do we have any timeline when this will be attempted again?

Comment by Faustin Lammler [ 2023-01-10 ]

Hi amitnarkhede, yes we plan it for the next release (probably before the end of the month).

Comment by Faustin Lammler [ 2023-01-31 ]

Hi! The new key is now in place on our root mirrors, it is going to be synced in the next hours to all mirrors.

Please note that the v2 key is still available:

  • PublicKey_v2
  • yum/RPM-GPG-KEY-MariaDB_v2

But, we will probably remove the key in the next month since people should use the following:

  • PublicKey
  • yum/RPM-GPG-KEY-MariaDB
Comment by Amit Narkhede [ 2023-02-01 ]

Hi Faustin, Thanks for the update.

I tried to point a VM so that it will fetch things from https://downloads.mariadb.com/MariaDB/mariadb-10.5/yum/sles/15/x86_64 and see that latest packages from that repo still had older signature.

Just to make sure I'm on the same page, is this expected behavior? Shouldn't older package also get re-signed with new key?

Comment by Faustin Lammler [ 2023-02-02 ]

Hi amitnarkhede, yes that's normal since only the next release (that has been delayed) will use the new signature key (so, 10.5.19 in your case).

Comment by Daniel Bartholomew [ 2023-02-06 ]

The releases have now been published.

Comment by Amit Narkhede [ 2023-02-07 ]

Thank you Daniel and Faustin for the help on this.
I can confirm that now things are working as expected and I was able to successfully deploy systems this morning. We can close this ticket.

Comment by Daniel Bartholomew [ 2023-02-07 ]

Great to hear. Closing.

Generated at Thu Feb 08 10:11:07 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.