Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29765

Packages signed with weak cipher causing issues with FIPS

Details

    Description

      We recently started testing deployments of SLES 15 SP4 and we found out that MariaDB repo doesn't work when FIPS is enabled on the machine.

      The following steps can be taken to reproduce the issue.
      1. Deploy a machine with SLES 15 SP4
      2. Add MariaDB package repository script from https://mariadb.com/kb/en/installing-mariadb-with-zypper/#using-the-mariadb-package-repository-setup-script
      3. Enable FIPS mode on SLES using steps documented at https://www.suse.com/support/kb/doc/?id=000019432
      4. Run command zypper ref on terminal.

      Error message we get from doing the above steps looks something like below.

      testsles15sp4vm:/tmp # zypper ref
      		 
       
      		 
      Note: The rpm database seems to contain old V3 version gpg keys which are meanwhile obsolete and
      		 
      considered insecure:
      		 
       
      		 
              gpg-pubkey-1bb943db-511147a9
      		 
       
      		 
          To see details about a key call 'rpm -qi GPG-PUBKEY-VERSION'.
      		 
       
      		 
          Unless you believe the key in question is still in use, you can remove it from the rpm database
      		 
          calling 'rpm -e GPG-PUBKEY-VERSION'.
      		 
       
      		 
      Repository 'SLE-Module-Basesystem15-SP4-Pool' is up to date.                                                                                                                                                         
      Repository 'SLE-Module-Basesystem15-SP4-Updates' is up to date.                                                                                                                                                      
      Repository 'SLE-Module-Containers15-SP4-Pool' is up to date.                                                                                                                                                         
      Repository 'SLE-Module-Containers15-SP4-Updates' is up to date.                                                                                                                                                      
      Repository 'SLE-Module-HPC15-SP4-Pool' is up to date.                                                                                                                                                                
      Repository 'SLE-Module-HPC15-SP4-Updates' is up to date.                                                                                                                                                             
      Repository 'SLE-Module-Legacy15-SP4-Pool' is up to date.                                                                                                                                                             
      Repository 'SLE-Module-Legacy15-SP4-Updates' is up to date.                                                                                                                                                          
      Repository 'SLE-Module-Packagehub-Subpackages15-SP4-Updates' is up to date.                                                                                                                                          
      Repository 'SLE-Module-Packagehub-Subpackages15-SP4-Pool' is up to date.                                                                                                                                             
      Repository 'SLE-Module-Public-Cloud15-SP4-Pool' is up to date.                                                                                                                                                       
      Repository 'SLE-Module-Public-Cloud15-SP4-Updates' is up to date.                                                                                                                                                    
      Repository 'SLE-Module-Server-Applications15-SP4-Pool' is up to date.                                                                                                                                                
      Repository 'SLE-Module-Server-Applications15-SP4-Updates' is up to date.                                                                                                                                             
      Repository 'SLE-Module-Web-Scripting15-SP4-Pool' is up to date.                                                                                                                                                      
      Repository 'SLE-Module-Web-Scripting15-SP4-Updates' is up to date.                                                                                                                                                   
      Repository 'SLE-Product-SLES15-SP4-Pool' is up to date.                                                                                                                                                              
      Repository 'SLE-Product-SLES15-SP4-Updates' is up to date.                                                                                                                                                           
      Repository 'SUSE-PackageHub-15-SP4-Backports-Pool' is up to date.                                                                                                                                                    
      Repository 'SUSE-PackageHub-15-SP4-Pool' is up to date.                                                                                                                                                                                                                                                                                                          
      Looking for gpg key ID 1BB943DB in cache /var/cache/zypp/pubkeys.
      		 
      Repository mariadb_repository_sles15 does not define additional 'gpgkey=' URLs.
      		 
      Warning: File 'repomd.xml' from repository 'mariadb_repository_sles15' is signed with an unknown key 'CBCB082A1BB943DB'.
      		 
       
      		 
          Note: Signing data enables the recipient to verify that no modifications occurred after the data
      		 
          were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
      		 
          and in extreme cases even to a system compromise.
      		 
       
      		 
          Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
      		 
          whole repo.
      		 
       
      		 
          Warning: We can't verify that no one meddled with this file, so it might not be trustworthy
      		 
          anymore! You should not continue unless you know it's safe.
      		 
       
      		 
      File 'repomd.xml' from repository 'mariadb_repository_sles15' is signed with an unknown key 'CBCB082A1BB943DB'. Continue? [yes/no] (no)

      Looking further why this is happening only with MariaDB and not other third party repositories we have, we noticed this must be weak signature that is probably being blocked with FIPS now.
      MariaDB is still using DSA/SHA1 for signing the packges while all other repositories we have on the system are using SHA256 at minimum.

      rpm -qip ./MariaDB-server-10.5.17-1.x86_64.rpm
      		 
      Name        : MariaDB-server
      		 
      Version     : 10.5.17
      		 
      Release     : 1
      		 
      Architecture: x86_64
      		 
      Install Date: (not installed)
      		 
      Group       : Applications/Databases
      		 
      Size        : 132794741
      		 
      License     : GPLv2
      		 
      Signature   : DSA/SHA1, Fri Aug 12 00:08:48 2022, Key ID cbcb082a1bb943db
      		 
      


      rpm -qip ./emacs-nox-27.2-150400.1.49.x86_64.rpm
      		 
      Name        : emacs-nox
      		 
      Version     : 27.2
      		 
      Release     : 150400.1.49
      		 
      Architecture: x86_64
      		 
      Install Date: (not installed)
      		 
      Group       : Productivity/Text/Editors
      		 
      Size        : 15789584
      		 
      License     : GPL-3.0-or-later
      		 
      Signature   : RSA/SHA256, Thu May 12 03:40:45 2022, Key ID 70af9e8139db7c82

      Are there existing tickets or plans on fixing this issue? We need FIPS for compliance reasons and can't deploy using SLES15 SP4 for this reason.

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          dbart Daniel Bartholomew added a comment -

          For a while now we've used a stronger key for our Debian and Ubuntu repositories and so I'll run some tests with it to see if that key will work on SLES 15 w/FIPS enabled. If it does then we can definitely migrate our SLES repositories to using it. And maybe we should also migrate our RHEL and Fedora repositories too. I don't know if those have a FIPS mode or not, but getting everything onto the same (stronger) key would be beneficial.

          dbart Daniel Bartholomew added a comment - For a while now we've used a stronger key for our Debian and Ubuntu repositories and so I'll run some tests with it to see if that key will work on SLES 15 w/FIPS enabled. If it does then we can definitely migrate our SLES repositories to using it. And maybe we should also migrate our RHEL and Fedora repositories too. I don't know if those have a FIPS mode or not, but getting everything onto the same (stronger) key would be beneficial.
          dbart Daniel Bartholomew added a comment -

          I haven't been able to reproduce the error. On a sles15 VM with fips enabled I did the following:

          sudo sysctl -a | grep fips
          		 
          # output showed fips was enabled
          		 
          curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup \
          		 
          | sudo bash -s -- --skip-maxscale --skip-tools --mariadb-server-version=10.5
          		 
          sudo zypper refresh
          		 
          sudo zypper install -y MariaDB-server
          		 
          sudo systemctl start mariadb
          		 
          sudo mariadb -e "SELECT @@version;"

          The mariadb.repo file looks like this:

          buildbot@sles15:~> cat /etc/zypp/repos.d/mariadb.repo
          		 
           
          		 
          [mariadb-server]
          		 
          name = MariaDB Server
          		 
          baseurl = https://dlm.mariadb.com/repo/mariadb-server/10.5/yum/sles/15/x86_64
          		 
          gpgkey = file:///etc/pki/trust/MariaDB-Server-GPG-KEY
          		 
          gpgcheck = 1
          		 
          type=rpm-md
          		 
          enabled = 1
          		 
          autorefresh=1
          		 
          priority=10

          And the gpgkey file, downloaded and saved to the /etc/pki/trust/ folder by the mariadb_repo_setup script from it's default source at https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY has a sha256sum of:

          buildbot@sles15:~> sha256sum /etc/pki/trust/MariaDB-Server-GPG-KEY
          		 
          4e0424bd1d917400afeffe6e014ae2133247c3cce687b038a408a571977a3440  /etc/pki/trust/MariaDB-Server-GPG-KEY

          Can you verify the above are in line with your machine? If so, and you're still seeing the error, then which SLES 15 SP are you on? Let me know. Thanks.

          dbart Daniel Bartholomew added a comment - I haven't been able to reproduce the error. On a sles15 VM with fips enabled I did the following: sudo sysctl -a | grep fips # output showed fips was enabled curl -LsS https: //r .mariadb.com /downloads/mariadb_repo_setup \ | sudo bash -s -- --skip-maxscale --skip-tools --mariadb-server-version=10.5 sudo zypper refresh sudo zypper install -y MariaDB-server sudo systemctl start mariadb sudo mariadb -e "SELECT @@version;" The mariadb.repo file looks like this: buildbot@sles15:~> cat /etc/zypp/repos .d /mariadb .repo   [mariadb-server] name = MariaDB Server baseurl = https: //dlm .mariadb.com /repo/mariadb-server/10 .5 /yum/sles/15/x86_64 gpgkey = file : ///etc/pki/trust/MariaDB-Server-GPG-KEY gpgcheck = 1 type =rpm-md enabled = 1 autorefresh=1 priority=10 And the gpgkey file, downloaded and saved to the /etc/pki/trust/ folder by the mariadb_repo_setup script from it's default source at https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY has a sha256sum of: buildbot@sles15:~> sha256sum /etc/pki/trust/MariaDB-Server-GPG-KEY 4e0424bd1d917400afeffe6e014ae2133247c3cce687b038a408a571977a3440 /etc/pki/trust/MariaDB-Server-GPG-KEY Can you verify the above are in line with your machine? If so, and you're still seeing the error, then which SLES 15 SP are you on? Let me know. Thanks.
          amitnarkhede Amit Narkhede added a comment -

          sha256 on the machine I have is also the same 

          4e0424bd1d917400afeffe6e014ae2133247c3cce687b038a408a571977a3440

          What service pack are you using for the test above? So the issue is not reproducible on SLES 15 SP3, because we did use that for some testing earlier this year. It is reproducible on SLES15 SP4. Can you verify that you are indeed using SLES15 SP4 with output of 

          cat /etc/os-release

          amitnarkhede Amit Narkhede added a comment - sha256 on the machine I have is also the same 4e0424bd1d917400afeffe6e014ae2133247c3cce687b038a408a571977a3440 What service pack are you using for the test above? So the issue is not reproducible on SLES 15 SP3, because we did use that for some testing earlier this year. It is reproducible on SLES15 SP4. Can you verify that you are indeed using SLES15 SP4 with output of cat /etc/os-release
          dbart Daniel Bartholomew added a comment -

          Yes, it turns out my VM was on SP3. So that's why I wasn't able to reproduce. I'll spin up an SP4 VM. Thanks.

          dbart Daniel Bartholomew added a comment - Yes, it turns out my VM was on SP3. So that's why I wasn't able to reproduce. I'll spin up an SP4 VM. Thanks.
          dbart Daniel Bartholomew added a comment -

          Confirmed it on SLES 15 SP4 with fips enabled.

          I've created a temporary test repository signed using the same (stronger) key we use for our Debian/Ubuntu repositories (f1656f24c74cd1d8).

          I've temporarily put this repository in a location you can access to test:

          baseurl = https://downloads.mariadb.com/custom/mariadb-10.5.17/yum/sles/15/x86_64

          To try it out, here are some instructions that should work for a stock SLES 15 (fips-enabled or not) system that doesn't have MariaDB installed. If a MariaDB repository is already configured, just change the baseurl line in your MariaDB repo file to the above and then skip the first two steps and jump to step 3 about importing the updated GPG-KEY file and continue on from there.

          #1 run mariadb_repo_setup
          		 
          curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup | sudo bash -s -- --skip-maxscale --skip-tools --mariadb-server-version=10.5
          		 
           
          		 
          #2 Change the base url to the temp test repo
          		 
          sudo sed -i s"/dlm.mariadb.com\/repo\/mariadb-server\/10.5/downloads.mariadb.com\/custom\/mariadb-10.5.17/" /etc/zypp/repos.d/mariadb.repo
          		 
           
          		 
          #3 Import an updated GPG-KEY file which has the f1656f24c74cd1d8 pub-key
          		 
          pushd /etc/pki/trust/ ; sudo curl -LsSO https://downloads.mariadb.com/custom/mariadb-10.5.17/MariaDB-Server-GPG-KEY ; popd
          		 
          sudo rpm --import /etc/pki/trust/MariaDB-Server-GPG-KEY
          		 
           
          		 
          #4 Remove the old keys, so zypper stops complaining about them
          		 
          sudo rpm -e gpg-pubkey-1bb943db-511147a9
          		 
          sudo rpm -e gpg-pubkey-307e3d54-5aaa90a5
          		 
           
          		 
          #5 Refresh repositories
          		 
          sudo zypper refresh
          		 
           
          		 
          #6 Install and Start MariaDB (to test that the repo is working properly with zypper)
          		 
          sudo zypper install -y MariaDB-server
          		 
           
          		 
          sudo systemctl start mariadb
          		 
           
          		 
          sudo mariadb -e "SELECT @@version;"

          Assuming things look good for using the f1656f24c74cd1d8 key, I'll start the planning process for an orderly transition to using this key, probably for all of our zypper and dnf repositories (it doesn't hurt for all of them to be using a stronger key).

          dbart Daniel Bartholomew added a comment - Confirmed it on SLES 15 SP4 with fips enabled. I've created a temporary test repository signed using the same (stronger) key we use for our Debian/Ubuntu repositories ( f1656f24c74cd1d8 ). I've temporarily put this repository in a location you can access to test: baseurl = https://downloads.mariadb.com/custom/mariadb-10.5.17/yum/sles/15/x86_64 To try it out, here are some instructions that should work for a stock SLES 15 (fips-enabled or not) system that doesn't have MariaDB installed. If a MariaDB repository is already configured, just change the baseurl line in your MariaDB repo file to the above and then skip the first two steps and jump to step 3 about importing the updated GPG-KEY file and continue on from there. #1 run mariadb_repo_setup curl -LsS https: //r .mariadb.com /downloads/mariadb_repo_setup | sudo bash -s -- --skip-maxscale --skip-tools --mariadb-server-version=10.5   #2 Change the base url to the temp test repo sudo sed -i s "/dlm.mariadb.com\/repo\/mariadb-server\/10.5/downloads.mariadb.com\/custom\/mariadb-10.5.17/" /etc/zypp/repos .d /mariadb .repo   #3 Import an updated GPG-KEY file which has the f1656f24c74cd1d8 pub-key pushd /etc/pki/trust/ ; sudo curl -LsSO https: //downloads .mariadb.com /custom/mariadb-10 .5.17 /MariaDB-Server-GPG-KEY ; popd sudo rpm -- import /etc/pki/trust/MariaDB-Server-GPG-KEY   #4 Remove the old keys, so zypper stops complaining about them sudo rpm -e gpg-pubkey-1bb943db-511147a9 sudo rpm -e gpg-pubkey-307e3d54-5aaa90a5   #5 Refresh repositories sudo zypper refresh   #6 Install and Start MariaDB (to test that the repo is working properly with zypper) sudo zypper install -y MariaDB-server   sudo systemctl start mariadb   sudo mariadb -e "SELECT @@version;" Assuming things look good for using the f1656f24c74cd1d8 key, I'll start the planning process for an orderly transition to using this key, probably for all of our zypper and dnf repositories (it doesn't hurt for all of them to be using a stronger key).
          amitnarkhede Amit Narkhede added a comment -

          Thanks for the detailed steps above. I created a VM on SLES15 SP4. Tried to reproduce the issue on the custom repo and was not able to. So it looks like this indeed fixes the issue here.
          Do we have any timeline to when the transition to the new key will be planned? It may be good idea to give organizations time to migration things over in all environments.

          amitnarkhede Amit Narkhede added a comment - Thanks for the detailed steps above. I created a VM on SLES15 SP4. Tried to reproduce the issue on the custom repo and was not able to. So it looks like this indeed fixes the issue here. Do we have any timeline to when the transition to the new key will be planned? It may be good idea to give organizations time to migration things over in all environments.
          amitnarkhede Amit Narkhede added a comment -

          Just to make sure, as things are working fine now can I assume here that the new key will be the one above? I can use that to prepare our migration as this affects VMs in our live environment even if they are not using SLES15 yet.

          amitnarkhede Amit Narkhede added a comment - Just to make sure, as things are working fine now can I assume here that the new key will be the one above? I can use that to prepare our migration as this affects VMs in our live environment even if they are not using SLES15 yet.
          dbart Daniel Bartholomew added a comment -

          Yes, the plan is to migrate to using that key for our yum/dnf/zypper repositories. I don't have a definite timeline yet on this, but I'm starting discussions with the relevant people so the roll-out can be as organized and smooth as possible.

          To kickstart things I've today updated the official location for the yum/dnf/zypper repository pub-key - https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY - to include both the legacy and 'new' pub keys, but there are other places that need to be updated that I'm working on. This is a transparent change for end users. When rpm imports a pub-key it doesn't care if the file has one or multiple keys in it. So adding the key to that file now has the effect of pre-staging it for those that set up repositories using our repository setup script from today.

          We'll also provide documentation in relevant release notes, the GPG page in the Knowledgebase (and other pages), and probably blog posts for updating existing repositories, which I think will be along the lines of:

          sudo rpm --import https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY
          		 
          # package-manager-specific commands to clean and refresh the package cache

          (At least, that's what I assume will be needed. I haven't tested migrations yet.)

          dbart Daniel Bartholomew added a comment - Yes, the plan is to migrate to using that key for our yum/dnf/zypper repositories. I don't have a definite timeline yet on this, but I'm starting discussions with the relevant people so the roll-out can be as organized and smooth as possible. To kickstart things I've today updated the official location for the yum/dnf/zypper repository pub-key - https://supplychain.mariadb.com/MariaDB-Server-GPG-KEY - to include both the legacy and 'new' pub keys, but there are other places that need to be updated that I'm working on. This is a transparent change for end users. When rpm imports a pub-key it doesn't care if the file has one or multiple keys in it. So adding the key to that file now has the effect of pre-staging it for those that set up repositories using our repository setup script from today. We'll also provide documentation in relevant release notes, the GPG page in the Knowledgebase (and other pages), and probably blog posts for updating existing repositories, which I think will be along the lines of: sudo rpm -- import https: //supplychain .mariadb.com /MariaDB-Server-GPG-KEY # package-manager-specific commands to clean and refresh the package cache (At least, that's what I assume will be needed. I haven't tested migrations yet.)
          faust Faustin Lammler added a comment - - edited

          FYI: https://github.com/MariaDB/buildbot/pull/68 has just been merged and I have triggered rebuild of knowing green install test on RPM platform (new BB: https://buildbot.mariadb.org/#/builders?tags=%2Binstall&tags=kvm&tags=%2Brpm) to test the v2 version of the key.

          faust Faustin Lammler added a comment - - edited FYI: https://github.com/MariaDB/buildbot/pull/68 has just been merged and I have triggered rebuild of knowing green install test on RPM platform (new BB: https://buildbot.mariadb.org/#/builders?tags=%2Binstall&tags=kvm&tags=%2Brpm ) to test the v2 version of the key.
          faust Faustin Lammler added a comment - - edited

          The old key has now been replaced on the root mirror with the new one.

          I have just roll-backed since I was not aware of:
          https://mariadb.com/kb/en/gpg/#rpm-source-key-2023

          faust Faustin Lammler added a comment - - edited The old key has now been replaced on the root mirror with the new one. I have just roll-backed since I was not aware of: https://mariadb.com/kb/en/gpg/#rpm-source-key-2023
          amitnarkhede Amit Narkhede added a comment -

          Faustin, I see you did get the migration back in December and rolled it back as it was planned for 2023. Do we have any timeline when this will be attempted again?

          amitnarkhede Amit Narkhede added a comment - Faustin, I see you did get the migration back in December and rolled it back as it was planned for 2023. Do we have any timeline when this will be attempted again?
          faust Faustin Lammler added a comment -

          Hi amitnarkhede, yes we plan it for the next release (probably before the end of the month).

          faust Faustin Lammler added a comment - Hi amitnarkhede , yes we plan it for the next release (probably before the end of the month).
          faust Faustin Lammler added a comment -

          Hi! The new key is now in place on our root mirrors, it is going to be synced in the next hours to all mirrors.

          Please note that the v2 key is still available:

          • PublicKey_v2
          • yum/RPM-GPG-KEY-MariaDB_v2

          But, we will probably remove the key in the next month since people should use the following:

          • PublicKey
          • yum/RPM-GPG-KEY-MariaDB
          faust Faustin Lammler added a comment - Hi! The new key is now in place on our root mirrors, it is going to be synced in the next hours to all mirrors. Please note that the v2 key is still available: PublicKey_v2 yum/RPM-GPG-KEY-MariaDB_v2 But, we will probably remove the key in the next month since people should use the following: PublicKey yum/RPM-GPG-KEY-MariaDB
          amitnarkhede Amit Narkhede added a comment -

          Hi Faustin, Thanks for the update.

          I tried to point a VM so that it will fetch things from https://downloads.mariadb.com/MariaDB/mariadb-10.5/yum/sles/15/x86_64 and see that latest packages from that repo still had older signature.

          Just to make sure I'm on the same page, is this expected behavior? Shouldn't older package also get re-signed with new key?

          amitnarkhede Amit Narkhede added a comment - Hi Faustin, Thanks for the update. I tried to point a VM so that it will fetch things from https://downloads.mariadb.com/MariaDB/mariadb-10.5/yum/sles/15/x86_64 and see that latest packages from that repo still had older signature. Just to make sure I'm on the same page, is this expected behavior? Shouldn't older package also get re-signed with new key?
          faust Faustin Lammler added a comment -

          Hi amitnarkhede, yes that's normal since only the next release (that has been delayed) will use the new signature key (so, 10.5.19 in your case).

          faust Faustin Lammler added a comment - Hi amitnarkhede , yes that's normal since only the next release (that has been delayed) will use the new signature key (so, 10.5.19 in your case).
          dbart Daniel Bartholomew added a comment -

          The releases have now been published.

          dbart Daniel Bartholomew added a comment - The releases have now been published.
          amitnarkhede Amit Narkhede added a comment -

          Thank you Daniel and Faustin for the help on this.
          I can confirm that now things are working as expected and I was able to successfully deploy systems this morning. We can close this ticket.

          amitnarkhede Amit Narkhede added a comment - Thank you Daniel and Faustin for the help on this. I can confirm that now things are working as expected and I was able to successfully deploy systems this morning. We can close this ticket.
          dbart Daniel Bartholomew added a comment -

          Great to hear. Closing.

          dbart Daniel Bartholomew added a comment - Great to hear. Closing.

          People

            dbart Daniel Bartholomew
            amitnarkhede Amit Narkhede
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.