The name "ssl_fips_mode" that MySQL uses for the system variable would seem to imply that it is only relevant for SSL (i.e. TLS). However, I think that FIPS mode would also affect things like hash functions, encryption functions, and encryption plugins (the equivalent of keyring plugins from MySQL).

If we port this implementation or if we implement something similar, would we want to use the same name as MySQL, even if it is kind of a misnomer?

I've documented the current state of support for FIPS mode here: https://mariadb.com/kb/en/library/tls-and-cryptography-libraries-used-by-mariadb/#fips-certification

Hi,

Goeff's comment was correct, enabling FIPS mode would affect more than just TLS related usage of cryptographic functions. When dynamically linked with a fips OpenSSL and if FIPS is also enabled at the kernel level, low level API calls with fail.

Please find below an example of this issue:

Error:

sha_locl.h(128): OpenSSL internal error, assertion failed: Low level API call to digest SHA1 forbidden in FIPS mode!

210719 15:47:03 [ERROR] mysqld got signal 6 ;

StackTrace:

stack_bottom = 0x7fb79571ee88 thread_stack 0x49000

/opt/bitnami/mariadb/sbin/mysqld(my_print_stacktrace+0x29)[0x556d5f4231b9]

/opt/bitnami/mariadb/sbin/mysqld(handle_fatal_signal+0x4ed)[0x556d5ef6743d]

/lib/libpthread.so.0(+0x122a0)[0x7fb7acd572a0]

/lib/libc.so.6(gsignal+0xca)[0x7fb7acbc27ea]

/lib/libc.so.6(abort+0x141)[0x7fb7acbc3881]

/lib/libcrypto.so.1.0.0(+0xc9e2f)[0x7fb7ad157e2f]

/lib/libcrypto.so.1.0.0(SHA1_Init+0x25)[0x7fb7ad160b65]

/opt/bitnami/mariadb/sbin/mysqld(my_sha1+0x1e)[0x556d5f405dde]

/opt/bitnami/mariadb/sbin/mysqld(my_make_scrambled_password+0x21)[0x556d5f06bbb1]

/opt/bitnami/mariadb/sbin/mysqld(+0x53a942)[0x556d5ed2a942]

/opt/bitnami/mariadb/sbin/mysqld(_Z17mysql_create_userP3THDR4ListI8LEX_USEREb+0x1b0)[0x556d5ed35cd0]

/opt/bitnami/mariadb/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x16f0)[0x556d5ed9df50]

/opt/bitnami/mariadb/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1e9)[0x556d5eda3ff9]

/opt/bitnami/mariadb/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xb95)[0x556d5eda5865]

/opt/bitnami/mariadb/sbin/mysqld(_Z10do_commandP3THD+0x11a)[0x556d5eda730a]

/opt/bitnami/mariadb/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x314)[0x556d5ee6fad4]

/opt/bitnami/mariadb/sbin/mysqld(handle_one_connection+0x34)[0x556d5ee6fbb4]

/lib/libpthread.so.0(+0x7f87)[0x7fb7acd4cf87]

/lib/libc.so.6(clone+0x3f)[0x7fb7acc805ef]

The line referenced from the sha_locl.h header, indicates that the message was thrown while calling openssl/crypto/sha/sha_locl.h:128 → fips_md_init_ctx(SHA1, SHA). This is based on the currently FIPS supported branch of OpenSSL (OpenSSL_1_0_2_stable)

This function has an implementation through the following a macro in openssl/crypto/crypto.h:605

# define fips_md_init(alg) fips_md_init_ctx(alg, alg)

# ifdef OPENSSL_FIPS

#  define fips_md_init_ctx(alg, cx) \

        int alg##_Init(cx##_CTX *c) \

        { \

        if (FIPS_mode()) OpenSSLDie(__FILE__, __LINE__, \

                "Low level API call to digest " #alg " forbidden in FIPS mode!"); \

        return private_##alg##_Init(c); \

        } \

        int private_##alg##_Init(cx##_CTX *c)

#  define fips_cipher_abort(alg) \

        if (FIPS_mode()) OpenSSLDie(__FILE__, __LINE__, \ 

                "Low level API call to cipher " #alg " forbidden in FIPS mode!")

# else

#  define fips_md_init_ctx(alg, cx) \

        int alg##_Init(cx##_CTX *c)

#  define fips_cipher_abort(alg) while(0)

# endif

Looking at the OpenSSL User Guide 2.0 - https://www.openssl.org/docs/fips/UserGuide-2.0.pdf the following is stated in 2.6.2

The OpenSSL library attempts to disable non-FIPS algorithms. when in FIPS mode. The disabling occurs on the EVP_* APIs and most low level function calls.

Failure to check the return code from low level functions could result in unexpected behavior.

Note also that sufficiently creative or unusual use of the API may still allow the use of non-FIPS algorithms.

The non-FIPS algorithm disabling is intended as an aid to the developer in preventing the accidental use of non-FIPS algorithms in FIPS mode, and not as an absolute guarantee. It is the responsibility of the application developer to ensure that only FIPS algorithms are used when in FIPS mode.

So basically, if we want to be work properly in FIPS mode, we need to move to EVP_* APIs which allow for the OpenSSL FIPS library to call lower level APIs with FIPS compliant parameters.

The MySQL project has more details regarding the impact of such a change (not exhaustive), if you look at their WorkLog item 8102, in the High Level Architecture tab, section Impacted functionalities/codes

Some more references:

• https://wiki.openssl.org/index.php/EVP

I would like to suggest in the meantime to clarify the documentation, I initially assumed that the project was FIPS compliant, which is probably also the case of other users.

ralf.gebhardt@mariadb.com Are you working on this or do you plan to work on this? If not, then unassign yourself from the issue so somebody else can see that it is "free" and perhaps pick it up. I might know somebody who is interested in FIPS and security certification stuff for MariaDB.

Hi otto, thanks for your comment. I am not looking at it currently, so I have removed me as an assignee.

Yass, MariaDB used to work in the FIPS mode, as can be seen from numerous fips related bugs that were fixed in the past. Unfortunately, it seems like it was broken again, as we don't set FIPS mode in CI yet. So, the first step should be to start testing FIPS mode in CI, everything else is pointless until we do that. I've created MDEV-27729 for the CI part.

The documentation isn't exactly correct claiming that FIPS has to be enabled at the kernel level, it can be enabled specifically in MariaDB by starting the server with a MariaDB specific openssl.conf file, as documented here.

Hi @Sergei Golubchik, thanks for the follow-up.

I think there is a confusion around the scope of changes required to be considered FIPS compliant, the bugs you reference are not all related to FIPS and do cover a very small part of the work to be done.

I agree on the kernel level enablement, the kernel level enablement is checked by the openSSL library to enable FIPS Mode (wether or not MariaDB started in FIPS mode). MariaDB could also enable it through a specific call to the library, through a user exposed configuration or argument.

Looking at the code, I don't see many references to EVP_ functions or fips in general, I think the best two resources to start with are those I mentioned earlier

The MySQL project has more details regarding the impact of such a change (not exhaustive), if you look at their WorkLog item 8102, in the High Level Architecture tab, section Impacted functionalities/codes

• https://dev.mysql.com/worklog/task/?id=8102 (Requirements and High level architecture tabs)

I'm happy to discuss that in further details asynchronously and even help with this work.

Fixes for bugs like MDEV-14567, MDEV-14343, MDEV-14064, MDEV-7788, MDEV-7695, MDEV-8482, MDEV-8481, MDEV-8480, MDEV-8444, MDEV-7794 ensured that MariaDB started when fips was enabled system-wide, that it didn't crash, could be connected to, and all tests in our test harness passed.

First we need to make sure that FIPS bugs, once fixed, stay fixed. Meaning, we need MDEV-27729 done.

Can you advise whether we need to test FIPS mode both with OpenSSL 1.0.2 and 3.0? Is it significantly different, can one have one working and the other failing? If they either both fail or both succeed, then we can just test one.

I agree with the priority of having testing available to avoid regressions. My point was that we should use Approved Security Functions (SP 800-140C) for the usages specified. As far as I remember, on of the problem is that the OpenSSL library cannot determine the usage we make of a cryptographic function as it lacks the application context.

Moreover, we probably want to cover cases where the user initially configures MariaDB without fips mode but would then enable it with an incompatible configuration. She should then receive a proper error message to allow her to understand what configuration makes enabling fips impossible.

Regarding the use of OpenSSL 1.0.2 or OpenSSL 3.0, OpenSSL 1.0.2 is not supported anymore, see release strategy from the OpenSSL project. And OpenSSL 1.1.1 doesn't support the validated fips cryptographic modules (see https://wiki.openssl.org/index.php/FIPS_modules).

The OpenSSL team has a CMVP submission (see post - https://www.openssl.org/blog/blog/2021/09/22/OpenSSL3-fips-submission/) but it doesn't seem to have been validated yet. (see https://csrc.nist.gov/projects/cryptographic-module-validation-program/modules-in-process/modules-in-process-list).

one of the problem is that the OpenSSL library cannot determine the usage we make of a cryptographic function

Correct. This was for example, MDEV-7788. We use EVP_MD_CTX_FLAG_NON_FIPS_ALLOW to let md5 work even in the FIPS mode, as it's not used for any cryptographic purposes.

user initially configures MariaDB without fips mode but would then enable it with an incompatible configuration

What do you mean by that? What is an incompatible configuration?

Both these points are kind of covered in the OpenSSL FIPS User guide, in Section 2.6 - FIPS Mode of Operation: https://www.openssl.org/docs/fips/UserGuide-2.0.pdf

When I say the OpenSSL library is lacking context, I was referring to this part of the OpenSSL UserGuide:

Note also that sufficiently creative or unusual use of the API may still allow the use of non-FIPS algorithms. The non-FIPS algorithm disabling is intended as an aid to the developer in preventing the accidental use of non-FIPS algorithms in FIPS mode, and not as an absolute guarantee.

For my second point I was referring to the following:

Applications that utilize FIPS mode must call the FIPS_mode_set() function. After successful FIPS mode initialization, the non-FIPS algorithms will be disabled by default

To make up an example, let's say that

• At the 1st startup (without FIPS mode), a user configures an authentication hashing mechanism, a certificate, etc... that is not available in FIPS Mode
• At the 2nd startup, the user enables fips mode, but the previous configuration will fail, because the previously available algorithms are now restricted

Now I should state that I'm not a FIPS expert, what I'm sharing is based on previous reading and research and I haven't looked that the whole MariaDB code base to provide more an thorough analysis.

Thanks! That was enough guidance for us to take from there.

Please note that OpenSSL 3.0 has now been released in September 2021. It is no longer plagued by the "OpenSSL license exception" thing, but is pure Apache Licence 2.0 and can be used freely with in MariaDB.

OpenSSL 3.0 has been submitted for FIPS 140-2 certification: https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?product=14433

MariaDB should start using OpenSSL 3.0 and have the fips_mode variable implemented so that it requires OpenSSL 3.0 + the FIPS module in it. This way it would last time and be a modern FIPS 140-2 version that also has TLS 1.3 support. It is however not yet FIPS 140-3, so it isn't the most modern one.. but would meet the needs of most users who are interested in FIPS.

MariaDB uses OpenSSL 3.0 since MDEV-25785. See e.g. https://buildbot.askmonty.org/buildbot/builders/kvm-rpm-fedora666-amd64/builds/187

Thanks for pointing out MDEV-25785. It has 4 git commits linked, from which I learned that:

• OpenSSL 3.0 is disabled on all old MariaDB majors (https://github.com/MariaDB/server/commit/c9beef43154a199bfcd9f71049c011a2ed77ca74) and enabled only on 10.8 and 10.9+ (https://github.com/MariaDB/server/commit/64e358821e0e8b5a2134aa93ba9ddf24cd46ac7c).
• Support for OpenSSL 3.0 was committed in https://github.com/MariaDB/server/commit/d42c2efbaa06a0307c2f0fd8fa87819ff50bbd7e on mariadb-10.8.1+.

Yes. It's disabled before 10.8 because these builds crash (see numerous linked bug reports).

It is quite possible that we'll backport 10.8 OpenSSL 3.0 patches to earlier versions eventually.

Backporting has already been done in CentOS and Debian, so submitted that patch for 10.6 in https://github.com/MariaDB/server/pull/2036

serg What are the next steps on this? If somebody had extra time on their hands, what should they do? Do you want to have a design/planning meeting or is the implementation basically just to copy the flag from MySQL, and make it so that ssl-fips-mode=on errors out if the server detects that MariaDB was not compiled with OpenSSL 3.0?

At the moment we have fips tests running in buildbot, with openssl-1.0.2 and with openssl-3.0, see MDEV-27729 for details.

As for the ssl-fips-mode, I'm not sure it'll be very helpful. There're lots of OpenSSL aspects one can configure, they all can be configured in a config file and with OPENSSL_CONF one can use a dedicated config file that will only apply to MariaDB. One can enable fips mode that way (our tests do that), but also everything else OpenSSL related. Singling out just one particular openssl setting and creating a MariaDB option for it doesn't look very logical to me. Why not to create an option for the security level, for example? or for protocols? or for tsl1.3 ciphers?

I think it's reasonable to be able to configure anything OpenSSL related, and do it in a separate file using the well known and documented syntax. And not to duplicate arbitrary selected small bits of OpenSSL config in the my.cnf file.

MySQL has actually deprecated ssl_fips_mode in favor of OpenSSL side configuration starting with MySQL 8.0.34