to test FIPS mode in 3.0 we'd need rawhide build to work and currently it has some selinux (?) related failures

#3  <signal handler called>

#4  0x0000000000000000 in ?? ()

#5  0x00007f7bfaf6a441 in EVP_DigestUpdate (ctx=0x7f7be414e050, data=0x7f7ba8018298, count=15) at crypto/evp/digest.c:397

#6  0x000055c538eab836 in md5_input (context=0x7f7be414e050, buf=0x7f7ba8018298 "select 1 AS `1`", len=15) at /home/buildbot/maria-slave/10.8/mysys_ssl/my_md5.cc:65

#7  0x000055c538eab8c8 in my_md5 (digest=0x7f7be414e0d0 "^\350\024\344{\177", buf=0x7f7ba8018298 "select 1 AS `1`", len=15) at /home/buildbot/maria-slave/10.8/mysys_ssl/my_md5.cc:90

#8  0x000055c53896ada9 in TABLE_LIST::calc_md5 (this=0x7f7ba8016658, buffer=0x7f7be414e1f0 "koC9\305U") at /home/buildbot/maria-slave/10.8/sql/table.cc:5658

#9  0x000055c5389532de in mysql_register_view (thd=0x7f7ba8000db8, ddl_log_state=0x7f7be4150980, view=0x7f7ba8016658, mode=VIEW_CREATE_NEW, backup_file_name=0x7f7be4150b40 "") at /home/buildbot/maria-slave/10.8/sql/sql_view.cc:1024

#10 0x000055c5389521c1 in mysql_create_view (thd=0x7f7ba8000db8, views=0x7f7ba8016658, mode=VIEW_CREATE_NEW) at /home/buildbot/maria-slave/10.8/sql/sql_view.cc:662

#11 0x000055c5387fb618 in mysql_execute_command (thd=0x7f7ba8000db8, is_called_from_prepared_stmt=false) at /home/buildbot/maria-slave/10.8/sql/sql_parse.cc:5796

#12 0x000055c5388022b0 in mysql_parse (thd=0x7f7ba8000db8, rawbuf=0x7f7ba8016560 "CREATE VIEW v AS SELECT 1", length=25, parser_state=0x7f7be4151480) at /home/buildbot/maria-slave/10.8/sql/sql_parse.cc:8027

#13 0x000055c5387ee774 in dispatch_command (command=COM_QUERY, thd=0x7f7ba8000db8, packet=0x7f7ba800b889 "CREATE VIEW v AS SELECT 1", packet_length=25, blocking=true) at /home/buildbot/maria-slave/10.8/sql/sql_parse.cc:1894

#14 0x000055c5387ed110 in do_command (thd=0x7f7ba8000db8, blocking=true) at /home/buildbot/maria-slave/10.8/sql/sql_parse.cc:1402

#15 0x000055c5389c5755 in do_handle_one_connection (connect=0x55c53b37b6a8, put_in_cache=true) at /home/buildbot/maria-slave/10.8/sql/sql_connect.cc:1418

#16 0x000055c5389c53e5 in handle_one_connection (arg=0x55c53b37b6a8) at /home/buildbot/maria-slave/10.8/sql/sql_connect.cc:1312

#17 0x000055c538ed4109 in pfs_spawn_thread (arg=0x55c53b29e398) at /home/buildbot/maria-slave/10.8/storage/perfschema/pfs.cc:2201

#18 0x00007f7bfad43609 in start_thread (arg=<optimized out>) at pthread_create.c:477

#19 0x00007f7bfa916293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

bt_full

Not sure whether it's our bug or openssl's, but md5 doesn't work with openssl3 in fips mode.

The above is from enabling FIPS via configuration file, in case it matters.

For 1.0.2 via configuration file – do we have any indication that it ever worked, that is, that it would be taken into account if enabled?
I've checked fips-related bugs mentioned in MDEV-20260, but none says that FIPS mode was set via configuration file, they either suggest that it was enabled at the kernel level, or have no specifics at all.

The manual says that the application would need to call either FIPS_mode_set or OPENSSL_config, I don't see either in the server (there is an OPENSSL_config(NULL) call in libmariadb, but I don't know whether it's relevant).
And if I configure fips through the default openssl_conf section, I see it noticed by openssl, but not by the server.

With the patch from MDEV-27778 applied to 10.8, the crash on bootstrap with OpenSSL 3 doesn't occur, instead a bunch of tests fail with

2022-02-21 16:13:18 0 [Warning] SSL error: SSL_CTX_new failed

2022-02-21 16:13:18 0 [Warning] SSL error: error:0308010C:digital envelope routines::unsupported

note that mtr overwrites OPENSSL_CONF for openssl > 1.1.1, so 1.0.2 would be easier, but 3.0 would be future-proof.

I don't think we can give up on 1.0.2 just yet, since we only have 3.0 in 10.8+. Besides, even though 1.0.2 is out of general support, it's still in extended support for who-knows-how long.
And it doesn't look like 1.0.2 only would be sufficient, as there are clearly issues which are specific to 3.0 (e.g. MDEV-27778).

So,

for 1.0.2:

A new step "mtr-fips" has been added to kvm-rpm-centos74-amd64 builder. It uses a VM image with CentOS 7.9, OpenSSL 1.0.2k, and FIPS enabled at the kernel level, installs the RPM packages built by the standard build (MariaDB-server, MariaDB-test and dependencies) and runs the default MTR set. We can tune the set of packages and the set of tests if needed. The builder, including the step, runs for all server branches.
A few tests which aren't shielded by include/have_des.inc currently fail due to result differences from encrypt/decrypt functions. However, the crash described in the comment to MDEV-20260 is not reproducible (at least not on 10.2 or 10.8), so either we don't have a test which covers the problem, or something else is missing.

for 3.0:

I have re-used the openssl3 builders which were originally created for feature testing, but haven't been used since.
There are two builders, openssl3-fips-debug and openssl3-nofips-debug. They run on all 10.8+ branches and build a debug server (currently a small build, with many engines disabled) linking with debug openssl 3.0.1 installed locally.
openssl3-fips-debug links and runs with fips-enabled openssl3. It has 2 MTR steps, "mtr_nm_fips" and "mtr_nm_fips_disabled".
"mtr_nm_fips" runs with fips enabled via config file. It currently fails with MDEV-27778 or, if the patch is applied, with errors mentioned in the previous comment.
"mtr_nm_fips_disabled" runs with the libraries built with fips, but without enabling it in the config file. We can discard it if it turns out to be redundant, but there seems to be many ifdefs for FIPS in the openssl3 code, so I wasn't sure.

Other notes

I didn't set up tests with openssl 1.0.2 enabled at the configuration level, as it doesn't seem to be doing anything as mentioned in a comment above. If needed, we can set it up the same way as it's done for openssl3, the logic is expandable. However, I would expect that the majority of users who pay for the extended support of openssl 1.0.2 have it enabled at the kernel level.

For openssl 3 enabled at the kernel level, the situation is tricky at the moment. I don't think we should use Rawhide for such tests. We needed it for feature testing and it served the purpose, but it cannot be used for regression testing as it's unpredictable by design. It uses non-GA kernels, too.
I did try to switch it to FIPS mode, but I couldn't even start up the image after that. It can be due to configuration issues or problems in the system itself, I think we will only keep wasting time investigating it.

Fedora 36 has just reached the beta stage and is expected to be GA in 2 months, I assume it will have openssl 3, we can set up the kernel-level test when we add the fc36 builder to the buildbot.

For the question of MTR overriding OPENSSL_CONF file, we could benefit from something like

@@ -2081,7 +2081,7 @@ sub environment_setup {

   $ENV{'LC_COLLATE'}=         "C";

   $ENV{'OPENSSL_CONF'}= $mysqld_variables{'version-ssl-library'} gt 'OpenSSL 1.1.1'

-                       ? "$glob_mysql_test_dir/lib/openssl.cnf" : '/dev/null';

+                       ? ($ENV{'MTR_OPENSSL_CONF'} || "$glob_mysql_test_dir/lib/openssl.cnf") : '/dev/null';

   $ENV{'MYSQL_TEST_DIR'}=     $glob_mysql_test_dir;

   $ENV{'DEFAULT_MASTER_PORT'}= $mysqld_variables{'port'};

or alike; but for now the builder will run without it, it will patch the config file which MTR uses.