(gdb) bt

#0  0x00007fcf1676b75b in kill () at ../sysdeps/unix/syscall-template.S:120

#1  0x0000560999302c2d in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:372

#2  <signal handler called>

#3  0x0000000000000000 in ?? ()

#4  0x0000560999511821 in md5_input (len=4455, 

    buf=0x56099bd67858 "select `mysql`.`global_priv`.`Host` AS `Host`,`mysql`.`global_priv`.`User` AS `User`,if(json_value(`mysql`.`global_priv`.`Priv`,'$.plugin') in ('mysql_native_password','mysql_old_password'),ifnull(jso"..., context=0x7fff7066bcb0) at ./mysys_ssl/my_md5.cc:76

#5  my_md5 (digest=0x7fff7066bd50 "\001", 

    buf=0x56099bd67858 "select `mysql`.`global_priv`.`Host` AS `Host`,`mysql`.`global_priv`.`User` AS `User`,if(json_value(`mysql`.`global_priv`.`Priv`,'$.plugin') in ('mysql_native_password','mysql_old_password'),ifnull(jso"..., len=4455) at ./mysys_ssl/my_md5.cc:101

#6  0x0000560999195557 in TABLE_LIST::calc_md5 (this=this@entry=0x56099bd077b0, buffer=buffer@entry=0x7fff7066bf30 "") at ./sql/table.cc:5910

#7  0x0000560999185455 in mysql_register_view (backup_file_name=0x7fff7066c580 "", mode=VIEW_CREATE_NEW, view=0x56099bd077b0, 

    ddl_log_state=0x7fff7066be50, thd=0x56099bce4048) at ./sql/sql_view.cc:1035

#8  mysql_create_view (thd=thd@entry=0x56099bce4048, views=views@entry=0x56099bd077b0, mode=VIEW_CREATE_NEW) at ./sql/sql_view.cc:664

#9  0x000056099909b313 in mysql_execute_command (thd=0x56099bce4048, is_called_from_prepared_stmt=<optimized out>) at ./sql/sql_parse.cc:5833

#10 0x000056099909e867 in mysql_parse (thd=0x56099bce4048, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>)

    at ./sql/sql_parse.cc:8030

#11 0x000056099909ec8f in bootstrap (file=0x56099aaea8e0 <instrumented_stdin>) at ./sql/sql_class.h:243

#12 0x0000560998f8f27e in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:5959

#13 0x00007fcf16752d90 in __libc_start_call_main (main=main@entry=0x560998f42020 <main(int, char**)>, argc=argc@entry=18, 

    argv=argv@entry=0x7fff7066fc58) at ../sysdeps/nptl/libc_start_call_main.h:58

#14 0x00007fcf16752e40 in __libc_start_main_impl (main=0x560998f42020 <main(int, char**)>, argc=18, argv=0x7fff7066fc58, init=<optimized out>, 

    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff7066fc48) at ../csu/libc-start.c:392

#15 0x0000560998f83335 in _start ()

Seems related https://github.com/MariaDB/server/commit/f23f45413fd02c180182cd341b800e2b9fa169f4

That seems like it makes sense. If I'm understanding the comments in that code correctly (admittedly a big "if"), it sounds like it's trying to go ahead and use MD5 despite FIPS mode since it's not using it for a crypto operation. But I don't think OpenSSL can provide MD5 in FIPS mode since it's not included in the "base" (non-crypto) or FIPS providers.

(To be fair, there's a fair amount I don't understand here, including the entirety of the internals of MariaDB, so I could be way off base.)

serg, know what's going on here?

Not really. cory.mccarty is right that that code is using MD5 despite FIPS mode since it's not using it for a crypto operation. It works, because it explicitly tells OpenSSL that it's ok to do it despite FIPS. We have tests, on a dedicated openssl3+fips builder to verify that MariaDB works in such a setup.

Have you confirmed that the test builder has OpenSSL fully configured for FIPS mode? As before, there are significant limitations to my understanding of how this works, so I apologize if this I've got something wrong or this isn't helpful, but I don't think it should be possible to use MD5 from OpenSSL's high-level APIs if it's in FIPS mode because it won't actually have a provider that includes that algorithm. (I gather there are ways to bypass the FIPS configuration using lower-level APIs, so maybe that's what you're doing?) The best mechanism I've found to check whether OpenSSL is fully in FIPS mode is to run openssl list -digest-algorithms and make sure all the algorithms listed under "Provided" end with "@ fips", indicating that they're from the FIPS provider. (You can use any of the other algorithm types as well, not just digest.)

apparently, according to the patch, it indeed explicitly asks for a non-fips algorithm:

md5 = EVP_MD_fetch(NULL, "MD5", "fips=no");

And according to https://www.openssl.org/docs/manmaster/man7/ossl-guide-libraries-introduction.html

The FIPS provider may also contain non-approved algorithm implementations and these can be selected with the property "fips=no".

Does this "may" means that FIPS provider also "may" not contain them? How it can be done?

possible, stack trace looks a null pointer call.

https://github.com/openssl/openssl/blob/master/providers/fips/fipsprov.c#L276 - on digest there's no FIPS_UNAPPROVED_PROPERTIES

Oh, so, indeed, may be the builder doesn't properly enable fips?

Perhaps using "provider=default" will work better then "fips=no". Or using a non-fips context. And ff nothing else works we can bundle MD5 implementation with the server.

Is this supposed to work as described at all? Dockerfile builds own copy of openssl with fips, then ovewrites system libraries with own openssl. Not sure it can work like that. That assumes libraries are ABI compatible, but I do not believe there is a guarantee. The way to handle it, as I see it, is to build openssl with fips first, then build server against openssl-with-fips libraries and headers.

Which is what I did, and it worked fine. Maybe I'm missing something? Below is what I did, on ubuntu-22.04

  sudo apt-get update

  sudo apt-get install -y wget build-essential 

  wget https://www.openssl.org/source/openssl-3.0.0.tar.gz 

  tar zxpf openssl-3.0.0.tar.gz

  cd openssl-3.0.0

  ./Configure enable-fips

  make

  sudo make install # installs into /usr/local

  cd ..

  # build server against openssl we just built

  sudo apt install cmake bison ncurses-dev libz-dev

  git clone https://github.com/mariadb/server --depth=1

  cd server

  mkdir bld

  cd bld

  cmake .. -DOPENSSL_ROOT_DIR=/usr/local -DOPENSSL_SSL_LIBRARY=/usr/local/lib64/libssl.so -DOPENSSL_CRYPTO_LIBRARY=/usr/local/lib64/libcrypto.so 

  cmake --build . -j32 --target minbuild

Now I test it like this

  cd mysql-test

  perl mysql-test-run --suite=main openssl_1

Now I check that mariadbd really links against my own openssl, and yes, it does link against /usr/local/lib64/ SSL libraries

wlad@desktop:~/server/bld$ ldd sql/mariadbd

        linux-vdso.so.1 (0x00007fffd449f000)

        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fcf3a340000)

        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fcf3a320000)

        libssl.so.3 => /usr/local/lib64/libssl.so.3 (0x00007fcf3a273000)

        libcrypto.so.3 => /usr/local/lib64/libcrypto.so.3 (0x00007fcf39dfb000)

        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fcf39bc0000)

        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcf39ad0000)

        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcf39ab0000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcf39880000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fcf3c363000)

You can also just build the OpenSSL FIPS module and configure the system version of OpenSSL that's included in the original MariaDB image to be in FIPS mode (which is a configuration that is recognized/recommended by OpenSSL), and the result is the same. In that case, MariaDB is still linking against the system OpenSSL, and the image still crashes on launch. (Also, note that just running ./Configure enable-fips doesn't actually update the OpenSSL configuration file to turn on FIPS mode.

To expand on my previous comment, I've attached a different Dockerfile and openssl.cnf (maybe not linked correctly from this comment since they have the same names as the original attachments) that use the system OpenSSL and just build and install the FIPS module (and configure OpenSSL to use it, since just installing it doesn't mean it gets used). It's somewhat of an odd configuration because it's using a slightly older version of the FIPS module than the version of OpenSSL because only certain versions of the FIPS module are actually FIPS validated (3.0.0 and 3.0.8; this configuration uses 3.0.0). The OpenSSL documentation suggests that this is a valid configuration. I strongly suspect that you would get the same results from just building the version of the FIPS module that matches the OpenSSL version, but that version wouldn't actually be FIPS validated.

Dockerfile openssl.cnf