Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32368

Docker image crashes on launch with OpenSSL 3 FIPS activated

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11.5, 11.1.2
    • 10.11
    • Docker
    • None
    • Official Docker image

    Description

      I've built a docker image based on the official MariaDB image, but with the OpenSSL FIPS provider (v3.0.0) built and configured for use (but no other changes). When I try to run this image, it crashes on launch after logging [Entrypoint]: Initializing database files. I've tried with both mariadb:11-jammy and mariadb:10-jammy with the same results. I've also tried doing a full build of OpenSSL 3.0.8 with its FIPS provider and configuring the system to use that OpenSSL, also with the same results.

      I assume based on the documentation here that MariaDB is intended to function correctly with OpenSSL 3 using the FIPS provider. I have confirmed using the ldd command on that page that the server is dynamically linking against the expected OpenSSL libraries.

      I've attached the output (both stdout and stderr) from running docker logs on the container (crashlog.txt) as well as the Dockerfile and openssl.cnf files used to build the container.

      For convenience, here are the full instructions to reproduce. First, put the attached Dockerfile and opensl.cnf in a directory. From that directory, build the image:

      docker build -t mariadb-fips .
      

      Then run the image:

      docker run --detach --name mariadb-fips --env MARIADB_USER=example-user --env MARIADB_PASSWORD=my_cool_secret --env MARIADB_ROOT_PASSWORD=my-secret-pw  mariadb-fips
      

      After a couple of seconds, observe that the container has stopped and check the logs:

      docker ps -a
      docker logs mariadb-fips
      

      Attachments

        1. crashlog.txt
          9 kB
          Cory McCarty
        2. openssl.cnf
          12 kB
          Cory McCarty
        3. Dockerfile
          0.6 kB
          Cory McCarty
        4. Dockerfile-1
          0.6 kB
          Cory McCarty
        5. openssl-1.cnf
          12 kB
          Cory McCarty

        Issue Links

          Activity

            danblack Daniel Black added a comment -

            possible, stack trace looks a null pointer call.

            https://github.com/openssl/openssl/blob/master/providers/fips/fipsprov.c#L276 - on digest there's no FIPS_UNAPPROVED_PROPERTIES

            danblack Daniel Black added a comment - possible, stack trace looks a null pointer call. https://github.com/openssl/openssl/blob/master/providers/fips/fipsprov.c#L276 - on digest there's no FIPS_UNAPPROVED_PROPERTIES

            Oh, so, indeed, may be the builder doesn't properly enable fips?

            Perhaps using "provider=default" will work better then "fips=no". Or using a non-fips context. And ff nothing else works we can bundle MD5 implementation with the server.

            serg Sergei Golubchik added a comment - Oh, so, indeed, may be the builder doesn't properly enable fips? Perhaps using "provider=default" will work better then "fips=no". Or using a non-fips context. And ff nothing else works we can bundle MD5 implementation with the server.
            wlad Vladislav Vaintroub added a comment - - edited

            Is this supposed to work as described at all? Dockerfile builds own copy of openssl with fips, then ovewrites system libraries with own openssl. Not sure it can work like that. That assumes libraries are ABI compatible, but I do not believe there is a guarantee. The way to handle it, as I see it, is to build openssl with fips first, then build server against openssl-with-fips libraries and headers.

            Which is what I did, and it worked fine. Maybe I'm missing something? Below is what I did, on ubuntu-22.04

              sudo apt-get update
              sudo apt-get install -y wget build-essential 
              wget https://www.openssl.org/source/openssl-3.0.0.tar.gz 
              tar zxpf openssl-3.0.0.tar.gz
              cd openssl-3.0.0
              ./Configure enable-fips
              make
              sudo make install # installs into /usr/local
              cd ..
              # build server against openssl we just built
              sudo apt install cmake bison ncurses-dev libz-dev
              git clone https://github.com/mariadb/server --depth=1
              cd server
              mkdir bld
              cd bld
              cmake .. -DOPENSSL_ROOT_DIR=/usr/local -DOPENSSL_SSL_LIBRARY=/usr/local/lib64/libssl.so -DOPENSSL_CRYPTO_LIBRARY=/usr/local/lib64/libcrypto.so 
              cmake --build . -j32 --target minbuild
            

            Now I test it like this

              cd mysql-test
              perl mysql-test-run --suite=main openssl_1
            

            Now I check that mariadbd really links against my own openssl, and yes, it does link against /usr/local/lib64/ SSL libraries

            wlad@desktop:~/server/bld$ ldd sql/mariadbd
                    linux-vdso.so.1 (0x00007fffd449f000)
                    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fcf3a340000)
                    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fcf3a320000)
                    libssl.so.3 => /usr/local/lib64/libssl.so.3 (0x00007fcf3a273000)
                    libcrypto.so.3 => /usr/local/lib64/libcrypto.so.3 (0x00007fcf39dfb000)
                    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fcf39bc0000)
                    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcf39ad0000)
                    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcf39ab0000)
                    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcf39880000)
                    /lib64/ld-linux-x86-64.so.2 (0x00007fcf3c363000)
            

            wlad Vladislav Vaintroub added a comment - - edited Is this supposed to work as described at all? Dockerfile builds own copy of openssl with fips, then ovewrites system libraries with own openssl. Not sure it can work like that. That assumes libraries are ABI compatible, but I do not believe there is a guarantee. The way to handle it, as I see it, is to build openssl with fips first, then build server against openssl-with-fips libraries and headers . Which is what I did, and it worked fine. Maybe I'm missing something? Below is what I did, on ubuntu-22.04 sudo apt-get update sudo apt-get install -y wget build-essential wget https://www.openssl.org/source/openssl-3.0.0.tar.gz tar zxpf openssl-3.0.0.tar.gz cd openssl-3.0.0 ./Configure enable-fips make sudo make install # installs into /usr/local cd .. # build server against openssl we just built sudo apt install cmake bison ncurses-dev libz-dev git clone https://github.com/mariadb/server --depth=1 cd server mkdir bld cd bld cmake .. -DOPENSSL_ROOT_DIR=/usr/local -DOPENSSL_SSL_LIBRARY=/usr/local/lib64/libssl.so -DOPENSSL_CRYPTO_LIBRARY=/usr/local/lib64/libcrypto.so cmake --build . -j32 --target minbuild Now I test it like this cd mysql-test perl mysql-test-run --suite=main openssl_1 Now I check that mariadbd really links against my own openssl, and yes, it does link against /usr/local/lib64/ SSL libraries wlad@desktop:~/server/bld$ ldd sql/mariadbd linux-vdso.so.1 (0x00007fffd449f000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fcf3a340000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fcf3a320000) libssl.so.3 => /usr/local/lib64/libssl.so.3 (0x00007fcf3a273000) libcrypto.so.3 => /usr/local/lib64/libcrypto.so.3 (0x00007fcf39dfb000) libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fcf39bc0000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcf39ad0000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcf39ab0000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcf39880000) /lib64/ld-linux-x86-64.so.2 (0x00007fcf3c363000)
            cory.mccarty Cory McCarty added a comment -

            You can also just build the OpenSSL FIPS module and configure the system version of OpenSSL that's included in the original MariaDB image to be in FIPS mode (which is a configuration that is recognized/recommended by OpenSSL), and the result is the same. In that case, MariaDB is still linking against the system OpenSSL, and the image still crashes on launch. (Also, note that just running ./Configure enable-fips doesn't actually update the OpenSSL configuration file to turn on FIPS mode.

            cory.mccarty Cory McCarty added a comment - You can also just build the OpenSSL FIPS module and configure the system version of OpenSSL that's included in the original MariaDB image to be in FIPS mode (which is a configuration that is recognized/recommended by OpenSSL), and the result is the same. In that case, MariaDB is still linking against the system OpenSSL, and the image still crashes on launch. (Also, note that just running ./Configure enable-fips doesn't actually update the OpenSSL configuration file to turn on FIPS mode.
            cory.mccarty Cory McCarty added a comment -

            To expand on my previous comment, I've attached a different Dockerfile and openssl.cnf (maybe not linked correctly from this comment since they have the same names as the original attachments) that use the system OpenSSL and just build and install the FIPS module (and configure OpenSSL to use it, since just installing it doesn't mean it gets used). It's somewhat of an odd configuration because it's using a slightly older version of the FIPS module than the version of OpenSSL because only certain versions of the FIPS module are actually FIPS validated (3.0.0 and 3.0.8; this configuration uses 3.0.0). The OpenSSL documentation suggests that this is a valid configuration. I strongly suspect that you would get the same results from just building the version of the FIPS module that matches the OpenSSL version, but that version wouldn't actually be FIPS validated.

            Dockerfile openssl.cnf

            cory.mccarty Cory McCarty added a comment - To expand on my previous comment, I've attached a different Dockerfile and openssl.cnf (maybe not linked correctly from this comment since they have the same names as the original attachments) that use the system OpenSSL and just build and install the FIPS module (and configure OpenSSL to use it, since just installing it doesn't mean it gets used). It's somewhat of an odd configuration because it's using a slightly older version of the FIPS module than the version of OpenSSL because only certain versions of the FIPS module are actually FIPS validated (3.0.0 and 3.0.8; this configuration uses 3.0.0). The OpenSSL documentation suggests that this is a valid configuration. I strongly suspect that you would get the same results from just building the version of the FIPS module that matches the OpenSSL version, but that version wouldn't actually be FIPS validated. Dockerfile openssl.cnf

            People

              wlad Vladislav Vaintroub
              cory.mccarty Cory McCarty
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.