[MXS-4198] MaxScale fails to validate its own certificate when the chain of trust is unknown to OpenSSL Created: 2022-07-08  Updated: 2022-11-04  Resolved: 2022-07-15

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 6.2.4
Fix Version/s: 2.5.22, 6.4.2

Type: Bug Priority: Major
Reporter: Assen Totin (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None


 Description   

When a listener is created with encryption enabled, MaxScale may load extended authentication options from external files: a certificate, its key and a chain of trust. The chain of trust is provided to allow the verification of certificates, issued by authorities that are not known to the underlying encryption layer (e.g., an authority which chain of trust is unknown to OpenSSL).

During this process there are two errors:

First, MaxScale tries to verify its own server certificate. This is useless, because this certificate will be presented to the client, which should verify it (if desired). What is important for MaxScale is to be able to verify clients' certificates, not its own. Hence the below check is wrong, because the context that is passed to SSL_CTX_build_cert_chain() is the one with the MaxScale's own certificate (the previous check, calling SSL_CTX_check_private_key() for a match between the certificate and the key, is OK and should remain).

https://github.com/mariadb-corporation/MaxScale/blob/68800753d4b540ecedb5aefb012805d1a974f31c/server/core/ssl.cc#L217

Second, if the CA used to issue the MaxScale certificate is unknown to OpenSSL, it will produce an error - even if the appropriate CA chain of trust is loaded by MaxScale from the provided file. There is even a vain attempt to handle this, converting the error to a notice (which is still wrong, because MaxScale does have the proper chain of trust, but OpenSSL does not):

https://github.com/mariadb-corporation/MaxScale/blob/68800753d4b540ecedb5aefb012805d1a974f31c/server/core/ssl.cc#L222

I see few ways out of this:

  • Completely remove the call to SSL_CTX_build_cert_chain() as this is wrong to do it in this place for the context specified.
  • Alternatively, make sure OpenSSL is passed the CA chain of trust that MaxScale has loaded so that the validation does succeed.

PoC:

  • Generate a self-signed CA certificate and key.
  • Generate a CSR for a new certificate.
  • Sign the certificate with the CA.
  • Manually validate the certificate with the CA:

[root@a1w1 assen.totin]# cd /etc/test/
[root@a1w1 assen.totin]# ls -l
total 12
rw-rr-. 1 root root 2065 Jul 1 13:28 ca.pem
rw-rr-. 1 root root 1939 Jul 1 13:28 certificate.pem
rw-rr-. 1 root root 3272 Jul 1 13:28 key.pem
[root@a1w1 test]# openssl verify -CAfile ca.pem certificate.pem
certificate.pem: OK

  • Use maxctrl to create a listener with encryption enabled and the certificate, key and chain of trust loaded:

maxctrl create listener service-3 listener-3 20003 --interface=172.20.2.41 --protocol=mariadbclient ssl=true ssl_cert=/etc/test/certificate.pem ssl_key=/etc/test/key.pem ssl_ca_cert=/etc/test/ca.pem

  • Observe the notice printed to MaxScale error log:

2022-07-08 13:07:58 notice : (listener-3); OpenSSL reported problems in the certificate chain: error:1414C086:SSL routines:ssl_build_cert_chain:certificate verify failed. This is expected for certificates that do not contain the whole certificate chain.

 Comments   
Comment by markus makela [ 2022-07-11 ]

Does the verification work if you install the custom CA certificate into the system certificates?

Comment by Assen Totin (Inactive) [ 2022-07-11 ]

Yes, it does.

Comment by markus makela [ 2022-07-11 ]

Can you add the affected versions to the issue?

Comment by markus makela [ 2022-07-11 ]

On Fedora 35 it seems to always report the verification failure even if the CA certificate is installed into the system certificate with trust anchor. The verification will succeed with no errors from OpenSSL if the certificate in ssl_cert contains the complete certificate chain from the root CA to the individual certificate. That is, the output of cat certificate.pem ca.pem > chained.pem.

Comment by Assen Totin (Inactive) [ 2022-07-11 ]

Well, if you put the chain of trust into the same file as the certificate, verification will, naturally, succeed. However, this beats the purpose of having a separate CA parameter. Moreover, the CA is necessary to validate the certificates of the SQL clients, not of MaxScale itself - and only if the OpenSSL does not have the necessary chain of trust (e.g., when using a private CA).

(The very same issue may well exists with "server" entries, where a CA is only necessary to validate the certificate of the MariaDB Server backend.)

Comment by markus makela [ 2022-07-11 ]

What OS did you test this on? So far I haven't been able to make it so that OpenSSL accepts the certificate chain from the ssl_cert if it's partial and the system default CA files are being used.

My config for the listener was this:

[RW-Split-Listener]
 
type=listener
 
service=RW-Split-Router
 
protocol=MariaDBClient
 
port=4006
 
ssl=true
 
ssl_cert=/home/markusjm/ssl-certs/server-cert.pem
 
ssl_key=/home/markusjm/ssl-certs/server-key.pem

Comment by markus makela [ 2022-07-11 ]

I think we can just make it so that the cert chain is built but any problems that OpenSSL are ignored by using SSL_BUILD_CHAIN_FLAG_IGNORE_ERROR in the SSL_CTX_build_cert_chain call. This should still make it perform better as the chain is built during construction instead of once per TLS session.

From the OpenSSL manual:

The functions SSL_CTX_build_cert_chain() and SSL_build_cert_chain() can be used to check application configuration and to ensure any necessary subordinate CAs are sent in the correct order. Misconfigured applications sending incorrect certificate chains often cause problems with peers.

For example an application can add any set of certificates using SSL_CTX_use_certificate_chain_file() then call SSL_CTX_build_cert_chain() with the option SSL_BUILD_CHAIN_FLAG_CHECK to check and reorder them.

Applications can issue non fatal warnings when checking chains by setting the flag SSL_BUILD_CHAIN_FLAG_IGNORE_ERRORS and checking the return value.

Calling SSL_CTX_build_cert_chain() or SSL_build_cert_chain() is more efficient than the automatic chain building as it is only performed once. Automatic chain building is performed on each new session.

Comment by Assen Totin (Inactive) [ 2022-07-11 ]

I guess my first test was on RHEL-9 where our RHEL-8 builds for MaxScale 6.2 run fine.

My original suggestion to remove a check that is useless still stands. What is the point of trying to verify the local certificate? Are we going to fail the whole start-up if the verification fails? If yes, why - what does it mean to us to have the local certificate unverified? If not, than remove the unnecessary check altogether.

Comment by markus makela [ 2022-07-11 ]

The call to SSL_CTX_build_cert_chain() isn't for verifying the certificate chain, it is to build it which is otherwise done once per client connection (at least based on that OpenSSL documentation). I guess in some sense these errors were ignore before so the sensible thing is to ignore them again.

Generated at Thu Feb 08 04:26:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.