I based demonstration_of_CONC-648_vulnerability against MariaDB server 10.11 and client v3.3.5, but it's trivial to demonstrate that earlier versions are equally vulnerable.

Do you think it's a problem? An "untrusted on-path attacker" can send TCP RST flag to abort the connection, wouldn't it basically have the same effect?

Do you think it's a problem?

Yes, I think it's a major vulnerability if a client is intended not to trust unencrypted communications, and yet it turns out to trusts unencrypted communication.

I've merely made an extremely simple demo of the vulnerability here, but it's quite likely that it's worse than this, and that the client will also trust other types of traffic before completing the TLS handshake.

An "untrusted on-path attacker" can send TCP RST flag to abort the connection, wouldn't it basically have the same effect?

An attacker interfering with layers below a TLS-based channel can disrupt communication between the peers, but should not be able to inject traffic that either peer accepts as genuine. A deviation from that (such as CVE-2009-3555 in which an attacker can inject TLS renegotiations along with unauthenticated data) is a vulnerability.

Due to this vulnerability in MariaDB, an on-path attacker can signal an arbitrary error to a client, which might (for example) cause the client to alter its retry behavior or to misdiagnose a connection problem.

And it can get much worse than that: I discovered this vulnerability in MariaDB because I'm working on an approach to MDEV-15935 ("server redirection mechanism") in which the server tells the client to switch to another server using an error packet. If error packets are indeed chosen as the mechanism for server redirection, this means that attackers would be able to redirect clients to arbitary attacker-controlled servers even when the clients think they are using TLS.

The client always has to accept error packages before TLS handshake is completed, since the handshake itself may fail. Everything which happens before the TLS handshake has completed succesfully should be considered as insecure.

In the comments of MDEV-15935 I suggested that both client and server should indicate redirection via capability flag, additionally there should be an option to allow redirection only for secure connections.

Everything which happens before the TLS handshake has completed succesfully should be considered as insecure.

I completely agree with that, but the MySQL/MariaDB Connector/C libraries fail in this regard, because they do not allow consumers to distinguish errors that happen after the TLS handshake from errors that happen before/during the TLS handshake. Contrast this with basically every other TLS client.

Every modern HTTP client clearly distinguishes errors before TLS handshake from errors after

Errors before/during establishment of TLS

Try cURLing {expired,untrusted-root,self-signed,revoked}.badssl.com as a useful source of reproducible TLS errors …

$ curl -f https://expired.badssl.com; echo $?

curl: (60) SSL certificate problem: certificate has expired

More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not

establish a secure connection to it. To learn more about this situation and

how to fix it, please visit the web page mentioned above.

60      # ← This error code indicates that 

        # "Peer certificate cannot be authenticated with known CA certificates."

Application-level errors after establishment of TLS

$ curl -f 'https://google.com/path/that/does/not/exist'; echo $?

curl: (22) The requested URL returned error: 404 

22     # ← This error code indicates that

       # "HTTP page not retrieved. The requested url was not found or returned another error with the HTTP error code being 400 or above. This return code only appears if -f, --fail is used."

or

$ curl -f https://httpbin.org/status/500; echo $?

curl: (22) The requested URL returned error: 500 

22      # ← This error code indicates that

        # "HTTP page not retrieved. The requested url was not found or returned another error with the HTTP error code being 400 or above. This return code only appears if -f, --fail is used."

mariadb CLI does not distinguish errors sent before or after TLS establishment

Error that is sent by the server, but clearly should not be trusted by the client:

(with my trivially modified server from dlenski/mariadb-server:demonstration_of_CONC-648_vulnerability)

$ mariadb --ssl --ssl-verify-server-cert -uUsername -pVerySecretPassword -h 127.0.0.1

ERROR 1815 (HY000): Internal error: Client will accept this error as genuine even if running with --ssl --ssl-verify-server-cert, and even though this error is sent in plaintext PRIOR TO TLS HANDSHAKE.

Error that should be trusted because it is sent after the TLS handshake:

$ mariadb --ssl --ssl-verify-server-cert -uUsername -pVerySecretPassword -h $NORMAL_MARIADB10.6.14_SERVER

ERROR 1045 (28000): Access denied for user 'Username'@'A.B.C.D' (using password: YES)

As far as I can tell, there is no way for a consumer of MariaDB Connector/C to distinguish these untrustworthy pre-handshake and trustworthy post-handshake errors, so it would be completely unsafe to base any automated redirection mechanism on the interpretation of an error packet.

In the comments of MDEV-15935 I suggested … there should be an option to allow redirection only for secure connections.

If the client libraries continue to handle untrustworthy pre-TLS-handshake errors in a way that's indistinguishable from trustworthy post-TLS-handshake errors, any flag sent by the server to indicate (only allow redirection for secure connections) will be ineffectual.

I've submitted mariadb-connector-c PR #223 which mitigates at least one avenue by which a TLS-enabled client can be tricked into improperly trusting an application-layer error packet received before completion of the TLS handshake.

(ping otto)

Do you really think it's more safe to send another error message and to hide the original one (which makes it even more difficult to find reasons for connection errors) ? Then it would be also logical to change the message when a TLS alert has occurred before the handshake has completed, since an alert could be sent by a MIM. This would make it impossible to analyze possible TLS problems.

Any error packet which the client retrieves before the handshake completed (e.g. "No more connections available", "Server can't create new thread", including all critical TLS alerts) forces the client to abort the connection. So I don't really see any vulnerability here.

Do you really think it's more safe to send another error message and to hide the original one (which makes it even more difficult to find reasons for connection errors) ?

Yes, I do. If the server inadvisably sends such error responses prior to the completion of the TLS handshake, then yes I think it is unsafe for the error to be relayed to the client in a form where the client could vary its behavior based on the contents of the error response. Replacing the untrusted-origin error with a generic one seems like the only safe mitigation.

Clearly, a much better approach would be to handshake and switch to TLS handshake immediately upon connection rather than having it happen several packets later. That's how TLS is supposed to be used and how almost every other TLS-secured protocol works.

Any error packet which the client retrieves before the handshake completed (e.g. "No more connections available", "Server can't create new thread", including all critical TLS alerts) forces the client to abort the connection.

The fact that the connection is aborted following the error does not in any way mitigate the fact that the error conveys information to the client, which the client should not trust.

In https://github.com/mariadb-corporation/mariadb-connector-c/pull/223, I give a brief description of a very plausible scenario in which clients can be induced to modify their behavior by sending such packets.

And as I explained above, this vulnerability is going make it completely unsafe to build any automated redirection mechanism based on the handling of error packets immediate following initial connection.

So I don't really see any vulnerability here.

I'm very surprised that you don't see this as a serious vulnerability.

Suppose that $SPY_AGENCY or $INTERNET_CENSOR wants to write a tool to degrade and interfere with TLS-secured MariaDB client→server connections. The issue discovered here makes it almost trivial to do so. Just detect the unencrypted pre-handshake MySQL server/client greeting packets and inject whichever application-layer error packets you think will interfere.

I was asked to review this by dlenski, I have been through it and here are my comments:

This does appear to be a cut and dry MITM attack, and if MDEV-15935 is implemented in such a way that the feature is possible to use before TLS handshake it becomes an even more serious MITM attack.

Right now it is important but I don't think critical. It is possible to make an application behave badly due to it being developed to respond a certain way based on an error code. For example, connect to server B because A returned a max connections error. This may sound benign, but I suspect there are ways to abuse this.

All that being said, I'm not sure the current PR is 100% the correct approach. You do want the error codes to remain as-is because this will break applications. I don't think an additional error code / message at the same time is possible, so the library should probably inject a prefix to the message which states that this message came before the handshake. If well documented, this could be something applications can watch for. Even better if it can be some kind of flag or API call to check if the message came prior to encryption.

TheLinuxJedi wrote:

All that being said, I'm not sure the current PR is 100% the correct approach. You do want the error codes to remain as-is because this will break applications.

Disagree. I think that (1) the security concern should trump any backwards-compatibility concern, and (2) there is no evidence that there is any real backwards-compatibility concern.

For (2), I don't find any evidence that real/non-malicious servers (MySQL, MariaDB, Percona, etc.) actually send pre-handshake application-level error messages.

As far as I can tell:

• Extant, working versions of the MariaDB server do not ever send error messages to TLS-using clients prior to the TLS handshake. (Although it's extremely hard to audit the code to be certain of this given its structure.)
• The fact that the MariaDB client operating in -ssl mode will accept a pre(TLS handshake) error message is simply a client-side vulnerability.

I don't think an additional error code / message at the same time is possible, so the library should probably inject a prefix to the message which states that this message came before the handshake.

What you're proposing here is to push the burden of verifying the transport-layer security onto the client application logic.

If well documented, this could be something applications can watch for.

In practice, applications won't watch for it, and pre-existing applications certainly won't watch for it.

This is why when a vulnerability like this one is discovered (supposedly TLS-secured client is vulnerable to incorrectly trust non-TLS-secured inputs), it's generally good practice to err on the side of breaking backwards-compatibility.

And anyway, as I describe above, there's no evidence that existing MySQL/MariaDB server implementations will actually send meaningful error packets to TLS-secured clients prior to the TLS handshake, so the backwards compatibility concern appears to be a theoretical one.

• Extant, working versions of the MariaDB server do not ever send error messages to TLS-using clients prior to the TLS handshake. (Although it's extremely hard to audit the code to be certain of this given its structure.)

And anyway, as I describe above, there's no evidence that existing MySQL/MariaDB server implementations will actually send meaningful error packets to TLS-secured clients prior to the TLS handshake, so the backwards compatibility concern appears to be a theoretical one.

The only exception I can find to this is that if the client requests an auth plugin which the server can't load, then the server will send this back to the client as an unencrypted error packet even if the client wants TLS/SSL: https://github.com/MariaDB/server/blob/11.0/sql/sql_acl.cc#L14331-L14338

There are 3 possible errors which the server can send before the handshake succeeded:

• too many connections
• server ran out of threads
• a TLS error/alert, e.g. no matching protocol, unsupported cipher suite, ....

plugin errors will be sent after the handshake, since the client hello packet will be encrypted.

There are 3 possible errors which the server can send before the handshake succeeded:

too many connections
server ran out of threads
a TLS error/alert, e.g. no matching protocol, unsupported cipher suite, ....

plugin errors will be sent after the handshake, since the client hello packet will be encrypted.

That is fewer than I thought, which is good to know. I do think it is important to know which one of these has been hit, particularly a differentiator between the first two and the last one. At least logged somewhere for debugging / post-mortem purposes. Even if the original error code is not maintained.

georg wrote:

There are 3 possible errors which the server can send before the handshake succeeded:

• too many connections
• server ran out of threads
• a TLS error/alert, e.g. no matching protocol, unsupported cipher suite, ....

These cases need to be distinguished:

The "too many connections" and "out of threads" errors are application-level errors: they are sent *by the server to the client.

The fact that these are sent by the server before the TLS handshake happens is bad. As I described above, the client may vary its retry behavior based on exactly what error it receives, and this could enable a variety of attacks… which rapidly get much, much more serious if the client starts automatically trying to connect to different servers based on errors it receives.

• TLS errors/alerts are transport-level errors: they are generated by the client as a result of an inability to construct a satisfactorily-secure TLS channel between it and the server. Assuming they all get the same error code (CR_SSL_CONNECTION_ERROR, I think?) then an attacker can prevent a connection from succeeding, but can't cause a client to vary its behavior based on the contents of the error. (If a client receives a CR_SSL_CONNECTION_ERROR, it shouldn't vary its behavior based on the detailed error message; "TLS alert X" vs "not allowed cipher Y", etc.)

TheLinuxJedi, perhaps my statements above about this previously were unclear: it's not that the client needs to hide the details of any error which it encounters before the TLS handshake. It's just that the client should hide the details of an application-level error which it appears to have received from the server before the TLS handshake (because that error could actually be from a MITM attacker).

And indeed that is what my PR does in its current form: "Do not trust error packets received from the server prior to TLS handshake completion". It does not prevent the client from returning unique error codes for other conditions which are unrelated to application-layer traffic ("hostname not found", "TLS alert", "conflicting connection parameters don't make sense", etc).

I've counted 13 errors that a server can legally send before TLS is established.

Four of them can only happen before authentication (before TLS, if requested):

• ER_BAD_HOST_ERROR
• ER_CANT_CREATE_THREAD
• ER_HOST_IS_BLOCKED
• ER_HOST_NOT_PRIVILEGED

Others can happen before or after TLS:

• EE_OUTOFMEMORY
• ER_CONNECTION_KILLED
• ER_HANDSHAKE_ERROR
• ER_NET_FCNTL_ERROR
• ER_NET_PACKET_TOO_LARGE
• ER_NET_READ_ERROR
• ER_NET_READ_INTERRUPTED
• ER_NET_UNCOMPRESS_ERROR
• ER_OUT_OF_RESOURCES

Even if we only consider the first four errors, ER_CANT_CREATE_THREAD is transient and the client might want to retry, ER_HOST_IS_BLOCKED and ER_HOST_NOT_PRIVILEGED are permanent. A client cannot just ignore those errors. And in the MitM case they can be spoofed indeed. I don't see how you can possibly solve this.

ER_ACCESS_DENIED can never come before TLS and a client can safely ignore it and all errors that aren't in the list above. But errors from the above list are more than enough for MitM to use.

I've counted 13 errors that a server can legally send before TLS is established.

serg, thanks for researching this!

Even if we only consider the first four errors, ER_CANT_CREATE_THREAD is transient and the client might want to retry, ER_HOST_IS_BLOCKED and ER_HOST_NOT_PRIVILEGED are permanent. A client cannot just ignore those errors. And in the MitM case they can be spoofed indeed. I don't see how you can possibly solve this.

You can solve this by designing the applications (client+server) and the protocol to ensure an appropriate separation of concerns between the transport layer (TLS to ensure authenticity of the server and end-to-end encryption of the communications with it) and the application layer (validating the user's credentials to access the database).

If a TCP-based client-server protocol wants to use TLS, then wrapping the TCP socket in a TLS socket should be the very first thing that happens, before any communication over the socket. There shouldn't be any exchange of plaintext packets prior to the TLS handshake.

Modern web browsers using TLS (especially TLSv1.3 with ECH), and VoIP apps, and TLS-based VPNs (like those supported by openconnect, which I contribute to) basically get this right, and leak no application-layer information prior to TLS establishment… but MariaDB does this quite badly and leaks a ton of information. This seems to be largely a consequence of the unstructured approach to bolting TLS on to the client and server code. The description of the connection protocol appears to have been written after-the-fact to reflect how MariaDB server and Connector/C use TLS, rather than designed in advance.

Even in the happy case where the server doesn't send any pre-handshake error to the client, the server sends a "greeting" packet to the client (containing server version information) and the client sends back a login request packet (with no credentials, but with the client's charset and flags) in plaintext before the TLS handshake starts:

This makes MariaDB client-server connections an exploitable and target-rich environment for pervasive MITM attackers. A government agency could, for example, fingerprint the plaintext client+server greeting packets to determine the exact versions, pull out the ones that appear to be from interesting parts of the world based on the plaintext preferred client charset, and manipulate them in various ways with MITM and downgrade attacks using this vulnerability, as well as the long-known MDEV-28634… and all of that without needing to actually do any TLS cracking.

For all I know, the NSA or CSIC or GCHQ or יחידה 8200 or the Chinese/Iranian/Indian/Russian/$COUNTRY equivalents have already figured this out themselves, and have been MITM'ing MariaDB connections on the Internet at massive scale for years.

(UPDATE: Spun this off into CONC-654 and MDEV-31585; it turns out that there's a server-side mistake in TLS setup, which makes it impossible to fix the client-side leakage without a server-side fix as well.)

I've been studying the client-server protocol and implementations pretty carefully for a couple weeks now, and I'm convinced that these vulnerabilities are entirely solvable and in a backwards-compatible way, but it'd require a concerted effort to prioritize the code and design changes.

Connector/C PR #223 has now been merged to the 3.3 branch, but not yet tagged in a release.