Uploaded image for project: 'MariaDB Connector/C'
  1. MariaDB Connector/C
  2. CONC-648

Client improperly accepts error packets prior to TLS handshake

Details

    Description

      If the MariaDB client is running with --ssl --ssl-verify-server-cert, it should not trust any application-level traffic received prior to the completion of the TLS handshake and the validation of the server's TLS certificate.

      The MariaDB client (as built from [v3.3.5](https://github.com/mariadb-corporation/mariadb-connector-c/releases/tag/v3.3.5)) violates this expectation, making it trivially susceptible to DOS by untrusted on-path attackers, even when the user has explicitly specified --ssl --ssl-verify-server-cert.

      Demonstration:

      1. Build dlenski/mariadb-server:demonstration_of_CONC-648_vulnerability
        • This commit modifies the server to unconditionally send an error packet to the client, prior to authentication and prior to TLS handshake and server certificate validation:

          diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc
          index 80d52ce18fc..283b095f2eb 100644
          --- a/sql/sql_acl.cc
          +++ b/sql/sql_acl.cc
          @@ -14457,6 +14457,14 @@ bool acl_authenticate(THD *thd, uint com_change_user_pkt_len)
               DBUG_ASSERT(mpvio.status == MPVIO_EXT::RESTART ||
                           mpvio.status == MPVIO_EXT::SUCCESS);
             }
          +  else if (1)
          +  {
          +    my_error(ER_INTERNAL_ERROR, MYF(0),
          +             "Client will accept this error as genuine even if running with "
          +             "--ssl --ssl-verify-server-cert, and even though this error is "
          +             "sent in plaintext PRIOR TO TLS HANDSHAKE.");
          +    res= CR_ERROR;
          +  }
             else
             {
               /* mark the thd as having no scramble yet */

      2. Start the server, e.g.

        $ DIR=$(mktemp -d); mkdir -p $DIR/data; sql/mariadbd --no-defaults --datadir=$DIR/data --socket=$DIR/mysql.sock --skip-grant-tables --debug
        2023-06-05 15:24:07 0 [Note] sql/mariadbd: ready for connections.
        Version: '10.11.4-MariaDB-debug'  socket: '/tmp/tmp.P4FvcEcKrH/mysql.sock'  port: 3306  Source distribution

      3. Attempt to connect to it with --ssl --ssl-verify-server-cert:

        $ client/mariadb -h 127.0.0.1
        ERROR 1815 (HY000): Internal error: Client will accept this error as genuine even if running with --ssl --ssl-verify-server-cert, and even though this error is sent in plaintext PRIOR TO TLS HANDSHAKE.

      Running tcpdump in the background confirms that the client is improperly accepting the error packet, even though it has been sent in plaintext and without a TLS handshake:

      $ sudo tcpdump -n -X -i lo tcp port 3306
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
      15:24:46.182853 IP 127.0.0.1.40234 > 127.0.0.1.3306: Flags [S], seq 1546762979, win 65495, options [mss 65495,sackOK,TS val 113496632 ecr 0,nop,wscale 7], length 0
              0x0000:  4500 003c 2b03 4000 8006 d1b6 7f00 0001  E..<+.@.........       # TCP handshake, no application-level content
              0x0010:  7f00 0001 9d2a 0cea 5c31 bae3 0000 0000  .....*..\1......
              0x0020:  a002 ffd7 fe30 0000 0204 ffd7 0402 080a  .....0..........
              0x0030:  06c3 d238 0000 0000 0103 0307            ...8........
      15:24:46.182917 IP 127.0.0.1.3306 > 127.0.0.1.40234: Flags [S.], seq 3238764927, ack 1546762980, win 65483, options [mss 65495,sackOK,TS val 113496632 ecr 113496632,nop,wscale 7], length 0
              0x0000:  4500 003c 0000 4000 8006 fcb9 7f00 0001  E..<..@.........       # TCP handshake, no application-level content
              0x0010:  7f00 0001 0cea 9d2a c10b a17f 5c31 bae4  .......*....\1..
              0x0020:  a012 ffcb fe30 0000 0204 ffd7 0402 080a  .....0..........
              0x0030:  06c3 d238 06c3 d238 0103 0307            ...8...8....
      15:24:46.182957 IP 127.0.0.1.40234 > 127.0.0.1.3306: Flags [.], ack 1, win 512, options [nop,nop,TS val 113496632 ecr 113496632], length 0
              0x0000:  4500 0034 2b04 4000 8006 d1bd 7f00 0001  E..4+.@.........       # TCP handshake, no application-level content
              0x0010:  7f00 0001 9d2a 0cea 5c31 bae4 c10b a180  .....*..\1......
              0x0020:  8010 0200 fe28 0000 0101 080a 06c3 d238  .....(.........8
              0x0030:  06c3 d238                                ...8
      15:24:46.185305 IP 127.0.0.1.3306 > 127.0.0.1.40234: Flags [P.], seq 1:189, ack 1, win 512, options [nop,nop,TS val 113496635 ecr 113496632], length 188
              0x0000:  4508 00f0 622c 4000 8006 99d1 7f00 0001  E...b,@.........
              0x0010:  7f00 0001 0cea 9d2a c10b a180 5c31 bae4  .......*....\1..
              0x0020:  8018 0200 fee4 0000 0101 080a 06c3 d23b  ...............;
              0x0030:  06c3 d238 b800 0000 ff17 0749 6e74 6572  ...8.......Inter       # [b8 00 00 00]: header { [b8 00 00] = length following header, 0xb8 bytes, [00]: sequence number (https://mariadb.com/kb/en/0-packet) }
              0x0040:  6e61 6c20 6572 726f 723a 2043 6c69 656e  nal.error:.Clien       #     [ff]: ERR_Packet (https://mariadb.com/kb/en/err_packet/)
              0x0050:  7420 7769 6c6c 2061 6363 6570 7420 7468  t.will.accept.th       #     [17 07]: error ER_INTERNAL_ERROR (https://mariadb.com/kb/en/mariadb-error-codes)
              0x0060:  6973 2065 7272 6f72 2061 7320 6765 6e75  is.error.as.genu       #     [remaining 0xb5 bytes] human-readable message = "Internal error: Client … TLS HANDSHAKE."
              0x0070:  696e 6520 6576 656e 2069 6620 7275 6e6e  ine.even.if.runn
              0x0080:  696e 6720 7769 7468 202d 2d73 736c 202d  ing.with.--ssl.-
              0x0090:  2d73 736c 2d76 6572 6966 792d 7365 7276  -ssl-verify-serv
              0x00a0:  6572 2d63 6572 742c 2061 6e64 2065 7665  er-cert,.and.eve
              0x00b0:  6e20 7468 6f75 6768 2074 6869 7320 6572  n.though.this.er
      	0x00c0:  726f 7220 6973 2073 656e 7420 696e 2070  ror.is.sent.in.p
      	0x00d0:  6c61 696e 7465 7874 2050 5249 4f52 2054  laintext.PRIOR.T
      	0x00e0:  4f20 544c 5320 4841 4e44 5348 414b 452e  O.TLS.HANDSHAKE.
      

      Attachments

        Issue Links

          Activity

            There are 3 possible errors which the server can send before the handshake succeeded:

            too many connections
            server ran out of threads
            a TLS error/alert, e.g. no matching protocol, unsupported cipher suite, ....

            plugin errors will be sent after the handshake, since the client hello packet will be encrypted.

            That is fewer than I thought, which is good to know. I do think it is important to know which one of these has been hit, particularly a differentiator between the first two and the last one. At least logged somewhere for debugging / post-mortem purposes. Even if the original error code is not maintained.

            TheLinuxJedi Andrew Hutchings (Inactive) added a comment - There are 3 possible errors which the server can send before the handshake succeeded: too many connections server ran out of threads a TLS error/alert, e.g. no matching protocol, unsupported cipher suite, .... plugin errors will be sent after the handshake, since the client hello packet will be encrypted. That is fewer than I thought, which is good to know. I do think it is important to know which one of these has been hit, particularly a differentiator between the first two and the last one. At least logged somewhere for debugging / post-mortem purposes. Even if the original error code is not maintained.

            georg wrote:

            There are 3 possible errors which the server can send before the handshake succeeded:

            • too many connections
            • server ran out of threads
            • a TLS error/alert, e.g. no matching protocol, unsupported cipher suite, ....

            These cases need to be distinguished:

            The "too many connections" and "out of threads" errors are application-level errors: they are sent *by the server to the client.

            The fact that these are sent by the server before the TLS handshake happens is bad. As I described above, the client may vary its retry behavior based on exactly what error it receives, and this could enable a variety of attacks… which rapidly get much, much more serious if the client starts automatically trying to connect to different servers based on errors it receives.

            • TLS errors/alerts are transport-level errors: they are generated by the client as a result of an inability to construct a satisfactorily-secure TLS channel between it and the server. Assuming they all get the same error code (CR_SSL_CONNECTION_ERROR, I think?) then an attacker can prevent a connection from succeeding, but can't cause a client to vary its behavior based on the contents of the error. (If a client receives a CR_SSL_CONNECTION_ERROR, it shouldn't vary its behavior based on the detailed error message; "TLS alert X" vs "not allowed cipher Y", etc.)

            TheLinuxJedi, perhaps my statements above about this previously were unclear: it's not that the client needs to hide the details of any error which it encounters before the TLS handshake. It's just that the client should hide the details of an application-level error which it appears to have received from the server before the TLS handshake (because that error could actually be from a MITM attacker).

            And indeed that is what my PR does in its current form: "Do not trust error packets received from the server prior to TLS handshake completion". It does not prevent the client from returning unique error codes for other conditions which are unrelated to application-layer traffic ("hostname not found", "TLS alert", "conflicting connection parameters don't make sense", etc).

            dlenski Daniel Lenski (Inactive) added a comment - georg wrote: There are 3 possible errors which the server can send before the handshake succeeded: too many connections server ran out of threads a TLS error/alert, e.g. no matching protocol, unsupported cipher suite, .... These cases need to be distinguished: The "too many connections" and "out of threads" errors are application -level errors: they are sent *by the server to the client. The fact that these are sent by the server before the TLS handshake happens is bad. As I described above , the client may vary its retry behavior based on exactly what error it receives, and this could enable a variety of attacks… which rapidly get much, much more serious if the client starts automatically trying to connect to different servers based on errors it receives. TLS errors/alerts are transport -level errors: they are generated by the client as a result of an inability to construct a satisfactorily-secure TLS channel between it and the server. Assuming they all get the same error code ( CR_SSL_CONNECTION_ERROR , I think?) then an attacker can prevent a connection from succeeding, but can't cause a client to vary its behavior based on the contents of the error. (If a client receives a CR_SSL_CONNECTION_ERROR , it shouldn't vary its behavior based on the detailed error message; "TLS alert X" vs "not allowed cipher Y", etc.) TheLinuxJedi , perhaps my statements above about this previously were unclear: it's not that the client needs to hide the details of any error which it encounters before the TLS handshake. It's just that the client should hide the details of an application-level error which it appears to have received from the server before the TLS handshake (because that error could actually be from a MITM attacker). And indeed that is what my PR does in its current form: "Do not trust error packets received from the server prior to TLS handshake completion". It does not prevent the client from returning unique error codes for other conditions which are unrelated to application-layer traffic ("hostname not found", "TLS alert", "conflicting connection parameters don't make sense", etc).

            I've counted 13 errors that a server can legally send before TLS is established.

            Four of them can only happen before authentication (before TLS, if requested):

            • ER_BAD_HOST_ERROR
            • ER_CANT_CREATE_THREAD
            • ER_HOST_IS_BLOCKED
            • ER_HOST_NOT_PRIVILEGED

            Others can happen before or after TLS:

            • EE_OUTOFMEMORY
            • ER_CONNECTION_KILLED
            • ER_HANDSHAKE_ERROR
            • ER_NET_FCNTL_ERROR
            • ER_NET_PACKET_TOO_LARGE
            • ER_NET_READ_ERROR
            • ER_NET_READ_INTERRUPTED
            • ER_NET_UNCOMPRESS_ERROR
            • ER_OUT_OF_RESOURCES

            Even if we only consider the first four errors, ER_CANT_CREATE_THREAD is transient and the client might want to retry, ER_HOST_IS_BLOCKED and ER_HOST_NOT_PRIVILEGED are permanent. A client cannot just ignore those errors. And in the MitM case they can be spoofed indeed. I don't see how you can possibly solve this.

            ER_ACCESS_DENIED can never come before TLS and a client can safely ignore it and all errors that aren't in the list above. But errors from the above list are more than enough for MitM to use.

            serg Sergei Golubchik added a comment - I've counted 13 errors that a server can legally send before TLS is established. Four of them can only happen before authentication (before TLS, if requested): ER_BAD_HOST_ERROR ER_CANT_CREATE_THREAD ER_HOST_IS_BLOCKED ER_HOST_NOT_PRIVILEGED Others can happen before or after TLS: EE_OUTOFMEMORY ER_CONNECTION_KILLED ER_HANDSHAKE_ERROR ER_NET_FCNTL_ERROR ER_NET_PACKET_TOO_LARGE ER_NET_READ_ERROR ER_NET_READ_INTERRUPTED ER_NET_UNCOMPRESS_ERROR ER_OUT_OF_RESOURCES Even if we only consider the first four errors, ER_CANT_CREATE_THREAD is transient and the client might want to retry, ER_HOST_IS_BLOCKED and ER_HOST_NOT_PRIVILEGED are permanent. A client cannot just ignore those errors. And in the MitM case they can be spoofed indeed. I don't see how you can possibly solve this. ER_ACCESS_DENIED can never come before TLS and a client can safely ignore it and all errors that aren't in the list above. But errors from the above list are more than enough for MitM to use.
            dlenski Daniel Lenski (Inactive) added a comment - - edited

            I've counted 13 errors that a server can legally send before TLS is established.

            serg, thanks for researching this!

            Even if we only consider the first four errors, ER_CANT_CREATE_THREAD is transient and the client might want to retry, ER_HOST_IS_BLOCKED and ER_HOST_NOT_PRIVILEGED are permanent. A client cannot just ignore those errors. And in the MitM case they can be spoofed indeed. I don't see how you can possibly solve this.

            You can solve this by designing the applications (client+server) and the protocol to ensure an appropriate separation of concerns between the transport layer (TLS to ensure authenticity of the server and end-to-end encryption of the communications with it) and the application layer (validating the user's credentials to access the database).

            If a TCP-based client-server protocol wants to use TLS, then wrapping the TCP socket in a TLS socket should be the very first thing that happens, before any communication over the socket. There shouldn't be any exchange of plaintext packets prior to the TLS handshake.

            Modern web browsers using TLS (especially TLSv1.3 with ECH), and VoIP apps, and TLS-based VPNs (like those supported by openconnect, which I contribute to) basically get this right, and leak no application-layer information prior to TLS establishment… but MariaDB does this quite badly and leaks a ton of information. This seems to be largely a consequence of the unstructured approach to bolting TLS on to the client and server code. The description of the connection protocol appears to have been written after-the-fact to reflect how MariaDB server and Connector/C use TLS, rather than designed in advance.

            Even in the happy case where the server doesn't send any pre-handshake error to the client, the server sends a "greeting" packet to the client (containing server version information) and the client sends back a login request packet (with no credentials, but with the client's charset and flags) in plaintext before the TLS handshake starts:

            This makes MariaDB client-server connections an exploitable and target-rich environment for pervasive MITM attackers. A government agency could, for example, fingerprint the plaintext client+server greeting packets to determine the exact versions, pull out the ones that appear to be from interesting parts of the world based on the plaintext preferred client charset, and manipulate them in various ways with MITM and downgrade attacks using this vulnerability, as well as the long-known MDEV-28634… and all of that without needing to actually do any TLS cracking.

            For all I know, the NSA or CSIC or GCHQ or יחידה 8200 or the Chinese/Iranian/Indian/Russian/$COUNTRY equivalents have already figured this out themselves, and have been MITM'ing MariaDB connections on the Internet at massive scale for years.

            (UPDATE: Spun this off into CONC-654 and MDEV-31585; it turns out that there's a server-side mistake in TLS setup, which makes it impossible to fix the client-side leakage without a server-side fix as well.)


            I've been studying the client-server protocol and implementations pretty carefully for a couple weeks now, and I'm convinced that these vulnerabilities are entirely solvable and in a backwards-compatible way, but it'd require a concerted effort to prioritize the code and design changes.

            dlenski Daniel Lenski (Inactive) added a comment - - edited I've counted 13 errors that a server can legally send before TLS is established. serg , thanks for researching this! Even if we only consider the first four errors, ER_CANT_CREATE_THREAD is transient and the client might want to retry, ER_HOST_IS_BLOCKED and ER_HOST_NOT_PRIVILEGED are permanent. A client cannot just ignore those errors. And in the MitM case they can be spoofed indeed. I don't see how you can possibly solve this. You can solve this by designing the applications (client+server) and the protocol to ensure an appropriate separation of concerns between the transport layer (TLS to ensure authenticity of the server and end-to-end encryption of the communications with it) and the application layer (validating the user's credentials to access the database). If a TCP-based client-server protocol wants to use TLS, then wrapping the TCP socket in a TLS socket should be the very first thing that happens , before any communication over the socket. There shouldn't be any exchange of plaintext packets prior to the TLS handshake. Modern web browsers using TLS (especially TLSv1.3 with ECH ), and VoIP apps, and TLS-based VPNs (like those supported by openconnect , which I contribute to) basically get this right , and leak no application-layer information prior to TLS establishment… but MariaDB does this quite badly and leaks a ton of information. This seems to be largely a consequence of the unstructured approach to bolting TLS on to the client and server code. The description of the connection protocol appears to have been written after-the-fact to reflect how MariaDB server and Connector/C use TLS, rather than designed in advance. Even in the happy case where the server doesn't send any pre-handshake error to the client, the server sends a "greeting" packet to the client ( containing server version information ) and the client sends back a login request packet (with no credentials, but with the client's charset and flags ) in plaintext before the TLS handshake starts : This makes MariaDB client-server connections an exploitable and target-rich environment for pervasive MITM attackers. A government agency could, for example, fingerprint the plaintext client+server greeting packets to determine the exact versions, pull out the ones that appear to be from interesting parts of the world based on the plaintext preferred client charset , and manipulate them in various ways with MITM and downgrade attacks using this vulnerability, as well as the long-known MDEV-28634 … and all of that without needing to actually do any TLS cracking. For all I know, the NSA or CSIC or GCHQ or יחידה 8200 or the Chinese/Iranian/Indian/Russian/$COUNTRY equivalents have already figured this out themselves, and have been MITM'ing MariaDB connections on the Internet at massive scale for years. (UPDATE: Spun this off into CONC-654 and MDEV-31585 ; it turns out that there's a server-side mistake in TLS setup, which makes it impossible to fix the client-side leakage without a server-side fix as well.) I've been studying the client-server protocol and implementations pretty carefully for a couple weeks now, and I'm convinced that these vulnerabilities are entirely solvable and in a backwards-compatible way, but it'd require a concerted effort to prioritize the code and design changes.

            Connector/C PR #223 has now been merged to the 3.3 branch, but not yet tagged in a release.

            dlenski Daniel Lenski (Inactive) added a comment - Connector/C PR #223 has now been merged to the 3.3 branch, but not yet tagged in a release.

            People

              serg Sergei Golubchik
              dlenski Daniel Lenski (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.