Since a picture is worth 1,000 words.

Unpatched client talking to unpatched server

This is the plaintext client hello packet sent by an unpatched MariaDB client. It contains a great deal of information that can be used to fingerprint the client version, and it identifies the client's preferred character set (Russian, in this case!) which can be used to geographically identify the client. Trivially exploitable by every intelligence agency, ISP, or middlebox.

Client patched for CONC-654 talking to server patched for MDEV-31585

This is the plaintext client hello packet sent by a MariaDB client with my fix for CONC-654, talking to a server with my fix for MDEV-31585. It contains no meaningful information at all, other than a single bit which tells the server to expect a TLS handshake.

You can change client, and server, but how much did you test that you did not break the protocol, and that MySQL clients (drivers) and 3rd party clients(drivers) can talk to MariaDB server, and that and that MariaDB clients (other than C) can talk to MariaDB servers or MySQL servers.

Breaking this is worse than alleged catastrophe leaking the extremely meaningful charset information which in 99.9% will be some form of UTF8.

Changing server in a way that it can talk only to C client of the last version, and nothing else, is much worse than leaking "understands 4.1 protocol" flag.

Changing client in a way that it can talk only to own server in the last version, and nothing else, is much worse than leaking "understands multiple result sets" flag.

Having software that is allegedly more secure, means nothing if your application does not work at all, because the client can't connect.

Changing a single bit in a protocol requires a huge effort of compatibility testing, that's a little more than "my patched client can talk to my patched server".

wlad, did you see the corresponding server-side Jira, MDEV-31585?

The server-side problem is just a straight-up bug: the server reads the client's login request packet twice (once before TLS handshake, once after), but it sets the capability flags based only on the pre-TLS packet, and ignores the capability flags sent in the post-TLS-handshake… even though all client software that I can find resends them:

• The server-side issue can be fixed without any loss of backwards-compatibility to older clients based on my testing thus far.
• The client-side issue (this one) cannot be resolved in a way that is fully backwards-compatible with existing buggy servers; it requires "the introduction of a new extended capability bit [on the server] to inform the client that it has been fixed", as I wrote in MDEV-31585.

Breaking this is worse than alleged catastrophe leaking the extremely meaningful charset information…

For at least 10 years, the widely-understood threat model for Internet traffic involves pervasive attackers that can statefully inspect and react to any plaintext information travelling over the network. A character set field can give a strong hint about the end-user's geographic location.

There is longstanding precedent for this. About 15 years ago, the Conficker worm tried to restrict its targeting to specific countries by detecting the keyboard layouts of the systems it was infecting. Other suspected APT-authored malware has done similarly.

… which in 99.9% will be some form of UTF8.

There are millions of installations of MariaDB worldwide, so your 99.9% estimate would imply there are tens of thousands of MariaDB installations using non-UTF8-based charsets by default.

Having software that is allegedly more secure, means nothing if your application does not work at all, because the client can't connect.

I think that this orientation of "backwards-compatibility, even at the cost of security" largely explains why MariaDB still has TLS vulnerabilities such as this one, CONC-648, and MDEV-28634… which other types of client-server software made a concerted effort to eliminate many years ago.

Changing a single bit in a protocol requires a huge effort of compatibility testing, that's a little more than "my patched client can talk to my patched server".

Absolutely. I look forward to sharing the code and the accompanying compatibility testing.

• Server-side fix (should be 100% backwards-compatible with older clients): is in https://github.com/MariaDB/server/pull/2684
• Connector/C fix (requires updated server to be fully effective) is in https://github.com/mariadb-corporation/mariadb-connector-c/pull/227

If you like to eliminate different plaintext-leaks that happen before TLS handshake, then there should be a dedicated TLS port, on which TLS handshake happens before anything else, I think.

Otherwise, there will be some plaintext information travelling over the network,

This would require a more concerted effort to bring all clients (Oracle MySQL and 3rd party connectors, MariaDB connectors, and all possible own and 3rd party CLI and GUI tools) to handle additional TLS-dedicated-port, but I seriously doubt that any half-measure, and a very gentle massage of the current connection phase logic with new capability bit would be able to fix various "server reports its version and capability bits in plaintext in welcome packet", or "client inappropriately gets error packet 'Too many connections' in plaintext" that you also reported.

Of course, the old logic of switching from plaintext to SSL after server's "welcome" packet would still have to be present, for the backward compatibility.
At least this, and maybe a good OSS citizen would try reporting and fixing the same in MySQL, too. It is MySQL client-server protocol after all.

If you like to eliminate different plaintext-leaks that happen before TLS handshake, then there should be a dedicated TLS port, on which TLS handshake happens before anything else, I think.

That's the HTTP approach: port 80 for plain HTTP, port 443 for HTTPS.

Otherwise, there will be some plaintext information travelling over the network,

Sure, but there can be much less of it than there is currently.

When TLS support was added to the MySQL/MariaDB connection protocol, even a very small amount of design review should have made it clear that:

1. The pre-(TLS handshake) client packet (called "SSLRequest packet" here, though not in the code) is largely redundant with the post-(TLS handshake) client packet.
2. When the client is preparing to do a TLS handshake, it signals this simply by setting the CLIENT_SSL bit in the "SSLRequest" packet
3. The server should react to the initial plaintext packet received from the client, as follow…
   • When the CLIENT_SSL bit is set, that means that this is the "SSLRequest packet". Its only purpose is to signal the forthcoming TLS handshake. The server should not pay attention to any other information from this packet, because the client is going to send its "real" login request packet over the TLS channel.
   • When the CLIENT_SSL bit is unset, this is the "real" login request packet.
4. The code should be written in such a way as to cleanly separate these two paths, and to ensure that information send and received pre-TLS cannot be inadvertently mixed with information sent and received post-TLS.
   • The current structure of parse_client_reply_packet completely fails at this, with the yes-TLS and the no-TLS paths interleaved in an extremely hard-to-audit fashion.

My https://github.com/MariaDB/server/pull/2684 is an initial step towards (#3), but it doesn't fully refactor the design (#4) to make it obvious that the code is free of such improper pre-TLS and post-TLS mixing.

maybe a good OSS citizen would try reporting and fixing the same in MySQL, too. It is MySQL client-server protocol after all.

Is the MariaDB project committed to maintaining "bug-for-bug" bidirectional compatibility with MySQL? Even for security-related vulnerabilities?

Is the MariaDB project committed to maintaining "bug-for-bug" bidirectional compatibility with MySQL? Even for security-related vulnerabilities?

I do not speak on behalf of the project, only for myself

I believe MariaDB project is committed to protocol compatibility, and it claims that all clients that work with MySQL also work with MariaDB (unfortunately, due to missing testing, that claim breaks every now and then). As for security-related vulnerabilities, there were already cases where added security-related checks broke connectivity. MDEV-19838 broke all versions of MySQL JDBC, Python connectors, older PHP, and perhaps more, so that re-release was necessary that relaxed some of those checks. Yes, some bidirectional compatibility is required, until MariaDB takes over the world, and owns all possible client driver implementations, and tests them properly on the CI.

I do agree that SSL initiation has more info than it needs, I'm not convinced however that leaking utf8, or latin1_swe, or even cp850 to the world is a problem . We'd know by now if this was a problem. (a for Conficker, it guessed country information, only to exclude own country server's from its own virus, not to attack specific countries, and only in its first variant).

On the other side, leaking server version in plaintext (server welcome packet) helps to detect unpatched servers with specific unfixed vulnerability. and I believe it is more serious when it comes to security, but this approach does not attempt to fix that.

So my opinion is that a problem exists in the protocol(not the charset one, but leaking version of the server, and trusting server's error messages prior to verifying server's identity) but this patch does not fix that. It fixes something, which might not even be a problem, for MariaDB only, and uses up a precious bit in the extended capability flags for that.