Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.4.6
Description
We're running Ubuntu 18.04 LTS with MariaDB 10.3.16 configured with SSL/TLS support and `REQUIRE SSL` on all accounts. We have a lots of Mac clients accessing our MariaDB servers and most of them are using nightly builds Sequel Pro, arguably the most popular MySQL client in the macOS world. (Nightly builds because the Sequel Pro project is struggling with release management and maintenance).
Today I attempted to upgrade one of our MariaDB servers to MariaDB 10.4.6 only to find out that Sequel Pro no longer was able to connect with SSL/TLS (only establishing an SSL/TLS session and verifying the remote peer certificate; the server is not using client cert authentication).
Unfortunately I no longer have the exact error message at hand that Sequel Pro reported but it said something about the connection failed and that the SSL error number was unknown (or zero).
Typically this means that the SSL/TLS handshake failed, and most often this is related to cipher suite selection and/or key-exchanges.
Here's the relevant SSL/TLS configuration on our MariaDB server:
[mysqld]
ssl = On
ssl-ca = /etc/mysql/xxx-mysql-ca.pem
ssl-cert = /etc/mysql/replica-XYZ.example.com.crt
ssl-key = /etc/mysql/replica-XYZ.example.com.key
We use TLS extensively in our network to ensure privacy (encryption in transit), authentication and integrity, so the SSL/TLS issues in MySQL clients are a blocker for us.
I noticed there was a mention of the SSL/TLS library in the most recent release notes:
- https://mariadb.com/kb/en/library/mariadb-1046-release-notes/
- https://mariadb.com/kb/en/library/mariadb-1046-changelog/ (more details)
I also note that you link to a page on your SSL/TLS support (kudos!) here:
It seems that you've switched to wolfSSL from yaSSL (good riddance!) in MariaDB 10.4.6. While I haven't tried earlier versions of the 10.4 series, perhaps the change to wolfSSL might be related to the issues with Sequel Pro. (That doesn't mean I'd like you to switch back to yaSSL because most likely the problem is related to the selection of cipher suites and/or keyexchange.)
FWIW, over the years we've run into many SSL/TLS related issues with both MySQL servers and MySQL clients. MySQL's use of yaSSL has been a problem (it's under developed and doesn't support modern TLS very well). On the Windows side, SChannel has a bug with respect to DH (Diffie-Hellman) key-exchanges and some cipher suites (I think this is mentioned in Georg's comment in MDEV-13492). Default installations of HeidiSQL often experience SSL/TLS handshake failures with until you replace the shipped libmysql.dll with a specific version and drop the bundled libmariadb.dll file (see e.g. https://www.heidisql.com/forum.php?t=19494 and https://github.com/HeidiSQL/HeidiSQL/issues/519).
It would be great if the SSL/TLS issues with popular MySQL clients (Sequel Pro on macOS and HeidiSQL on Windows) could be worked out so that it's possible to connect to MariaDB 10.4. (Note that I didn't test HeidiSQL with MariaDB 10.4.)
Perhaps any workarounds found for this issue can be posted here?
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Fix Version/s | 10.4 [ 22408 ] | |
| Assignee | Oleksandr Byelkin [ sanja ] |
| Component/s | Documentation [ 10903 ] |
greenman, at least #1 and #2 from above look like documentation issues, please reassign back when done.
| Assignee | Oleksandr Byelkin [ sanja ] | Ian Gilfillan [ greenman ] |
https://mariadb.com/kb/en/library/upgrading-from-mariadb-103-to-mariadb-104/ and https://mariadb.com/kb/en/library/ssltls-system-variables/#tls_version updated.
| Assignee | Ian Gilfillan [ greenman ] | Oleksandr Byelkin [ sanja ] |
| Assignee | Oleksandr Byelkin [ sanja ] | Georg Richter [ georg ] |
| Assignee | Georg Richter [ georg ] | Ian Gilfillan [ greenman ] |
greenman Why was it assigned to me? I tried to unassign it, but then it was automatically assigned to you
| Comment | [ So, you did not test HeidiSQL on Windows, but you file bug about an alleged Schannel bug? ] |
| Attachment | worksforme.png [ 59100 ] |
ZPtbrdqkIEdBg
It works for me, with HeidiSQL. The latest 10.2, and its bundled HeidiSQL(11.3), or also HeidiSQL (11.1) preinstalled in Program Files. The server and client certificates, key, and cacert come from https://github.com/MariaDB/server/blob/10.7/mysql-test/std_data
ZPtbrdqkIEdBg, "SChannel has a bug with respect to DH", maybe, but I doubt you're affected. MDEV-13492 was really an embarassing bug in Connector/C, rather than in schannel itself.
The older MariaDB 10.2 did not with newer TLS versions, due to a yassl bug MDEV-12190(YASSL isn't able to negotiate TLS version correctly), but this bug was fixed in bundled yassl, 3.5 years ago
| Labels | regression | need_feedback regression |
| Assignee | Ian Gilfillan [ greenman ] |
Thanks for looking into this.
Looking at https://jira.mariadb.org/browse/MDEV-19823?focusedCommentId=129818&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-129818, it seems that #1 is fixed.
#2 might still be an issue (whitespace still present) assuming that MariaDB still want a list without whitespace.
I don't know the status of #3 as I haven't kept up with neither Windows nor SChannel. A quick Google "windows channel dh bug" suggests that these link may contain more information about the problem: https://support.microsoft.com/en-us/topic/transport-layer-security-tls-connections-might-fail-or-timeout-when-connecting-or-attempting-a-resumption-326bd5b1-52a1-b367-8179-b154e5c01e90 or https://cdn.f5.com/product/bugtracker/ID517202.html.
Judging from your comment, I assume #4 is fixed as well.
I no longer work with MariaDB 10.4 or SQL clients so I have neither time nor interest in following up on this. Feel free to close this issue. Thanks!
I do not really know much about SequelPro though, and have no macOS access. I'll assign to someone else to verify, who might have a Mac. cvicentiu, could you verify claims that SequelPro either works or does not , with 10.2 and 10.3 server (I guess build from source with bundled YASSL), with SSL connection. The question is about older TLS versions, really.
| Assignee | Vicențiu Ciorbaru [ cvicentiu ] |
| Attachment | screenshot-1.png [ 59106 ] |
ZPtbrdqkIEdBg, the 3. was the Yassl negotiation bug, that was MDEV-12190, and 2. is about documentation . It looks like there are white spaces between enum members, but there are none, if you copy and paste, it is "TLSv1.0,TLSv1.1,TLSv1.2,TLSv1.3" , but if you read, definitely some whitespace greenman^
| Labels | need_feedback regression | regression |
| Assignee | Vicențiu Ciorbaru [ cvicentiu ] | Dmitry Shulga [ JIRAUSER47315 ] |
| Assignee | Dmitry Shulga [ JIRAUSER47315 ] | Ian Gilfillan [ greenman ] |
| Fix Version/s | N/A [ 14700 ] | |
| Fix Version/s | 10.4 [ 22408 ] |
It's unfortunate that the KB style adds what appear to be whitespaces, although they're not present. I've updated the text to make this clearer, and used a different style so that the phantom whitespaces are not present. https://mariadb.com/kb/en/ssltls-system-variables/#tls_version
| Assignee | Ian Gilfillan [ greenman ] |
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Closed [ 6 ] |
| Assignee | Ian Gilfillan [ greenman ] |
| Workflow | MariaDB v3 [ 97659 ] | MariaDB v4 [ 156375 ] |
I fired up Wireshark and noticed that Sequel Pro is attempting to negotiate TLSv1.0, which is disabled by default starting with MariaDB 10.4.6.
1. The change to supported TLS versions should really be documented on https://mariadb.com/kb/en/library/upgrading-from-mariadb-103-to-mariadb-104/
2. The docs on https://mariadb.com/kb/en/library/ssltls-system-variables/#tls_version suggests that the default value is "TLSv1.1,TLSv1.2,TLSv1.3". Furthermore it says that valid values are "TLSv1.0, TLSv1.1, TLSv1.2, TLSv1.3". If copy-pasting the valid values part, you get a spaced list which will cause MySQL to fail startup with an unhelpful message such as "[ERROR] /usr/sbin/mysqld: Error while setting value 'TLSv1.0, TLSv1.1, TLSv1.2, TLSv1.3' to 'tls_version'". The documentation should be more clear that the list mustn't contain whitespace.
3. A default install of HeidiSQL 10.2 on Windows 10 attempts to negotiate TLS v1.2, with the server selecting TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039). Unfortunately the connection fails after the Client Key Change+Change Chiper Spec messages. It seems the server aborts the connection with a fatal TLS error of "unknown message". The only thing logged by MariaDB is "[Warning] Aborted connection 47 to db: 'unconnected' user: 'unauthenticated' host: '192.168.2.2' (This connection closed normally without authentication)".
4. By removing libmariadb.dll from C:\Program files\HeidiSQL the client will fall back to using the bundled libmysql 5.6.6.0. Now the client attempts to negotiate TLS 1.0 with the server selecting the cipher TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039). Unfortunately the connection fails in a similar manner as when relying on libmariadb.dll. With MariaDB 10.3 installations I've had success connecting from HeidiSQL with libmariadb.dll removed and using a libmysql.dll version 5.7.17.