|
Apparently it started here:
http://buildbot.askmonty.org/buildbot/builders/win32-debug/builds/4549
|
|
Also happens with main.openssl_1
http://buildbot.askmonty.org/buildbot/builders/winx64-debug/builds/4856/steps/test/logs/stdio
main.openssl_1 w2 [ fail ]
|
Test ended at 2017-08-15 12:21:47
|
|
CURRENT_TEST: main.openssl_1
|
mysqldump.exe: Got error: 2026: "Unknown SSL error (0x80090308)" when trying to connect
|
mysqltest: At line 208: exec of 'D:\winx64-debug\build\client\Debug\mysqldump.exe --defaults-file=D:/winx64-debug/build/mysql-test/var/2/my.cnf --defaults-group-suffix=.1 --skip-create-options --skip-comments --ssl-ca=D:/winx64-debug/build/src/mysql-test/std_data/cacert.pem --ssl-key=D:/winx64-debug/build/src/mysql-test/std_data/client-key.pem --ssl-cert=D:/winx64-debug/build/src/mysql-test/std_data/client-cert.pem test' failed, error: 2, status: 2, errno: 0
|
Output from before failure:
|
|
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
|
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
|
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
|
/*!40101 SET NAMES utf8 */;
|
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
|
/*!40103 SET TIME_ZONE='+00:00' */;
|
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
|
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
|
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
|
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
|
DROP TABLE IF EXISTS `t1`;
|
/*!40101 SET @saved_cs_client = @@character_set_client */;
|
/*!40101 SET character_set_client = utf8 */;
|
CREATE TABLE `t1` (
|
`a` int(11) DEFAULT NULL
|
);
|
/*!40101 SET character_set_client = @saved_cs_client */;
|
|
LOCK TABLES `t1` WRITE;
|
/*!40000 ALTER TABLE `t1` DISABLE KEYS */;
|
INSERT INTO `t1` VALUES (1),(2);
|
/*!40000 ALTER TABLE `t1` ENABLE KEYS */;
|
UNLOCK TABLES;
|
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
|
|
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
|
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
|
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
|
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
|
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
|
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
|
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
|
|
|
|
|
The result from queries just before the failure was:
|
< snip >
|
CREATE TABLE `t1` (
|
`a` int(11) DEFAULT NULL
|
);
|
/*!40101 SET character_set_client = @saved_cs_client */;
|
|
LOCK TABLES `t1` WRITE;
|
/*!40000 ALTER TABLE `t1` DISABLE KEYS */;
|
INSERT INTO `t1` VALUES (1),(2);
|
/*!40000 ALTER TABLE `t1` ENABLE KEYS */;
|
UNLOCK TABLES;
|
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
|
|
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
|
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
|
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
|
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
|
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
|
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
|
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
|
|
|
Also happens with main.mysql_upgrade_ssl
http://buildbot.askmonty.org/buildbot/builders/win32-debug/builds/4991/steps/test/logs/stdio
main.mysql_upgrade_ssl 'innodb' w2 [ fail ]
|
Test ended at 2017-08-15 15:46:08
|
|
CURRENT_TEST: main.mysql_upgrade_ssl
|
mysqltest: At line 10: exec of 'D:\win32-debug\build\client\Debug\mysql_upgrade.exe --defaults-file=D:/win32-debug/build/mysql-test/var/2/my.cnf --skip-verbose --skip-silent --ssl --force 2>&1' failed, error: 1, status: 1, errno: 2
|
Output from before failure:
|
Phase 1/7: Checking and upgrading mysql database
|
Processing databases
|
mysql
|
mysql.column_stats OK
|
mysql.columns_priv OK
|
mysql.db OK
|
mysql.event OK
|
mysql.func OK
|
mysql.gtid_slave_pos OK
|
mysql.help_category OK
|
mysql.help_keyword OK
|
mysql.help_relation OK
|
mysql.help_topic OK
|
mysql.host OK
|
mysql.index_stats OK
|
mysql.innodb_index_stats OK
|
mysql.innodb_table_stats OK
|
mysql.plugin OK
|
mysql.proc OK
|
mysql.procs_priv OK
|
mysql.proxies_priv OK
|
mysql.roles_mapping OK
|
mysql.servers OK
|
mysql.table_stats OK
|
mysql.tables_priv OK
|
mysql.time_zone OK
|
mysql.time_zone_leap_second OK
|
mysql.time_zone_name OK
|
mysql.time_zone_transition OK
|
mysql.time_zone_transition_type OK
|
mysql.user OK
|
Phase 2/7: Installing used storage engines... Skipped
|
Phase 3/7: Fixing views
|
D:\win32-debug\build\client\Debug\mysqlcheck.exe: Got error: 2026: Unknown SSL error (0x80090308) when trying to connect
|
FATAL ERROR: Upgrade failed
|
|
|
|
The result from queries just before the failure was:
|
#
|
# Bug#55672 mysql_upgrade dies with internal error
|
|
|
During TLS handshake the server (yassl) chooses cipher 0x0039 = DHE-RSA-AES256-SHA but is sometimes not able to decrypt the 64-byte client packet during handshake.
Instead of sending an alert yassl just drops the connection (see attached wireshark screenshot).
Running ssl_connect against MariaDB Server 10.2 Windows with OpenSSL (by specifying cipher DHE-RSA-AES256-SHA in ssl_connect.test) passed with --repeat=300.
|
|
Georg, in your wireshark log, what is send with "TCP fragment of reassembed PDU" at 420473 ?
Yassl does not drop connections. t might return handshake error, and server sends "bad handshake" error packet, and server closes the socket.
|
|
there is no MariaDB server built with openssl though. openssl is not part of OS, and openssl license is not compatible to GPLv2.
|
|
Sorry, I was wrong:
After KB Update (to fix problems with screen resolution in VirtualBox) I didn't notice that the cipher suites DHE-RSA-AES128/256-SHA disappeared over night, so the tests ran without DH algorithm and (of course) passed.
Wlad, the TCP fragment you were asking for is a "08S01 Bad Handshake" error message, sent by server, which followed immediately after the client sent certificate and encrypted packet. According to TLS 1.1 protocol (RFC 4346) in case of an error an tls alert code has to be sent. This doesn't happen, so the client is trying to interpret the server error message as part of the handshake and ends up in unknown error (unexpected token).
After doing some research I found out that Microsoft recommends to remove DHE-RSA-AES_128/256-SHA from the registry due to failing handshake errors. This obviously happened by KB update on mariadb-05 and also my local box, but for some reason not on mariadb-02 box. After removing both cipher suites from the registry of mariadb-02 all tests passed (--repeat=30).
I'm leaving this report open, since the yassl problem (not sending an alert code) still exists,
Links:
Transport Layer Security (TLS) Handshake Failing
TLS DH handshake fails when Premaster secret contains leading zeros
|
|
georg, as far as I understand, it's a yassl bug, because yassl does not send tls alert code.
If this is fixed, will Windows be able to connect? Or it'll just get a better error message?
|
|
I believe, C/C just needs to retry the full authentication after 0x80090308 .
This will pop up from time to time (I read somewhere once out of about 256 attempts), on different machines, different ciphers with schannel. Whether it is a bug in Schannel or not, but there is no need to present user with the error, ask user to disable ciphers in registry, ask user upgrade to Windows 10.
If C /C retries once more inside connector, on exactly this error, and hides this fact from C/C, this is much more user-friendly.
|
|
Someone I am helping is seeing this exact "Unknown SSL error (0x80090308)" error message. In their case, yaSSL isn't a component at all. They are seeing the problem while using MariaDB Connector/ODBC 3.0.9 on Windows (so Schannel is the client library) to connect to MariaDB 10.3.13 on RHEL 6 (so OpenSSL is the server library). Based on georg's comment above, it sounds like maybe this user's Windows system may be trying to use DHE-RSA-AES_128/256-SHA, and the handshake is failing. Is that still the current hypothesis? So they just need to update Windows and/or explicitly set ssl_cipher to a value that is not DHE-RSA-AES_128/256-SHA?
Is CONC-391 related to this? That one was closed as "Not a Bug", but it looks similar.
|
|
I would like to add that setting MARIADB_OPT_TLS_VERSION to 'TLSv1,TLSv1.1,TLSv1.2,TLSv1.3' via mysql_options() changed something here. Without that TLS option in the connection logic in HeidiSQL, I get the above mentioned "Unknown SSL error 0x80090308" when connecting to a MySQL 8.0.16 server. Note that also mysql_ssl_set() is called, and CLIENT_SSL is passed to mysql_real_connect(). Now, by setting MARIADB_OPT_TLS_VERSION, I can connect to MySQL 8.0.16, but no longer to a MySQL 5.7 server.
I have not yet tested connecting to an SSL enabled MariaDB server, sorry for that.
|
|
Adding to this report to this MDEV as the error code is the same.
When using MariaDB 10.4.8 client (mysql Ver 15.1 Distrib 10.4.8-MariaDB, for Win64 (AMD64), source revision 4c2464b87d58a43d1292e166bae6720b51f4b000) to connect to MariaDB server (mysqld Ver 10.3.14-MariaDB-log for Linux on x86_64 (MariaDB Server) using SSL, the connection is failing with error message "ERROR 2026 (HY000): Unknown SSL error (0x80090308)".
Environment details:
Client: Windows 10 Version 1809 (IP:192.168.56.1)
Server: CentOS Linux release 7.6.1810 (Core) (IP:192.168.56.104)
The SSL certificate was generated from an intermediate CA as per guide OpenSSL Certificate Authority. I can provide all the commands used to generate CA if required. Below are the commands used on the CA to generate the database server certificate:
#!/bin/bash
|
|
DESTDIR="../ca"
|
DBHOSTNAME="srv01.dbservers.internal.local"
|
|
cd ${DESTDIR}
|
openssl genrsa \
|
-out intermediate/private/${DBHOSTNAME}.key.pem 2048
|
chmod 400 intermediate/private/${DBHOSTNAME}.key.pem
|
|
openssl req -config intermediate/openssl.cnf \
|
-key intermediate/private/${DBHOSTNAME}.key.pem \
|
-new -sha256 -out intermediate/csr/${DBHOSTNAME}.csr.pem
|
|
openssl ca -config intermediate/openssl.cnf \
|
-extensions server_cert -days 375 -notext -md sha256 \
|
-in intermediate/csr/${DBHOSTNAME}.csr.pem \
|
-out intermediate/certs/${DBHOSTNAME}.cert.pem
|
chmod 444 intermediate/certs/${DBHOSTNAME}.cert.pem
|
|
Certificate details:
[penzk001@mariabackup ssl]$ cd /etc/my.cnf.d/ssl
|
[penzk001@mariabackup ssl]$ openssl x509 -noout -text -in srv01.dbservers.internal.local.cert.pem
|
Certificate:
|
Data:
|
Version: 3 (0x2)
|
Serial Number: 4096 (0x1000)
|
Signature Algorithm: sha256WithRSAEncryption
|
Issuer: C=MT, ST=Malta, O=NA, OU=NA, CN=My intermediate CA
|
Validity
|
Not Before: Oct 27 07:51:26 2019 GMT
|
Not After : Nov 5 07:51:26 2020 GMT
|
Subject: C=MT, ST=Malta, L=Santa Venera, O=NA, OU=NA, CN=srv01.dbservers.internal.local
|
Subject Public Key Info:
|
Public Key Algorithm: rsaEncryption
|
Public-Key: (2048 bit)
|
Modulus:
|
00:ba:b9:2b:1f:1b:0f:88:92:b7:48:d3:50:83:c0:
|
...
|
93:25
|
Exponent: 65537 (0x10001)
|
X509v3 extensions:
|
X509v3 Basic Constraints:
|
CA:FALSE
|
Netscape Cert Type:
|
SSL Server
|
Netscape Comment:
|
OpenSSL Generated Server Certificate
|
X509v3 Subject Key Identifier:
|
17:1D:13:1D:05:94:77:50:97:F4:57:42:1B:FB:61:A3:F0:AF:65:E0
|
X509v3 Authority Key Identifier:
|
keyid:DF:7B:4F:C1:5C:8F:94:19:0D:B4:39:2A:1C:C1:0E:A6:F5:FD:A3:89
|
DirName:/C=MT/ST=Malta/L=Santa Venera/O=NA/OU=NA/CN=My root CA
|
serial:10:00
|
|
X509v3 Key Usage: critical
|
Digital Signature, Key Encipherment
|
X509v3 Extended Key Usage:
|
TLS Web Server Authentication
|
Signature Algorithm: sha256WithRSAEncryption
|
67:71:17:d1:0c:7d:21:85:47:d7:b1:8d:aa:d7:07:47:1e:25:
|
...
|
3e:c5:68:8c:ea:ea:dc:ca
|
[penzk001@mariabackup ssl]$
|
|
Server startup options:
mysqld would have been started with the following arguments:
|
--port=3306 --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql --thread_stack=256K --sql_mode=STRICT_ALL_TABLES,ONLY_FULL_GROUP_BY --default_storage_engine=InnoDB --default_tmp_storage_engine=Aria --plugin-load=auth_pam.so --pam-use-cleartext-plugin=1 --ssl-ca=/etc/my.cnf.d/ssl/ca-chain.cert.pem --ssl-cert=/etc/my.cnf.d/ssl/srv01.dbservers.internal.local.cert.pem --ssl-key=/etc/my.cnf.d/ssl/srv01.dbservers.internal.local.key.pem --ssl_cipher=HIGH:+TLSv1.2 --key_buffer_size=16K --read_buffer_size=256K --read_rnd_buffer_size=256K --aria_pagecache_buffer_size=8M --table_open_cache=400 --sort_buffer_size=512K --myisam_sort_buffer_size=8388608 --query_cache_size=64M --query_cache_type=ON --innodb_buffer_pool_size=134217728 --innodb_strict_mode=1 --innodb_file_format=Barracuda --innodb_file_per_table=1 --innodb_flush_method=O_DIRECT --innodb_flush_log_at_trx_commit=1 --innodb_large_prefix=1 --innodb_autoinc_lock_mode=2 --innodb_log_group_home_dir=/var/lib/mysqlinnoredo --innodb_log_files_in_group=8 --innodb_log_file_size=78643200 --log_basename=mariadbsrv1 --log_bin=/var/lib/mysql-binlogs/mariadblog --max_binlog_size=10240000 --expire_logs_days=5 --binlog_format=ROW --tmpdir=/var/lib/mysql-tmp --innodb_buffer_pool_load_at_startup=ON --innodb_buffer_pool_dump_at_shutdown=ON --innodb_buffer_pool_filename=innodb_buffer_pool --max_allowed_packet=64M --net_buffer_length=8K --bind-address=0.0.0.0 --slow_query_log=1 --slow_query_log_file=mysql-slow.log --long_query_time=5 --log-queries-not-using-indexes=on --thread_handling=pool-of-threads --thread_pool_idle_timeout=60 --thread_pool_max_threads=1000 --thread_pool_size=8 --extra-port=3307 --extra-max-connections=500 --tmp_table_size=256M --max_heap_table_size=256M --innodb_undo_directory=/var/lib/mysqlundo --innodb_undo_logs=128 --innodb_undo_tablespaces=8 --slow_query_log=1 --slow_query_log_file=mysql-slow.log --long_query_time=5 --log-queries-not-using-indexes=on --lower_case_table_names=1 --server-id=1 --sync_binlog=1 --relay-log=mariadb-relay-bin --performance_schema=off --histogram_size=255 --histogram_type=SINGLE_PREC_HB --use_stat_tables=preferably --optimizer_use_condition_selectivity=4 --log_warnings=4 --log_error
|
[root@mariabackup my.cnf.d]#
|
|
MariaDB client connection attempt:
C:\temp\mariadb-10.4.8-winx64\bin>mysql --user=penzk001 --password --host=srv01.dbservers.internal.local --port=3306 --ssl-ca=c:\temp\ca-chain.cert.pem
|
Enter password: ********
|
ERROR 2026 (HY000): Unknown SSL error (0x80090308)
|
|
C:\temp\mariadb-10.4.8-winx64\bin>
|
In MariaDB server the following message is logged
2019-10-27 10:03:36 10 [Warning] Aborted connection 10 to db: 'unconnected' user: 'unauthenticated' host: '192.168.56.1' (CLOSE_CONNECTION)
|
Using wireshark on the database server, I have captured all the communication using filter (tcp port 3306), capture available in file (mariadbclt.pcapng)
MySQL client connection attempt:
C:\temp\mysql-8.0.18-winx64\bin>mysql --user=penzk001 --password --host=srv01.dbservers.internal.local --port=3306 --ssl-ca=c:\temp\ca-chain.cert.pem
|
Enter password: ********
|
Welcome to the MySQL monitor. Commands end with ; or \g.
|
...
|
mysql>
|
Using wireshark on the database server, I have captured all the communication using filter (tcp port 3306), capture available in file (mysqlclt.pcapng).
I have replicated the issue on a test environment, thus I can run additional tests if required.
|
|
Just an update, I tried different MariaDB clients (latest and just after GA for versions 10.3 & 10.2) on Windows 10 and can confirm that the issue is reproducible. Versions tested are: 10.3.7, 10.3.18, 10.2.12, 10.2.27.
|
|
kpenza, would it be possible to provide full input to your shell script. It did not work for me, because openssl.cnf was missing along with several directories.
|
|
Vladislav Vaintroub I have attached myca.tar that I used to generate the CA certificates.
Note that the tar file needs to be extracted in the root of your home directory and you need to update the dir value in both intermediateca.cnf and rootca.cnf to reflect your home directory. Just execute the scripts in sequence of their prefix.
|
|
Unfortunately, there is some missing piece of the puzzle
wlad@workpc:~$ ./01_bootintermediateca.sh
Generating RSA private key, 4096 bit long modulus
...................................................................................................................................................................................++++
...............................................................................................................++++
e is 65537 (0x010001)
Enter pass phrase for intermediate/private/intermediate.key.pem:
Verifying - Enter pass phrase for intermediate/private/intermediate.key.pem:
Enter pass phrase for intermediate/private/intermediate.key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [GB]:
State or Province Name [England]:
Locality Name []:
Organization Name [Alice Ltd]:
Organizational Unit Name []:
Common Name []:
Email Address []:
Using configuration from openssl.cnf
Can't open openssl.cnf for reading, No such file or directory
140217668604352:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('openssl.cnf','r')
140217668604352:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
chmod: cannot access 'intermediate/certs/intermediate.cert.pem': No such file or directory
cat: intermediate/certs/intermediate.cert.pem: No such file or directory
cat: certs/ca.cert.pem: No such file or directory
I created the intermediate and certs subdirectory, and maybe some other subdirectories it complained about,
but no luck
|
|
kpenza maybe you can just share the certficates generated with your script, that should be enough, right?
|
|
Vladislav Vaintroub Yes, sure. I have attached the generated certificates in tar file certificates.tar.
|
|
Thanks kpenza, those certificates were really helpful.
It appears that openssl (but not on some reason wolfssl), sends a lot of data in the server handshake, at once, and there was a bug reading large messages during the handshake in connector's code. This will be fixed with the next 10.2+
|
Screenshot from 2017-09-01 15-55-36.png
certificates.tar