[#MXS-1462] MaxScale erroneously connects to MySQLBackend servers via TLS1.0 rather than TLS1.2

[MXS-1462] MaxScale erroneously connects to MySQLBackend servers via TLS1.0 rather than TLS1.2 Created: 2017-10-05 Updated: 2017-11-30 Resolved: 2017-11-24
Status:	Closed
Project:	MariaDB MaxScale
Component/s:	mariadbbackend
Affects Version/s:	2.1.9
Fix Version/s:	N/A

Type:

Bug

Priority:

Major

Reporter:

Pak Chan

Assignee:

markus makela

Resolution:

Incomplete

Votes:

Labels:

None

Environment:

Ubuntu 16.04, MariaDB Galera, MariaDB 10.1.28

Attachments:

maxscale.log

Issue Links:

PartOf
includes	~~MXS-1019~~	permit using self-signed certificate ...	Closed
Relates
relates to	~~CONC-296~~	Unexpected TLSv1.0 usage when TLSv1.2...	Closed

Sprint:

2017-45, 2017-46

Description

Hi,

I'm in the process of setting up MaxScale on Ubuntu 16.04 fronting a Galera cluster where the MariaDB database nodes (also on Ubuntu 16.04) are set to use TLSv1.2. I have another Ubuntu 16.04 server, which is where I'm testing from (using a MariaDB "mysql" client, set up with the same client certificate as is used by MaxScale to connect to the Galera cluster). There is a "test" user (requiring SSL) and a "galeramon" user on the database.

According to the documentation, I can configure this in MaxScale as follows:

[dbnode1]

type=server

address=172.16.1.22

port=3306

protocol=MySQLBackend

ssl=required

ssl_version=TLSv12

ssl_cert=/etc/mysql/ssl/db-client-cert.pem

ssl_key=/etc/mysql/ssl/db-client-key.pem

ssl_ca_cert=/etc/mysql/ssl/ca-cert.pem

[dbnode2]

type=server

address=172.16.1.23

port=3306

protocol=MySQLBackend

ssl=required

ssl_version=TLSv12

ssl_cert=/etc/mysql/ssl/db-client-cert.pem

ssl_key=/etc/mysql/ssl/db-client-key.pem

ssl_ca_cert=/etc/mysql/ssl/ca-cert.pem

[Galera Monitor]

type=monitor

module=galeramon

servers=dbnode1,dbnode2

user=galeramon

passwd=galeramon

monitor_interval=1000

[Galera Service]

type=service

router=readwritesplit

servers=dbnode1,dbnode2

user=galeramon

passwd=galeramon

[MaxAdmin Service]

type=service

router=cli

[Galera Listener]

type=listener

service=Galera Service

protocol=MySQLClient

port=3306

authenticator=MySQL

ssl=required

ssl_version=TLSv12

ssl_cert=/etc/mysql/ssl/server-cert.pem

ssl_key=/etc/mysql/ssl/server-key.pem

ssl_ca_cert=/etc/mysql/ssl/ca-cert.pem

ssl_cert_verify_depth=9

[MaxAdmin Listener]

type=listener

service=MaxAdmin Service

protocol=maxscaled

socket=default

However, this never successfully connects. I ran a tcpdump packet capture on the connection, and found that the reason it was failing was that MaxScale was trying to connect using TLSv1.0 despite the specification. Changing the "ssl_version" setting in the "dbnode*" sections to "MAX" had no effect; neither did commenting out that setting altogether.

I've attached a log of a sample session.

The version of openssl and libssl1.0.0 on the server are both 1.0.2g-1ubuntu4.8, so it should support TLSv1.2. I installed MaxScale with:

curl -sS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash -s -- --mariadb-server-version=mariadb-10.1

sudo apt install maxscale

I can disable the TLS requirement for the "galeramon" user, which allows MaxScale to start up, but the moment I log into the database via MaxScale as the "test" user, the connection fails, as the following transcript (from a different server) shows:

test@dbclient01:~$ mysql -h 172.16.2.1 -u test -p

Enter password:

Welcome to the MariaDB monitor.  Commands end with ; or \g.

Your MySQL connection id is 31200

Server version: 10.0.0 2.1.9-maxscale

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> show databases;

ERROR 2006 (HY000): MySQL server has gone away

No connection. Trying to reconnect...

Connection id:    31200

Current database: *** NONE ***

ERROR 2003 (HY000): Authentication with backend failed. Session will be closed.

MySQL [(none)]>

I can connect to a database instance over TLSv1.2 from the mysql client on another machine using the mysql client with the same ("db-client-*") certificate as specified above, and I can connect to the MaxScale "Galera Listener" using the same mysql client, also over TLSv1.2 (as indicated via tcpdump packet captures), so the "ssl_version" setting is being honoured by the MaxScale listener, just not the MySQLBackend server.

Comments

Comment by markus makela [ 2017-11-08 ]

Managed to reproduce it:

2017-11-08 10:06:27   error  : (4) SSL operation failed in dcb_connect_SSL, dcb 0x7f93cc025a10 in state DCB_STATE_POLLING fd 21 return code -1. More details may follow.

2017-11-08 10:06:27   error  : (4) error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Generated certs with:

#!/bin/sh

# Creates CA key and certificate

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem -subj "/CN=certificate-authority/"

# The server key and certificate

openssl req -newkey rsa:2048 -days 365000 -nodes -keyout server-key.pem -out server-req.pem -subj "/CN=server/"

openssl rsa -in server-key.pem -out server-key.pem

openssl x509 -req -in server-req.pem -days 365000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem

# The client key and certificate

openssl req -newkey rsa:2048 -days 365000 -nodes -keyout client-key.pem -out client-req.pem -subj "/CN=client/"

openssl rsa -in client-key.pem -out client-key.pem

openssl x509 -req -in client-req.pem -days 365000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem

Configuration:

[maxscale]

threads=4

log_info=1

log_debug=1

[MySQL Monitor]

type=monitor

module=mysqlmon

servers=server1,server2,server3,server4

user=maxskysql

passwd=skysql

monitor_interval=1000

[RW Split Router]

type=service

router=readwritesplit

servers=server1,server2,server3,server4

user=maxskysql

passwd=skysql

[RW Split Listener]

type=listener

service=RW Split Router

protocol=MySQLClient

port=4006

ssl=required

ssl_key=/ssl/server-key.pem

ssl_cert=/ssl/server-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[CLI]

type=service

router=cli

[CLI Listener]

type=listener

service=CLI

protocol=maxscaled

socket=default

[server1]

type=server

address=192.168.121.230

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[server2]

type=server

address=192.168.121.202

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[server3]

type=server

address=192.168.121.173

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[server4]

type=server

address=192.168.121.253

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

Comment by markus makela [ 2017-11-08 ]

Sadly, the earlier error was due to faulty server SSL configuration. These are my results after testing with MaxScale 2.1.10 on CentOS 7 with MariaDB 10.0.32. This would point towards this being a problem with the OS SSL version. I'll continue investigations on Ubuntu.

No TLS version specified
[markusjm@localhost ssl]$ mysql -ss -umaxskysql -pskysql -h $maxscale_IP -P 4006 --ssl --ssl-ca=ca-cert.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem -e "SHOW SESSION STATUS LIKE 'ssl_version'"
Ssl_version TLSv1.2
[vagrant@maxscale ~]$ sudo maxadmin show server server1
Server 0x1549ed0 (server1)
Server: 192.168.121.230
Status: Master, Running
Protocol: MySQLBackend
Port: 3306
Server Version: 10.0.32-MariaDB-wsrep
Node Id: 1
Master Id: -1
Slave Ids: 2, 3 , 4
Repl Depth: 0
Number of connections: 1
Current no. of conns: 0
Current no. of operations: 0
SSL initialized: yes
SSL method type: MAX
SSL certificate verification depth: 9
SSL certificate: /ssl/client-cert.pem
SSL key: /ssl/client-key.pem
SSL CA certificate: /ssl/ca-cert.pem

TLSv1.2
[markusjm@localhost ssl]$ mysql -ss -umaxskysql -pskysql -h $maxscale_IP -P 4006 --ssl --ssl-ca=ca-cert.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem -e "SHOW SESSION STATUS LIKE 'ssl_version'"
Ssl_version TLSv1.2
[vagrant@maxscale ~]$ sudo maxadmin show server server1
Server 0x160b050 (server1)
Server: 192.168.121.230
Status: Master, Running
Protocol: MySQLBackend
Port: 3306
Server Version: 10.0.32-MariaDB-wsrep
Node Id: 1
Master Id: -1
Slave Ids: 2, 3 , 4
Repl Depth: 0
Number of connections: 0
Current no. of conns: 0
Current no. of operations: 0
SSL initialized: yes
SSL method type: TLSV12
SSL certificate verification depth: 9
SSL certificate: /ssl/client-cert.pem
SSL key: /ssl/client-key.pem
SSL CA certificate: /ssl/ca-cert.pem

TLSv1.1
[markusjm@localhost ssl]$ mysql -ss -umaxskysql -pskysql -h $maxscale_IP -P 4006 --ssl --ssl-ca=ca-cert.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem -e "SHOW SESSION STATUS LIKE 'ssl_version'"
Ssl_version TLSv1.1
Server 0x1514050 (server1)
Server: 192.168.121.230
Status: Master, Running
Protocol: MySQLBackend
Port: 3306
Server Version: 10.0.32-MariaDB-wsrep
Node Id: 1
Master Id: -1
Slave Ids: 2, 3 , 4
Repl Depth: 0
Number of connections: 1
Current no. of conns: 0
Current no. of operations: 0
SSL initialized: yes
SSL method type: TLSV11
SSL certificate verification depth: 9
SSL certificate: /ssl/client-cert.pem
SSL key: /ssl/client-key.pem
SSL CA certificate: /ssl/ca-cert.pem

TLSv1.0
[markusjm@localhost ssl]$ mysql -ss -umaxskysql -pskysql -h $maxscale_IP -P 4006 --ssl --ssl-ca=ca-cert.pem --ssl-cert=client-cert.pem --ssl-key=client-key.pem -e "SHOW SESSION STATUS LIKE 'ssl_version'"
Ssl_version TLSv1
[vagrant@maxscale ~]$ sudo maxadmin show server server1
Server 0x6d9050 (server1)
Server: 192.168.121.230
Status: Master, Running
Protocol: MySQLBackend
Port: 3306
Server Version: 10.0.32-MariaDB-wsrep
Node Id: 1
Master Id: -1
Slave Ids: 2, 3 , 4
Repl Depth: 0
Number of connections: 1
Current no. of conns: 0
Current no. of operations: 0
SSL initialized: yes
SSL method type: TLSV10
SSL certificate verification depth: 9
SSL certificate: /ssl/client-cert.pem
SSL key: /ssl/client-key.pem
SSL CA certificate: /ssl/ca-cert.pem

Here's the MaxScale configuration.

[maxscale]

threads=4

log_info=1

[MySQL Monitor]

type=monitor

module=mysqlmon

servers=server1,server2,server3,server4

user=maxskysql

passwd=skysql

monitor_interval=1000

[RW Split Router]

type=service

router=readwritesplit

servers=server1,server2,server3,server4

user=maxskysql

passwd=skysql

[RW Split Listener]

type=listener

service=RW Split Router

protocol=MySQLClient

port=4006

ssl=required

ssl_key=/ssl/server-key.pem

ssl_cert=/ssl/server-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[CLI]

type=service

router=cli

[CLI Listener]

type=listener

service=CLI

protocol=maxscaled

socket=default

[server1]

type=server

address=192.168.121.230

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[server2]

type=server

address=192.168.121.202

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[server3]

type=server

address=192.168.121.173

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

[server4]

type=server

address=192.168.121.253

port=3306

protocol=MySQLBackend

ssl=required

ssl_key=/ssl/client-key.pem

ssl_cert=/ssl/client-cert.pem

ssl_ca_cert=/ssl/ca-cert.pem

ssl_version=TLSv12

Comment by markus makela [ 2017-11-09 ]

PC Please try with ssl_version=TLSv11 and ssl_version=TLSv10 to see if either one of them works.

Comment by Pak Chan [ 2017-11-09 ]

It works with ssl_version=TLSv10; I couldn't find a setting to set the MariaDB server to use TLSv1.1, so I couldn't test that.

I've updated to the latest release of openssl (1.0.2g-1ubuntu4.9) on Ubuntu as that was released recently, but it hasn't made any difference to the TLSv1.2 backend issue.

Comment by markus makela [ 2017-11-09 ]

Thanks, I'll continue investigating why it doesn't work. You can define MaxScale to offer only TLSv1.1 with the ssl_version=TLSv11 parameter regardless of what the server is configured with.

Comment by markus makela [ 2017-11-17 ]

PC Just to confirm that the server also uses and recognizes the TLSv1.0, please execute SHOW STATUS LIKE 'Ssl_version' through MaxScale and with a direct connection. If MaxScale truly connects with TLSv1.0, then it should produce the following output when queried through MaxScale.

+---------------+---------+

| Variable_name | Value   |

+---------------+---------+

| Ssl_version   | TLSv1.0 |

+---------------+---------+

Comment by Pak Chan [ 2017-11-20 ]

Hi Markus,

If I set "ssl_version=MAX" in the server definitions, and set the backend MariaDB database cipher suite to TLSv1 ("ssl-cipher=TLSv1" in the "mysqld" section), the MySQL client does connect via TLSv1.2; running "show status like 'ssl_version';show status like 'ssl_cipher';" results in the following:

+---------------+---------+

| Variable_name | Value   |

+---------------+---------+

| Ssl_version   | TLSv1.2 |

+---------------+---------+

+---------------+--------------------+

| Variable_name | Value              |

+---------------+--------------------+

| Ssl_cipher    | DHE-RSA-AES256-SHA |

+---------------+--------------------+

However, the Galera Monitor is only using TLSv1.0 when connecting with TLS, and so if I change the MariaDB configuration to only offer TLSv1.2 ("ssl-cipher=TLSv1.2"), the Galera Monitor is unable to connect as MariaDB rejects the connection, so MaxScale doesn't list any servers as being available, and thus doesn't allow any connections to them. That's probably the issue.

Comment by markus makela [ 2017-11-20 ]

OK, then we have ruled out MaxScale itself as a source of problems and now we can focus on the Connector-C that MaxScale uses to connect to the server. We'll continue investigating this.

Comment by Georg Richter [ 2017-11-24 ]

Hi.

forcing the server or client to use a certain tls protocol version is not possible yet (see ~~MDEV-14101~~).

If you specify a cipher like DHE-RSA-AES256-SHA which is an extended TLSv1.0 cipher, you can't expect that the selected protocol will be TLSv1.0. If both client and server supports TLSv1.2 (or TLSv1.1), the protocol will be TLSv1.2 and the cipher in use will be the TLSv1.0 cipher DHE-RSA-AES256-SHA.

Comment by markus makela [ 2017-11-24 ]

Closing this as Incomplete since the Connector-C appears to be functioning correctly.

Comment by Pak Chan [ 2017-11-27 ]

If I configure the MariaDB backend databases offer both a TLSv1.2-only cipher and a TLSv1.0 cipher via "ssl-cipher=DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA", and monitor the network traffic, I get the following results:

Galera Monitor connects to MariaDB via TLSv1.0 using the "DHE-RSA-AES256-SHA" cipher
MaxScale connects to MariaDB via TLSv1.2 using the "DHE-RSA-AES256-GCM-SHA384" cipher for a mysql client connecting with "ssl-cipher=TLSv1.2"

Now, there's no apparent (to me) reason why Galera Monitor will choose a different TLS version from the MaxScale database connection, but this is a problem for me, given that Galera Monitor uses a relatively insecure TLS version and cipher suite.

Comment by markus makela [ 2017-11-27 ]

PCI've opened a bug report for the Connector-C about the usage of TLSv1.0 when TLSv1.2 is available: ~~CONC-296~~

Comment by Pak Chan [ 2017-11-27 ]

Thanks Markus.

Generated at Thu Feb 08 04:06:55 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.

[MXS-1462] MaxScale erroneously connects to MySQLBackend servers via TLS1.0 rather than TLS1.2 Created: 2017-10-05 Updated: 2017-11-30 Resolved: 2017-11-24