Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.32, 10.3.6
-
None
Description
Today I spent 4 hours testing this scenario again, jumping here and there after hitting the same point as before. The new thing is about the SSL certificates creation considering the new socat version which has the --verify option as 0. If you create the CA and the client certs in one server, send to all of them and then, on each server, you create the server cert with CN as the servername, it's going to work pretty well.
#: CA certificates - Whatever method you use to generate the certificate and key files,
|
#: the Common Name value used for the server and client certificates/keys must each differ |
#: from the Common Name value used for the CA certificate. Otherwise, the certificate and key |
#: files will not work for servers compiled using OpenSSL. |
openssl genrsa 2048 > ca-key.pem |
openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem |
|
|
#: client certs
|
openssl req -newkey rsa:2048 -days 365000 -nodes -keyout client-key.pem -out client-req.pem |
openssl rsa -in client-key.pem -out client-key.pem
|
openssl x509 -req -in client-req.pem -days 365000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem |
The below certs creation will be based on the CA created previously and sent to all servers being part of the same cluster.
#: server certs
|
openssl req -newkey rsa:2048 -days 365000 -nodes -keyout server-key.pem -out server-req.pem |
openssl rsa -in server-key.pem -out server-key.pem
|
openssl x509 -req -in server-req.pem -days 365000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem |
=> https://jira.mariadb.org/browse/MDEV-15568
It was all configured in a very simplistic and by the book way, as this is just a test of functionality, not a test of performance. SSL certificates created, a user with ALL PRIVS and REQUIRE SSL, certificates configured for client and server, for wsrep_provider_options and the sst. Boom, we confirmed after a lot of tests that, "mariabackup +sst + ssl = problem".
In all tests the joiner crashes the same way:
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20180504 21:29:55.717) |
2018-05-04 21:29:56 139657601873664 [Note] WSREP: (e68b8351, 'ssl://0.0.0.0:4567') connection to peer e68b8351 with addr ssl://192.168.50.15:4567 timed out, no messages seen in PT3S |
2018-05-04 21:29:56 139657601873664 [Note] WSREP: (e68b8351, 'ssl://0.0.0.0:4567') turning message relay requesting off |
WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20180504 21:30:05.724) |
WSREP_SST: [ERROR] Cleanup after exit with status:2 (20180504 21:30:05.728) |
2018-05-04 21:30:05 139657564124928 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '192.168.50.15' --datadir '/var/lib/mysql/' --parent '12843' --binlog 'mariadb-bin' : 2 (No such file or directory) |
2018-05-04 21:30:05 139657564124928 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script. |
2018-05-04 21:30:05 139657904482496 [ERROR] WSREP: SST failed: 2 (No such file or directory) |
2018-05-04 21:30:05 139657904482496 [ERROR] Aborting |
|
|
2018-05-04 21:30:05 139657593480960 [Warning] WSREP: 0.0 (box04): State transfer to 1.0 (box05) failed: -22 (Invalid argument) |
2018-05-04 21:30:05 139657593480960 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort. |
While in the donor:
[root@box04 ~]# tail -f /var/lib/mysql/innobackup.backup.log |
180504 20:47:24 innobackupex: Starting the backup operation |
|
|
IMPORTANT: Please check that the backup run completes successfully.
|
At the end of a successful backup run innobackupex
|
prints "completed OK!". |
|
|
180504 20:47:24 Connecting to MySQL server host: localhost, user: maas, password: set, port: not set, socket: /var/lib/mysql/mysql.sock |
Failed to connect to MySQL server: Access denied for user 'maas'@'localhost' (using password: YES). |
The SST user configured is maas:123456 (wsrep_sst_auth=maas:123456)
box04 [(none)]> show grants for maas@localhost\G |
*************************** 1. row *************************** |
Grants for maas@localhost: GRANT ALL PRIVILEGES ON *.* TO 'maas'@'localhost' IDENTIFIED BY PASSWORD '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9' REQUIRE SSL |
1 row in set (0.00 sec) |
If I try the mysql locally:
[root@box04 ~]# mysql -umaas -p123456 -hlocalhost |
Welcome to the MariaDB monitor. Commands end with ; or \g.
|
Your MariaDB connection id is 7 |
Server version: 10.1.32-MariaDB MariaDB Server |
|
|
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. |
|
|
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. |
|
|
box04 [(none)]> \s
|
--------------
|
mysql Ver 15.1 Distrib 10.1.32-MariaDB, for Linux (x86_64) using readline 5.1 |
|
|
Connection id: 7 |
Current database:
|
Current user: maas@localhost |
SSL: Cipher in use is DHE-RSA-AES256-GCM-SHA384
|
Current pager: stdout
|
Using outfile: '' |
Using delimiter: ;
|
Server: MariaDB
|
Server version: 10.1.32-MariaDB MariaDB Server |
Protocol version: 10 |
Connection: Localhost via UNIX socket
|
Server characterset: latin1
|
Db characterset: latin1
|
Client characterset: utf8
|
Conn. characterset: utf8
|
UNIX socket: /var/lib/mysql/mysql.sock
|
Uptime: 39 min 24 sec |
|
|
Threads: 3 Questions: 10 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 11 Queries per second avg: 0.004 |
--------------
|
But the mariabackup does not work and give me the same authentication error happening with the SST
[root@box04 ~]# mariabackup --innobackupex --galera-info -u maas -p 123456 --host=localhost /opt |
180504 22:10:00 innobackupex: Starting the backup operation |
|
|
IMPORTANT: Please check that the backup run completes successfully.
|
At the end of a successful backup run innobackupex
|
prints "completed OK!". |
|
|
180504 22:10:00 Connecting to MySQL server host: localhost, user: maas, password: set, port: not set, socket: not set |
Failed to connect to MySQL server: Access denied for user 'maas'@'localhost' (using password: YES). |
Going further and summarizing it, what we need is the support of SSL on mariabackup.
[vagrant@box04 ~]$ mariabackup --version |
mariabackup based on MariaDB server 10.1.32-MariaDB Linux (x86_64) |
|
|
[vagrant@box04 ~]$ mariabackup --help | grep -i ssl |
mariabackup based on MariaDB server 10.1.32-MariaDB Linux (x86_64) |
|
|
[vagrant@box04 ~]$ xtrabackup --help | grep -i ssl |
xtrabackup version 2.3.6 based on MySQL server 5.6.24 Linux (x86_64) (revision id: ) |
--ssl Enable SSL for connection (automatically enabled with |
--ssl-ca=name CA file in PEM format (check OpenSSL docs, implies
|
--ssl).
|
--ssl-capath=name CA directory (check OpenSSL docs, implies --ssl).
|
--ssl-cert=name X509 cert in PEM format (implies --ssl).
|
--ssl-cipher=name SSL cipher to use (implies --ssl).
|
--ssl-key=name X509 key in PEM format (implies --ssl).
|
--ssl-crl=name Certificate revocation list (implies --ssl).
|
--ssl-crlpath=name Certificate revocation list path (implies --ssl).
|
--ssl-verify-server-cert
|
ssl TRUE
|
ssl-ca /etc/my.cnf.d/certs/ca-cert.pem
|
ssl-capath (No default value) |
ssl-cert /etc/my.cnf.d/certs/client-cert.pem
|
ssl-cipher (No default value) |
ssl-key /etc/my.cnf.d/certs/client-key.pem
|
ssl-crl (No default value) |
ssl-crlpath (No default value) |
ssl-verify-server-cert FALSE
|
Thanks!!