Details

      Description

      Today I spent 4 hours testing this scenario again, jumping here and there after hitting the same point as before. The new thing is about the SSL certificates creation considering the new socat version which has the --verify option as 0. If you create the CA and the client certs in one server, send to all of them and then, on each server, you create the server cert with CN as the servername, it's going to work pretty well.

      #: CA certificates - Whatever method you use to generate the certificate and key files,
      #: the Common Name value used for the server and client certificates/keys must each differ
      #: from the Common Name value used for the CA certificate. Otherwise, the certificate and key
      #: files will not work for servers compiled using OpenSSL.
      openssl genrsa 2048 > ca-key.pem
      openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem
       
      #: client certs
      openssl req -newkey rsa:2048 -days 365000 -nodes -keyout client-key.pem -out client-req.pem
      openssl rsa -in client-key.pem -out client-key.pem
      openssl x509 -req -in client-req.pem -days 365000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem
      

      The below certs creation will be based on the CA created previously and sent to all servers being part of the same cluster.

      #: server certs
      openssl req -newkey rsa:2048 -days 365000 -nodes -keyout server-key.pem -out server-req.pem
      openssl rsa -in server-key.pem -out server-key.pem
      openssl x509 -req -in server-req.pem -days 365000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
      

      => https://jira.mariadb.org/browse/MDEV-15568

      It was all configured in a very simplistic and by the book way, as this is just a test of functionality, not a test of performance. SSL certificates created, a user with ALL PRIVS and REQUIRE SSL, certificates configured for client and server, for wsrep_provider_options and the sst. Boom, we confirmed after a lot of tests that, "mariabackup +sst + ssl = problem".

      In all tests the joiner crashes the same way:

      WSREP_SST: [INFO] Waiting for SST streaming to complete! (20180504 21:29:55.717)
      2018-05-04 21:29:56 139657601873664 [Note] WSREP: (e68b8351, 'ssl://0.0.0.0:4567') connection to peer e68b8351 with addr ssl://192.168.50.15:4567 timed out, no messages seen in PT3S
      2018-05-04 21:29:56 139657601873664 [Note] WSREP: (e68b8351, 'ssl://0.0.0.0:4567') turning message relay requesting off
      WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20180504 21:30:05.724)
      WSREP_SST: [ERROR] Cleanup after exit with status:2 (20180504 21:30:05.728)
      2018-05-04 21:30:05 139657564124928 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '192.168.50.15' --datadir '/var/lib/mysql/' --parent '12843' --binlog 'mariadb-bin' : 2 (No such file or directory)
      2018-05-04 21:30:05 139657564124928 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
      2018-05-04 21:30:05 139657904482496 [ERROR] WSREP: SST failed: 2 (No such file or directory)
      2018-05-04 21:30:05 139657904482496 [ERROR] Aborting
       
      2018-05-04 21:30:05 139657593480960 [Warning] WSREP: 0.0 (box04): State transfer to 1.0 (box05) failed: -22 (Invalid argument)
      2018-05-04 21:30:05 139657593480960 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.

      While in the donor:

      [root@box04 ~]# tail -f /var/lib/mysql/innobackup.backup.log
      180504 20:47:24 innobackupex: Starting the backup operation
       
      IMPORTANT: Please check that the backup run completes successfully.
      At the end of a successful backup run innobackupex
      prints "completed OK!".
       
      180504 20:47:24 Connecting to MySQL server host: localhost, user: maas, password: set, port: not set, socket: /var/lib/mysql/mysql.sock
      Failed to connect to MySQL server: Access denied for user 'maas'@'localhost' (using password: YES).
      

      The SST user configured is maas:123456 (wsrep_sst_auth=maas:123456)

      box04 [(none)]> show grants for maas@localhost\G
      *************************** 1. row ***************************
      Grants for maas@localhost: GRANT ALL PRIVILEGES ON *.* TO 'maas'@'localhost' IDENTIFIED BY PASSWORD '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9' REQUIRE SSL
      1 row in set (0.00 sec)
      

      If I try the mysql locally:

      [root@box04 ~]# mysql -umaas -p123456 -hlocalhost
      Welcome to the MariaDB monitor. Commands end with ; or \g.
      Your MariaDB connection id is 7
      Server version: 10.1.32-MariaDB MariaDB Server
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      box04 [(none)]> \s
      --------------
      mysql Ver 15.1 Distrib 10.1.32-MariaDB, for Linux (x86_64) using readline 5.1
       
      Connection id:	7
      Current database:
      Current user:	maas@localhost
      SSL:	Cipher in use is DHE-RSA-AES256-GCM-SHA384
      Current pager:	stdout
      Using outfile:	''
      Using delimiter:	;
      Server:	MariaDB
      Server version:	10.1.32-MariaDB MariaDB Server
      Protocol version:	10
      Connection:	Localhost via UNIX socket
      Server characterset:	latin1
      Db characterset:	latin1
      Client characterset:	utf8
      Conn. characterset:	utf8
      UNIX socket:	/var/lib/mysql/mysql.sock
      Uptime:	39 min 24 sec
       
      Threads: 3 Questions: 10 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 11 Queries per second avg: 0.004
      --------------
      

      But the mariabackup does not work and give me the same authentication error happening with the SST

      [root@box04 ~]# mariabackup --innobackupex --galera-info -u maas -p 123456 --host=localhost /opt
      180504 22:10:00 innobackupex: Starting the backup operation
       
      IMPORTANT: Please check that the backup run completes successfully.
      At the end of a successful backup run innobackupex
      prints "completed OK!".
       
      180504 22:10:00 Connecting to MySQL server host: localhost, user: maas, password: set, port: not set, socket: not set
      Failed to connect to MySQL server: Access denied for user 'maas'@'localhost' (using password: YES).
      

      Going further and summarizing it, what we need is the support of SSL on mariabackup.

      [vagrant@box04 ~]$ mariabackup --version
      mariabackup based on MariaDB server 10.1.32-MariaDB Linux (x86_64)
       
      [vagrant@box04 ~]$ mariabackup --help | grep -i ssl
      mariabackup based on MariaDB server 10.1.32-MariaDB Linux (x86_64)
       
      [vagrant@box04 ~]$ xtrabackup --help | grep -i ssl
      xtrabackup version 2.3.6 based on MySQL server 5.6.24 Linux (x86_64) (revision id: )
        --ssl               Enable SSL for connection (automatically enabled with
        --ssl-ca=name       CA file in PEM format (check OpenSSL docs, implies
                            --ssl).
        --ssl-capath=name   CA directory (check OpenSSL docs, implies --ssl).
        --ssl-cert=name     X509 cert in PEM format (implies --ssl).
        --ssl-cipher=name   SSL cipher to use (implies --ssl).
        --ssl-key=name      X509 key in PEM format (implies --ssl).
        --ssl-crl=name      Certificate revocation list (implies --ssl).
        --ssl-crlpath=name  Certificate revocation list path (implies --ssl).
        --ssl-verify-server-cert
      ssl                               TRUE
      ssl-ca                            /etc/my.cnf.d/certs/ca-cert.pem
      ssl-capath                        (No default value)
      ssl-cert                          /etc/my.cnf.d/certs/client-cert.pem
      ssl-cipher                        (No default value)
      ssl-key                           /etc/my.cnf.d/certs/client-key.pem
      ssl-crl                           (No default value)
      ssl-crlpath                       (No default value)
      ssl-verify-server-cert            FALSE
      

      Thanks!!

        Attachments

          Activity

            People

            • Assignee:
              wlad Vladislav Vaintroub
              Reporter:
              wagnerbianchi Wagner Bianchi
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: