Debian 9.2

Implementing the workaround in https://jira.mariadb.org/browse/MDEV-14256, results in even more issues.

SST cannot start because it cannot properly set up the encryption for it:

Nov  2 20:03:15 -wsrep-sst-joiner: 2017/11/02 20:03:15 socat[8884] E SSL_accept(): error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Nov  2 20:03:15 -wsrep-sst-joiner: Error while getting data from donor node:  exit codes: 139 0

it also showed a deprecated warning:

Nov  2 19:23:45 -wsrep-sst-joiner: WSREP_SST: [WARNING] **** WARNING **** encrypt=3 is deprecated and will be removed in a future release (20171102 19:23:45.134)

That's weird, taking a look into /usr/bin/wsrep_sst_xtrabackup-v2:

        elif [[ $encrypt -eq 4 ]]; then

            wsrep_log_info "Using openssl based encryption with socat: with key, crt, and ca"

            verify_file_exists "$ssl_ca" "CA, certificate, and key files are required." \

                                         "Please check the 'ssl-ca' option.           "

            verify_file_exists "$ssl_cert" "CA, certificate, and key files are required." \

                                           "Please check the 'ssl-cert' option.         "

            verify_file_exists "$ssl_key" "CA, certificate, and key files are required." \

                                          "Please check the 'ssl-key' option.          "

            # Check to see that the key matches the cert

            verify_cert_matches_key $ssl_cert $ssl_key

            stagemsg+="-OpenSSL-Encrypted-4"

            if [[ "$WSREP_SST_OPT_ROLE"  == "joiner" ]]; then

                wsrep_log_info "Decrypting with CERT: $ssl_cert, KEY: $ssl_key, CA: $ssl_ca"

                tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert=${ssl_cert},key=${ssl_key},cafile=${ssl_ca},verify=1${joiner_extra}${sockopt} stdio"

            else

                wsrep_log_info "Encrypting with CERT: $ssl_cert, KEY: $ssl_key, CA: $ssl_ca"

                tcmd="socat -u stdio openssl-connect:${REMOTEIP}:${TSST_PORT},cert=${ssl_cert},key=${ssl_key},cafile=${ssl_ca},verify=1${donor_extra}${sockopt}"

            fi

OK, new way to do things. The "encrypt = 4" option is present in 10.2.9 but there is no deprecation warning, let's do it the new way then. Check the release notes. Nada. Check the changelog. Nada. Had to go to Percona's documentation to get the info. And since the documentation is missing, here's what you need to change in your configuration (just for future reference, I'm almost positive nobody else uses encrypted SST on this planet):

[sst]

encrypt=4

ssl-ca="/your/ca-cert.pem"

ssl-key="/your/node.key"

ssl-cert="/your/node.pem"

Even with those changes, the wrong ssl version error still shows up, as shown above. I will restate that the only change performed on this node was updating MariaDB. All other nodes in the cluster are exactly up to the same point as this node, except the MariaDB update. Forcing a different node to perform an SST works as expected.

I cannot reproduce error in 'routines:ssl3_get_record:wrong version number' on jessie with 10.2.10 when generating certificates like in https://github.com/AndriiNikitin/mariadb-environs/blob/master/_plugin/configure/m-all/configure_ssl.sh - maybe it is some problem with your SSL certificates? Both encrypt=3 and encrypt=4 does work in tests I tried with mentioned fix from MDEV-14256

Could you confirm exact algorithm or commands used while creating certificates or provide non-sensitive output of command like:
openssl version
openssl x509 -inform pem -in client-cert.pem -noout -text

If you wish I can share full exact commands used to generate docker image(s), so you may spot eventual difference.

Tried in debian stretch 9.2 as well - both encrypt=3 and encrypt=4 work as expected.

Revisiting this since after another server was brought to 10.2.10, that server dropped off the cluster with the wrong ssl version error.

openssl version
OpenSSL 1.1.0f 25 May 2017

openssl x509 -inform pem -in client-cert.pem -noout -text
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 37 (0x25)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = [redacted], ST = [redacted], O = [redacted], OU = [redacted]
Validity
Not Before: Sep 28 22:06:27 2017 GMT
Not After : Sep 4 22:06:27 2117 GMT
Subject: C = [redacted], ST = [redacted], L = [redacted], O = [redacted], CN = [redacted]
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[redacted]
Exponent: 65537 (0x10001)
Signature Algorithm: sha256WithRSAEncryption
[redacted]

the used Environment:Debian 9.2 is obsoleted by Debian 9.3

Does this problem exists on more recent version of 10.2 e.g. on 10.2.29 ?