[MDEV-9403] When using xtrabackup-v2 SST, socat + SSL fails on CentOS/RHEL 6 Created: 2016-01-12  Updated: 2017-11-06  Resolved: 2016-06-28

Status: Closed
Project: MariaDB Server
Component/s: Galera SST, Scripts & Clients
Affects Version/s: 10.1.10, 10.0.23-galera
Fix Version/s: 10.1.15

Type: Bug Priority: Major
Reporter: Geoff Montee (Inactive) Assignee: Andrii Nikitin (Inactive)
Resolution: Fixed Votes: 0
Labels: contribution, foundation, galera, sst, wsrep

Issue Links:
Relates
relates to MDEV-10934 SST fails when SSL is enabled Closed
Sprint: 10.1.15

 Description   

The wsrep_sst_xtrabackup-v2 script supports 3 different types of encryption. An encryption type is chosen via the encrypt option:

  • XtraBackup-based encryption (set encrypt=1 in [sst]).
  • OpenSSL encryption via socat (set encrypt=2 in [sst]).
  • Galera-compatible OpenSSL encryption via socat (set encrypt=3 in [sst]).

Two of the encryption options listed above use SSL functionality provided by socat. These encryption options currently fail on CentOS/RHEL 6 with the following error:

2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

The error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error seems to be caused by using export-grade ciphers by default or using default DH parameters as mentioned here.

Reproducing

The configuration file that is used on both joiner and donor is:

[mariadb-10.0]
 
binlog_format=ROW
 
default_storage_engine=InnoDB
 
innodb_autoinc_lock_mode=2
 
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
 
wsrep_cluster_address="gcomm://172.31.33.135,172.31.32.117,172.31.32.116"
 
wsrep_sst_auth="sst:password"
 
wsrep_sst_method=xtrabackup-v2

Since there are two different encryption options that use socat, there are two primary ways to reproduce this problem. However, I've also included a couple variants of those cases that also fail.

encrypt=3, no sockopt

First, let's generate the keys:

# CA
 
openssl genrsa 2048 > ca-key.pem
 
openssl req -new -x509 -nodes -days 365000 \
 
-key ca-key.pem -out ca-cert.pem
 
 
 
# server1
 
openssl req -newkey rsa:2048 -days 365000 \
 
-nodes -keyout server1-key.pem -out server1-req.pem
 
openssl rsa -in server1-key.pem -out server1-key.pem
 
openssl x509 -req -in server1-req.pem -days 365000 \
 
-CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 \
 
-out server1-cert.pem

And then configure the [sst] section for the joiner and donor:

[sst]
 
encrypt=3
 
tkey=/home/ec2-user/certs/server1-key.pem
 
tcert=/home/ec2-user/certs/server1-cert.pem

After the server starts, the joiner attempts to SST, but sees errors like the following:

160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' '
 
WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.147)
 
WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.148)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.153)
 
WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.154)
 
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0 stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.178)
 
160112 12:43:02 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
 
160112 12:43:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
160112 12:43:02 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
160112 12:43:02 [Note] WSREP: Service thread queue flushed.
 
160112 12:43:02 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
 
160112 12:43:02 [Note] WSREP: Service thread queue flushed.
 
160112 12:43:02 [Note] WSREP: Prepared IST receiver, listening at: tcp://172.31.32.116:4568
 
160112 12:43:02 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-33-135.us-west-2.compute.internal)(SYNCED) as donor.
 
160112 12:43:02 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
 
160112 12:43:02 [Note] WSREP: Requesting state transfer: success, donor: 0
 
2016/01/12 12:43:02 socat[3553] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
 
160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)
 
160112 12:43:02 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
 
160112 12:43:02 [Note] WSREP: gcomm: terminating thread
 
160112 12:43:02 [Note] WSREP: gcomm: joining thread
 
160112 12:43:02 [Note] WSREP: gcomm: closing backend
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160112 12:43:02.638)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.646)
 
160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' : 32 (Broken pipe)
 
160112 12:43:02 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
 
160112 12:43:02 [ERROR] WSREP: SST failed: 32 (Broken pipe)
 
160112 12:43:02 [ERROR] Aborting

And the donor sees errors like the following:

160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass'
 
160112 12:43:02 [Note] WSREP: sst_donor_thread signaled with 0
 
160112 12:43:02 [Note] WSREP: async IST sender starting to serve tcp://172.31.32.116:4568 sending 1-1
 
WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.803)
 
WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.805)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.809)
 
WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.810)
 
WSREP_SST: [INFO] Bypassing the SST for IST (20160112 12:43:02.813)
 
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} ${IST_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.815)
 
2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160112 12:43:02.825)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.827)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20160112 12:43:02.829)
 
160112 12:43:02 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass
 
160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass: 32 (Broken pipe)
 
160112 12:43:02 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass
 
160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=3, cipher=3DES

It looks like the sockopt option could be used to test different ciphers. I tried the following configuration file as a test:

[sst]
 
encrypt=3
 
tkey=/home/ec2-user/certs/server1-key.pem
 
tcert=/home/ec2-user/certs/server1-cert.pem
 
sockopt=",cipher=3DES"

But that is giving the same error.

The joiner log:

160115 17:56:21 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' '
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:21.953)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:21.954)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:21.958)
 
WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:21.959)
 
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 17:56:21.962)
 
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:21.980)
 
160115 17:56:22 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
 
160115 17:56:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
160115 17:56:22 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
160115 17:56:22 [Note] WSREP: Service thread queue flushed.
 
160115 17:56:22 [Note] WSREP: Assign initial position for certification: 5, protocol version: 3
 
160115 17:56:22 [Note] WSREP: Service thread queue flushed.
 
160115 17:56:22 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)
 
         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
 
160115 17:56:22 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.
 
160115 17:56:22 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5)
 
160115 17:56:22 [Note] WSREP: Requesting state transfer: success, donor: 0
 
2016/01/15 17:56:22 socat[4632] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 17:56:22.405)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.406)
 
160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' : 32 (Broken pipe)
 
160115 17:56:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
 
160115 17:56:22 [ERROR] WSREP: SST failed: 32 (Broken pipe)
 
160115 17:56:22 [ERROR] Aborting

The donor log:

160115 17:56:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''
 
160115 17:56:22 [Note] WSREP: sst_donor_thread signaled with 0
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:22.770)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:22.772)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:22.781)
 
WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:22.783)
 
WSREP_SST: [INFO] Using /tmp/tmp.C5BjWXUoJ6 as xtrabackup temporary directory (20160115 17:56:22.793)
 
WSREP_SST: [INFO] Using /tmp/tmp.RgEy8onXWJ as innobackupex temporary directory (20160115 17:56:22.795)
 
WSREP_SST: [INFO] Streaming GTID file before SST (20160115 17:56:22.797)
 
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:22.798)
 
2016/01/15 17:56:22 socat[4819] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 17:56:22.810)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.811)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20160115 17:56:22.813)
 
160115 17:56:22 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)
 
160115 17:56:22 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 17:56:22 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=3, cipher=EDH, dhparam

I'm wondering if maybe I also need to change the DH key size somewhere to make this work. I tried to generate new DH parameters with openssl's dhparam command:

openssl dhparam -out dhparams.pem 2048

And then I tried the configuration file:

[sst]
 
encrypt=3
 
tkey=/home/ec2-user/certs/server1-key.pem
 
tcert=/home/ec2-user/certs/server1-cert.pem
 
sockopt=",cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem"

For some reason, it's saying that dhparams isn't a valid option, despite that the option is present in the socat manual.

The joiner log:

160115 18:13:26 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' '
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:26.538)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:26.539)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:26.543)
 
WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:26.544)
 
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 18:13:26.547)
 
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:26.564)
 
2016/01/15 18:13:26 socat[5547] E parseopts(): unknown option "dhparams"
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 18:13:26.568)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:26.569)
 
160115 18:13:28 [Note] WSREP: (944378db, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
160115 18:13:36 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' : 32 (Broken pipe)
 
160115 18:13:36 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
 
160115 18:13:36 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
 
160115 18:13:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
160115 18:13:36 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
160115 18:13:36 [ERROR] WSREP: SST failed: 32 (Broken pipe)
 
160115 18:13:36 [ERROR] Aborting

The donor log:

160115 18:13:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''
 
160115 18:13:37 [Note] WSREP: sst_donor_thread signaled with 0
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:37.264)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:37.265)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:37.269)
 
WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:37.270)
 
WSREP_SST: [INFO] Using /tmp/tmp.j44LNEMrOc as xtrabackup temporary directory (20160115 18:13:37.280)
 
WSREP_SST: [INFO] Using /tmp/tmp.jaTNqw7gtU as innobackupex temporary directory (20160115 18:13:37.282)
 
WSREP_SST: [INFO] Streaming GTID file before SST (20160115 18:13:37.283)
 
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:37.285)
 
2016/01/15 18:13:37 socat[5490] E parseopts(): unknown option "dhparams"
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 18:13:37.288)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:37.290)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20160115 18:13:37.292)
 
160115 18:13:37 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 18:13:37 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)
 
160115 18:13:37 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 18:13:37 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=2, no sockopt

I also tried out encrypt=2, which uses SSL via socat in a slightly different way than encrypt=3. This failed in the same way as encrypt=3 on RHEL 6.

To test this method, first I generated certificates and keys using these instructions. e.g.:

FILENAME=xtrabackup
 
openssl genrsa -out $FILENAME.key 1024
 
openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt
 
cat $FILENAME.key $FILENAME.crt >$FILENAME.pem
 
chmod 600 $FILENAME.key $FILENAME.pem

I copied the keys and certificates to both donor and joiner.

After that, I set this in the configuration files for donor and joiner:

[sst]
 
encrypt=2
 
tca=/home/ec2-user/certs/xtrabackup.crt
 
tcert=/home/ec2-user/certs/xtrabackup.pem

The SST failed with the same error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error.

The joiner log:

160114 17:18:29 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' '
 
WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:29.573)
 
WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:29.574)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:29.578)
 
WSREP_SST: [INFO] Decrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:29.579)
 
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160114 17:18:29.582)
 
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:29.601)
 
160114 17:18:29 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
 
160114 17:18:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
160114 17:18:29 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
160114 17:18:29 [Note] WSREP: Service thread queue flushed.
 
160114 17:18:29 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
 
160114 17:18:29 [Note] WSREP: Service thread queue flushed.
 
160114 17:18:29 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)
 
         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
 
160114 17:18:29 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.
 
160114 17:18:29 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
 
160114 17:18:29 [Note] WSREP: Requesting state transfer: success, donor: 0
 
2016/01/14 17:18:29 socat[4924] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160114 17:18:29.943)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:29.945)
 
160114 17:18:29 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' : 32 (Broken pipe)
 
160114 17:18:29 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
 
160114 17:18:29 [ERROR] WSREP: SST failed: 32 (Broken pipe)
 
160114 17:18:29 [ERROR] Aborting

The donor log:

160114 17:18:30 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0''
 
160114 17:18:30 [Note] WSREP: sst_donor_thread signaled with 0
 
WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:30.913)
 
WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:30.914)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:30.918)
 
WSREP_SST: [INFO] Encrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:30.919)
 
WSREP_SST: [INFO] Using /tmp/tmp.QF43wiwOR9 as xtrabackup temporary directory (20160114 17:18:30.929)
 
WSREP_SST: [INFO] Using /tmp/tmp.J2GHRRhFOg as innobackupex temporary directory (20160114 17:18:30.931)
 
WSREP_SST: [INFO] Streaming GTID file before SST (20160114 17:18:30.932)
 
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:30.934)
 
2016/01/14 17:18:30 socat[4053] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 0 1 (20160114 17:18:30.941)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:30.942)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20160114 17:18:30.944)
 
160114 17:18:30 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'
 
160114 17:18:30 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0': 32 (Broken pipe)
 
160114 17:18:30 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'
 
160114 17:18:30 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)



 Comments   
Comment by Geoff Montee (Inactive) [ 2016-01-15 ]

By the way, the certificates and keys used for the above were generated with the following commands:

# CA
 
openssl genrsa 2048 > ca-key.pem
 
openssl req -new -x509 -nodes -days 365000 \
 
-key ca-key.pem -out ca-cert.pem
 
 
 
# server1
 
openssl req -newkey rsa:2048 -days 365000 \
 
-nodes -keyout server1-key.pem -out server1-req.pem
 
openssl rsa -in server1-key.pem -out server1-key.pem
 
openssl x509 -req -in server1-req.pem -days 365000 \
 
-CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 \
 
-out server1-cert.pem

Comment by Geoff Montee (Inactive) [ 2016-01-15 ]

I also tried out encrypt=2, which uses SSL via socat in a slightly different way than encrypt=3. This failed in the same way as encrypt=3 on RHEL 6.

To test this method, first I generated certificates and keys using these instructions. e.g.:

FILENAME=xtrabackup
 
openssl genrsa -out $FILENAME.key 1024
 
openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt
 
cat $FILENAME.key $FILENAME.crt >$FILENAME.pem
 
chmod 600 $FILENAME.key $FILENAME.pem

I copied the keys and certificates to both donor and joiner.

After that, I set this in the configuration files for donor and joiner:

[sst]
 
encrypt=2
 
tca=/home/ec2-user/certs/xtrabackup.crt
 
tcert=/home/ec2-user/certs/xtrabackup.pem

The SST failed with the same error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error.

The joiner log:

160114 17:18:29 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' '
 
WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:29.573)
 
WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:29.574)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:29.578)
 
WSREP_SST: [INFO] Decrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:29.579)
 
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160114 17:18:29.582)
 
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:29.601)
 
160114 17:18:29 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
 
160114 17:18:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
160114 17:18:29 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
160114 17:18:29 [Note] WSREP: Service thread queue flushed.
 
160114 17:18:29 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
 
160114 17:18:29 [Note] WSREP: Service thread queue flushed.
 
160114 17:18:29 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)
 
         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
 
160114 17:18:29 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.
 
160114 17:18:29 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
 
160114 17:18:29 [Note] WSREP: Requesting state transfer: success, donor: 0
 
2016/01/14 17:18:29 socat[4924] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160114 17:18:29.943)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:29.945)
 
160114 17:18:29 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' : 32 (Broken pipe)
 
160114 17:18:29 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
 
160114 17:18:29 [ERROR] WSREP: SST failed: 32 (Broken pipe)
 
160114 17:18:29 [ERROR] Aborting

The donor log:

160114 17:18:30 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0''
 
160114 17:18:30 [Note] WSREP: sst_donor_thread signaled with 0
 
WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:30.913)
 
WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:30.914)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:30.918)
 
WSREP_SST: [INFO] Encrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:30.919)
 
WSREP_SST: [INFO] Using /tmp/tmp.QF43wiwOR9 as xtrabackup temporary directory (20160114 17:18:30.929)
 
WSREP_SST: [INFO] Using /tmp/tmp.J2GHRRhFOg as innobackupex temporary directory (20160114 17:18:30.931)
 
WSREP_SST: [INFO] Streaming GTID file before SST (20160114 17:18:30.932)
 
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:30.934)
 
2016/01/14 17:18:30 socat[4053] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 0 1 (20160114 17:18:30.941)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:30.942)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20160114 17:18:30.944)
 
160114 17:18:30 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'
 
160114 17:18:30 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0': 32 (Broken pipe)
 
160114 17:18:30 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'
 
160114 17:18:30 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Comment by Geoff Montee (Inactive) [ 2016-01-15 ]

The wsrep_sst_xtrabackup-v2 script does not seem to set socat's cipher address option at all when encrypt is set to 2 or 3. For more information about socat's address options, check the manual.

Comment by Geoff Montee (Inactive) [ 2016-01-16 ]

It looks like the sockopt option could be used for this. I tried the following configuration file as a test:

[sst]
 
encrypt=3
 
tkey=/home/ec2-user/certs/server1-key.pem
 
tcert=/home/ec2-user/certs/server1-cert.pem
 
sockopt=",cipher=3DES"

But that is giving the same error.

The joiner log:

160115 17:56:21 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' '
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:21.953)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:21.954)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:21.958)
 
WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:21.959)
 
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 17:56:21.962)
 
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:21.980)
 
160115 17:56:22 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
 
160115 17:56:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
160115 17:56:22 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
160115 17:56:22 [Note] WSREP: Service thread queue flushed.
 
160115 17:56:22 [Note] WSREP: Assign initial position for certification: 5, protocol version: 3
 
160115 17:56:22 [Note] WSREP: Service thread queue flushed.
 
160115 17:56:22 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)
 
         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
 
160115 17:56:22 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.
 
160115 17:56:22 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5)
 
160115 17:56:22 [Note] WSREP: Requesting state transfer: success, donor: 0
 
2016/01/15 17:56:22 socat[4632] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 17:56:22.405)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.406)
 
160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' : 32 (Broken pipe)
 
160115 17:56:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
 
160115 17:56:22 [ERROR] WSREP: SST failed: 32 (Broken pipe)
 
160115 17:56:22 [ERROR] Aborting

The donor log:

160115 17:56:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''
 
160115 17:56:22 [Note] WSREP: sst_donor_thread signaled with 0
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:22.770)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:22.772)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:22.781)
 
WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:22.783)
 
WSREP_SST: [INFO] Using /tmp/tmp.C5BjWXUoJ6 as xtrabackup temporary directory (20160115 17:56:22.793)
 
WSREP_SST: [INFO] Using /tmp/tmp.RgEy8onXWJ as innobackupex temporary directory (20160115 17:56:22.795)
 
WSREP_SST: [INFO] Streaming GTID file before SST (20160115 17:56:22.797)
 
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:22.798)
 
2016/01/15 17:56:22 socat[4819] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 17:56:22.810)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.811)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20160115 17:56:22.813)
 
160115 17:56:22 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)
 
160115 17:56:22 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 17:56:22 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Comment by Geoff Montee (Inactive) [ 2016-01-16 ]

I'm wondering if maybe I also need to change the DH key size somewhere to make this work. I tried to generate new DH parameters with openssl's dhparam command:

openssl dhparam -out dhparams.pem 2048

And then I tried the configuration file:

[sst]
 
encrypt=3
 
tkey=/home/ec2-user/certs/server1-key.pem
 
tcert=/home/ec2-user/certs/server1-cert.pem
 
sockopt=",cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem"

For some reason, it's saying that dhparams isn't a valid option, despite that the option is present in the socat manual.

The joiner log:

160115 18:13:26 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' '
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:26.538)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:26.539)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:26.543)
 
WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:26.544)
 
WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 18:13:26.547)
 
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:26.564)
 
2016/01/15 18:13:26 socat[5547] E parseopts(): unknown option "dhparams"
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 18:13:26.568)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:26.569)
 
160115 18:13:28 [Note] WSREP: (944378db, 'tcp://0.0.0.0:4567') turning message relay requesting off
 
160115 18:13:36 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' : 32 (Broken pipe)
 
160115 18:13:36 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
 
160115 18:13:36 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
 
160115 18:13:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
 
160115 18:13:36 [Note] WSREP: REPL Protocols: 7 (3, 2)
 
160115 18:13:36 [ERROR] WSREP: SST failed: 32 (Broken pipe)
 
160115 18:13:36 [ERROR] Aborting

The donor log:

160115 18:13:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''
 
160115 18:13:37 [Note] WSREP: sst_donor_thread signaled with 0
 
WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:37.264)
 
WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:37.265)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:37.269)
 
WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:37.270)
 
WSREP_SST: [INFO] Using /tmp/tmp.j44LNEMrOc as xtrabackup temporary directory (20160115 18:13:37.280)
 
WSREP_SST: [INFO] Using /tmp/tmp.jaTNqw7gtU as innobackupex temporary directory (20160115 18:13:37.282)
 
WSREP_SST: [INFO] Streaming GTID file before SST (20160115 18:13:37.283)
 
WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:37.285)
 
2016/01/15 18:13:37 socat[5490] E parseopts(): unknown option "dhparams"
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 18:13:37.288)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:37.290)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20160115 18:13:37.292)
 
160115 18:13:37 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 18:13:37 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)
 
160115 18:13:37 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
 
160115 18:13:37 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Comment by Geoff Montee (Inactive) [ 2016-01-20 ]

Adding DH parameters to the certificate fixes the issue.

e.g. generate keys:

FILENAME=xtrabackup
 
openssl genrsa -out $FILENAME.key 1024
 
openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt
 
cat $FILENAME.key $FILENAME.crt >$FILENAME.pem
 
chmod 600 $FILENAME.key $FILENAME.pem

Then generate DH parameters:

openssl dhparam -out dhparams.pem 2048

Then add them to the certificate:

cat dhparams.pem >> xtrabackup.pem

Then configure SST:

[sst]
 
encrypt=2
 
tca=/home/ec2-user/certs/xtrabackup.crt
 
tcert=/home/ec2-user/certs/xtrabackup.pem

Comment by Martin Štefany [ 2016-06-14 ]

I did some further investigation on this, and (1) problem here is that RHEL6/RHEL7 are using outdated socat in version 1.7.2.4 which uses DH of 768 bits and doesn't even support TLSv1.1 or TLSv1.2. I have created COPR repository for RHEL7 https://copr.fedorainfracloud.org/coprs/mstefany/socat/ with more fresh socat 1.7.3.1, and tested it so no such error appears again and SST works.

Well, in my case almost. Another (2) problem is that WSREP_SST automatically passes IP address to wsrep_sst_xtrabackup-v2.sh script, so that it uses IP addresses everywhere in the script - this is an issue if you have certificates only with hostnames in CN and SubjectAltName fields - either because you don't want to or can't use them (FreeIPA Dogtag CA doesn't allow IP addresses in SAN). In such case you will get

... mysqld[531]: 2016/06/14 21:10:11 socat[5799] E certificate is valid but its commonName does not match hostname

error and SST again fails. I will submit pull request with patch for this IP address / hostname thing.

Anyway, try if COPR socat solves the issue for you (do you want build also for RHEL6?), and consider putting +1 in RH Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1021946 to get socat rebased officially for next RHEL7.3 or later release.

Comment by Nirbhay Choubey (Inactive) [ 2016-06-28 ]

This is related to logjam attack.

openssl dhparam -out dhparams.pem 2048
 
cat dhparams.pem >> cert.pem

Generated at Thu Feb 08 07:34:24 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.