[MDEV-9403] When using xtrabackup-v2 SST, socat + SSL fails on CentOS/RHEL 6 - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.1.10, 10.0.23-galera
Fix Version/s: 10.1.15
Component/s: Galera SST, Scripts & Clients
Labels:
- contribution
- foundation
- galera
- sst
- wsrep

Sprint:
10.1.15

Description

The wsrep_sst_xtrabackup-v2 script supports 3 different types of encryption. An encryption type is chosen via the encrypt option:

XtraBackup-based encryption (set encrypt=1 in [sst]).
OpenSSL encryption via socat (set encrypt=2 in [sst]).
Galera-compatible OpenSSL encryption via socat (set encrypt=3 in [sst]).

Two of the encryption options listed above use SSL functionality provided by socat. These encryption options currently fail on CentOS/RHEL 6 with the following error:

2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

The error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error seems to be caused by using export-grade ciphers by default or using default DH parameters as mentioned here.

Reproducing

The configuration file that is used on both joiner and donor is:

[mariadb-10.0]

binlog_format=ROW

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_address="gcomm://172.31.33.135,172.31.32.117,172.31.32.116"

wsrep_sst_auth="sst:password"

wsrep_sst_method=xtrabackup-v2

Since there are two different encryption options that use socat, there are two primary ways to reproduce this problem. However, I've also included a couple variants of those cases that also fail.

encrypt=3, no sockopt

First, let's generate the keys:

# CA

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 \

-key ca-key.pem -out ca-cert.pem

# server1

openssl req -newkey rsa:2048 -days 365000 \

-nodes -keyout server1-key.pem -out server1-req.pem

openssl rsa -in server1-key.pem -out server1-key.pem

openssl x509 -req -in server1-req.pem -days 365000 \

-CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 \

-out server1-cert.pem

And then configure the [sst] section for the joiner and donor:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

After the server starts, the joiner attempts to SST, but sees errors like the following:

160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.147)

WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.148)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.153)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.154)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0 stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.178)

160112 12:43:02 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160112 12:43:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160112 12:43:02 [Note] WSREP: REPL Protocols: 7 (3, 2)

160112 12:43:02 [Note] WSREP: Service thread queue flushed.

160112 12:43:02 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3

160112 12:43:02 [Note] WSREP: Service thread queue flushed.

160112 12:43:02 [Note] WSREP: Prepared IST receiver, listening at: tcp://172.31.32.116:4568

160112 12:43:02 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-33-135.us-west-2.compute.internal)(SYNCED) as donor.

160112 12:43:02 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)

160112 12:43:02 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/12 12:43:02 socat[3553] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

160112 12:43:02 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.

160112 12:43:02 [Note] WSREP: gcomm: terminating thread

160112 12:43:02 [Note] WSREP: gcomm: joining thread

160112 12:43:02 [Note] WSREP: gcomm: closing backend

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160112 12:43:02.638)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.646)

160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' : 32 (Broken pipe)

160112 12:43:02 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160112 12:43:02 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160112 12:43:02 [ERROR] Aborting

And the donor sees errors like the following:

160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass'

160112 12:43:02 [Note] WSREP: sst_donor_thread signaled with 0

160112 12:43:02 [Note] WSREP: async IST sender starting to serve tcp://172.31.32.116:4568 sending 1-1

WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.803)

WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.805)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.809)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.810)

WSREP_SST: [INFO] Bypassing the SST for IST (20160112 12:43:02.813)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} ${IST_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.815)

2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160112 12:43:02.825)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.827)

WSREP_SST: [INFO] Cleaning up temporary directories (20160112 12:43:02.829)

160112 12:43:02 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass

160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass: 32 (Broken pipe)

160112 12:43:02 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass

160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=3, cipher=3DES

It looks like the sockopt option could be used to test different ciphers. I tried the following configuration file as a test:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

sockopt=",cipher=3DES"

But that is giving the same error.

The joiner log:

160115 17:56:21 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:21.953)

WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:21.954)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:21.958)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:21.959)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 17:56:21.962)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:21.980)

160115 17:56:22 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160115 17:56:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160115 17:56:22 [Note] WSREP: REPL Protocols: 7 (3, 2)

160115 17:56:22 [Note] WSREP: Service thread queue flushed.

160115 17:56:22 [Note] WSREP: Assign initial position for certification: 5, protocol version: 3

160115 17:56:22 [Note] WSREP: Service thread queue flushed.

160115 17:56:22 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)

         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.

160115 17:56:22 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.

160115 17:56:22 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5)

160115 17:56:22 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/15 17:56:22 socat[4632] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 17:56:22.405)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.406)

160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' : 32 (Broken pipe)

160115 17:56:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160115 17:56:22 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160115 17:56:22 [ERROR] Aborting

The donor log:

160115 17:56:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''

160115 17:56:22 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:22.770)

WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:22.772)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:22.781)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:22.783)

WSREP_SST: [INFO] Using /tmp/tmp.C5BjWXUoJ6 as xtrabackup temporary directory (20160115 17:56:22.793)

WSREP_SST: [INFO] Using /tmp/tmp.RgEy8onXWJ as innobackupex temporary directory (20160115 17:56:22.795)

WSREP_SST: [INFO] Streaming GTID file before SST (20160115 17:56:22.797)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:22.798)

2016/01/15 17:56:22 socat[4819] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 17:56:22.810)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.811)

WSREP_SST: [INFO] Cleaning up temporary directories (20160115 17:56:22.813)

160115 17:56:22 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)

160115 17:56:22 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 17:56:22 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=3, cipher=EDH, dhparam

I'm wondering if maybe I also need to change the DH key size somewhere to make this work. I tried to generate new DH parameters with openssl's dhparam command:

openssl dhparam -out dhparams.pem 2048

And then I tried the configuration file:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

sockopt=",cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem"

For some reason, it's saying that dhparams isn't a valid option, despite that the option is present in the socat manual.

The joiner log:

160115 18:13:26 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:26.538)

WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:26.539)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:26.543)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:26.544)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 18:13:26.547)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:26.564)

2016/01/15 18:13:26 socat[5547] E parseopts(): unknown option "dhparams"

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 18:13:26.568)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:26.569)

160115 18:13:28 [Note] WSREP: (944378db, 'tcp://0.0.0.0:4567') turning message relay requesting off

160115 18:13:36 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' : 32 (Broken pipe)

160115 18:13:36 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160115 18:13:36 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160115 18:13:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160115 18:13:36 [Note] WSREP: REPL Protocols: 7 (3, 2)

160115 18:13:36 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160115 18:13:36 [ERROR] Aborting

The donor log:

160115 18:13:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''

160115 18:13:37 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:37.264)

WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:37.265)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:37.269)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:37.270)

WSREP_SST: [INFO] Using /tmp/tmp.j44LNEMrOc as xtrabackup temporary directory (20160115 18:13:37.280)

WSREP_SST: [INFO] Using /tmp/tmp.jaTNqw7gtU as innobackupex temporary directory (20160115 18:13:37.282)

WSREP_SST: [INFO] Streaming GTID file before SST (20160115 18:13:37.283)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:37.285)

2016/01/15 18:13:37 socat[5490] E parseopts(): unknown option "dhparams"

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 18:13:37.288)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:37.290)

WSREP_SST: [INFO] Cleaning up temporary directories (20160115 18:13:37.292)

160115 18:13:37 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 18:13:37 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)

160115 18:13:37 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 18:13:37 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=2, no sockopt

I also tried out encrypt=2, which uses SSL via socat in a slightly different way than encrypt=3. This failed in the same way as encrypt=3 on RHEL 6.

To test this method, first I generated certificates and keys using these instructions. e.g.:

FILENAME=xtrabackup

openssl genrsa -out $FILENAME.key 1024

openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt

cat $FILENAME.key $FILENAME.crt >$FILENAME.pem

chmod 600 $FILENAME.key $FILENAME.pem

I copied the keys and certificates to both donor and joiner.

After that, I set this in the configuration files for donor and joiner:

[sst]

encrypt=2

tca=/home/ec2-user/certs/xtrabackup.crt

tcert=/home/ec2-user/certs/xtrabackup.pem

The SST failed with the same error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error.

The joiner log:

160114 17:18:29 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:29.573)

WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:29.574)

WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:29.578)

WSREP_SST: [INFO] Decrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:29.579)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160114 17:18:29.582)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:29.601)

160114 17:18:29 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160114 17:18:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160114 17:18:29 [Note] WSREP: REPL Protocols: 7 (3, 2)

160114 17:18:29 [Note] WSREP: Service thread queue flushed.

160114 17:18:29 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3

160114 17:18:29 [Note] WSREP: Service thread queue flushed.

160114 17:18:29 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)

         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.

160114 17:18:29 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.

160114 17:18:29 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)

160114 17:18:29 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/14 17:18:29 socat[4924] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160114 17:18:29.943)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:29.945)

160114 17:18:29 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' : 32 (Broken pipe)

160114 17:18:29 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160114 17:18:29 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160114 17:18:29 [ERROR] Aborting

The donor log:

160114 17:18:30 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0''

160114 17:18:30 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:30.913)

WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:30.914)

WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:30.918)

WSREP_SST: [INFO] Encrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:30.919)

WSREP_SST: [INFO] Using /tmp/tmp.QF43wiwOR9 as xtrabackup temporary directory (20160114 17:18:30.929)

WSREP_SST: [INFO] Using /tmp/tmp.J2GHRRhFOg as innobackupex temporary directory (20160114 17:18:30.931)

WSREP_SST: [INFO] Streaming GTID file before SST (20160114 17:18:30.932)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:30.934)

2016/01/14 17:18:30 socat[4053] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 0 1 (20160114 17:18:30.941)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:30.942)

WSREP_SST: [INFO] Cleaning up temporary directories (20160114 17:18:30.944)

160114 17:18:30 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'

160114 17:18:30 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0': 32 (Broken pipe)

160114 17:18:30 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'

160114 17:18:30 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Attachments

Issue Links

relates to

MDEV-10934 SST fails when SSL is enabled

Closed

links to

GitHub Pull Request #184

Activity

Ascending order - Click to sort in descending order

Geoff Montee (Inactive) added a comment - 2016-01-15 00:20

By the way, the certificates and keys used for the above were generated with the following commands:

# CA

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 \

-key ca-key.pem -out ca-cert.pem

# server1

openssl req -newkey rsa:2048 -days 365000 \

-nodes -keyout server1-key.pem -out server1-req.pem

openssl rsa -in server1-key.pem -out server1-key.pem

openssl x509 -req -in server1-req.pem -days 365000 \

-CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 \

-out server1-cert.pem

Geoff Montee (Inactive) added a comment - 2016-01-15 00:20 By the way, the certificates and keys used for the above were generated with the following commands: # CA openssl genrsa 2048 > ca-key.pem openssl req -new -x509 -nodes -days 365000 \ -key ca-key.pem -out ca-cert.pem # server1 openssl req -newkey rsa:2048 -days 365000 \ -nodes -keyout server1-key.pem -out server1-req.pem openssl rsa -in server1-key.pem -out server1-key.pem openssl x509 -req -in server1-req.pem -days 365000 \ -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 \ -out server1-cert.pem

Geoff Montee (Inactive) added a comment - 2016-01-15 00:28

I also tried out encrypt=2, which uses SSL via socat in a slightly different way than encrypt=3. This failed in the same way as encrypt=3 on RHEL 6.

To test this method, first I generated certificates and keys using these instructions. e.g.:

FILENAME=xtrabackup

openssl genrsa -out $FILENAME.key 1024

openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt

cat $FILENAME.key $FILENAME.crt >$FILENAME.pem

chmod 600 $FILENAME.key $FILENAME.pem

I copied the keys and certificates to both donor and joiner.

After that, I set this in the configuration files for donor and joiner:

[sst]

encrypt=2

tca=/home/ec2-user/certs/xtrabackup.crt

tcert=/home/ec2-user/certs/xtrabackup.pem

The SST failed with the same error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error.

The joiner log:

160114 17:18:29 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:29.573)

WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:29.574)

WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:29.578)

WSREP_SST: [INFO] Decrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:29.579)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160114 17:18:29.582)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:29.601)

160114 17:18:29 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160114 17:18:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160114 17:18:29 [Note] WSREP: REPL Protocols: 7 (3, 2)

160114 17:18:29 [Note] WSREP: Service thread queue flushed.

160114 17:18:29 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3

160114 17:18:29 [Note] WSREP: Service thread queue flushed.

160114 17:18:29 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)

         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.

160114 17:18:29 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.

160114 17:18:29 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)

160114 17:18:29 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/14 17:18:29 socat[4924] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160114 17:18:29.943)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:29.945)

160114 17:18:29 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' : 32 (Broken pipe)

160114 17:18:29 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160114 17:18:29 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160114 17:18:29 [ERROR] Aborting

The donor log:

160114 17:18:30 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0''

160114 17:18:30 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:30.913)

WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:30.914)

WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:30.918)

WSREP_SST: [INFO] Encrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:30.919)

WSREP_SST: [INFO] Using /tmp/tmp.QF43wiwOR9 as xtrabackup temporary directory (20160114 17:18:30.929)

WSREP_SST: [INFO] Using /tmp/tmp.J2GHRRhFOg as innobackupex temporary directory (20160114 17:18:30.931)

WSREP_SST: [INFO] Streaming GTID file before SST (20160114 17:18:30.932)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:30.934)

2016/01/14 17:18:30 socat[4053] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 0 1 (20160114 17:18:30.941)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:30.942)

WSREP_SST: [INFO] Cleaning up temporary directories (20160114 17:18:30.944)

160114 17:18:30 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'

160114 17:18:30 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0': 32 (Broken pipe)

160114 17:18:30 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'

160114 17:18:30 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Geoff Montee (Inactive) added a comment - 2016-01-15 00:28 I also tried out encrypt=2 , which uses SSL via socat in a slightly different way than encrypt=3 . This failed in the same way as encrypt=3 on RHEL 6. To test this method, first I generated certificates and keys using these instructions . e.g.: FILENAME=xtrabackup openssl genrsa -out $FILENAME.key 1024 openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt cat $FILENAME.key $FILENAME.crt >$FILENAME.pem chmod 600 $FILENAME.key $FILENAME.pem I copied the keys and certificates to both donor and joiner. After that, I set this in the configuration files for donor and joiner: [sst] encrypt=2 tca=/home/ec2-user/certs/xtrabackup.crt tcert=/home/ec2-user/certs/xtrabackup.pem The SST failed with the same error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error. The joiner log: 160114 17:18:29 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672' '' ' WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:29.573) WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:29.574) WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:29.578) WSREP_SST: [INFO] Decrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:29.579) WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160114 17:18:29.582) WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:29.601) 160114 17:18:29 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1 160114 17:18:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 160114 17:18:29 [Note] WSREP: REPL Protocols: 7 (3, 2) 160114 17:18:29 [Note] WSREP: Service thread queue flushed. 160114 17:18:29 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3 160114 17:18:29 [Note] WSREP: Service thread queue flushed. 160114 17:18:29 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable. 160114 17:18:29 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor. 160114 17:18:29 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0) 160114 17:18:29 [Note] WSREP: Requesting state transfer: success, donor: 0 2016/01/14 17:18:29 socat[4924] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20160114 17:18:29.943) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:29.945) 160114 17:18:29 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672' '' : 32 (Broken pipe) 160114 17:18:29 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 160114 17:18:29 [ERROR] WSREP: SST failed: 32 (Broken pipe) 160114 17:18:29 [ERROR] Aborting The donor log: 160114 17:18:30 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'' 160114 17:18:30 [Note] WSREP: sst_donor_thread signaled with 0 WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:30.913) WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:30.914) WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:30.918) WSREP_SST: [INFO] Encrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:30.919) WSREP_SST: [INFO] Using /tmp/tmp.QF43wiwOR9 as xtrabackup temporary directory (20160114 17:18:30.929) WSREP_SST: [INFO] Using /tmp/tmp.J2GHRRhFOg as innobackupex temporary directory (20160114 17:18:30.931) WSREP_SST: [INFO] Streaming GTID file before SST (20160114 17:18:30.932) WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:30.934) 2016/01/14 17:18:30 socat[4053] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 0 1 (20160114 17:18:30.941) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:30.942) WSREP_SST: [INFO] Cleaning up temporary directories (20160114 17:18:30.944) 160114 17:18:30 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' 160114 17:18:30 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0': 32 (Broken pipe) 160114 17:18:30 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' 160114 17:18:30 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Geoff Montee (Inactive) added a comment - 2016-01-15 00:36

The wsrep_sst_xtrabackup-v2 script does not seem to set socat's cipher address option at all when encrypt is set to 2 or 3. For more information about socat's address options, check the manual.

Geoff Montee (Inactive) added a comment - 2016-01-15 00:36 The wsrep_sst_xtrabackup-v2 script does not seem to set socat's cipher address option at all when encrypt is set to 2 or 3. For more information about socat's address options, check the manual .

Geoff Montee (Inactive) added a comment - 2016-01-16 01:01

It looks like the sockopt option could be used for this. I tried the following configuration file as a test:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

sockopt=",cipher=3DES"

But that is giving the same error.

The joiner log:

160115 17:56:21 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:21.953)

WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:21.954)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:21.958)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:21.959)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 17:56:21.962)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:21.980)

160115 17:56:22 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160115 17:56:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160115 17:56:22 [Note] WSREP: REPL Protocols: 7 (3, 2)

160115 17:56:22 [Note] WSREP: Service thread queue flushed.

160115 17:56:22 [Note] WSREP: Assign initial position for certification: 5, protocol version: 3

160115 17:56:22 [Note] WSREP: Service thread queue flushed.

160115 17:56:22 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)

         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.

160115 17:56:22 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.

160115 17:56:22 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5)

160115 17:56:22 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/15 17:56:22 socat[4632] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 17:56:22.405)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.406)

160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' : 32 (Broken pipe)

160115 17:56:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160115 17:56:22 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160115 17:56:22 [ERROR] Aborting

The donor log:

160115 17:56:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''

160115 17:56:22 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:22.770)

WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:22.772)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:22.781)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:22.783)

WSREP_SST: [INFO] Using /tmp/tmp.C5BjWXUoJ6 as xtrabackup temporary directory (20160115 17:56:22.793)

WSREP_SST: [INFO] Using /tmp/tmp.RgEy8onXWJ as innobackupex temporary directory (20160115 17:56:22.795)

WSREP_SST: [INFO] Streaming GTID file before SST (20160115 17:56:22.797)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:22.798)

2016/01/15 17:56:22 socat[4819] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 17:56:22.810)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.811)

WSREP_SST: [INFO] Cleaning up temporary directories (20160115 17:56:22.813)

160115 17:56:22 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)

160115 17:56:22 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 17:56:22 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Geoff Montee (Inactive) added a comment - 2016-01-16 01:01 It looks like the sockopt option could be used for this. I tried the following configuration file as a test: [sst] encrypt=3 tkey=/home/ec2-user/certs/server1-key.pem tcert=/home/ec2-user/certs/server1-cert.pem sockopt=",cipher=3DES" But that is giving the same error. The joiner log: 160115 17:56:21 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380' '' ' WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:21.953) WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:21.954) WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:21.958) WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:21.959) WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 17:56:21.962) WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:21.980) 160115 17:56:22 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1 160115 17:56:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 160115 17:56:22 [Note] WSREP: REPL Protocols: 7 (3, 2) 160115 17:56:22 [Note] WSREP: Service thread queue flushed. 160115 17:56:22 [Note] WSREP: Assign initial position for certification: 5, protocol version: 3 160115 17:56:22 [Note] WSREP: Service thread queue flushed. 160115 17:56:22 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable. 160115 17:56:22 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor. 160115 17:56:22 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5) 160115 17:56:22 [Note] WSREP: Requesting state transfer: success, donor: 0 2016/01/15 17:56:22 socat[4632] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20160115 17:56:22.405) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.406) 160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380' '' : 32 (Broken pipe) 160115 17:56:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 160115 17:56:22 [ERROR] WSREP: SST failed: 32 (Broken pipe) 160115 17:56:22 [ERROR] Aborting The donor log: 160115 17:56:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'' 160115 17:56:22 [Note] WSREP: sst_donor_thread signaled with 0 WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:22.770) WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:22.772) WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:22.781) WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:22.783) WSREP_SST: [INFO] Using /tmp/tmp.C5BjWXUoJ6 as xtrabackup temporary directory (20160115 17:56:22.793) WSREP_SST: [INFO] Using /tmp/tmp.RgEy8onXWJ as innobackupex temporary directory (20160115 17:56:22.795) WSREP_SST: [INFO] Streaming GTID file before SST (20160115 17:56:22.797) WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:22.798) 2016/01/15 17:56:22 socat[4819] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 141 1 (20160115 17:56:22.810) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.811) WSREP_SST: [INFO] Cleaning up temporary directories (20160115 17:56:22.813) 160115 17:56:22 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5' 160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe) 160115 17:56:22 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5' 160115 17:56:22 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Geoff Montee (Inactive) added a comment - 2016-01-16 01:22

I'm wondering if maybe I also need to change the DH key size somewhere to make this work. I tried to generate new DH parameters with openssl's dhparam command:

openssl dhparam -out dhparams.pem 2048

And then I tried the configuration file:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

sockopt=",cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem"

For some reason, it's saying that dhparams isn't a valid option, despite that the option is present in the socat manual.

The joiner log:

160115 18:13:26 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:26.538)

WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:26.539)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:26.543)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:26.544)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 18:13:26.547)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:26.564)

2016/01/15 18:13:26 socat[5547] E parseopts(): unknown option "dhparams"

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 18:13:26.568)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:26.569)

160115 18:13:28 [Note] WSREP: (944378db, 'tcp://0.0.0.0:4567') turning message relay requesting off

160115 18:13:36 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' : 32 (Broken pipe)

160115 18:13:36 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160115 18:13:36 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160115 18:13:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160115 18:13:36 [Note] WSREP: REPL Protocols: 7 (3, 2)

160115 18:13:36 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160115 18:13:36 [ERROR] Aborting

The donor log:

160115 18:13:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''

160115 18:13:37 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:37.264)

WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:37.265)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:37.269)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:37.270)

WSREP_SST: [INFO] Using /tmp/tmp.j44LNEMrOc as xtrabackup temporary directory (20160115 18:13:37.280)

WSREP_SST: [INFO] Using /tmp/tmp.jaTNqw7gtU as innobackupex temporary directory (20160115 18:13:37.282)

WSREP_SST: [INFO] Streaming GTID file before SST (20160115 18:13:37.283)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:37.285)

2016/01/15 18:13:37 socat[5490] E parseopts(): unknown option "dhparams"

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 18:13:37.288)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:37.290)

WSREP_SST: [INFO] Cleaning up temporary directories (20160115 18:13:37.292)

160115 18:13:37 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 18:13:37 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)

160115 18:13:37 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 18:13:37 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Geoff Montee (Inactive) added a comment - 2016-01-16 01:22 I'm wondering if maybe I also need to change the DH key size somewhere to make this work. I tried to generate new DH parameters with openssl's dhparam command : openssl dhparam -out dhparams.pem 2048 And then I tried the configuration file: [sst] encrypt=3 tkey=/home/ec2-user/certs/server1-key.pem tcert=/home/ec2-user/certs/server1-cert.pem sockopt=",cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem" For some reason, it's saying that dhparams isn't a valid option, despite that the option is present in the socat manual. The joiner log: 160115 18:13:26 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295' '' ' WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:26.538) WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:26.539) WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:26.543) WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:26.544) WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 18:13:26.547) WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:26.564) 2016/01/15 18:13:26 socat[5547] E parseopts(): unknown option "dhparams" WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20160115 18:13:26.568) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:26.569) 160115 18:13:28 [Note] WSREP: (944378db, 'tcp://0.0.0.0:4567') turning message relay requesting off 160115 18:13:36 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295' '' : 32 (Broken pipe) 160115 18:13:36 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 160115 18:13:36 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1 160115 18:13:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 160115 18:13:36 [Note] WSREP: REPL Protocols: 7 (3, 2) 160115 18:13:36 [ERROR] WSREP: SST failed: 32 (Broken pipe) 160115 18:13:36 [ERROR] Aborting The donor log: 160115 18:13:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'' 160115 18:13:37 [Note] WSREP: sst_donor_thread signaled with 0 WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:37.264) WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:37.265) WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:37.269) WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:37.270) WSREP_SST: [INFO] Using /tmp/tmp.j44LNEMrOc as xtrabackup temporary directory (20160115 18:13:37.280) WSREP_SST: [INFO] Using /tmp/tmp.jaTNqw7gtU as innobackupex temporary directory (20160115 18:13:37.282) WSREP_SST: [INFO] Streaming GTID file before SST (20160115 18:13:37.283) WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:37.285) 2016/01/15 18:13:37 socat[5490] E parseopts(): unknown option "dhparams" WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 141 1 (20160115 18:13:37.288) WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:37.290) WSREP_SST: [INFO] Cleaning up temporary directories (20160115 18:13:37.292) 160115 18:13:37 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5' 160115 18:13:37 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe) 160115 18:13:37 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5' 160115 18:13:37 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Geoff Montee (Inactive) added a comment - 2016-01-20 21:30

Adding DH parameters to the certificate fixes the issue.

e.g. generate keys:

FILENAME=xtrabackup

openssl genrsa -out $FILENAME.key 1024

openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt

cat $FILENAME.key $FILENAME.crt >$FILENAME.pem

chmod 600 $FILENAME.key $FILENAME.pem

Then generate DH parameters:

openssl dhparam -out dhparams.pem 2048

Then add them to the certificate:

cat dhparams.pem >> xtrabackup.pem

Then configure SST:

[sst]

encrypt=2

tca=/home/ec2-user/certs/xtrabackup.crt

tcert=/home/ec2-user/certs/xtrabackup.pem

Geoff Montee (Inactive) added a comment - 2016-01-20 21:30 Adding DH parameters to the certificate fixes the issue. e.g. generate keys: FILENAME=xtrabackup openssl genrsa -out $FILENAME.key 1024 openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt cat $FILENAME.key $FILENAME.crt >$FILENAME.pem chmod 600 $FILENAME.key $FILENAME.pem Then generate DH parameters: openssl dhparam -out dhparams.pem 2048 Then add them to the certificate: cat dhparams.pem >> xtrabackup.pem Then configure SST: [sst] encrypt=2 tca=/home/ec2-user/certs/xtrabackup.crt tcert=/home/ec2-user/certs/xtrabackup.pem

Martin Štefany added a comment - 2016-06-14 19:41

I did some further investigation on this, and (1) problem here is that RHEL6/RHEL7 are using outdated socat in version 1.7.2.4 which uses DH of 768 bits and doesn't even support TLSv1.1 or TLSv1.2. I have created COPR repository for RHEL7 https://copr.fedorainfracloud.org/coprs/mstefany/socat/ with more fresh socat 1.7.3.1, and tested it so no such error appears again and SST works.

Well, in my case almost. Another (2) problem is that WSREP_SST automatically passes IP address to wsrep_sst_xtrabackup-v2.sh script, so that it uses IP addresses everywhere in the script - this is an issue if you have certificates only with hostnames in CN and SubjectAltName fields - either because you don't want to or can't use them (FreeIPA Dogtag CA doesn't allow IP addresses in SAN). In such case you will get

... mysqld[531]: 2016/06/14 21:10:11 socat[5799] E certificate is valid but its commonName does not match hostname

error and SST again fails. I will submit pull request with patch for this IP address / hostname thing.

Anyway, try if COPR socat solves the issue for you (do you want build also for RHEL6?), and consider putting +1 in RH Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1021946 to get socat rebased officially for next RHEL7.3 or later release.

Martin Štefany added a comment - 2016-06-14 19:41 I did some further investigation on this, and (1) problem here is that RHEL6/RHEL7 are using outdated socat in version 1.7.2.4 which uses DH of 768 bits and doesn't even support TLSv1.1 or TLSv1.2. I have created COPR repository for RHEL7 https://copr.fedorainfracloud.org/coprs/mstefany/socat/ with more fresh socat 1.7.3.1, and tested it so no such error appears again and SST works. Well, in my case almost. Another (2) problem is that WSREP_SST automatically passes IP address to wsrep_sst_xtrabackup-v2.sh script, so that it uses IP addresses everywhere in the script - this is an issue if you have certificates only with hostnames in CN and SubjectAltName fields - either because you don't want to or can't use them (FreeIPA Dogtag CA doesn't allow IP addresses in SAN). In such case you will get ... mysqld[531]: 2016/06/14 21:10:11 socat[5799] E certificate is valid but its commonName does not match hostname error and SST again fails. I will submit pull request with patch for this IP address / hostname thing. Anyway, try if COPR socat solves the issue for you (do you want build also for RHEL6?), and consider putting +1 in RH Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1021946 to get socat rebased officially for next RHEL7.3 or later release.

Nirbhay Choubey (Inactive) added a comment - 2016-06-28 03:10

This is related to logjam attack.

openssl dhparam -out dhparams.pem 2048

cat dhparams.pem >> cert.pem

Nirbhay Choubey (Inactive) added a comment - 2016-06-28 03:10 This is related to logjam attack. openssl dhparam -out dhparams.pem 2048 cat dhparams.pem >> cert.pem

People

Assignee:: Andrii Nikitin (Inactive)

Reporter:: Geoff Montee (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2016-01-12 21:28

Updated:: 2017-11-06 09:51

Resolved:: 2016-06-28 03:10

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Reproducing

encrypt=3, no sockopt

encrypt=3, cipher=3DES

encrypt=3, cipher=EDH, dhparam

encrypt=2, no sockopt

Attachments

Issue Links

Activity

People

Dates

Git Integration