[MDEV-9403] When using xtrabackup-v2 SST, socat + SSL fails on CentOS/RHEL 6 - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.1.10, 10.0.23-galera
Fix Version/s: 10.1.15
Component/s: Galera SST, Scripts & Clients
Labels:
- contribution
- foundation
- galera
- sst
- wsrep

Sprint:
10.1.15

Description

The wsrep_sst_xtrabackup-v2 script supports 3 different types of encryption. An encryption type is chosen via the encrypt option:

XtraBackup-based encryption (set encrypt=1 in [sst]).
OpenSSL encryption via socat (set encrypt=2 in [sst]).
Galera-compatible OpenSSL encryption via socat (set encrypt=3 in [sst]).

Two of the encryption options listed above use SSL functionality provided by socat. These encryption options currently fail on CentOS/RHEL 6 with the following error:

2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

The error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error seems to be caused by using export-grade ciphers by default or using default DH parameters as mentioned here.

Reproducing

The configuration file that is used on both joiner and donor is:

[mariadb-10.0]

binlog_format=ROW

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_address="gcomm://172.31.33.135,172.31.32.117,172.31.32.116"

wsrep_sst_auth="sst:password"

wsrep_sst_method=xtrabackup-v2

Since there are two different encryption options that use socat, there are two primary ways to reproduce this problem. However, I've also included a couple variants of those cases that also fail.

encrypt=3, no sockopt

First, let's generate the keys:

# CA

openssl genrsa 2048 > ca-key.pem

openssl req -new -x509 -nodes -days 365000 \

-key ca-key.pem -out ca-cert.pem

# server1

openssl req -newkey rsa:2048 -days 365000 \

-nodes -keyout server1-key.pem -out server1-req.pem

openssl rsa -in server1-key.pem -out server1-key.pem

openssl x509 -req -in server1-req.pem -days 365000 \

-CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 \

-out server1-cert.pem

And then configure the [sst] section for the joiner and donor:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

After the server starts, the joiner attempts to SST, but sees errors like the following:

160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.147)

WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.148)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.153)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.154)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0 stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.178)

160112 12:43:02 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160112 12:43:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160112 12:43:02 [Note] WSREP: REPL Protocols: 7 (3, 2)

160112 12:43:02 [Note] WSREP: Service thread queue flushed.

160112 12:43:02 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3

160112 12:43:02 [Note] WSREP: Service thread queue flushed.

160112 12:43:02 [Note] WSREP: Prepared IST receiver, listening at: tcp://172.31.32.116:4568

160112 12:43:02 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-33-135.us-west-2.compute.internal)(SYNCED) as donor.

160112 12:43:02 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)

160112 12:43:02 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/12 12:43:02 socat[3553] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

160112 12:43:02 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.

160112 12:43:02 [Note] WSREP: gcomm: terminating thread

160112 12:43:02 [Note] WSREP: gcomm: joining thread

160112 12:43:02 [Note] WSREP: gcomm: closing backend

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160112 12:43:02.638)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.646)

160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' : 32 (Broken pipe)

160112 12:43:02 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160112 12:43:02 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160112 12:43:02 [ERROR] Aborting

And the donor sees errors like the following:

160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass'

160112 12:43:02 [Note] WSREP: sst_donor_thread signaled with 0

160112 12:43:02 [Note] WSREP: async IST sender starting to serve tcp://172.31.32.116:4568 sending 1-1

WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.803)

WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.805)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.809)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.810)

WSREP_SST: [INFO] Bypassing the SST for IST (20160112 12:43:02.813)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} ${IST_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.815)

2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160112 12:43:02.825)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.827)

WSREP_SST: [INFO] Cleaning up temporary directories (20160112 12:43:02.829)

160112 12:43:02 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass

160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass: 32 (Broken pipe)

160112 12:43:02 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass

160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=3, cipher=3DES

It looks like the sockopt option could be used to test different ciphers. I tried the following configuration file as a test:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

sockopt=",cipher=3DES"

But that is giving the same error.

The joiner log:

160115 17:56:21 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:21.953)

WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:21.954)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:21.958)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:21.959)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 17:56:21.962)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:21.980)

160115 17:56:22 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160115 17:56:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160115 17:56:22 [Note] WSREP: REPL Protocols: 7 (3, 2)

160115 17:56:22 [Note] WSREP: Service thread queue flushed.

160115 17:56:22 [Note] WSREP: Assign initial position for certification: 5, protocol version: 3

160115 17:56:22 [Note] WSREP: Service thread queue flushed.

160115 17:56:22 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)

         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.

160115 17:56:22 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.

160115 17:56:22 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5)

160115 17:56:22 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/15 17:56:22 socat[4632] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 17:56:22.405)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.406)

160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' : 32 (Broken pipe)

160115 17:56:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160115 17:56:22 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160115 17:56:22 [ERROR] Aborting

The donor log:

160115 17:56:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''

160115 17:56:22 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:22.770)

WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:22.772)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:22.781)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:22.783)

WSREP_SST: [INFO] Using /tmp/tmp.C5BjWXUoJ6 as xtrabackup temporary directory (20160115 17:56:22.793)

WSREP_SST: [INFO] Using /tmp/tmp.RgEy8onXWJ as innobackupex temporary directory (20160115 17:56:22.795)

WSREP_SST: [INFO] Streaming GTID file before SST (20160115 17:56:22.797)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:22.798)

2016/01/15 17:56:22 socat[4819] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 17:56:22.810)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.811)

WSREP_SST: [INFO] Cleaning up temporary directories (20160115 17:56:22.813)

160115 17:56:22 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)

160115 17:56:22 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 17:56:22 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=3, cipher=EDH, dhparam

I'm wondering if maybe I also need to change the DH key size somewhere to make this work. I tried to generate new DH parameters with openssl's dhparam command:

openssl dhparam -out dhparams.pem 2048

And then I tried the configuration file:

[sst]

encrypt=3

tkey=/home/ec2-user/certs/server1-key.pem

tcert=/home/ec2-user/certs/server1-cert.pem

sockopt=",cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem"

For some reason, it's saying that dhparams isn't a valid option, despite that the option is present in the socat manual.

The joiner log:

160115 18:13:26 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:26.538)

WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:26.539)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:26.543)

WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:26.544)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 18:13:26.547)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:26.564)

2016/01/15 18:13:26 socat[5547] E parseopts(): unknown option "dhparams"

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 18:13:26.568)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:26.569)

160115 18:13:28 [Note] WSREP: (944378db, 'tcp://0.0.0.0:4567') turning message relay requesting off

160115 18:13:36 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' : 32 (Broken pipe)

160115 18:13:36 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160115 18:13:36 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160115 18:13:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160115 18:13:36 [Note] WSREP: REPL Protocols: 7 (3, 2)

160115 18:13:36 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160115 18:13:36 [ERROR] Aborting

The donor log:

160115 18:13:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''

160115 18:13:37 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:37.264)

WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:37.265)

WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:37.269)

WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:37.270)

WSREP_SST: [INFO] Using /tmp/tmp.j44LNEMrOc as xtrabackup temporary directory (20160115 18:13:37.280)

WSREP_SST: [INFO] Using /tmp/tmp.jaTNqw7gtU as innobackupex temporary directory (20160115 18:13:37.282)

WSREP_SST: [INFO] Streaming GTID file before SST (20160115 18:13:37.283)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:37.285)

2016/01/15 18:13:37 socat[5490] E parseopts(): unknown option "dhparams"

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 18:13:37.288)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:37.290)

WSREP_SST: [INFO] Cleaning up temporary directories (20160115 18:13:37.292)

160115 18:13:37 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 18:13:37 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)

160115 18:13:37 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'

160115 18:13:37 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

encrypt=2, no sockopt

I also tried out encrypt=2, which uses SSL via socat in a slightly different way than encrypt=3. This failed in the same way as encrypt=3 on RHEL 6.

To test this method, first I generated certificates and keys using these instructions. e.g.:

FILENAME=xtrabackup

openssl genrsa -out $FILENAME.key 1024

openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt

cat $FILENAME.key $FILENAME.crt >$FILENAME.pem

chmod 600 $FILENAME.key $FILENAME.pem

I copied the keys and certificates to both donor and joiner.

After that, I set this in the configuration files for donor and joiner:

[sst]

encrypt=2

tca=/home/ec2-user/certs/xtrabackup.crt

tcert=/home/ec2-user/certs/xtrabackup.pem

The SST failed with the same error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error.

The joiner log:

160114 17:18:29 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' '

WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:29.573)

WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:29.574)

WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:29.578)

WSREP_SST: [INFO] Decrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:29.579)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160114 17:18:29.582)

WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:29.601)

160114 17:18:29 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1

160114 17:18:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

160114 17:18:29 [Note] WSREP: REPL Protocols: 7 (3, 2)

160114 17:18:29 [Note] WSREP: Service thread queue flushed.

160114 17:18:29 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3

160114 17:18:29 [Note] WSREP: Service thread queue flushed.

160114 17:18:29 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)

         at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.

160114 17:18:29 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.

160114 17:18:29 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)

160114 17:18:29 [Note] WSREP: Requesting state transfer: success, donor: 0

2016/01/14 17:18:29 socat[4924] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160114 17:18:29.943)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:29.945)

160114 17:18:29 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' : 32 (Broken pipe)

160114 17:18:29 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

160114 17:18:29 [ERROR] WSREP: SST failed: 32 (Broken pipe)

160114 17:18:29 [ERROR] Aborting

The donor log:

160114 17:18:30 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0''

160114 17:18:30 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:30.913)

WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:30.914)

WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:30.918)

WSREP_SST: [INFO] Encrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:30.919)

WSREP_SST: [INFO] Using /tmp/tmp.QF43wiwOR9 as xtrabackup temporary directory (20160114 17:18:30.929)

WSREP_SST: [INFO] Using /tmp/tmp.J2GHRRhFOg as innobackupex temporary directory (20160114 17:18:30.931)

WSREP_SST: [INFO] Streaming GTID file before SST (20160114 17:18:30.932)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:30.934)

2016/01/14 17:18:30 socat[4053] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 0 1 (20160114 17:18:30.941)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:30.942)

WSREP_SST: [INFO] Cleaning up temporary directories (20160114 17:18:30.944)

160114 17:18:30 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'

160114 17:18:30 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0': 32 (Broken pipe)

160114 17:18:30 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'

160114 17:18:30 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

Attachments

Issue Links

relates to

MDEV-10934 SST fails when SSL is enabled

Closed

links to

GitHub Pull Request #184

Activity

People

Assignee:: Andrii Nikitin (Inactive)

Reporter:: Geoff Montee (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2016-01-12 21:28

Updated:: 2017-11-06 09:51

Resolved:: 2016-06-28 03:10

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.