Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9403

When using xtrabackup-v2 SST, socat + SSL fails on CentOS/RHEL 6

    Details

    • Sprint:
      10.1.15

      Description

      The wsrep_sst_xtrabackup-v2 script supports 3 different types of encryption. An encryption type is chosen via the encrypt option:

      • XtraBackup-based encryption (set encrypt=1 in [sst]).
      • OpenSSL encryption via socat (set encrypt=2 in [sst]).
      • Galera-compatible OpenSSL encryption via socat (set encrypt=3 in [sst]).

      Two of the encryption options listed above use SSL functionality provided by socat. These encryption options currently fail on CentOS/RHEL 6 with the following error:

      2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

      The error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error seems to be caused by using export-grade ciphers by default or using default DH parameters as mentioned here.

      Reproducing

      The configuration file that is used on both joiner and donor is:

      [mariadb-10.0]
      binlog_format=ROW
      default_storage_engine=InnoDB
      innodb_autoinc_lock_mode=2
      wsrep_provider=/usr/lib64/galera/libgalera_smm.so
      wsrep_cluster_address="gcomm://172.31.33.135,172.31.32.117,172.31.32.116"
      wsrep_sst_auth="sst:password"
      wsrep_sst_method=xtrabackup-v2

      Since there are two different encryption options that use socat, there are two primary ways to reproduce this problem. However, I've also included a couple variants of those cases that also fail.

      encrypt=3, no sockopt

      First, let's generate the keys:

      # CA
      openssl genrsa 2048 > ca-key.pem
      openssl req -new -x509 -nodes -days 365000 \
      -key ca-key.pem -out ca-cert.pem
       
      # server1
      openssl req -newkey rsa:2048 -days 365000 \
      -nodes -keyout server1-key.pem -out server1-req.pem
      openssl rsa -in server1-key.pem -out server1-key.pem
      openssl x509 -req -in server1-req.pem -days 365000 \
      -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 \
      -out server1-cert.pem

      And then configure the [sst] section for the joiner and donor:

      [sst]
      encrypt=3
      tkey=/home/ec2-user/certs/server1-key.pem
      tcert=/home/ec2-user/certs/server1-cert.pem

      After the server starts, the joiner attempts to SST, but sees errors like the following:

      160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' '
      WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.147)
      WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.148)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.153)
      WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.154)
      WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0 stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.178)
      160112 12:43:02 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
      160112 12:43:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      160112 12:43:02 [Note] WSREP: REPL Protocols: 7 (3, 2)
      160112 12:43:02 [Note] WSREP: Service thread queue flushed.
      160112 12:43:02 [Note] WSREP: Assign initial position for certification: 1, protocol version: 3
      160112 12:43:02 [Note] WSREP: Service thread queue flushed.
      160112 12:43:02 [Note] WSREP: Prepared IST receiver, listening at: tcp://172.31.32.116:4568
      160112 12:43:02 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-33-135.us-west-2.compute.internal)(SYNCED) as donor.
      160112 12:43:02 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1)
      160112 12:43:02 [Note] WSREP: Requesting state transfer: success, donor: 0
      2016/01/12 12:43:02 socat[3553] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
      160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)
      160112 12:43:02 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
      160112 12:43:02 [Note] WSREP: gcomm: terminating thread
      160112 12:43:02 [Note] WSREP: gcomm: joining thread
      160112 12:43:02 [Note] WSREP: gcomm: closing backend
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160112 12:43:02.638)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.646)
      160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '3300'  '' : 32 (Broken pipe)
      160112 12:43:02 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
      160112 12:43:02 [ERROR] WSREP: SST failed: 32 (Broken pipe)
      160112 12:43:02 [ERROR] Aborting

      And the donor sees errors like the following:

      160112 12:43:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass'
      160112 12:43:02 [Note] WSREP: sst_donor_thread signaled with 0
      160112 12:43:02 [Note] WSREP: async IST sender starting to serve tcp://172.31.32.116:4568 sending 1-1
      WSREP_SST: [INFO] Streaming with xbstream (20160112 12:43:02.803)
      WSREP_SST: [INFO] Using socat as streamer (20160112 12:43:02.805)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160112 12:43:02.809)
      WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160112 12:43:02.810)
      WSREP_SST: [INFO] Bypassing the SST for IST (20160112 12:43:02.813)
      WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} ${IST_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0; RC=( ${PIPESTATUS[@]} ) (20160112 12:43:02.815)
      2016/01/12 12:43:02 socat[2347] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160112 12:43:02.825)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160112 12:43:02.827)
      WSREP_SST: [INFO] Cleaning up temporary directories (20160112 12:43:02.829)
      160112 12:43:02 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass
      160112 12:43:02 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass: 32 (Broken pipe)
      160112 12:43:02 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0' --bypass
      160112 12:43:02 [Warning] WSREP: 0.0 (ip-172-31-33-135.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

      encrypt=3, cipher=3DES

      It looks like the sockopt option could be used to test different ciphers. I tried the following configuration file as a test:

      [sst]
      encrypt=3
      tkey=/home/ec2-user/certs/server1-key.pem
      tcert=/home/ec2-user/certs/server1-cert.pem
      sockopt=",cipher=3DES"

      But that is giving the same error.

      The joiner log:

      160115 17:56:21 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' '
      WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:21.953)
      WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:21.954)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:21.958)
      WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:21.959)
      WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 17:56:21.962)
      WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:21.980)
      160115 17:56:22 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
      160115 17:56:22 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      160115 17:56:22 [Note] WSREP: REPL Protocols: 7 (3, 2)
      160115 17:56:22 [Note] WSREP: Service thread queue flushed.
      160115 17:56:22 [Note] WSREP: Assign initial position for certification: 5, protocol version: 3
      160115 17:56:22 [Note] WSREP: Service thread queue flushed.
      160115 17:56:22 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)
               at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
      160115 17:56:22 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.
      160115 17:56:22 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5)
      160115 17:56:22 [Note] WSREP: Requesting state transfer: success, donor: 0
      2016/01/15 17:56:22 socat[4632] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 17:56:22.405)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.406)
      160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4380'  '' : 32 (Broken pipe)
      160115 17:56:22 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
      160115 17:56:22 [ERROR] WSREP: SST failed: 32 (Broken pipe)
      160115 17:56:22 [ERROR] Aborting

      The donor log:

      160115 17:56:22 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''
      160115 17:56:22 [Note] WSREP: sst_donor_thread signaled with 0
      WSREP_SST: [INFO] Streaming with xbstream (20160115 17:56:22.770)
      WSREP_SST: [INFO] Using socat as streamer (20160115 17:56:22.772)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 17:56:22.781)
      WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 17:56:22.783)
      WSREP_SST: [INFO] Using /tmp/tmp.C5BjWXUoJ6 as xtrabackup temporary directory (20160115 17:56:22.793)
      WSREP_SST: [INFO] Using /tmp/tmp.RgEy8onXWJ as innobackupex temporary directory (20160115 17:56:22.795)
      WSREP_SST: [INFO] Streaming GTID file before SST (20160115 17:56:22.797)
      WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=3DES; RC=( ${PIPESTATUS[@]} ) (20160115 17:56:22.798)
      2016/01/15 17:56:22 socat[4819] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 17:56:22.810)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 17:56:22.811)
      WSREP_SST: [INFO] Cleaning up temporary directories (20160115 17:56:22.813)
      160115 17:56:22 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
      160115 17:56:22 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)
      160115 17:56:22 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
      160115 17:56:22 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

      encrypt=3, cipher=EDH, dhparam

      I'm wondering if maybe I also need to change the DH key size somewhere to make this work. I tried to generate new DH parameters with openssl's dhparam command:

      openssl dhparam -out dhparams.pem 2048

      And then I tried the configuration file:

      [sst]
      encrypt=3
      tkey=/home/ec2-user/certs/server1-key.pem
      tcert=/home/ec2-user/certs/server1-cert.pem
      sockopt=",cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem"

      For some reason, it's saying that dhparams isn't a valid option, despite that the option is present in the socat manual.

      The joiner log:

      160115 18:13:26 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' '
      WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:26.538)
      WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:26.539)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:26.543)
      WSREP_SST: [INFO] Decrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:26.544)
      WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160115 18:13:26.547)
      WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:26.564)
      2016/01/15 18:13:26 socat[5547] E parseopts(): unknown option "dhparams"
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160115 18:13:26.568)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:26.569)
      160115 18:13:28 [Note] WSREP: (944378db, 'tcp://0.0.0.0:4567') turning message relay requesting off
      160115 18:13:36 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '5295'  '' : 32 (Broken pipe)
      160115 18:13:36 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
      160115 18:13:36 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
      160115 18:13:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      160115 18:13:36 [Note] WSREP: REPL Protocols: 7 (3, 2)
      160115 18:13:36 [ERROR] WSREP: SST failed: 32 (Broken pipe)
      160115 18:13:36 [ERROR] Aborting

      The donor log:

      160115 18:13:37 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5''
      160115 18:13:37 [Note] WSREP: sst_donor_thread signaled with 0
      WSREP_SST: [INFO] Streaming with xbstream (20160115 18:13:37.264)
      WSREP_SST: [INFO] Using socat as streamer (20160115 18:13:37.265)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20160115 18:13:37.269)
      WSREP_SST: [INFO] Encrypting with certificate /home/ec2-user/certs/server1-cert.pem, key /home/ec2-user/certs/server1-key.pem (20160115 18:13:37.270)
      WSREP_SST: [INFO] Using /tmp/tmp.j44LNEMrOc as xtrabackup temporary directory (20160115 18:13:37.280)
      WSREP_SST: [INFO] Using /tmp/tmp.jaTNqw7gtU as innobackupex temporary directory (20160115 18:13:37.282)
      WSREP_SST: [INFO] Streaming GTID file before SST (20160115 18:13:37.283)
      WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/server1-cert.pem,key=/home/ec2-user/certs/server1-key.pem,verify=0,cipher=EDH,dhparams=/home/ec2-user/certs/dhparams.pem; RC=( ${PIPESTATUS[@]} ) (20160115 18:13:37.285)
      2016/01/15 18:13:37 socat[5490] E parseopts(): unknown option "dhparams"
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 141 1 (20160115 18:13:37.288)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160115 18:13:37.290)
      WSREP_SST: [INFO] Cleaning up temporary directories (20160115 18:13:37.292)
      160115 18:13:37 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
      160115 18:13:37 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5': 32 (Broken pipe)
      160115 18:13:37 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:5'
      160115 18:13:37 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

      encrypt=2, no sockopt

      I also tried out encrypt=2, which uses SSL via socat in a slightly different way than encrypt=3. This failed in the same way as encrypt=3 on RHEL 6.

      To test this method, first I generated certificates and keys using these instructions. e.g.:

      FILENAME=xtrabackup
      openssl genrsa -out $FILENAME.key 1024
      openssl req -new -key $FILENAME.key -x509 -days 3653 -out $FILENAME.crt
      cat $FILENAME.key $FILENAME.crt >$FILENAME.pem
      chmod 600 $FILENAME.key $FILENAME.pem

      I copied the keys and certificates to both donor and joiner.

      After that, I set this in the configuration files for donor and joiner:

      [sst]
      encrypt=2
      tca=/home/ec2-user/certs/xtrabackup.crt
      tcert=/home/ec2-user/certs/xtrabackup.pem

      The SST failed with the same error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small error.

      The joiner log:

      160114 17:18:29 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' '
      WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:29.573)
      WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:29.574)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:29.578)
      WSREP_SST: [INFO] Decrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:29.579)
      WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20160114 17:18:29.582)
      WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u openssl-listen:4444,reuseaddr,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:29.601)
      160114 17:18:29 [Note] WSREP: Prepared SST request: xtrabackup-v2|172.31.32.116:4444/xtrabackup_sst//1
      160114 17:18:29 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      160114 17:18:29 [Note] WSREP: REPL Protocols: 7 (3, 2)
      160114 17:18:29 [Note] WSREP: Service thread queue flushed.
      160114 17:18:29 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
      160114 17:18:29 [Note] WSREP: Service thread queue flushed.
      160114 17:18:29 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (fbeb045d-b94f-11e5-8504-4ae80b892690): 1 (Operation not permitted)
               at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
      160114 17:18:29 [Note] WSREP: Member 1.0 (ip-172-31-32-116.us-west-2.compute.internal) requested state transfer from '*any*'. Selected 0.0 (ip-172-31-32-117.us-west-2.compute.internal)(SYNCED) as donor.
      160114 17:18:29 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
      160114 17:18:29 [Note] WSREP: Requesting state transfer: success, donor: 0
      2016/01/14 17:18:29 socat[4924] E SSL_accept(): error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20160114 17:18:29.943)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:29.945)
      160114 17:18:29 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.31.32.116' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '4672'  '' : 32 (Broken pipe)
      160114 17:18:29 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
      160114 17:18:29 [ERROR] WSREP: SST failed: 32 (Broken pipe)
      160114 17:18:29 [ERROR] Aborting

      The donor log:

      160114 17:18:30 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0''
      160114 17:18:30 [Note] WSREP: sst_donor_thread signaled with 0
      WSREP_SST: [INFO] Streaming with xbstream (20160114 17:18:30.913)
      WSREP_SST: [INFO] Using socat as streamer (20160114 17:18:30.914)
      WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20160114 17:18:30.918)
      WSREP_SST: [INFO] Encrypting with PEM /home/ec2-user/certs/xtrabackup.pem, CA: /home/ec2-user/certs/xtrabackup.crt (20160114 17:18:30.919)
      WSREP_SST: [INFO] Using /tmp/tmp.QF43wiwOR9 as xtrabackup temporary directory (20160114 17:18:30.929)
      WSREP_SST: [INFO] Using /tmp/tmp.J2GHRRhFOg as innobackupex temporary directory (20160114 17:18:30.931)
      WSREP_SST: [INFO] Streaming GTID file before SST (20160114 17:18:30.932)
      WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio openssl-connect:172.31.32.116:4444,cert=/home/ec2-user/certs/xtrabackup.pem,cafile=/home/ec2-user/certs/xtrabackup.crt; RC=( ${PIPESTATUS[@]} ) (20160114 17:18:30.934)
      2016/01/14 17:18:30 socat[4053] E SSL_connect(): error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small
      WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 0 1 (20160114 17:18:30.941)
      WSREP_SST: [ERROR] Cleanup after exit with status:32 (20160114 17:18:30.942)
      WSREP_SST: [INFO] Cleaning up temporary directories (20160114 17:18:30.944)
      160114 17:18:30 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'
      160114 17:18:30 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0': 32 (Broken pipe)
      160114 17:18:30 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '172.31.32.116:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix ''   '' --gtid 'fbeb045d-b94f-11e5-8504-4ae80b892690:0'
      160114 17:18:30 [Warning] WSREP: 0.0 (ip-172-31-32-117.us-west-2.compute.internal): State transfer to 1.0 (ip-172-31-32-116.us-west-2.compute.internal) failed: -32 (Broken pipe)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                anikitin Andrii Nikitin (Inactive)
                Reporter:
                GeoffMontee Geoff Montee
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: