The issue lies with socat.
Using socat from jessie (Debian) doesn't work and always results in 100% CPU usage with processes that need to be killed with a -9 signal in order to terminate (100% reproducible).
Using socat from jessie-backports results in a hostname lookup failure (joiner does not have a DNS A record pointed to it, but IS declared in /etc/hosts).
Editing /usr/bin/wsrep_sst_xtrabackup-v2:
Line 235:
tcmd="socat -u stdio openssl-connect:${REMOTEHOST}:${TSST_PORT},cert=${tpem},key=${tkey},cafile=${tcert}${sockopt}"
Change to:
tcmd="socat -u stdio openssl-connect:${REMOTEIP}:${TSST_PORT},cert=${tpem},key=${tkey},cafile=${tcert}${sockopt}"
This gets the SST progress further. Since the certificates where generated with a hostname as their CN, their verification fails and SST stops again.
Generating a new set of certificates with IPs as CNs (but leaving the old ones still used by wsrep, just not for SST), and setting those under a [sst] segment (keeping the old tca, since the new set is also signed by the same CA) allows the SST to complete, fully encrypted 
Edited to add:
using encrypt = 3 under [sst]
Thank you for your analysis and confirmation.
Just for reference - 'Broken pipe' usually means that some of chained command was aborted (e.g. by kill) . Especially in sst - if some of parties meets any problem - communication will be closed and thus 'Broken pipe' is very possible. So, that may mean that real error message might be on the other node or not included into provided logs.
I could reproduce similar issue with encrypt=1 (without SSL-certificated nodes) in 10.1.18 with message on joiner:
2017-10-05 13:41:13 140227855251200 [Note] WSREP: (150a3819, 'tcp://0.0.0.0:4569') turning message relay requesting off
|
xbcrypt:xb_crypt_read_chunk: wrong chunk magic at offset 0x0.
|
2017/10/05 13:41:21 socat[11977] E write(1, 0x55b852ddb5d0, 8192): Broken pipe
|
Just the same command succeeds in 10.1.28 , so while problem really exists in previous version, there is nothing to fix at the moment.
Below is script which downloads and unpacks corresponding tar package, generates templates and manages three local nodes:
set -e |
ver=${1:-10.1.28}
|
|
|
# just use current directory if called from framework
|
if [ ! -f common.sh ] ; then |
[ -d mariadb-environs ] || git clone http://github.com/AndriiNikitin/mariadb-environs |
cd mariadb-environs |
./get_plugin.sh galera |
fi
|
|
|
function onExit { |
[ "$passed" == yes ] && echo TEST PASSED && exit |
cluster1/tail_log.sh 100 |
# uncomment if you wish docker build hang on failire (to attach to container and troubleshoot)
|
# sleep 10000
|
}
|
trap onExit EXIT |
|
|
_template/plant_cluster.sh cluster1 |
echo m1 > cluster1/nodes.lst |
echo m2 >> cluster1/nodes.lst |
echo m3 >> cluster1/nodes.lst |
cluster1/replant.sh ${ver} |
|
|
./build_or_download.sh m1 |
|
|
cluster1/gen_cnf.sh general_log=1 |
cluster1/install_db.sh |
cluster1/configure_ssl.sh |
cluster1/galera_setup_acl.sh |
cluster1/galera_start_new.sh $WSREP_EXTRA_OPT |
|
|
sleep 5 |
cluster1/galera_cluster_size.sh |
cluster1/sql.sh 'show variables like "wsrep_sst_method"' |
|
|
grep -A10 -B10 -i "\[ERROR\]" m1*/dt/error.log || : |
|
|
cluster_size=$(m1*/sql.sh 'show status like "wsrep_cluster_size"') |
[[ "${cluster_size}" =~ 3 ]] || exit 1 |
|
|
cluster1/sql.sh status | grep 'SSL: Cipher in use' && passed=yes |
To force sst xtrabackup-v2 with encryption I paste command below into shell (use both encrypt=1 and encrypt=3):
export SST_OPT="
|
[sst]
|
encrypt=1 |
encrypt-algo=AES128
|
encrypt-key=A1EDC73815467C083B0869508406637E
|
tca=$(pwd)/_depot/ca.pem
|
tcert=$(pwd)/_depot/ca.pem
|
tkey=$(pwd)/_depot/ca-key.pem
|
"
|
|
|
WSREP_EXTRA_OPT="wsrep_sst_method=xtrabackup-v2 $SST_OPT" bash example_ssl.sh 10.1.28 |
To make it work in 10.1.28 I had to address several bugs and patch related scripts in unpacked tar image:
$ diff 10.1.28.orig/bin 10.1.28.fixed/bin |
diff 10.1.28.orig/bin/mysqld_safe 10.1.28.fixed/bin/mysqld_safe |
249c249
|
< local wr_logfile=$(mktemp wsrep_recovery.XXXXXX) |
---
|
> local wr_logfile=$(mktemp -t wsrep_recovery.XXXXXX) |
271c271
|
< eval_log_error "$mysqld_cmd --wsrep_recover $wr_options > $wr_logfile" |
---
|
> eval "$mysqld_cmd --wsrep_recover $wr_options 2> $wr_logfile" |
diff 10.1.28.orig/bin/wsrep_sst_common 10.1.28.fixed/bin/wsrep_sst_common |
48c48
|
< readonly WSREP_SST_OPT_PORT=$(echo $WSREP_SST_OPT_ADDR | \ |
---
|
> WSREP_SST_OPT_PORT=$(echo $WSREP_SST_OPT_ADDR | \ |
89c89
|
< readonly WSREP_SST_OPT_PORT="$2" |
---
|
> WSREP_SST_OPT_PORT="$2" |
diff 10.1.28.orig/bin/wsrep_sst_mysqldump 10.1.28.fixed/bin/wsrep_sst_mysqldump |
120c120
|
< MYSQL="$MYSQL_CLIENT --defaults-extra-file=$WSREP_SST_OPT_CONF "\ |
---
|
> MYSQL="$MYSQL_CLIENT $WSREP_SST_OPT_CONF "\ |
diff 10.1.28.orig/bin/wsrep_sst_xtrabackup-v2 10.1.28.fixed/bin/wsrep_sst_xtrabackup-v2 |
361,362c361
|
< tcmd="socat -u openssl-listen:${TSST_PORT},reuseaddr,cert=${tpem},key=${tkey},verify=0${sockopt} stdio" |
< else |
---
|
> else |
457c456
|
< cpat=$(parse_cnf sst cpat '.*\.pem$\|.*init\.ok$\|.*galera\.cache$\|.*sst_in_progress$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$') |
---
|
> cpat=$(parse_cnf sst cpat '.*\.pem$\|.*init\.ok$\|.*galera\.cache$\|.*sst_in_progress$\|.*\.sst$\|.*gvwstate\.dat$\|.*grastate\.dat$\|.*\.err$\|.*\.log$\|.*RPM_UPGRADE_MARKER$\|.*RPM_UPGRADE_HISTORY$\|.*innobackup*log$') |
870c869
|
< INNOMOVE="${INNOBACKUPEX_BIN} --defaults-file=${WSREP_SST_OPT_CONF} $disver $impts --datadir=${DATA} --move-back --force-non-empty-directories \${DATA} 2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-move " |
---
|
> INNOMOVE="${INNOBACKUPEX_BIN} ${WSREP_SST_OPT_CONF} $disver $impts --datadir=${DATA} --move-back --force-non-empty-directories \${DATA} 2>&1 | logger -p daemon.err -t ${ssystag}innobackupex-move " |
876c875
|
< INNOMOVE="${INNOBACKUPEX_BIN} --defaults-file=${WSREP_SST_OPT_CONF} --defaults-group=mysqld${WSREP_SST_OPT_CONF_SUFFIX} $disver $impts --datadir=${DATA} --move-back --force-non-empty-directories \${DATA} &>\${DATA}/innobackup.move.log" |
---
|
> INNOMOVE="${INNOBACKUPEX_BIN} ${WSREP_SST_OPT_CONF} --defaults-group=mysqld${WSREP_SST_OPT_CONF_SUFFIX} $disver $impts --datadir=${DATA} --move-back --force-non-empty-directories \${DATA} &>\${DATA}/innobackup.move.log" |
So, since original problem looks solved - I will close this call with resolution 'fixed'.
It is hard work to find exact version/commit where it was fixed - so please let me know if somebody really needs this info and/or use script above to verify particular versions