[MDEV-26211] Cluster joiner node is failed to start when using TLS mariabackup SST encryption built into socat Created: 2021-07-22  Updated: 2021-12-07  Resolved: 2021-08-16

Status: Closed
Project: MariaDB Server
Component/s: Galera SST
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6, 10.7
Fix Version/s: 10.2.41, 10.3.32, 10.4.22, 10.5.13, 10.6.5, 10.7.0

Type: Bug Priority: Major
Reporter: Ramesh Sivaraman Assignee: Julius Goryavsky
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates

 Description   

Start 2 node cluster using TLS mariabackup SST encryption built into socat.

config info
 
 
 
wsrep_provider_options='gmcast.listen_addr=tcp://127.0.0.1:4808;'
 
ssl-ca = /dev/shm/qa/cert/ca.pem
 
ssl-cert = /dev/shm/qa/cert/server-cert.pem
 
ssl-key = /dev/shm/qa/cert/server-key.pem
 
[sst]
 
encrypt = 2
 
tca = /dev/shm/qa/cert/sst_encypt2.crt
 
tcert = /dev/shm/qa/cert/sst_encypt2.pem
 
$

Error info from donor node.

2021-07-21 16:49:12 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address 'ax3win:4444/xtrabackup_sst//1' --local-port '4800' --socket '/dev/shm/qa/node1/mysql.sock' --datadir '/dev/shm/qa/node1/' --defaults-file '/dev/shm/qa/conf/node1.cnf' --gtid '92245b05-ea43-11eb-89cc-ba6f0392455f:0' --gtid-domain-id '0' --mysqld-args --defaults-file=/dev/shm/qa/conf/node1.cnf --wsrep-new-cluster'
 
2021-07-21 16:49:12 2 [Note] WSREP: sst_donor_thread signaled with 0
 
WSREP_SST: [INFO] SSL configuration: CA='/dev/shm/qa/cert/sst_encypt2.crt', CERT='/dev/shm/qa/cert/sst_encypt2.pem', KEY='', MODE='DISABLED', encrypt='2' (20210721 16:49:12.800)
 
WSREP_SST: [INFO] Streaming with mbstream (20210721 16:49:12.950)
 
WSREP_SST: [INFO] Using socat as streamer (20210721 16:49:12.952)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with crt and pem (20210721 16:49:12.959)
 
WSREP_SST: [INFO] Encrypting with cert=/dev/shm/qa/cert/sst_encypt2.pem, cafile=/dev/shm/qa/cert/sst_encypt2.crt (20210721 16:49:12.968)
 
WSREP_SST: [INFO] Using '/tmp/tmp.PqLfYMKEGR' as mariabackup temporary directory (20210721 16:49:12.990)
 
WSREP_SST: [INFO] Using '/tmp/tmp.wt3mCpRqiI' as mariabackup working directory (20210721 16:49:12.995)
 
WSREP_SST: [INFO] Streaming GTID file before SST (20210721 16:49:12.999)
 
WSREP_SST: [INFO] Evaluating '/home/ramesh/framework/GAL_MD200721-mariadb-10.3.30-linux-x86_64-opt//bin/mbstream' -c 'xtrabackup_galera_info' | socat -u stdio openssl-connect:ax3win:4444,cert='/dev/shm/qa/cert/sst_encypt2.pem',cafile='/dev/shm/qa/cert/sst_encypt2.crt'; RC=( ${PIPESTATUS[@]} ) (20210721 16:49:13.002)
 
2021/07/21 16:49:13 socat[2709097] E certificate is valid but its commonName does not match hostname
 
WSREP_SST: [ERROR] Error while sending data to joiner node:  exit codes: 0 1 (20210721 16:49:13.020)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20210721 16:49:13.022)
 
WSREP_SST: [INFO] Cleaning up temporary directories (20210721 16:49:13.025)



 Comments   
Comment by Ramesh Sivaraman [ 2021-08-10 ]

The issue is present even if we use galera-compatible certificates and keys

WSREP_SST: [INFO] SSL configuration: CA='/home/vagrant/ca.pem', CERT='/home/vagrant/server-cert.pem', KEY='/home/vagrant/server-key.pem', MODE='DISABLED', encrypt='3' (20210809 10:42:32.645)
 
WSREP_SST: [INFO] Moving '/home/vagrant/data/mariabackup.prepare.log' to '/tmp/sst_log_archive/mariabackup.prepare.log.2021.08.09-10.42.32.714837449' (20210809 10:42:32.717)
 
WSREP_SST: [INFO] Moving '/home/vagrant/data/mariabackup.move.log' to '/tmp/sst_log_archive/mariabackup.move.log.2021.08.09-10.42.32.714837449' (20210809 10:42:32.721)
 
WSREP_SST: [INFO] Streaming with mbstream (20210809 10:42:32.726)
 
WSREP_SST: [INFO] Using socat as streamer (20210809 10:42:32.727)
 
WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20210809 10:42:32.731)
 
WSREP_SST: [INFO] Decrypting with cert=/home/vagrant/server-cert.pem, key=/home/vagrant/server-key.pem, cafile=/home/vagrant/ca.pem (20210809 10:42:32.746)
 
WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u openssl-listen:4444,reuseaddr,cert='/home/vagrant/server-cert.pem',key='/home/vagrant/server-key.pem',cafile='/home/vagrant/ca.pem',commonname=localhost stdio | '/home/vagrant/mariadb-10.6.4-1-linux-x86_64//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20210809 10:42:32.762)
 
2021-08-09 10:42:32 1 [Note] WSREP: ####### IST uuid:3cb6d88c-f8fd-11eb-9496-7a09349f8515 f: 9, l: 11, STRv: 3
 
2021-08-09 10:42:32 1 [Note] WSREP: IST receiver addr using ssl://192.168.100.20:4568
 
2021-08-09 10:42:32 1 [Note] WSREP: IST receiver using ssl
 
2021-08-09 10:42:32 1 [Note] WSREP: Prepared IST receiver for 9-11, listening at: ssl://192.168.100.20:4568
 
2021-08-09 10:42:32 0 [Note] WSREP: Member 1.0 (galera-node2) requested state transfer from '*any*'. Selected 0.0 (galera-node1)(SYNCED) as donor.
 
2021-08-09 10:42:32 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 11)
 
2021-08-09 10:42:32 1 [Note] WSREP: Requesting state transfer: success, donor: 0
 
2021/08/09 10:42:32 socat[4181] E certificate is valid but its commonName does not match hostname
 
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20210809 10:42:32.974)
 
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20210809 10:42:32.976)

Config info

wsrep_provider_options="socket.ssl_cert=/home/vagrant/server-cert.pem;socket.ssl_key=/home/vagrant/server-key.pem;socket.ssl_ca=/home/vagrant/ca.pem"
 
ssl-ca=/home/vagrant/ca.pem
 
ssl-key=/home/vagrant/server-key.pem
 
ssl-cert=/home/vagrant/server-cert.pem
 
 
 
[sst]
 
encrypt=3
 
tca=/home/vagrant/ca.pem
 
tkey=/home/vagrant/server-key.pem
 
tcert=/home/vagrant/server-cert.pem

Comment by Jan Lindström (Inactive) [ 2021-08-16 ]

ok to push

Comment by Julius Goryavsky [ 2021-08-16 ]

Fixed, https://github.com/MariaDB/server/commit/d1a948cfaaab67e699674af4c11efad3868a629d

Generated at Thu Feb 08 09:43:35 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.