|
Checking for "is_local_ip" on the local hostname forces the common name to be "localhost" which is not allowed as a common name on third-party certificate authorities. This causes certificate validation to fail. It also fails when the second argument is set to "1" (on line 387 of wsrep_sst_mariabackup) to disable hostname checking. The common name does correctly match the hostname of the server, so I'm not quite sure where it is failing here, unless it's trying to match the donor server's common name.
For debugging purposes I have added a debug line that prints the hostname it is checking. This will be visible in the logs.
Both cases fail with the below error:
Unmodified
Aug 13 07:57:28 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:28 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '23351' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186'
|
Aug 13 07:57:28 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:28 0 [Note] WSREP: Joiner monitor thread started to monitor
|
Aug 13 07:57:28 vc-galera03 mariadbd[23407]: WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/certs/ca.pem', CERT='/etc/mysql/certs/server-cert.pem', KEY='/etc/mysql/certs/server-key.pem', MODE='DISABLED', encrypt='3' (20210813 07:57:28.688)
|
Aug 13 07:57:28 vc-galera03 mariadbd[23407]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20210813 07:57:28.846)
|
Aug 13 07:57:28 vc-galera03 -wsrep-sst-joiner[23556]: Streaming with mbstream
|
Aug 13 07:57:28 vc-galera03 -wsrep-sst-joiner[23557]: Using socat as streamer
|
Aug 13 07:57:29 vc-galera03 -wsrep-sst-joiner[23561]: Using openssl based encryption with socat: with key and crt
|
Aug 13 07:57:29 vc-galera03 -wsrep-sst-joiner[23573]: Decrypting with cert=/etc/mysql/certs/server-cert.pem, key=/etc/mysql/certs/server-key.pem, cafile=/etc/mysql/certs/ca.pem
|
Aug 13 07:57:29 vc-galera03 -wsrep-sst-joiner[23592]: Evaluating timeout -k 310 300 socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname=localhost stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
|
Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:29 2 [Note] WSREP: ####### IST uuid:9795eb17-c967-11eb-896e-32dd10aa7427 f: 27643187, l: 27650534, STRv: 3
|
Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:29 2 [Note] WSREP: IST receiver addr using ssl://vc-galera03.my.domain.com:4568
|
Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:29 2 [Note] WSREP: IST receiver using ssl
|
Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:29 2 [Note] WSREP: Prepared IST receiver for 27643187-27650534, listening at: ssl://10.22.0.38:4568
|
Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:29 0 [Note] WSREP: Member 0.0 (vc-galera03) requested state transfer from 'vc-galera01,'. Selected 1.0 (vc-galera01)(SYNCED) as donor.
|
Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:29 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 27650544)
|
Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:29 2 [Note] WSREP: Requesting state transfer: success, donor: 1
|
Aug 13 07:57:31 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:31 0 [Note] WSREP: (0366274e-9c1d, 'ssl://0.0.0.0:4567') turning message relay requesting off
|
Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23554]: 2021/08/13 07:57:32 socat[23597] E certificate is valid but its commonName does not match hostname
|
Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23601]: Error while getting data from donor node: exit codes: 1 0
|
Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23602]: Cleanup after exit with status:32
|
Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23603]: Removing the sst_in_progress file
|
Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23605]: Cleaning up temporary directories
|
Aug 13 07:57:32 vc-galera03 mariadbd[23351]: 2021-08-13 7:57:32 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '23351' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186: 32 (Broken pipe)
|
Modified with disabled hostname checking for "is_local_ip"
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '28976' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186'
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 0 [Note] WSREP: Joiner monitor thread started to monitor
|
Aug 13 08:20:48 vc-galera03 mariadbd[28986]: WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/certs/ca.pem', CERT='/etc/mysql/certs/server-cert.pem', KEY='/etc/mysql/certs/server-key.pem', MODE='DISABLED', encrypt='3' (20210813 08:20:48.389)
|
Aug 13 08:20:48 vc-galera03 mariadbd[28986]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20210813 08:20:48.548)
|
Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29134]: Streaming with mbstream
|
Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29135]: Using socat as streamer
|
Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29139]: Using openssl based encryption with socat: with key and crt
|
Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29148]: Host is 'vc-galera03.my.domain.com'
|
Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29153]: Decrypting with cert=/etc/mysql/certs/server-cert.pem, key=/etc/mysql/certs/server-key.pem, cafile=/etc/mysql/certs/ca.pem
|
Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29172]: Evaluating timeout -k 310 300 socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname='vc-galera03.my.domain.com' stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 1 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 27658042, STRv: 3
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 1 [Note] WSREP: IST receiver addr using ssl://vc-galera03.my.domain.com:4568
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 1 [Note] WSREP: IST receiver using ssl
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 1 [Note] WSREP: Prepared IST receiver for 0-27658042, listening at: ssl://10.22.0.38:4568
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 0 [Note] WSREP: Member 0.0 (vc-galera03) requested state transfer from 'vc-galera01,'. Selected 2.0 (vc-galera02)(SYNCED) as donor.
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 27658043)
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 1 [Note] WSREP: Requesting state transfer: success, donor: 2
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 1 [Note] WSREP: Resetting GCache seqno map due to different histories.
|
Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:48 1 [Note] WSREP: GCache history reset: 9795eb17-c967-11eb-896e-32dd10aa7427:0 -> 9795eb17-c967-11eb-896e-32dd10aa7427:27658042
|
Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29132]: 2021/08/13 08:20:49 socat[29178] E certificate is valid but its commonName does not match hostname
|
Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29180]: Error while getting data from donor node: exit codes: 1 0
|
Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29181]: Cleanup after exit with status:32
|
Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29182]: Removing the sst_in_progress file
|
Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29184]: Cleaning up temporary directories
|
Aug 13 08:20:49 vc-galera03 mariadbd[28976]: 2021-08-13 8:20:49 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '28976' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186: 32 (Broken pipe)
|
Working with encrypt=4
Aug 13 08:32:42 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:42 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '29358' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186'
|
Aug 13 08:32:42 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:42 0 [Note] WSREP: Joiner monitor thread started to monitor
|
Aug 13 08:32:42 vc-galera03 mariadbd[29368]: WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/certs/ca.pem', CERT='/etc/mysql/certs/server-cert.pem', KEY='/etc/mysql/certs/server-key.pem', MODE='DISABLED', encrypt='4' (20210813 08:32:42.991)
|
Aug 13 08:32:43 vc-galera03 mariadbd[29368]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20210813 08:32:43.154)
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29516]: Streaming with mbstream
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29517]: Using socat as streamer
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29521]: Using openssl based encryption with socat: with key and crt
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29530]: Host is 'vc-galera03.my.domain.com'
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29531]: Decrypting with cert=/etc/mysql/certs/server-cert.pem, key=/etc/mysql/certs/server-key.pem, cafile=/etc/mysql/certs/ca.pem
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29550]: Evaluating timeout -k 310 300 socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname='' stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 27662078, STRv: 3
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 2 [Note] WSREP: IST receiver addr using ssl://vc-galera03.my.domain.com:4568
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 2 [Note] WSREP: IST receiver using ssl
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 2 [Note] WSREP: Prepared IST receiver for 0-27662078, listening at: ssl://10.22.0.38:4568
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 0 [Note] WSREP: Member 2.0 (vc-galera03) requested state transfer from 'vc-galera01'. Selected 0.0 (vc-galera01)(SYNCED) as donor.
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 27662082)
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 2 [Note] WSREP: Requesting state transfer: success, donor: 0
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
|
Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13 8:32:43 2 [Note] WSREP: GCache history reset: 9795eb17-c967-11eb-896e-32dd10aa7427:0 -> 9795eb17-c967-11eb-896e-32dd10aa7427:27662078
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29566]: Proceeding with SST
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29567]: Evaluating socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname='' stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
|
Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29568]: Cleaning the existing datadir and innodb-data/log directories
|
My modified wsrep_sst_mariabackup (starting at line 381)
verify_ca_matches_cert "$tcert" "$tpem"
|
wsrep_log_info "Host is '$WSREP_SST_OPT_HOST_UNESCAPED'"
|
if [ -n "$WSREP_SST_OPT_REMOTE_USER" ]; then
|
CN_option=",commonname='$WSREP_SST_OPT_REMOTE_USER'"
|
elif [ $encrypt -eq 4 ]; then
|
CN_option=",commonname=''"
|
elif is_local_ip "$WSREP_SST_OPT_HOST_UNESCAPED" 1; then
|
CN_option=',commonname=localhost'
|
else
|
CN_option=",commonname='$WSREP_SST_OPT_HOST_UNESCAPED'"
|
fi
|
tcmd="$tcmd,cert='$tpem',key='$tkey',cafile='$tcert'$CN_option$sockopt"
|
wsrep_log_info "$action with cert=$tpem, key=$tkey, cafile=$tcert"
|
Server Certificate
root@vc-galera03:~# openssl x509 -in /etc/mysql/certs/server-cert.pem -noout -text
|
Certificate:
|
Data:
|
Version: 3 (0x2)
|
Serial Number:
|
59:00:00:00:92:ac:30:fe:b2:b3:c3:1d:05:00:00:00:00:00:92
|
Signature Algorithm: sha256WithRSAEncryption
|
Issuer: C = US, ST = Illinois, O = My Company, OU = My OU, CN = My CA
|
Validity
|
Not Before: Jun 9 21:25:16 2021 GMT
|
Not After : Jun 8 21:25:16 2026 GMT
|
Subject: C = US, ST = Illinois, L = Chicago, O = My Company, OU = My OU, CN = vc-galera03.my.domain.com
|
Subject Public Key Info:
|
------- Skipped Lines -------
|
X509v3 extensions:
|
X509v3 Key Usage: critical
|
Digital Signature, Key Encipherment
|
X509v3 Extended Key Usage:
|
TLS Web Client Authentication, TLS Web Server Authentication
|
X509v3 Subject Alternative Name:
|
DNS:vc-galera03.my.domain.com, DNS:galeracluster.my.domain.com
|
The only solution is to use encrypt=4 which clears the common name.
|