Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26360

Using hostnames for MariaBackup SSTs breaks certificate validation with encrypt=3

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 10.2.40, 10.3.31, 10.4.21, 10.5.11, 10.5.12, 10.6.4
    • Fix Version/s: 10.2.41, 10.3.32, 10.4.22, 10.5.13, 10.6.5, 10.7.1
    • Component/s: Galera SST
    • Environment:
      Linux vc-galera03 5.4.114-1-pve #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200) x86_64 x86_64 x86_64 GNU/Linux
      Proxmox LXC Container

      Description

      Checking for "is_local_ip" on the local hostname forces the common name to be "localhost" which is not allowed as a common name on third-party certificate authorities. This causes certificate validation to fail. It also fails when the second argument is set to "1" (on line 387 of wsrep_sst_mariabackup) to disable hostname checking. The common name does correctly match the hostname of the server, so I'm not quite sure where it is failing here, unless it's trying to match the donor server's common name.

      For debugging purposes I have added a debug line that prints the hostname it is checking. This will be visible in the logs.

      Both cases fail with the below error:

      Unmodified

      Aug 13 07:57:28 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:28 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '23351' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186'
      Aug 13 07:57:28 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:28 0 [Note] WSREP: Joiner monitor thread started to monitor
      Aug 13 07:57:28 vc-galera03 mariadbd[23407]: WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/certs/ca.pem', CERT='/etc/mysql/certs/server-cert.pem', KEY='/etc/mysql/certs/server-key.pem', MODE='DISABLED', encrypt='3' (20210813 07:57:28.688)
      Aug 13 07:57:28 vc-galera03 mariadbd[23407]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20210813 07:57:28.846)
      Aug 13 07:57:28 vc-galera03 -wsrep-sst-joiner[23556]: Streaming with mbstream
      Aug 13 07:57:28 vc-galera03 -wsrep-sst-joiner[23557]: Using socat as streamer
      Aug 13 07:57:29 vc-galera03 -wsrep-sst-joiner[23561]: Using openssl based encryption with socat: with key and crt
      Aug 13 07:57:29 vc-galera03 -wsrep-sst-joiner[23573]: Decrypting with cert=/etc/mysql/certs/server-cert.pem, key=/etc/mysql/certs/server-key.pem, cafile=/etc/mysql/certs/ca.pem
      Aug 13 07:57:29 vc-galera03 -wsrep-sst-joiner[23592]: Evaluating timeout -k 310 300 socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname=localhost stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
      Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:29 2 [Note] WSREP: ####### IST uuid:9795eb17-c967-11eb-896e-32dd10aa7427 f: 27643187, l: 27650534, STRv: 3
      Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:29 2 [Note] WSREP: IST receiver addr using ssl://vc-galera03.my.domain.com:4568
      Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:29 2 [Note] WSREP: IST receiver using ssl
      Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:29 2 [Note] WSREP: Prepared IST receiver for 27643187-27650534, listening at: ssl://10.22.0.38:4568
      Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:29 0 [Note] WSREP: Member 0.0 (vc-galera03) requested state transfer from 'vc-galera01,'. Selected 1.0 (vc-galera01)(SYNCED) as donor.
      Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:29 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 27650544)
      Aug 13 07:57:29 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:29 2 [Note] WSREP: Requesting state transfer: success, donor: 1
      Aug 13 07:57:31 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:31 0 [Note] WSREP: (0366274e-9c1d, 'ssl://0.0.0.0:4567') turning message relay requesting off
      Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23554]: 2021/08/13 07:57:32 socat[23597] E certificate is valid but its commonName does not match hostname
      Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23601]: Error while getting data from donor node:  exit codes: 1 0
      Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23602]: Cleanup after exit with status:32
      Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23603]: Removing the sst_in_progress file
      Aug 13 07:57:32 vc-galera03 -wsrep-sst-joiner[23605]: Cleaning up temporary directories
      Aug 13 07:57:32 vc-galera03 mariadbd[23351]: 2021-08-13  7:57:32 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '23351' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186: 32 (Broken pipe)
      

      Modified with disabled hostname checking for "is_local_ip"

      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '28976' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186'
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 0 [Note] WSREP: Joiner monitor thread started to monitor
      Aug 13 08:20:48 vc-galera03 mariadbd[28986]: WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/certs/ca.pem', CERT='/etc/mysql/certs/server-cert.pem', KEY='/etc/mysql/certs/server-key.pem', MODE='DISABLED', encrypt='3' (20210813 08:20:48.389)
      Aug 13 08:20:48 vc-galera03 mariadbd[28986]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20210813 08:20:48.548)
      Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29134]: Streaming with mbstream
      Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29135]: Using socat as streamer
      Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29139]: Using openssl based encryption with socat: with key and crt
      Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29148]: Host is 'vc-galera03.my.domain.com'
      Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29153]: Decrypting with cert=/etc/mysql/certs/server-cert.pem, key=/etc/mysql/certs/server-key.pem, cafile=/etc/mysql/certs/ca.pem
      Aug 13 08:20:48 vc-galera03 -wsrep-sst-joiner[29172]: Evaluating timeout -k 310 300 socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname='vc-galera03.my.domain.com' stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 1 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 27658042, STRv: 3
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 1 [Note] WSREP: IST receiver addr using ssl://vc-galera03.my.domain.com:4568
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 1 [Note] WSREP: IST receiver using ssl
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 1 [Note] WSREP: Prepared IST receiver for 0-27658042, listening at: ssl://10.22.0.38:4568
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 0 [Note] WSREP: Member 0.0 (vc-galera03) requested state transfer from 'vc-galera01,'. Selected 2.0 (vc-galera02)(SYNCED) as donor.
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 27658043)
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 1 [Note] WSREP: Requesting state transfer: success, donor: 2
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 1 [Note] WSREP: Resetting GCache seqno map due to different histories.
      Aug 13 08:20:48 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:48 1 [Note] WSREP: GCache history reset: 9795eb17-c967-11eb-896e-32dd10aa7427:0 -> 9795eb17-c967-11eb-896e-32dd10aa7427:27658042
      Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29132]: 2021/08/13 08:20:49 socat[29178] E certificate is valid but its commonName does not match hostname
      Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29180]: Error while getting data from donor node:  exit codes: 1 0
      Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29181]: Cleanup after exit with status:32
      Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29182]: Removing the sst_in_progress file
      Aug 13 08:20:49 vc-galera03 -wsrep-sst-joiner[29184]: Cleaning up temporary directories
      Aug 13 08:20:49 vc-galera03 mariadbd[28976]: 2021-08-13  8:20:49 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '28976' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186: 32 (Broken pipe)
      

      Working with encrypt=4

      Aug 13 08:32:42 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:42 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address 'vc-galera03.my.domain.com' --datadir '/var/lib/mysql/' --parent '29358' --mysqld-args --wsrep_start_position=9795eb17-c967-11eb-896e-32dd10aa7427:27643186'
      Aug 13 08:32:42 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:42 0 [Note] WSREP: Joiner monitor thread started to monitor
      Aug 13 08:32:42 vc-galera03 mariadbd[29368]: WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/certs/ca.pem', CERT='/etc/mysql/certs/server-cert.pem', KEY='/etc/mysql/certs/server-key.pem', MODE='DISABLED', encrypt='4' (20210813 08:32:42.991)
      Aug 13 08:32:43 vc-galera03 mariadbd[29368]: WSREP_SST: [INFO] Logging all stderr of SST/mariabackup to syslog (20210813 08:32:43.154)
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29516]: Streaming with mbstream
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29517]: Using socat as streamer
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29521]: Using openssl based encryption with socat: with key and crt
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29530]: Host is 'vc-galera03.my.domain.com'
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29531]: Decrypting with cert=/etc/mysql/certs/server-cert.pem, key=/etc/mysql/certs/server-key.pem, cafile=/etc/mysql/certs/ca.pem
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29550]: Evaluating timeout -k 310 300 socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname='' stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 27662078, STRv: 3
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 2 [Note] WSREP: IST receiver addr using ssl://vc-galera03.my.domain.com:4568
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 2 [Note] WSREP: IST receiver using ssl
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 2 [Note] WSREP: Prepared IST receiver for 0-27662078, listening at: ssl://10.22.0.38:4568
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 0 [Note] WSREP: Member 2.0 (vc-galera03) requested state transfer from 'vc-galera01'. Selected 0.0 (vc-galera01)(SYNCED) as donor.
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 27662082)
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 2 [Note] WSREP: Requesting state transfer: success, donor: 0
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
      Aug 13 08:32:43 vc-galera03 mariadbd[29358]: 2021-08-13  8:32:43 2 [Note] WSREP: GCache history reset: 9795eb17-c967-11eb-896e-32dd10aa7427:0 -> 9795eb17-c967-11eb-896e-32dd10aa7427:27662078
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29566]: Proceeding with SST
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29567]: Evaluating socat -u openssl-listen:4444,reuseaddr,cert='/etc/mysql/certs/server-cert.pem',key='/etc/mysql/certs/server-key.pem',cafile='/etc/mysql/certs/ca.pem',commonname='' stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} )
      Aug 13 08:32:43 vc-galera03 -wsrep-sst-joiner[29568]: Cleaning the existing datadir and innodb-data/log directories
      

      My modified wsrep_sst_mariabackup (starting at line 381)

      verify_ca_matches_cert "$tcert" "$tpem"
      wsrep_log_info "Host is '$WSREP_SST_OPT_HOST_UNESCAPED'"
      if [ -n "$WSREP_SST_OPT_REMOTE_USER" ]; then
      	CN_option=",commonname='$WSREP_SST_OPT_REMOTE_USER'"
      elif [ $encrypt -eq 4 ]; then
      	CN_option=",commonname=''"
      elif is_local_ip "$WSREP_SST_OPT_HOST_UNESCAPED" 1; then
      	CN_option=',commonname=localhost'
      else
      	CN_option=",commonname='$WSREP_SST_OPT_HOST_UNESCAPED'"
      fi
      tcmd="$tcmd,cert='$tpem',key='$tkey',cafile='$tcert'$CN_option$sockopt"
      wsrep_log_info "$action with cert=$tpem, key=$tkey, cafile=$tcert"
      

      Server Certificate

      root@vc-galera03:~# openssl x509 -in /etc/mysql/certs/server-cert.pem -noout -text
      Certificate:
          Data:
              Version: 3 (0x2)
              Serial Number:
                  59:00:00:00:92:ac:30:fe:b2:b3:c3:1d:05:00:00:00:00:00:92
              Signature Algorithm: sha256WithRSAEncryption
              Issuer: C = US, ST = Illinois, O = My Company, OU = My OU, CN = My CA
              Validity
                  Not Before: Jun  9 21:25:16 2021 GMT
                  Not After : Jun  8 21:25:16 2026 GMT
              Subject: C = US, ST = Illinois, L = Chicago, O = My Company, OU = My OU, CN = vc-galera03.my.domain.com
              Subject Public Key Info:
      ------- Skipped Lines -------
              X509v3 extensions:
                  X509v3 Key Usage: critical
                      Digital Signature, Key Encipherment
                  X509v3 Extended Key Usage:
                      TLS Web Client Authentication, TLS Web Server Authentication
                  X509v3 Subject Alternative Name:
                      DNS:vc-galera03.my.domain.com, DNS:galeracluster.my.domain.com
      

      The only solution is to use encrypt=4 which clears the common name.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              sysprg Julius Goryavsky
              Reporter:
              ib-mlatin Matthew Latin
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration