[MDEV-32342] WSREP_SST_OPT_REMOTE_AUTH bad value, causes bad socat commonname, causes SST to fail - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 10.3.39, 10.6.15, 10.11.5
Fix Version/s: 10.4.34, 10.5.25, 10.11.8, 10.6.18, 11.0.6, 11.1.5, 11.2.4, 11.4.2, 11.5.1
Component/s: wsrep
Labels:
None

Description

(edited) NOTE: Fixed by 288ea9e146a238872998d7089070e82f39272728 – see comments from 2024-07-04

Recently I ran into this:

WSREP_SST: [INFO] Evaluating '/usr//bin/mbstream' -c 'xtrabackup_galera_info' | socat stdio openssl-connect:JOINER_IP:4444,linger=10,cert='/etc/mysql/ssl/galera.crt',key='/etc/mysql/ssl/galera.key',cafile='/etc/mysql/ssl/ca.crt',commonname='CN '; RC=( ${PIPESTATUS[@]} ) (20231002 18:20:06.014)

2023/10/02 14:16:55 socat[1316] E certificate is valid but its commonName does not match hostname

So, the donor refuses to connect to the joiner because the commonname does not match. And the common name is CN<space>, which is unexpected and wrong.

The commonname should be autodetected by the joiner and passed to the donor, here (10.3 branch, at mariadb-10.3.39):
./scripts/wsrep_sst_mariabackup.sh

wait_for_listen()

    for i in {1..150}; do

        if check_port "" "$SST_PORT" 'socat|nc'; then

            break

fi

        sleep 0.2

    done

    echo "ready $ADDR:$SST_PORT/$MODULE/$lsn/$sst_ver"

The donor turns that response into:

2023-10-02 18:53:19 1 [Note] WSREP: Prepared SST request: mariabackup|CN :e4083cfc25e917fca34c140c56397c60@JOINER_IP:4444/xtrabackup_sst//1

The parts before the "@" get collected by sst_prepare_other into wsrep_sst_donate_cb where it is passed to ./scripts/wsrep_sst_mariabackup.sh (in ./scripts/wsrep_sst_common.sh) as WSREP_SST_OPT_REMOTE_AUTH:

WSREP_SST_OPT_REMOTE_AUTH="${WSREP_SST_OPT_REMOTE_AUTH:-}"

...

    readonly WSREP_SST_OPT_REMOTE_USER="${WSREP_SST_OPT_REMOTE_AUTH%%:*}"

    readonly WSREP_SST_OPT_REMOTE_PSWD="${WSREP_SST_OPT_REMOTE_AUTH#*:}"

Here it is finally used in ./scripts/wsrep_sst_mariabackup.sh:

           else

                # CA verification

                verify_ca_matches_cert "$tpem" "$tcert" "$tcap"

                if [ -n "$WSREP_SST_OPT_REMOTE_USER" ]; then

                    CN_option=",commonname='$WSREP_SST_OPT_REMOTE_USER'"

                elif [ "$WSREP_SST_OPT_ROLE" = 'joiner' -o $encrypt -eq 4 ]

                then

                    CN_option=",commonname=''"

                elif is_local_ip "$WSREP_SST_OPT_HOST_UNESCAPED"; then

                    CN_option=',commonname=localhost'

                else

                    CN_option=",commonname='$WSREP_SST_OPT_HOST_UNESCAPED'"

fi

Where we get the strange "CN" value. Disabling commonname checking with encrypt=4 is not even possible.

Working back again, we see where the "ready $ADDR" comes from. If tmode starts with VERIFY (on the joiner), it runs (./scripts/wsrep_sst_mariabackup.sh):

    if [ "${tmode#VERIFY}" != "$tmode" ]; then

        # backward-incompatible behavior:

        CN=""

        if [ -n "$tpem" ]; then

            # find out my Common Name

            get_openssl

            if [ -z "$OPENSSL_BINARY" ]; then

                wsrep_log_error \

                    'openssl not found but it is required for authentication'

                exit 42

fi

            CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$tpem" | \

                 tr ',' '\n' | grep -F 'CN =' | cut -d '=' -f2 | sed s/^\ // | \

                 sed s/\ %//)

fi

        MY_SECRET="$(wsrep_gen_secret)"

        # Add authentication data to address

        ADDR="$CN:$MY_SECRET@$ADDR"

And that CN generation is broken.

The trouble arises with the following example strings:

(openssl 1.1 and openssl 3.0)

subject=CN = galera-node1, O = OSSO B.V., C = NL, ST = Groningen, L = Groningen

(openssl 1.0)

subject= /CN=galera-node1/O=OSSO B.V./C=NL/ST=Groningen/L=Groningen

I don't expect the openssl-1.0 string to work. But the openssl-1.1/3.0 strings don't work either when CN is the first keyword of the subject.

See these:

$ echo 'subject= /CN=galera-node1/O=OSSO B.V./C=NL/ST=Groningen/L=Groningen' | tr ',' '\n' | grep -F 'CN =' | cut -d '=' -f2 | sed s/^\ // | sed s/\ %// | cat -A

$ echo 'subject=CN = galera-node1, O = OSSO B.V., C = NL, ST = Groningen, L = Groningen' | tr ',' '\n' | grep -F 'CN =' | cut -d '=' -f2 | sed s/^\ // | sed s/\ %// | cat -A

CN $

The only one that would work, would be something like:

$ echo 'subject=O = OSSO B.V., CN = galera-node1, C = NL, ST = Groningen, L = Groningen' | tr ',' '\n' | grep -F 'CN =' | cut -d '=' -f2 | sed s/^\ // | sed s/\ %// | cat -A

galera-node1$

I suggest replacing the tr+grep+cut+sed madness, with a single sed oneliner:

sed -e 's/^subject=//;s/^/, /;s@.*[/,][[:blank:]]*CN[[:blank:]]*=[[:blank:]]*@@;s@[/,].*@@'

That works on all mentioned -subject output lines.

Diff:

diff --git a/scripts/wsrep_sst_mariabackup.sh b/scripts/wsrep_sst_mariabackup.sh

index 7e26af83701..45dd1f3c4ff 100644

--- a/scripts/wsrep_sst_mariabackup.sh

+++ b/scripts/wsrep_sst_mariabackup.sh

@@ -1334,9 +1334,11 @@ else # joiner

                     'openssl not found but it is required for authentication'

                 exit 42

fi

-            CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$tpem" | \

-                 tr ',' '\n' | grep -F 'CN =' | cut -d '=' -f2 | sed s/^\ // | \

-                 sed s/\ %//)

+            CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$tpem" | sed -e '

+               s/^subject=//

+               s/^/, /

+               s@.*[/,][[:blank:]]*CN[[:blank:]]*=[[:blank:]]*@@

+               s@[/,].*@@')

fi

         MY_SECRET="$(wsrep_gen_secret)"

         # Add authentication data to address

This was tested with the following config:

[mysqld]

wsrep_sst_method=mariabackup

ssl-cert=/etc/mysql/ssl/galera.crt

ssl-key=/etc/mysql/ssl/galera.key

ssl-ca=/etc/mysql/ssl/ca.crt

[sst]

ssl-mode=VERIFY_CA

encrypt=3

This yields on donor (10.3.39+maria~ubu2004):

WSREP_SST: [INFO] SSL configuration: CA='/etc/mysql/ssl/ca.crt', CAPATH='', CERT='/etc/mysql/ssl/galera.crt', KEY='/etc/mysql/ssl/galera.key', MODE='VERIFY_CA', encrypt='3' (20231002 20:20:44.172)

...

WSREP_SST: [INFO] Evaluating '/usr//bin/mbstream' -c 'xtrabackup_galera_info' | socat stdio openssl-connect:JOINER_IP:4444,linger=10,cert='/etc/mysql/ssl/galera.crt',key='/etc/mysql/ssl/galera.key',cafile='/etc/mysql/ssl/ca.crt',commonname='galera-node3'; RC=( ${PIPESTATUS[@]} ) (20231002 20:20:44.323)

...

And everyone lives happily after.

Cheers,
Walter Doekes
OSSO B.V.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

issueCN_IN_REMOTE_AUTH.patch
0.8 kB
2023-10-03 12:11

Issue Links

relates to

MDEV-32344 IST "Donor does not know my secret" with ssl-mode=VERIFY_CA

Closed

MDEV-26360 Using hostnames for MariaBackup SSTs breaks certificate validation with encrypt=3

Closed

Activity

Ascending order - Click to sort in descending order

Walter Doekes added a comment - 2024-07-04 14:23 - edited

I guess this is superseded by 288ea9e146a238872998d7089070e82f39272728.

$ git branch --contains 288ea9e146a238872998d7089070e82f39272728 -a | grep -E 'origin/10...?$'

  remotes/origin/10.11

  remotes/origin/10.4

  remotes/origin/10.5

  remotes/origin/10.6

$ git tag --contains 288ea9e146a238872998d7089070e82f39272728

mariadb-11.5.1

mariadb-11.4.2

mariadb-11.2.4

mariadb-11.1.5

mariadb-11.0.6

mariadb-10.11.8

mariadb-10.6.18

mariadb-10.5.25

mariadb-10.4.34

Trimmed down version seems to work:

#!/bin/bash

commandex()

    if [ -n "$BASH_VERSION" ]; then

        command -v "$1" || :

    elif [ -x "$1" ]; then

        echo "$1"

    else

        which "$1" || :

fi

get_openssl()

    # If the OPENSSL_BINARY variable is already defined, just return:

    if [ -n "${OPENSSL_BINARY+x}" ]; then

        return

fi

    # Let's look for openssl:

    OPENSSL_BINARY=$(commandex 'openssl')

    if [ -z "$OPENSSL_BINARY" ]; then

        OPENSSL_BINARY='/usr/bin/openssl'

        if [ ! -x "$OPENSSL_BINARY" ]; then

            OPENSSL_BINARY=""

fi

fi

    readonly OPENSSL_BINARY

trim_left()

    if [ -n "$BASH_VERSION" ]; then

        local pattern="[![:space:]${2:-}]"

        local x="${1#*$pattern}"

        local z=${#1}

        x=${#x}

        if [ $x -ne $z ]; then

            x=$(( z-x-1 ))

            echo "${1:$x:$z}"

        else

            echo ''

fi

    else

        local pattern="[[:space:]${2:-}]"

        echo "$1" | sed -E "s/^$pattern+//g"

fi

trim_right()

    if [ -n "$BASH_VERSION" ]; then

        local pattern="[![:space:]${2:-}]"

        local z=${#1}

        local y="${1%$pattern*}"

        y=${#y}

        if [ $y -ne $z ]; then

            y=$(( y+1 ))

            echo "${1:0:$y}"

        else

            echo ''

fi

    else

        local pattern="[[:space:]${2:-}]"

        echo "$1" | sed -E "s/$pattern+\$//g"

fi

trim_string()

    if [ -n "$BASH_VERSION" ]; then

        local pattern="[![:space:]${2:-}]"

        local x="${1#*$pattern}"

        local z=${#1}

        x=${#x}

        if [ $x -ne $z ]; then

            local y="${1%$pattern*}"

            y=${#y}

            x=$(( z-x-1 ))

            y=$(( y-x+1 ))

            echo "${1:$x:$y}"

        else

            echo ''

fi

    else

        local pattern="[[:space:]${2:-}]"

        echo "$1" | sed -E "s/^$pattern+|$pattern+\$//g"

fi

# Get Common Name (CN) from the certificate:

openssl_getCN()

    get_openssl

    if [ -z "$OPENSSL_BINARY" ]; then

        wsrep_log_error \

            'openssl not found but it is required for authentication'

        exit 42

fi

    local bug=0

    local CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$1" 2>&1) || bug=1

    #local CN="$1"

    if [ $bug -ne 0 ]; then

        wsrep_log_info "run: \"$OPENSSL_BINARY\" x509 -noout -subject -in \"$1\""

        wsrep_log_info "output: $CN"

        wsrep_log_error "******** FATAL ERROR **********************************************"

        wsrep_log_error "* Unable to parse the certificate file to obtain the common name. *"

        wsrep_log_error "*******************************************************************"

        exit 22

fi

    CN=$(trim_string "$CN")

    if [ -n "$CN" ]; then

        # If the string begins with the "subject" prefix

        # then we need to remove it:

        local saved="$CN"

        local remain="${CN#subject}"

        if [ "$remain" != "$saved" ]; then

            remain=$(trim_left "$remain")

            # Now let's check for the presence of "=" character

            # after the "subject":

            saved="$remain"

            remain="${remain#=}"

            if [ "$remain" != "$saved" ]; then

                remain=$(trim_left "$remain")

            else

                remain=""

                bug=1

fi

fi

        while [ -n "$remain" ]; do

            local value=""

            # Let's extract the option name - all characters

            # up to the first '=' or ',' character (if present):

            local option="${remain%%[=,]*}"

            if [ "$option" != "$remain" ]; then

                option=$(trim_right "$option")

                # These variables will be needed to determine

                # which separator comes first:

                local x="${remain#*=}"

                local y="${remain#*,}"

                local z=${#remain}

                x=${#x}; [ $x -eq $z ] && x=0

                y=${#y}; [ $y -eq $z ] && y=0

                # The remaining string is everything that follows

                # the separator character:

                remain=$(trim_left "${remain#*[=,]}")

                # Let's check what we are dealing with - an equal

                # sign or a comma?

                if [ $x -gt $y ]; then

                    # If the remainder begins with a double quote,

                    # then there is a string containing commas and

                    # we need to parse it:

                    saved="$remain"

                    remain="${remain#\"}"

                    if [ "$remain" != "$saved" ]; then

                        while :; do

                            # We need to find the closing quote:

                            local prefix="$remain"

                            remain="${remain#*\"}"

                            # Let's check if there is a closing quote?

                            if [ "$remain" = "$prefix" ]; then

                                bug=1

                                break

fi

                            # Everything up to the closing quote is

                            # the next part of the value:

                            value="$value${prefix%%\"*}"

                            # But if the last character of the value

                            # is a backslash, then it is a quoted quotation

                            # mark and we need to add it to the value:

                            if [ "${value%\\}" != "$value" ]; then

                                value="$value\""

                            else

                                break

fi

                        done

                        [ $bug -ne 0 ] && break

                        # Now we have to remove "," if it is present

                        # in the string after the value:

                        saved=$(trim_left "$remain")

                        remain="${saved#,}"

                        if [ "$remain" != "$saved" ]; then

                            remain=$(trim_left "$remain")

                        elif [ -n "$remain" ]; then

                            bug=1

                            break

fi

                    else

                        # We are dealing with a simple unquoted string value,

                        # therefore we need to take everything up to the end

                        # of the string, or up to the next comma character:

                        value="${remain%%,*}"

                        if [ "$value" != "$remain" ]; then

                            remain=$(trim_left "${remain#*,}")

                        else

                            remain=""

fi

                        value=$(trim_right "$value")

fi

                    if [ "$option" = 'CN' -a -n "$value" ]; then

                        echo "$value"

                        return

fi

fi

            else

                remain=""

fi

        done

fi

    if [ $bug -ne 0 ]; then

        wsrep_log_error "******** FATAL ERROR **********************************************"

        wsrep_log_error "* Unable to parse the certificate options: '$CN'"

        wsrep_log_error "*******************************************************************"

        exit 22

fi

    echo ''

openssl_getCN "$1"

Or, if we replace

    local CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$1" 2>&1) || bug=1

with

    local CN="$1"

Then:

$ ./getcn 'subject=C = TR, L = Ankara, O = E-Tugra EBG A.S., OU = E-Tugra Trust Center, CN = E-Tugra Global Root CA RSA v3'

E-Tugra Global Root CA RSA v3

$ ./getcn 'subject=CN = E-Tugra Global Root CA RSA v3'

E-Tugra Global Root CA RSA v3

$ ./getcn 'subject=CN = E-Tugra Global Root CA RSA v3, OU = x'

E-Tugra Global Root CA RSA v3

All works.

So I think we can close this.

(P.S. Splitting off the subject-to-CN code from the openssl-x509-call into a separate function would've made testing slightly easier.)

(P.P.S. The openssl 1.0 strings now fail to parse. But I guess nobody uses openssl 1.0 anywhere where a recent mariadb is running.)

Walter Doekes added a comment - 2024-07-04 14:23 - edited I guess this is superseded by 288ea9e146a238872998d7089070e82f39272728. $ git branch --contains 288ea9e146a238872998d7089070e82f39272728 -a | grep -E 'origin/10...?$' remotes/origin/10.11 remotes/origin/10.4 remotes/origin/10.5 remotes/origin/10.6 $ git tag --contains 288ea9e146a238872998d7089070e82f39272728 mariadb-11.5.1 mariadb-11.4.2 mariadb-11.2.4 mariadb-11.1.5 mariadb-11.0.6 mariadb-10.11.8 mariadb-10.6.18 mariadb-10.5.25 mariadb-10.4.34 Trimmed down version seems to work: #!/bin/bash commandex() { if [ -n "$BASH_VERSION" ]; then command -v "$1" || : elif [ -x "$1" ]; then echo "$1" else which "$1" || : fi } get_openssl() { # If the OPENSSL_BINARY variable is already defined, just return: if [ -n "${OPENSSL_BINARY+x}" ]; then return fi # Let's look for openssl: OPENSSL_BINARY=$(commandex 'openssl') if [ -z "$OPENSSL_BINARY" ]; then OPENSSL_BINARY='/usr/bin/openssl' if [ ! -x "$OPENSSL_BINARY" ]; then OPENSSL_BINARY="" fi fi readonly OPENSSL_BINARY } trim_left() { if [ -n "$BASH_VERSION" ]; then local pattern="[![:space:]${2:-}]" local x="${1#*$pattern}" local z=${#1} x=${#x} if [ $x -ne $z ]; then x=$(( z-x-1 )) echo "${1:$x:$z}" else echo '' fi else local pattern="[[:space:]${2:-}]" echo "$1" | sed -E "s/^$pattern+//g" fi } trim_right() { if [ -n "$BASH_VERSION" ]; then local pattern="[![:space:]${2:-}]" local z=${#1} local y="${1%$pattern*}" y=${#y} if [ $y -ne $z ]; then y=$(( y+1 )) echo "${1:0:$y}" else echo '' fi else local pattern="[[:space:]${2:-}]" echo "$1" | sed -E "s/$pattern+\$//g" fi } trim_string() { if [ -n "$BASH_VERSION" ]; then local pattern="[![:space:]${2:-}]" local x="${1#*$pattern}" local z=${#1} x=${#x} if [ $x -ne $z ]; then local y="${1%$pattern*}" y=${#y} x=$(( z-x-1 )) y=$(( y-x+1 )) echo "${1:$x:$y}" else echo '' fi else local pattern="[[:space:]${2:-}]" echo "$1" | sed -E "s/^$pattern+|$pattern+\$//g" fi } # Get Common Name (CN) from the certificate: openssl_getCN() { get_openssl if [ -z "$OPENSSL_BINARY" ]; then wsrep_log_error \ 'openssl not found but it is required for authentication' exit 42 fi local bug=0 local CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$1" 2>&1) || bug=1 #local CN="$1" if [ $bug -ne 0 ]; then wsrep_log_info "run: \"$OPENSSL_BINARY\" x509 -noout -subject -in \"$1\"" wsrep_log_info "output: $CN" wsrep_log_error "******** FATAL ERROR **********************************************" wsrep_log_error "* Unable to parse the certificate file to obtain the common name. *" wsrep_log_error "*******************************************************************" exit 22 fi CN=$(trim_string "$CN") if [ -n "$CN" ]; then # If the string begins with the "subject" prefix # then we need to remove it: local saved="$CN" local remain="${CN#subject}" if [ "$remain" != "$saved" ]; then remain=$(trim_left "$remain") # Now let's check for the presence of "=" character # after the "subject": saved="$remain" remain="${remain#=}" if [ "$remain" != "$saved" ]; then remain=$(trim_left "$remain") else remain="" bug=1 fi fi while [ -n "$remain" ]; do local value="" # Let's extract the option name - all characters # up to the first '=' or ',' character (if present): local option="${remain%%[=,]*}" if [ "$option" != "$remain" ]; then option=$(trim_right "$option") # These variables will be needed to determine # which separator comes first: local x="${remain#*=}" local y="${remain#*,}" local z=${#remain} x=${#x}; [ $x -eq $z ] && x=0 y=${#y}; [ $y -eq $z ] && y=0 # The remaining string is everything that follows # the separator character: remain=$(trim_left "${remain#*[=,]}") # Let's check what we are dealing with - an equal # sign or a comma? if [ $x -gt $y ]; then # If the remainder begins with a double quote, # then there is a string containing commas and # we need to parse it: saved="$remain" remain="${remain#\"}" if [ "$remain" != "$saved" ]; then while :; do # We need to find the closing quote: local prefix="$remain" remain="${remain#*\"}" # Let's check if there is a closing quote? if [ "$remain" = "$prefix" ]; then bug=1 break fi # Everything up to the closing quote is # the next part of the value: value="$value${prefix%%\"*}" # But if the last character of the value # is a backslash, then it is a quoted quotation # mark and we need to add it to the value: if [ "${value%\\}" != "$value" ]; then value="$value\"" else break fi done [ $bug -ne 0 ] && break # Now we have to remove "," if it is present # in the string after the value: saved=$(trim_left "$remain") remain="${saved#,}" if [ "$remain" != "$saved" ]; then remain=$(trim_left "$remain") elif [ -n "$remain" ]; then bug=1 break fi else # We are dealing with a simple unquoted string value, # therefore we need to take everything up to the end # of the string, or up to the next comma character: value="${remain%%,*}" if [ "$value" != "$remain" ]; then remain=$(trim_left "${remain#*,}") else remain="" fi value=$(trim_right "$value") fi if [ "$option" = 'CN' -a -n "$value" ]; then echo "$value" return fi fi else remain="" fi done fi if [ $bug -ne 0 ]; then wsrep_log_error "******** FATAL ERROR **********************************************" wsrep_log_error "* Unable to parse the certificate options: '$CN'" wsrep_log_error "*******************************************************************" exit 22 fi echo '' } openssl_getCN "$1" Or, if we replace local CN=$("$OPENSSL_BINARY" x509 -noout -subject -in "$1" 2>&1) || bug=1 with local CN="$1" Then: $ ./getcn 'subject=C = TR, L = Ankara, O = E-Tugra EBG A.S., OU = E-Tugra Trust Center, CN = E-Tugra Global Root CA RSA v3' E-Tugra Global Root CA RSA v3 $ ./getcn 'subject=CN = E-Tugra Global Root CA RSA v3' E-Tugra Global Root CA RSA v3 $ ./getcn 'subject=CN = E-Tugra Global Root CA RSA v3, OU = x' E-Tugra Global Root CA RSA v3 All works. So I think we can close this. (P.S. Splitting off the subject-to-CN code from the openssl-x509-call into a separate function would've made testing slightly easier.) (P.P.S. The openssl 1.0 strings now fail to parse. But I guess nobody uses openssl 1.0 anywhere where a recent mariadb is running.)

Walter Doekes added a comment - 2024-07-05 08:49

Okay. In fact fixed everywhere relevant, except in 10.3. But I guess that's way past EOL.

Walter Doekes added a comment - 2024-07-05 08:49 Okay. In fact fixed everywhere relevant, except in 10.3. But I guess that's way past EOL.

Julius Goryavsky added a comment - 2024-07-05 14:26

Already fixed by https://github.com/MariaDB/server/commit/288ea9e146a238872998d7089070e82f39272728, thanks wdoekes for testing

Julius Goryavsky added a comment - 2024-07-05 14:26 Already fixed by https://github.com/MariaDB/server/commit/288ea9e146a238872998d7089070e82f39272728 , thanks wdoekes for testing

People

Assignee:: Julius Goryavsky

Reporter:: Walter Doekes

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2023-10-03 12:20

Updated:: 2024-07-05 14:26

Resolved:: 2024-07-05 14:26

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration