Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35812

Galera SST fails with TLS certificate containing an intermediate CA

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 11.4.4
    • None
    • Galera
    • None
    • MariaDB Community container running in Kubernetes v1.31

    Description

      Hey there! I am providing the following certificate to a MariaDB Community 11.4.4. It consists of 2 PEM blocks, in this order:

      • Leaf certificate
      • Intermediate CA certificate

      The certificates have been issued by cert-manager which using the go x509 library to issue certificates.

      It seems like Galera is not able to parse multiple PEM blocks, returning the following error when setting up the donor:

      mariadb 2025-01-09 18:02:20 0 [Note] WSREP: Failed to establish connection: unexpected eof while reading (SSL routines)

      The donor node comes up anyway and when the SST is requested by a joiner, a crash happens. (see attached logs)

      Including an intermediate CA as part of the certificate is valid and accepted by MariaDB. Standalone and replication topologies work normally with MariaDB 11.4.4. I have tested previous versions (11.4.3), but they also result in SSL errors (but not parsing errors) when this certificate structure is provided. It is also a very common PKI practice nowadays, as it allows to build trust in complex scenarios where multiple intermediate CAs are involved.

      I have found the following issues during my investigation, which might provide some context:

      I have attached the logs, configuration files and the PKI material.

      Attachments

        1. ca.crt
          0.5 kB
        2. client.crt
          1 kB
        3. client.key
          0.2 kB
        4. donor.log
          33 kB
        5. galera.cnf
          0.9 kB
        6. joiner.log
          17 kB
        7. mariabackup.backup.log
          0.3 kB
        8. server.crt
          2 kB
        9. server.key
          0.2 kB
        10. tls.cnf
          0.1 kB

        Activity

          Ascending order - Click to sort in descending order
          martin.montes Martin Montes added a comment -

          I have enabled encryption for the SST:

          [sst]
          		 
          encrypt=3
          		 
          tca=/etc/pki/ca.crt
          		 
          tcert=/etc/pki/client.crt
          		 
          tkey=/etc/pki/client.key

          Resulting in the following logs in the donor:

          mariadb 2025-01-10 12:43:37 0 [Note] WSREP: async IST sender starting to serve ssl://10.244.0.19:4568 send
          		 
          ing 6-6, preload starts from 6
          		 
          mariadb WSREP_SST: [INFO] new ssl configuration options (ssl-ca[path], ssl-cert and ssl-key) are ignored b
          		 
          y SST due to presence of the tca[path], tcert and/or tkey in the [sst] section (20250110 12:43:37.536)
          		 
          mariadb WSREP_SST: [INFO] SSL configuration: CA='/etc/pki/ca.crt', CAPATH='', CERT='/etc/pki/client.crt',
          		 
          KEY='/etc/pki/client.key', MODE='DISABLED', encrypt='3' (20250110 12:43:37.540)
          		 
          mariadb WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20250110 12:43:37.6
          		 
          76)
          		 
          mariadb WSREP_SST: [INFO] Evaluating '/usr//bin/mbstream' -c 'mariadb_backup_galera_info' 'donor_galera_in
          		 
          fo' | socat -u stdio openssl-connect:10.244.0.19:4444,no-sni=1,cert='/etc/pki/client.crt',key='/etc/pki/cl
          		 
          ient.key',cafile='/etc/pki/ca.crt',commonname='10.244.0.19'; RC=( ${PIPESTATUS[@]} ) (20250110 12:43:37.76
          		 
          4)
          		 
          mariadb 2025/01/10 12:43:37 socat[923] W OpenSSL: Warning: this implementation does not check CRLs
          		 
          mariadb 2025-01-10 12:43:38 0 [Note] WSREP: forgetting 821b0db3-a24b (ssl://10.244.0.19:4567)
          		 
          mariadb 2025-01-10 12:43:38 0 [Note] WSREP: forgetting 821b0db3-a24b (ssl://10.244.0.19:4567)
          		 
          mariadb 2025-01-10 12:43:39 0 [Note] WSREP: (622e7688-b998, 'ssl://0.0.0.0:4567') turning message relay re
          		 
          questing off
          		 
          mariadb 2025-01-10 12:43:43 0 [Note] WSREP:  cleaning up 821b0db3-a24b (ssl://10.244.0.19:4567)
          		 
          mariadb 2025-01-10 12:44:04 0 [Note] WSREP: (622e7688-b998, 'ssl://0.0.0.0:4567') connection established t
          		 
          o 936276b2-8d30 ssl://10.244.0.19:4567
          		 
          mariadb 2025-01-10 12:44:05 0 [Note] WSREP: declaring 936276b2-8d30 at ssl://10.244.0.19:4567 stable
          		 
          mariadb 2025-01-10 12:44:06 0 [Note] WSREP: async IST sender starting to serve ssl://10.244.0.19:4568 send
          		 
          ing 8-8, preload starts from 8
          		 
          mariadb WSREP_SST: [INFO] new ssl configuration options (ssl-ca[path], ssl-cert and ssl-key) are ignored b
          		 
          y SST due to presence of the tca[path], tcert and/or tkey in the [sst] section (20250110 12:44:06.470)
          		 
          mariadb WSREP_SST: [INFO] SSL configuration: CA='/etc/pki/ca.crt', CAPATH='', CERT='/etc/pki/client.crt',
          		 
          KEY='/etc/pki/client.key', MODE='DISABLED', encrypt='3' (20250110 12:44:06.473)
          		 
          mariadb WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt (20250110 12:44:06.6
          		 
          11)
          		 
          mariadb WSREP_SST: [INFO] Evaluating '/usr//bin/mbstream' -c 'mariadb_backup_galera_info' 'donor_galera_in
          		 
          fo' | socat -u stdio openssl-connect:10.244.0.19:4444,no-sni=1,cert='/etc/pki/client.crt',key='/etc/pki/cl
          		 
          ient.key',cafile='/etc/pki/ca.crt',commonname='10.244.0.19'; RC=( ${PIPESTATUS[@]} ) (20250110 12:44:06.71
          		 
          4)
          		 
          mariadb 2025/01/10 12:44:06 socat[1216] W OpenSSL: Warning: this implementation does not check CRLs
          		 
          mariadb 2025-01-10 12:44:07 0 [Note] WSREP: forgetting 936276b2-8d30 (ssl://10.244.0.19:4567)
          		 
          mariadb 2025-01-10 12:44:07 0 [Note] WSREP: forgetting 936276b2-8d30 (ssl://10.244.0.19:4567)
          		 
          mariadb 2025-01-10 12:44:08 0 [Note] WSREP: (622e7688-b998, 'ssl://0.0.0.0:4567') turning message relay re
          		 
          questing off
          		 
          mariadb 2025-01-10 12:44:12 0

          The crash still happens in the donor with the same error:

          mariadb 250110 12:49:06 [ERROR] mysqld got signal 11 ;
          		 
          mariadb Sorry, we probably made a mistake, and this is a bug.
          		 
          mariadb
          		 
          mariadb Your assistance in bug reporting will enable us to fix this for the next release.
          		 
          mariadb To report this bug, see https://mariadb.com/kb/en/reporting-bugs
          		 
          mariadb
          		 
          mariadb We will try our best to scrape up some info that will hopefully help
          		 
          mariadb diagnose the problem, but since we have already crashed,
          		 
          mariadb something is definitely wrong and this may fail.
          		 
          mariadb
          		 
          mariadb Server version: 11.4.4-MariaDB-ubu2404 source revision: e9a502df08bad16aa8a354e854f3c014b1380e32
          		 
          mariadb key_buffer_size=0
          		 
          mariadb read_buffer_size=131072
          		 
          mariadb max_used_connections=0
          		 
          mariadb max_threads=153
          		 
          mariadb thread_count=2
          		 
          mariadb It is possible that mysqld could use up to
          		 
          mariadb key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 336997 K  bytes of memory
          		 
          mariadb Hope that's ok; if not, decrease some variables in the equation.
          		 
          mariadb
          		 
          mariadb WSREP: Suppressing further logging
          		 
          mariadb WSREP: Shutting down network communications
          		 
          mariadb
          		 
          mariadb Thread pointer: 0x0
          		 
          mariadb Attempting backtrace. You can use the following information to find out
          		 
          mariadb where mysqld died. If you see no messages after this, something went
          		 
          mariadb terribly wrong...
          		 
          mariadb stack_bottom = 0x0 thread_stack 0x49000
          		 
          mariadb 2025-01-10 12:49:06 2 [Note] WSREP: gcomm: terminating thread
          		 
          mariadb 2025-01-10 12:49:06 2 [Note] WSREP: gcomm: joining thread
          		 
          mariadb 2025-01-10 12:49:06 2 [Note] WSREP: gcomm: closing backend

          I see that we are passing the container IP as `commonname='10.244.0.19';` , whereas the certificate is not valid for the container IP, as IPs are ephemeral in Kubernetes. Could this be related? In any case, it shouldn't be causing a crash I'm guessing?

          martin.montes Martin Montes added a comment - I have enabled encryption for the SST: [sst] encrypt= 3 tca=/etc/pki/ca.crt tcert=/etc/pki/client.crt tkey=/etc/pki/client.key Resulting in the following logs in the donor: mariadb 2025 - 01 - 10 12 : 43 : 37 0 [Note] WSREP: async IST sender starting to serve ssl: //10.244.0.19:4568 send ing 6 - 6 , preload starts from 6 mariadb WSREP_SST: [INFO] new ssl configuration options (ssl-ca[path], ssl-cert and ssl-key) are ignored b y SST due to presence of the tca[path], tcert and/or tkey in the [sst] section ( 20250110 12 : 43 : 37.536 ) mariadb WSREP_SST: [INFO] SSL configuration: CA= '/etc/pki/ca.crt' , CAPATH= '' , CERT= '/etc/pki/client.crt' , KEY= '/etc/pki/client.key' , MODE= 'DISABLED' , encrypt= '3' ( 20250110 12 : 43 : 37.540 ) mariadb WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt ( 20250110 12 : 43 : 37.6 76 ) mariadb WSREP_SST: [INFO] Evaluating '/usr//bin/mbstream' -c 'mariadb_backup_galera_info' 'donor_galera_in fo ' | socat -u stdio openssl-connect:10.244.0.19:4444,no-sni=1,cert=' /etc/pki/client.crt ',key=' /etc/pki/cl ient.key ',cafile=' /etc/pki/ca.crt ',commonname=' 10.244 . 0.19 '; RC=( ${PIPESTATUS[@]} ) ( 20250110 12 : 43 : 37.76 4 ) mariadb 2025 / 01 / 10 12 : 43 : 37 socat[ 923 ] W OpenSSL: Warning: this implementation does not check CRLs mariadb 2025 - 01 - 10 12 : 43 : 38 0 [Note] WSREP: forgetting 821b0db3-a24b (ssl: //10.244.0.19:4567) mariadb 2025 - 01 - 10 12 : 43 : 38 0 [Note] WSREP: forgetting 821b0db3-a24b (ssl: //10.244.0.19:4567) mariadb 2025 - 01 - 10 12 : 43 : 39 0 [Note] WSREP: (622e7688-b998, 'ssl://0.0.0.0:4567' ) turning message relay re questing off mariadb 2025 - 01 - 10 12 : 43 : 43 0 [Note] WSREP: cleaning up 821b0db3-a24b (ssl: //10.244.0.19:4567) mariadb 2025 - 01 - 10 12 : 44 : 04 0 [Note] WSREP: (622e7688-b998, 'ssl://0.0.0.0:4567' ) connection established t o 936276b2-8d30 ssl: //10.244.0.19:4567 mariadb 2025 - 01 - 10 12 : 44 : 05 0 [Note] WSREP: declaring 936276b2-8d30 at ssl: //10.244.0.19:4567 stable mariadb 2025 - 01 - 10 12 : 44 : 06 0 [Note] WSREP: async IST sender starting to serve ssl: //10.244.0.19:4568 send ing 8 - 8 , preload starts from 8 mariadb WSREP_SST: [INFO] new ssl configuration options (ssl-ca[path], ssl-cert and ssl-key) are ignored b y SST due to presence of the tca[path], tcert and/or tkey in the [sst] section ( 20250110 12 : 44 : 06.470 ) mariadb WSREP_SST: [INFO] SSL configuration: CA= '/etc/pki/ca.crt' , CAPATH= '' , CERT= '/etc/pki/client.crt' , KEY= '/etc/pki/client.key' , MODE= 'DISABLED' , encrypt= '3' ( 20250110 12 : 44 : 06.473 ) mariadb WSREP_SST: [INFO] Using openssl based encryption with socat: with key and crt ( 20250110 12 : 44 : 06.6 11 ) mariadb WSREP_SST: [INFO] Evaluating '/usr//bin/mbstream' -c 'mariadb_backup_galera_info' 'donor_galera_in fo ' | socat -u stdio openssl-connect:10.244.0.19:4444,no-sni=1,cert=' /etc/pki/client.crt ',key=' /etc/pki/cl ient.key ',cafile=' /etc/pki/ca.crt ',commonname=' 10.244 . 0.19 '; RC=( ${PIPESTATUS[@]} ) ( 20250110 12 : 44 : 06.71 4 ) mariadb 2025 / 01 / 10 12 : 44 : 06 socat[ 1216 ] W OpenSSL: Warning: this implementation does not check CRLs mariadb 2025 - 01 - 10 12 : 44 : 07 0 [Note] WSREP: forgetting 936276b2-8d30 (ssl: //10.244.0.19:4567) mariadb 2025 - 01 - 10 12 : 44 : 07 0 [Note] WSREP: forgetting 936276b2-8d30 (ssl: //10.244.0.19:4567) mariadb 2025 - 01 - 10 12 : 44 : 08 0 [Note] WSREP: (622e7688-b998, 'ssl://0.0.0.0:4567' ) turning message relay re questing off mariadb 2025 - 01 - 10 12 : 44 : 12 0 The crash still happens in the donor with the same error: mariadb 250110 12 : 49 : 06 [ERROR] mysqld got signal 11 ; mariadb Sorry, we probably made a mistake, and this is a bug. mariadb mariadb Your assistance in bug reporting will enable us to fix this for the next release. mariadb To report this bug, see https: //mariadb.com/kb/en/reporting-bugs mariadb mariadb We will try our best to scrape up some info that will hopefully help mariadb diagnose the problem, but since we have already crashed, mariadb something is definitely wrong and this may fail. mariadb mariadb Server version: 11.4 . 4 -MariaDB-ubu2404 source revision: e9a502df08bad16aa8a354e854f3c014b1380e32 mariadb key_buffer_size= 0 mariadb read_buffer_size= 131072 mariadb max_used_connections= 0 mariadb max_threads= 153 mariadb thread_count= 2 mariadb It is possible that mysqld could use up to mariadb key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 336997 K bytes of memory mariadb Hope that's ok; if not, decrease some variables in the equation. mariadb mariadb WSREP: Suppressing further logging mariadb WSREP: Shutting down network communications mariadb mariadb Thread pointer: 0x0 mariadb Attempting backtrace. You can use the following information to find out mariadb where mysqld died. If you see no messages after this , something went mariadb terribly wrong... mariadb stack_bottom = 0x0 thread_stack 0x49000 mariadb 2025 - 01 - 10 12 : 49 : 06 2 [Note] WSREP: gcomm: terminating thread mariadb 2025 - 01 - 10 12 : 49 : 06 2 [Note] WSREP: gcomm: joining thread mariadb 2025 - 01 - 10 12 : 49 : 06 2 [Note] WSREP: gcomm: closing backend I see that we are passing the container IP as `commonname='10.244.0.19';` , whereas the certificate is not valid for the container IP, as IPs are ephemeral in Kubernetes. Could this be related? In any case, it shouldn't be causing a crash I'm guessing?
          martin.montes Martin Montes added a comment -

          I've tried with certificates issued by a root CA, and with encrypt=3 I am running into the same issue. Maybe there is something wrong with SST TLS in general. Are ECDSA private keys supported?

          martin.montes Martin Montes added a comment - I've tried with certificates issued by a root CA, and with encrypt=3 I am running into the same issue. Maybe there is something wrong with SST TLS in general. Are ECDSA private keys supported?
          martin.montes Martin Montes added a comment -

          By the way, double checked that the certificates are valid:

          mysql@mariadb-galera-0:/$ openssl verify -CAfile /etc/pki/ca.crt /etc/pki/server.crt
          		 
          /etc/pki/server.crt: OK
          		 
          mysql@mariadb-galera-0:/$ openssl verify -CAfile /etc/pki/ca.crt /etc/pki/client.crt
          		 
          /etc/pki/client.crt: OK

          martin.montes Martin Montes added a comment - By the way, double checked that the certificates are valid: mysql @mariadb -galera- 0 :/$ openssl verify -CAfile /etc/pki/ca.crt /etc/pki/server.crt /etc/pki/server.crt: OK mysql @mariadb -galera- 0 :/$ openssl verify -CAfile /etc/pki/ca.crt /etc/pki/client.crt /etc/pki/client.crt: OK
          martin.montes Martin Montes added a comment - - edited

          Managed to make it work with certificates issued by a root CA and encrypt=3. The issue is that the script wsrep_sst_mariabackup.sh, by default, verifies that the certificate is valid for the container IP, something not valid for our case. Since in Kubernetes IPs are ephemeral, we cannot issue a certificate for an IP, as we cannot know it beforehand.

          To overcome this, after looking at wsrep_sst_common.sh as well, we have set WSREP_SST_OPT_REMOTE_AUTH="<client-cert-common-name>:". This way, instead of verifying against the IP, the commonName of the certificate is used. Source:
          https://github.com/codership/mariadb-server/blob/16394f1aa1b4097f897b8ab01ea2064726cca059/scripts/wsrep_sst_common.sh#L1064
          https://github.com/codership/mariadb-server/blob/16394f1aa1b4097f897b8ab01ea2064726cca059/scripts/wsrep_sst_mariabackup.sh#L407

          I couldn't find any documentation about the WSREP_SST_OPT_REMOTE_AUTH variable. Is this a valid approach for all the MariaDB versions?

          Unfortunately, when providing certificates issued by an intermediate CA the issue persists, therefore this JIRA still applies.

          martin.montes Martin Montes added a comment - - edited Managed to make it work with certificates issued by a root CA and encrypt=3. The issue is that the script wsrep_sst_mariabackup.sh, by default, verifies that the certificate is valid for the container IP, something not valid for our case. Since in Kubernetes IPs are ephemeral, we cannot issue a certificate for an IP, as we cannot know it beforehand. To overcome this, after looking at wsrep_sst_common.sh as well, we have set WSREP_SST_OPT_REMOTE_AUTH="<client-cert-common-name>:". This way, instead of verifying against the IP, the commonName of the certificate is used. Source: https://github.com/codership/mariadb-server/blob/16394f1aa1b4097f897b8ab01ea2064726cca059/scripts/wsrep_sst_common.sh#L1064 https://github.com/codership/mariadb-server/blob/16394f1aa1b4097f897b8ab01ea2064726cca059/scripts/wsrep_sst_mariabackup.sh#L407 I couldn't find any documentation about the WSREP_SST_OPT_REMOTE_AUTH variable. Is this a valid approach for all the MariaDB versions? Unfortunately, when providing certificates issued by an intermediate CA the issue persists, therefore this JIRA still applies.
          pasztorl Lenard Pasztor added a comment -

          We also use intermediate CA-s. Can you please support this case? Thanks!

          pasztorl Lenard Pasztor added a comment - We also use intermediate CA-s. Can you please support this case? Thanks!

          People

            Unassigned Unassigned
            martin.montes Martin Montes
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.