Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
11.4.4
-
None
-
None
-
MariaDB Community container running in Kubernetes v1.31
Description
Hey there! I am providing the following certificate to a MariaDB Community 11.4.4. It consists of 2 PEM blocks, in this order:
- Leaf certificate
- Intermediate CA certificate
The certificates have been issued by cert-manager which using the go x509 library to issue certificates.
It seems like Galera is not able to parse multiple PEM blocks, returning the following error when setting up the donor:
mariadb 2025-01-09 18:02:20 0 [Note] WSREP: Failed to establish connection: unexpected eof while reading (SSL routines) |
The donor node comes up anyway and when the SST is requested by a joiner, a crash happens. (see attached logs)
Including an intermediate CA as part of the certificate is valid and accepted by MariaDB. Standalone and replication topologies work normally with MariaDB 11.4.4. I have tested previous versions (11.4.3), but they also result in SSL errors (but not parsing errors) when this certificate structure is provided. It is also a very common PKI practice nowadays, as it allows to build trust in complex scenarios where multiple intermediate CAs are involved.
I have found the following issues during my investigation, which might provide some context:
I have attached the logs, configuration files and the PKI material.
Attachments
Activity
I've tried with certificates issued by a root CA, and with encrypt=3 I am running into the same issue. Maybe there is something wrong with SST TLS in general. Are ECDSA private keys supported?
By the way, double checked that the certificates are valid:
mysql@mariadb-galera-0:/$ openssl verify -CAfile /etc/pki/ca.crt /etc/pki/server.crt |
/etc/pki/server.crt: OK
|
mysql@mariadb-galera-0:/$ openssl verify -CAfile /etc/pki/ca.crt /etc/pki/client.crt |
/etc/pki/client.crt: OK
|
Managed to make it work with certificates issued by a root CA and encrypt=3. The issue is that the script wsrep_sst_mariabackup.sh, by default, verifies that the certificate is valid for the container IP, something not valid for our case. Since in Kubernetes IPs are ephemeral, we cannot issue a certificate for an IP, as we cannot know it beforehand.
To overcome this, after looking at wsrep_sst_common.sh as well, we have set WSREP_SST_OPT_REMOTE_AUTH="<client-cert-common-name>:". This way, instead of verifying against the IP, the commonName of the certificate is used. Source:
https://github.com/codership/mariadb-server/blob/16394f1aa1b4097f897b8ab01ea2064726cca059/scripts/wsrep_sst_common.sh#L1064
https://github.com/codership/mariadb-server/blob/16394f1aa1b4097f897b8ab01ea2064726cca059/scripts/wsrep_sst_mariabackup.sh#L407
I couldn't find any documentation about the WSREP_SST_OPT_REMOTE_AUTH variable. Is this a valid approach for all the MariaDB versions?
Unfortunately, when providing certificates issued by an intermediate CA the issue persists, therefore this JIRA still applies.
We also use intermediate CA-s. Can you please support this case? Thanks!
I have enabled encryption for the SST:
[sst]tca=/etc/pki/ca.crttcert=/etc/pki/client.crttkey=/etc/pki/client.keyResulting in the following logs in the donor:
questing offquesting offThe crash still happens in the donor with the same error:
mariadbmariadbmariadb diagnose the problem, but since we have already crashed,mariadbmariadb It is possible that mysqld could use up tomariadbmariadb WSREP: Suppressing further loggingmariadb WSREP: Shutting down network communicationsmariadbmariadb Attempting backtrace. You can use the following information to find outmariadb terribly wrong...I see that we are passing the container IP as `commonname='10.244.0.19';` , whereas the certificate is not valid for the container IP, as IPs are ephemeral in Kubernetes. Could this be related? In any case, it shouldn't be causing a crash I'm guessing?