[MDEV-30092] SSL Error after Update Created: 2022-11-25  Updated: 2023-03-07  Resolved: 2022-12-02

Status: Closed
Project: MariaDB Server
Component/s: Platform Debian, SSL
Affects Version/s: 10.3.37
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Christian Assignee: Sergei Golubchik
Resolution: Not a Bug Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-30270 ssl_cipher on Non-SSL system results ... Open

 Description   

Today my apt-cron installed the newest release for Debian:

Log started: 2022-11-25  06:16:41
(Reading database ... 125540 files and directories currently installed.)
Preparing to unpack .../mariadb-server-core-10.3_1%3a10.3.37-0ubuntu0.20.04.1_amd64.deb ...
Unpacking mariadb-server-core-10.3 (1:10.3.37-0ubuntu0.20.04.1) over (1:10.3.34-0ubuntu0.20.04.1) ...
Preparing to unpack .../mariadb-server-10.3_1%3a10.3.37-0ubuntu0.20.04.1_amd64.deb ...
/var/lib/mysql: found previous version 10.3
Unpacking mariadb-server-10.3 (1:10.3.37-0ubuntu0.20.04.1) over (1:10.3.34-0ubuntu0.20.04.1) ...
Preparing to unpack .../mariadb-server_1%3a10.3.37-0ubuntu0.20.04.1_all.deb ...
Unpacking mariadb-server (1:10.3.37-0ubuntu0.20.04.1) over (1:10.3.34-0ubuntu0.20.04.1) ...
Setting up mariadb-server-core-10.3 (1:10.3.37-0ubuntu0.20.04.1) ...
Setting up mariadb-server-10.3 (1:10.3.37-0ubuntu0.20.04.1) ...
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
Setting up mariadb-server (1:10.3.37-0ubuntu0.20.04.1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.19) ...
Log ended: 2022-11-25  06:17:07

Since this release the MariaDB Service will not start.

In Logs I can found:

tail /var/log/mysql/error.log
2022-11-25 11:51:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-11-25 11:51:02 0 [Note] InnoDB: Waiting for purge to start
2022-11-25 11:51:02 0 [Note] InnoDB: 10.3.37 started; log sequence number 17380273060; transaction id 59869107
2022-11-25 11:51:02 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-11-25 11:51:02 0 [Note] Plugin 'FEEDBACK' is disabled.
SSL error: Unable to get private key from '/etc/ssl/wildcard.key'
2022-11-25 11:51:02 0 [ERROR] Failed to setup SSL
2022-11-25 11:51:02 0 [ERROR] SSL error: Unable to get private key
2022-11-25 11:51:02 0 [ERROR] Aborting

When I deactivate SSL in config everything is fine again.
Rollback to a Backup is also working.



 Comments   
Comment by Sergei Golubchik [ 2022-11-27 ]

The error says it

Unable to get private key from '/etc/ssl/wildcard.key'

Do you have such a file? Does it have a valid private key?

This is likely the effect of MDEV-29811 — in previous releases the server ignored incorrect SSL setup and, depending on the client config, it was either pretending to be more secure than it was or it was impossible to connect to altogether.

Now the server refuses to start if the SSL was configured incorrectly, if SSL was requested, the server will not start without it.

Comment by Christian [ 2022-11-27 ]

Yes - file is at this path, was used by the service before.

Also it is used by other services like postfix - so I can confirm the certificate is valide and also my mailclient works with it.

Permission are for All readable (ugo+r).
In kernel and last log is nothing that indicates this error. apparmore is not active for this service.

Greetings
Christian

Comment by Sergei Golubchik [ 2022-12-02 ]

Do you use distro mariadb packages? They might be compiled with yassl, and yassl can refuse to load keys that openssl can. You've always had this error, just after in 10.3.37 it became visible, earlier versions simply disabled SSL and pretended than everything's fine, and 10.3.37 refuses to lie about it.

Search for "Unable to get private key" — I've got quite a few hits, both for MariaDB and MySQL, for older versions too. For example, https://bugs.mysql.com/bug.php?id=71271

Comment by Stefan [ 2022-12-20 ]

Same happened when upgrading from 10.5.15-0+deb11u1 to 10.5.18-0+deb11u1 on Debian 11. This is a really critical bug as our configs (managed through Puppet) include ssl = false in the config, so naturally one would expect that the ssl-cert etc. parameters would be ignored. For a patch update to change such a functionality to bring down all MySQL servers is not ideal.
I also tried skip-ssl and ssl = 0 but none of these options helped letting the server start up.

Comment by Faustin Lammler [ 2022-12-20 ]

I have just tested this and indeed, the server still tries to activate ssl (even if

ssl = false

or

skip-ssl

is used and further ssl directives are in the configuration file:

[client-server]
ssl = false
skip-ssl
[...]
ssl-ca = /etc/mysql/cacert.pem
ssl-cert = /etc/mysql/server-cert.pem
ssl-key = /etc/mysql/server-key.pem

I am not sure if this is on purpose though, my understanding from https://mariadb.com/kb/en/mysqld-options/#-ssl is that

skip-ssl

should disable any further ssl options but it also mention

automatically enabled with other flags

?

Anyway, I have forwarded https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026353 to this jira issue and regarding the puppet deployment tool, this comment is correct, there is no reason to deploy extra ssl directives if it's supposed to be disabled in the first place IMO.

Comment by Otto Kekäläinen [ 2023-03-07 ]

This was also related to https://bugs.launchpad.net/ubuntu/+source/mariadb-10.3/+bug/1997916 and MDEV-29811

Generated at Thu Feb 08 10:13:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.