AES_ENCRYPT() and KDF() broken with OpenSSL 4.0: EVP_CIPHER_CTX stack allocation hack fails for non-ECB cipher modes

Problem

Building MariaDB against OpenSSL 4.0.1 (shipped in Fedora 45 rawhide) causes AES_ENCRYPT() / AES_DECRYPT() with explicit cipher-name argument (4-arg form from MDEV-9069) and block_encryption_mode set to non-ECB modes (CBC, CTR) to silently return NULL instead of encrypted/decrypted data. KDF() output used with AES_ENCRYPT() also returns NULL.

Standard 2-arg AES_ENCRYPT() in default ECB mode still works. Standalone KDF() calls (both HKDF and PBKDF2) still work.

This is broken functionality, not a test-only issue. The SQL functions return NULL to the user where they previously returned valid data.

Failing tests

• main.func_str – aes_encrypt('foo', 'bar', '0123456789abcdef', 'aes-256-cbc') returns NULL instead of 42A3EB91E6DFC40A900D278F99E0726E
• main.func_kdf – kdf('password', 'salt', 2048) + aes_encrypt('secret', @key, '1234123412341234') returns NULL instead of 9EED553CDDEE426D5635EF559E015ECA

Root cause 1: EVP_CIPHER_CTX stack allocation in mysys_ssl/my_crypt.cc

The MyCTX class pre-allocates a 200-byte stack buffer (EVP_CIPHER_CTX_SIZE defined in include/ssl_compat.h) and casts it to EVP_CIPHER_CTX*, bypassing EVP_CIPHER_CTX_new():

#define CTX_ALIGN 16

class MyCTX

{

public:

  char ctx_buf[EVP_CIPHER_CTX_SIZE + CTX_ALIGN];

  EVP_CIPHER_CTX* ctx;

  MyCTX()

  {

    uintptr_t p= ((uintptr_t)ctx_buf + (CTX_ALIGN - 1)) & ~(CTX_ALIGN - 1);

    ctx = reinterpret_cast<EVP_CIPHER_CTX*>(p);

    EVP_CIPHER_CTX_init(ctx);  // memset(0, 200) + EVP_CIPHER_CTX_reset()

  }

This hack was introduced for OpenSSL 1.1 when EVP_CIPHER_CTX became an opaque struct. It worked in OpenSSL 1.1.x and 3.x because the internal struct size stayed within 200 bytes and EVP_CIPHER_CTX_reset() on a zeroed buffer happened to initialize it well enough.

In OpenSSL 4.0, EVP_CipherInit_ex() called on such a raw stack buffer returns 0 for non-ECB cipher modes, causing MY_AES_OPENSSL_ERROR and a NULL return from the SQL function.

Root cause 2: check_openssl_compatibility() bypass in mysys_ssl/openssl.c

The runtime safety check that validates EVP_CIPHER_CTX_new() allocates \<= 200 bytes is silently bypassed in OpenSSL 4.0:

int check_openssl_compatibility()

{

  if (!CRYPTO_set_mem_functions(coc_malloc, coc_realloc, coc_free))

    return 0;  // <-- bypassed here: returns "no error"

  ...

}

When CRYPTO_set_mem_functions() returns 0 (which happens in OpenSSL 4.0 when custom memory allocators cannot be set after library initialization), the function returns 0 (success) without performing any size check. The server starts without detecting the incompatibility.

Why upstream buildbot does not catch this

No upstream CI system uses OpenSSL 4.0 yet. The MariaDB buildbot uses Ubuntu, Debian, and other distributions that ship OpenSSL 3.x. Only Fedora 45 (rawhide) ships OpenSSL 4.0.1 as of 2026-07.

Proposed fix

1. mysys_ssl/my_crypt.cc: Replace the stack-allocated buffer with EVP_CIPHER_CTX_new() / EVP_CIPHER_CTX_free(). These functions have been available since OpenSSL 0.9.8, so this is fully backward compatible. Add a NULL check in MyCTX::init() to handle allocation failure.

   class MyCTX

   {

   public:

   EVP_CIPHER_CTX* ctx;

   MyCTX() : ctx(EVP_CIPHER_CTX_new()) {}

   virtual ~MyCTX()

   {

   EVP_CIPHER_CTX_free(ctx);

   ERR_remove_state(0);

   }

   virtual int init(...)

   {

   ...

   if (unlikely(!ctx))

   return MY_AES_OPENSSL_ERROR;

   ...

   }

2. mysys_ssl/openssl.c: Remove the EVP_CIPHER_CTX allocation size check from check_openssl_compatibility() since it is no longer relevant. Keep the EVP_MD_CTX check (still used by my_md5.cc).
3. sql/item_strfunc.cc (separate but related): The KDF() function uses EVP_sha512() which is a legacy API deprecated in OpenSSL 3.0. For OpenSSL >= 3.0, use EVP_MD_fetch(NULL, "SHA512", NULL) + EVP_MD_free() instead, guarded by OPENSSL_VERSION_NUMBER >= 0x30000000L. This applies to both the PBKDF2 path (PKCS5_PBKDF2_HMAC) and the HKDF path (EVP_PKEY_CTX_set_hkdf_md).

Reproducer

Build MariaDB 11.8.8 against OpenSSL 4.0.1 (e.g. Fedora 45 rawhide) and run:

SELECT HEX(AES_ENCRYPT('foo', 'bar', '0123456789abcdef', 'aes-256-cbc'));

-- Expected: 42A3EB91E6DFC40A900D278F99E0726E

-- Actual:   NULL

SET @key = KDF('password', 'salt', 2048);

SELECT HEX(AES_ENCRYPT('secret', @key, '1234123412341234'));

-- Expected: 9EED553CDDEE426D5635EF559E015ECA

-- Actual:   NULL

Related tickets

• MDEV-9069 – extend AES_ENCRYPT() and AES_DECRYPT() to support IV and the algorithm (introduced the 4-arg form that fails)
• MDEV-28339 – crashes with OpenSSL 3.0.2 (same EVP_CIPHER_CTX stack allocation root cause)
• MDEV-15587 – AES test fails, segfaults in EVP_CipherInit_ex (historical, same area)
• MDEV-25785 – add support for OpenSSL 3.0 (general OpenSSL compat tracking)