The size of DH (Diffie Hellmann) group parameter in MariaDB Server cannot be changed, since it uses a fixed size of 2048.
Citing OpenSSL Wiki:
"Your Diffie-Hellman group parameters should match the key size used in the server's certificate. If you use a 2048-bit RSA prime in the server's certificate, then use a 2048-bit Diffie-Hellman group for key agreement."
Citing manpage for SSL_CTX_set_tmp_dh
"Applications may supply their own DH parameters instead of using the built-in values.
This approach is discouraged and applications should in preference use the built-in parameter support described above.
If "auto" DH parameters are switched on then the parameters will be selected to be consistent with the size of the key associated with the server's certificate.”
So easiest solution would be to use SSL_CTX_set_auto_dh() instead.
Please also note that the current code for DH params doesn't work with OSSL3 anymore.