PAM authentication fails with ENOMEM

When using the authentication plugin PAM it fails in case the mariadb process uses more memory than is still free and memory overcommit is disabled.

The reason lies in fork() (https://github.com/MariaDB/server/blob/10.7/plugin/auth_pam/auth_pam.c#L62) which duplicates the process. As there is not enough free memory for a duplicate process fork() fails (silently) and the authentication does not succeed.

As the forked process is only used to exec() the pam tool, there is no need for a fork of the whole server process.

It can be reproduced by running a mariadb server with an innodb_buffer of more than half of the server's (best a VM with as few RAM as possible) RAM and filling it up by selecting a big table of test data.

This issue had already been reported in Percona server: https://jira.percona.com/browse/PS-3332

Cheers
Volker