https://jira.percona.com/browse/PS-3332 was closed with "Cannot Reproduce".

fork() does not duplicate the whole process memory, that would be too slow to be practically usable. It only copies changed pages, so if one of the processes rewrites the whole of innodb buffer pool — it'll be duplicated, yes. But the child process calls exec() immediately after the fork(). The parent cannot do much in that short time frame.

fork() does not duplicate the memory, but it kind of reserves it. Without overcommit this will be too much for big mariadb processes. I found a post which describes the issue quite good:

https://mail.gnome.org/archives/gtk-devel-list/2018-April/msg00000.html

When gnome-shell is launching apps (via glib) it ultimately comes down
to fork() + exec(). In this case the fork() fails with ENOMEM, because
the Linux kernel worries that the process being forked may end up
duplicating all of the memory allocations of the shell. By default the
memory map is set up so that the new process has a view on the exact
same pages as the parent process, however they are set up as
copy-on-write, so if the child writes to such memory it'll silently
cause new memory to be allocated. Under that limited perspective, the
kernel is not totally out of line in worrying about this situation,
especially because gnome-shell is a RAM-heavy process.

In reality we only want to fork() to immediately exec() which replaces
the child process memory map with a blank slate, but this
misinterpretation of intentions is a limitation of the fork() API
combined with a conflict with Linux's memory overcommit model.

I attached a patch that replaces fork() + exec() by posix_spawn() with which I could no longer reproduce the bug and could login via PAM even when the server process used more than 50% of available RAM.
However, as I am not very familiar with C, memory management and such, I cannot rule out side effects.

Cheers
Volker

Thanks. I've replaced fork() + exec() with posix_spawn()