max_session_mem_used setting does not prevent large memory allocations in some cases.

When max_session_mem_used is applied to limit the maximum memory use in a query, it is still possible for the mysqld memory footprint to go significantly past that limit. The reason is that when the limit is reached, my_malloc() does not return a failure and still performs the new allocation. The failure is registered by setting the KILLED state of the query thus relying on the code up the stack to check that state and abort the query. But this check may not happen for a while, and there are code paths that could perform significant memory allocations past the limit resulting in a very undesirable memory footprint of mysqld and potentially causing a crash on operating systems like Linux that use overcommit and OOM killer. I have a practical repro which I cannot yet post as I have not yet isolated sensitive customer data from it.

I understand the reason it was implemented this way. Overall the query execution engine code is not very robust against memory allocation failures. Thus returning a hard NULL from my_malloc() could in practice cause more harm by making it hit an edge case with higher probability than a delay in the query termination and the "bonus" memory allocations. But what this means is that we are tacitly acknowledging that the code is not tolerant against my_malloc() returning NULL, which is fundamentally bad. Ideally we should not be sweeping this problem under the rug, but rather take methodical steps to correct it. I propose that we add a special option - something like max_session_mem_used_hard_enforce - false by default, if true my_malloc() should return NULL when the limit is exceeded. And then start testing with this option and fix bugs adding robustness as we go.