[MDEV-26542] Memory allocation pre-flight check feature Created: 2021-09-05  Updated: 2021-09-07

Status: Open
Project: MariaDB Server
Component/s: Server
Fix Version/s: None

Type: Task Priority: Minor
Reporter: Roel Van de Paar Assignee: Sergei Golubchik
Resolution: Unresolved Votes: 1
Labels: affects-tests, memory


 Description   

A command like

SET @@GLOBAL.key_buffer_size=10000000000000000000;

Cannot possibly succeed when a limited amount of free memory is available.

However, both optimized and debug builds will immediately commence executing the command and attempt memory allocation, thereby quickly and needlessly filling up all available memory, leading to OOM termination by the kernel of either this process or any other running on the server, and all consequences thereof.

This feature request for a simple pre-flight check. Pseudo code:

IF requested_alloc_amount > (free_mem * 1.3) THEN print_error AND do_not_execute_request;
 
ELSEIF requested_alloc_amount > (free_mem * 0.87) THEN print_warning AND execute_request;
 
ELSE execute_request;



 Comments   
Comment by Daniel Black [ 2021-09-07 ]

Problems on "free memory" checks is it can be a far margin from what is measure to what is really free once normal operations begin. There's also a disparity between allocated (in a virtual memory space manner) and paged in. There's also the potential for VMs to be resized with more ram (balooning), and other effects.

With the amount of things that can go wrong, I think its over engineering to try to guess that amount upfrount.

I'd suggest a lazy allocation approach like MDEV-25340, lazily initialize each of the key blocks as needed. Its a quick check ahead of a moderately quick initialization and this would allow a very large allocation to not consume very much memory until its actually used. This would also help with MDEV-16607, by even with the default values, only consuming what is used rather than being wasted with amounts of initialized cache that won't be used.

Tied in with the mechanisms of MDEV-24670, we could start to purge off the key buffer size down to a frequently used threshold as pressure rises.

Generated at Thu Feb 08 09:46:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.