Details
-
Task
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
Description
Allocating sufficiently available memory but not too much has been a challenge for MariaDB and MySQL users for a significant time.
The overallocation results in a risk of OOM killing of the server process.
Linux kernel this interface:
https://www.kernel.org/doc/html/latest/accounting/psi.html
By receiving a memory event of memory pressure options are:
- madvice(DONT_NEED/FREE) on buf_pool.free
- Moving innodb buffer LRU pool chunks to free
- Resizing down innodb buffer pool
- Kill off thread cache thread(s)?
- table caches (are these large?)
So configuration would need a trigger values, "<some|full> <stall amount in us> <time window in us>", from the kernel interface. And a response for "some"/"full".
Attachments
Issue Links
- blocks
-
MDEV-31953 madvise(frame, srv_page_size, MADV_FREE) is causing a performance regression
-
- Closed
-
- causes
-
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Open
-
-
-
- Closed
-
-
-
- Open
-
-
-
- Needs Feedback
-
-
-
- Open
-
-
-
- Closed
-
-
-
- Open
-
-
-
- Open
-
-
-
- Closed
-
-
MDEV-36197 Implement Buffer Pool Auto-Scaling Based on RAM Availability
-
- Open
-
(8 relates to)
Activity
Looking to reuse the code as an alternate trigger to marko's fix in MDEV-31593.
Preliminary results of RQG testing
|
---------------------------------------------------
|
origin/bb-10.11-MDEV-24670-memory-pressure 7d941227f6b7031edceaa6abef90e178091ac172 2023-10-28T19:16:00+11:00
|
Build with debug+asan, PERFSCHEMA enabled
|
I got
|
1. quite frequent when running a test based on the RQG grammar conf/mariadb/innodb_compression_encryption.yy
|
mariadbd: /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/buf0buf.h:2031: void buf_page_t::clear_oldest_modification(): Assertion `oldest_modification()' failed.
|
(rr) bt
|
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=65511758501440) at ./nptl/pthread_kill.c:44
|
#1 __pthread_kill_internal (signo=6, threadid=65511758501440) at ./nptl/pthread_kill.c:78
|
#2 __GI___pthread_kill (threadid=65511758501440, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
|
#3 0x000000006fe17476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
|
#4 0x000000006fdfd7f3 in __GI_abort () at ./stdlib/abort.c:79
|
#5 0x000000006fdfd71b in __assert_fail_base (fmt=0x6ffb2150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5626d9f33680 "oldest_modification()",
|
file=0x5626d9f323a0 "/data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/buf0buf.h", line=2031, function=<optimized out>) at ./assert/assert.c:92
|
#6 0x000000006fe0ee96 in __GI___assert_fail (assertion=0x5626d9f33680 "oldest_modification()", file=0x5626d9f323a0 "/data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/buf0buf.h", line=2031,
|
function=0x5626d9f335e0 "void buf_page_t::clear_oldest_modification()") at ./assert/assert.c:101
|
#7 0x00005626d90408c4 in buf_page_t::clear_oldest_modification (this=this@entry=0x11696ad51700) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/buf0buf.h:2031
|
#8 0x00005626d902c845 in buf_pool_t::delete_from_flush_list (this=this@entry=0x5626dacadb40 <buf_pool>, bpage=bpage@entry=0x11696ad51700)
|
at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0flu.cc:170
|
#9 0x00005626d9001965 in buf_pool_t::garbage_collect (this=this@entry=0x5626dacadb40 <buf_pool>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:2130
|
#10 0x00005626d8fe4a68 in mem_pressure::trigger_collection (this=0x5626db7c7520 <mem_pressure_obj>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:906
|
#11 buf_resize_start () at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:2189
|
#12 0x00005626d891b434 in innodb_buffer_pool_size_update (save=<optimized out>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/handler/ha_innodb.cc:17430
|
#13 0x00005626d748cebb in sys_var_pluginvar::global_update (this=0x6210000d5488, thd=0x62c0003a0218, var=0x629002a0d4d8) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_plugin.cc:3677
|
#14 0x00005626d713c16b in sys_var::update (this=0x6210000d5488, thd=<optimized out>, var=0x629002a0d4d8) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/set_var.cc:207
|
#15 0x00005626d713d4e5 in set_var::update (this=<optimized out>, thd=<optimized out>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/set_var.cc:863
|
#16 0x00005626d7140a96 in sql_set_variables (thd=thd@entry=0x62c0003a0218, var_list=var_list@entry=0x62c0003a5630, free=free@entry=true) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/set_var.cc:745
|
#17 0x00005626d742f64e in mysql_execute_command (thd=thd@entry=0x62c0003a0218, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:5064
|
#18 0x00005626d74396b0 in mysql_parse (thd=thd@entry=0x62c0003a0218, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x3b952517f5a0)
|
at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:8030
|
#19 0x00005626d744053c in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x62c0003a0218,
|
packet=packet@entry=0x6290029f9219 " SET GLOBAL innodb_buffer_pool_size=@@innodb_buffer_pool_size /* E_R Thread1 QNO 2473 CON_ID 22 */ ", packet_length=packet_length@entry=99, blocking=blocking@entry=true)
|
at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:1894
|
#20 0x00005626d7445980 in do_command (thd=0x62c0003a0218, blocking=blocking@entry=true) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:1407
|
#21 0x00005626d78dbf63 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x608000003a38, put_in_cache=put_in_cache@entry=true) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_connect.cc:1416
|
#22 0x00005626d78dc779 in handle_one_connection (arg=arg@entry=0x608000003a38) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_connect.cc:1318
|
#23 0x00005626d87138f1 in pfs_spawn_thread (arg=0x617000007e98) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/perfschema/pfs.cc:2201
|
#24 0x000000006fe69b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
|
#25 0x000000006fefabb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
|
(rr)
|
sdp:/data1/results/1698783620/TBR-2080$ _RR_TRACE_DIR=./1/rr rr replay --mark-stdio
|
|
|
2. a bit less frequent when running a test based on the RQG grammar conf/mariadb/innodb_compression_encryption.yy
|
[rr 2146265 145385]mariadbd: /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:2105: void buf_pool_t::garbage_collect(): Assertion `state >= buf_page_t::FREED' failed.
|
(rr) bt
|
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140148681741888) at ./nptl/pthread_kill.c:44
|
#1 __pthread_kill_internal (signo=6, threadid=140148681741888) at ./nptl/pthread_kill.c:78
|
#2 __GI___pthread_kill (threadid=140148681741888, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
|
#3 0x00007f771ca5b476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
|
#4 0x00007f771ca417f3 in __GI_abort () at ./stdlib/abort.c:79
|
#5 0x00007f771ca4171b in __assert_fail_base (fmt=0x7f771cbf6150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55f5fe36f460 "state >= buf_page_t::FREED",
|
file=0x55f5fe368c80 "/data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc", line=2105, function=<optimized out>) at ./assert/assert.c:92
|
#6 0x00007f771ca52e96 in __GI___assert_fail (assertion=0x55f5fe36f460 "state >= buf_page_t::FREED", file=0x55f5fe368c80 "/data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc", line=2105,
|
function=0x55f5fe36f4a0 "void buf_pool_t::garbage_collect()") at ./assert/assert.c:101
|
#7 0x000055f5fd44c716 in buf_pool_t::garbage_collect (this=this@entry=0x55f5ff0f8b40 <buf_pool>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:2105
|
#8 0x000055f5fd42fa68 in mem_pressure::trigger_collection (this=0x55f5ffc12520 <mem_pressure_obj>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:906
|
#9 buf_resize_start () at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:2189
|
#10 0x000055f5fcd66434 in innodb_buffer_pool_size_update (save=<optimized out>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/handler/ha_innodb.cc:17430
|
#11 0x000055f5fb8d7ebb in sys_var_pluginvar::global_update (this=0x6210000d5488, thd=0x62c0001d0218, var=0x6290001184d8) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_plugin.cc:3677
|
#12 0x000055f5fb58716b in sys_var::update (this=0x6210000d5488, thd=<optimized out>, var=0x6290001184d8) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/set_var.cc:207
|
#13 0x000055f5fb5884e5 in set_var::update (this=<optimized out>, thd=<optimized out>) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/set_var.cc:863
|
#14 0x000055f5fb58ba96 in sql_set_variables (thd=thd@entry=0x62c0001d0218, var_list=var_list@entry=0x62c0001d5630, free=free@entry=true) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/set_var.cc:745
|
#15 0x000055f5fb87a64e in mysql_execute_command (thd=thd@entry=0x62c0001d0218, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:5064
|
#16 0x000055f5fb8846b0 in mysql_parse (thd=thd@entry=0x62c0001d0218, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7f76e863f5a0)
|
at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:8030
|
#17 0x000055f5fb88b53c in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x62c0001d0218,
|
packet=packet@entry=0x6290019a5219 " SET GLOBAL innodb_buffer_pool_size=@@innodb_buffer_pool_size /* E_R Thread1 QNO 2605 CON_ID 16 */ ", packet_length=packet_length@entry=99, blocking=blocking@entry=true)
|
at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:1894
|
#18 0x000055f5fb890980 in do_command (thd=0x62c0001d0218, blocking=blocking@entry=true) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_parse.cc:1407
|
#19 0x000055f5fbd26f63 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x608000003338, put_in_cache=put_in_cache@entry=true) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_connect.cc:1416
|
#20 0x000055f5fbd27779 in handle_one_connection (arg=arg@entry=0x608000003338) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/sql/sql_connect.cc:1318
|
#21 0x000055f5fcb5e8f1 in pfs_spawn_thread (arg=0x617000007098) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/perfschema/pfs.cc:2201
|
#22 0x00007f771caadb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
|
#23 0x00007f771cb3ebb4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
|
(rr)
|
sdp:/data1/results/1698783620/TBR-2081$ _RR_TRACE_DIR=./1/rr rr replay --mark-stdio
|
Thank you mleich for testing. Two executions of buf_pool_t::garbage_collect should be possible so something is unlocking buf_pool.mutex.
Since buf_pool_t::garbage_collect is only meant to be executed one at a time, maybe it needs it own mutex. Only a debug build can trigger this at the moment, so the mutex would be uncongested.
The assertion failure occurs in the buf_pool_t::garbage_collect() that I contributed. I checked the rr replay trace. It is invoking this function several times per second, most of the time freeing no pages. The trace involves the adaptive hash index, which was disabled by default in MDEV-20487. The last change to the block state was by the page cleaner thread:
#3 buf_page_t::set_state (s=0, this=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/buf0buf.h:1985
|
#4 buf_LRU_block_free_non_file_page (block=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0lru.cc:1016
|
#5 0x000055f5fd493f6d in buf_LRU_block_free_hashed_page (block=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0lru.cc:132
|
#6 0x000055f5fd49a9ab in buf_LRU_free_page (bpage=..., zip=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0lru.cc:995
|
#7 0x000055f5fd47ac1a in buf_flush_discard_page (bpage=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0flu.cc:1211
|
#8 0x000055f5fd483394 in buf_do_flush_list_batch (max_n=..., lsn=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0flu.cc:1472
|
#9 0x000055f5fd484349 in buf_flush_list_holding_mutex (max_n=..., lsn=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0flu.cc:1552
|
#10 0x000055f5fd489c88 in buf_flush_page_cleaner () at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0flu.cc:2559
|
At this time, the garbage collector was busy removing some adaptive hash index entries on a different block:
#0 0x000055f5fcdd6934 in ut_align_offset (ptr=..., alignment=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/ut0byte.h:88
|
#1 0x000055f5fd3fd607 in page_offset (ptr=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/page0page.h:216
|
#2 page_rec_check (rec=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/page0page.inl:310
|
#3 0x000055f5fd3ff852 in page_rec_get_next_low (rec=..., comp=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/include/page0page.inl:358
|
#4 0x000055f5fd406ab9 in btr_search_drop_page_hash_index (block=..., garbage_collect=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/btr/btr0sea.cc:1332
|
#5 0x000055f5fd49a94a in buf_LRU_free_page (bpage=..., zip=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0lru.cc:984
|
#6 0x000055f5fd44c9c7 in buf_pool_t::garbage_collect (this=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:2113
|
#7 0x000055f5fd42fa68 in mem_pressure::trigger_collection (this=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:906
|
#8 buf_resize_start () at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/buf/buf0buf.cc:2189
|
#9 0x000055f5fcd66434 in innodb_buffer_pool_size_update (save=...) at /data/Server/bb-10.11-MDEV-24670-memory-pressure/storage/innobase/handler/ha_innodb.cc:17430
|
I had overlooked the fact that buf_LRU_free_page() is indeed releasing and reacquiring the buf_pool.mutex in this code path:
#ifdef BTR_CUR_HASH_ADAPT
|
if (block->index) { |
mysql_mutex_unlock(&buf_pool.mutex);
|
|
|
/* Remove the adaptive hash index on the page. |
The page was declared uninitialized by
|
buf_LRU_block_remove_hashed(). We need to flag
|
the contents of the page valid (which it still is) in
|
order to avoid bogus Valgrind or MSAN warnings.*/
|
|
|
MEM_MAKE_DEFINED(block->page.frame, srv_page_size);
|
btr_search_drop_page_hash_index(block, false); |
MEM_UNDEFINED(block->page.frame, srv_page_size);
|
mysql_mutex_lock(&buf_pool.mutex);
|
}
|
#endif |
Due to this, we must use buf_pool.lru_hp to safely traverse the buf_pool.LRU list, similar to how buf_flush_LRU_list_batch() does it, but with a jump to the start of the eviction loop if the hazard pointer was invalidated. I believe that this handling is only needed #ifdef BTR_CUR_HASH_ADAPT.
I see that a subset of the "light variant" in my previous comment was implemented in
MDEV-25341. The last half of my comment is more relevant for the open tickets linked toMDEV-25341:MDEV-29445, MDEV-29432, MDEV-29429.