[MDEV-36880] Server gets stuck with extremely small aria_pagecache_buffer_size - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Minor
Resolution: Won't Fix
Affects Version/s: 10.6, 10.11, 11.4, 11.8
Fix Version/s: N/A
Component/s: Storage Engine - Aria
Labels:
None
Environment:

Hide
cmake . -DCMAKE_BUILD_TYPE=Debug -DENABLE_GCOV=ON -DCMAKE_C_FLAGS="--coverage -fno-inline -fno-inline-small-functions -fno-default-inline -O0" -DCMAKE_CXX_FLAGS="--coverage -fno-inline -fno-inline-small-functions -fno-default-inline -O0"

gcc (Debian 12.2.0-14) 12.2.0

Show
cmake . -DCMAKE_BUILD_TYPE=Debug -DENABLE_GCOV=ON -DCMAKE_C_FLAGS="--coverage -fno-inline -fno-inline-small-functions -fno-default-inline -O0" -DCMAKE_CXX_FLAGS="--coverage -fno-inline -fno-inline-small-functions -fno-default-inline -O0" gcc (Debian 12.2.0-14) 12.2.0

Bug Category:
Not for Release Notes

Description

--source include/have_sequence.inc

--let $restart_parameters= --aria_pagecache_buffer_size=128K

--source include/restart_mysqld.inc

CREATE TABLE t (

  f1 INT,

  f2 INT,

  f3 TEXT,

  f4 TEXT,

  f5 TEXT,

  f6 TEXT,

  f7 TEXT,

  f8 TEXT,

  f9 TEXT,

  KEY (f2)

) ENGINE=Aria;

INSERT INTO t SELECT seq, seq, REPEAT('x',1024), REPEAT('x',1024), REPEAT('x',1024), REPEAT('x',1024), REPEAT('x',1024), REPEAT('x',1024), REPEAT('x',1024) FROM seq_1_to_100;

SELECT * FROM t AS t1 JOIN t AS t2 ON (t2.f2 = t1.f1) ORDER BY t2.f1 LIMIT 10;

DROP TABLE t;

SELECT runs seemingly forever.
The process list:

10.6 e021a61b6fbf0b4c5929027e025d073e0c3418d5
+----+------+-----------------+------+---------+------+--------------+-------------------------------------------------------------------------------+----------+
\| Id \| User \| Host \| db \| Command \| Time \| State \| Info \| Progress \|
+----+------+-----------------+------+---------+------+--------------+-------------------------------------------------------------------------------+----------+
\| 3 \| root \| localhost \| test \| Query \| 2 \| Sending data \| SELECT * FROM t AS t1 JOIN t AS t2 ON (t2.f2 = t1.f1) ORDER BY t2.f1 LIMIT 10 \| 0.000 \|
\| 4 \| root \| localhost:57620 \| NULL \| Query \| 0 \| starting \| show processlist \| 0.000 \|
+----+------+-----------------+------+---------+------+--------------+-------------------------------------------------------------------------------+----------+

Stack trace from the running server:

10.6 e021a61b6fbf0b4c5929027e025d073e0c3418d5, waiting thread
Thread 6 (Thread 0x7f0b74f6b6c0 (LWP 1193128) "mariadbd"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x61300004ffb0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x61300004ffb0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007f0b7f2a4f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61300004ffb0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f0b7f2a75d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x556114231730 <maria_pagecache_var+240>, cond=0x61300004ff88) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=cond@entry=0x61300004ff88, mutex=mutex@entry=0x556114231730 <maria_pagecache_var+240>) at ./nptl/pthread_cond_wait.c:618
#5 0x000055610afa0d5f in safe_cond_wait (cond=cond@entry=0x61300004ff88, mp=mp@entry=0x556114231708 <maria_pagecache_var+200>, file=file@entry=0x55610bce4d80 "/data/bld/10.6-asan-ubsan/mysys/my_thr_init.c", line=line@entry=598) at /data/bld/10.6-asan-ubsan/mysys/thr_mutex.c:487
#6 0x000055610af8ba5c in psi_cond_wait (that=that@entry=0x61300004ff88, mutex=mutex@entry=0x556114231708 <maria_pagecache_var+200>, file=file@entry=0x55610b8625a0 "/data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c", line=line@entry=2149) at /data/bld/10.6-asan-ubsan/mysys/my_thr_init.c:598
#7 0x0000556109a79ed6 in inline_mysql_cond_wait (that=that@entry=0x61300004ff88, mutex=mutex@entry=0x556114231708 <maria_pagecache_var+200>, src_file=src_file@entry=0x55610b8625a0 "/data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c", src_line=src_line@entry=2149) at /data/bld/10.6-asan-ubsan/include/mysql/psi/mysql_thread.h:1070
#8 0x0000556109a860a5 in find_block (pagecache=pagecache@entry=0x556114231640 <maria_pagecache_var>, file=file@entry=0x629000127688, pageno=pageno@entry=7, init_hits_left=init_hits_left@entry=0, wrmode=wrmode@entry=1 '\001', block_is_copied=block_is_copied@entry=0 '\000', reg_req=<optimized out>, fast=<optimized out>, page_st=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c:2149
#9 0x0000556109a9c286 in pagecache_write_part (pagecache=pagecache@entry=0x556114231640 <maria_pagecache_var>, file=file@entry=0x629000127688, pageno=<optimized out>, level=level@entry=0, buff=<optimized out>, type=<optimized out>, lock=<optimized out>, pin=<optimized out>, write_mode=<optimized out>, page_link=0x7f0b74f66c60, first_REDO_LSN_for_page=<optimized out>, offset=<optimized out>, size=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c:4432
#10 0x0000556109b5a66f in write_tail (info=info@entry=0x629000127218, block=block@entry=0x61b000020f30, row_part=row_part@entry=0x62400006d918 'x' <repeats 200 times>..., org_length=org_length@entry=1024) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:2070
#11 0x0000556109b6b3db in write_block_record (info=info@entry=0x629000127218, old_record=old_record@entry=0x0, record=record@entry=0x62d00007b6b0 "", row=row@entry=0x629000127298, bitmap_blocks=bitmap_blocks@entry=0x629000127298, head_block_is_read=<optimized out>, row_pos=<optimized out>, undo_lsn=<optimized out>, old_record_checksum=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:2942
#12 0x0000556109b70988 in allocate_and_write_block_record (info=info@entry=0x629000127218, record=record@entry=0x62d00007b6b0 "", row=row@entry=0x629000127298, undo_lsn=undo_lsn@entry=1) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:3577
#13 0x0000556109b70ed2 in _ma_write_init_block_record (info=0x629000127218, record=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:3617
#14 0x0000556109ba17f7 in maria_write (info=<optimized out>, record=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_write.c:157
#15 0x00005561099e92a4 in ha_maria::write_row (this=0x62d00007aeb0, buf=0x62d00007b6b0 "") at /data/bld/10.6-asan-ubsan/storage/maria/ha_maria.cc:1255
#16 0x0000556107e53f35 in handler::ha_write_tmp_row (this=this@entry=0x62d00007aeb0, buf=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_class.h:7713
#17 0x0000556107e0ecdd in end_write (join=0x62d00005d308, join_tab=0x62d00006ebb8, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:23949
#18 0x0000556107d41ce9 in AGGR_OP::put_record (this=this@entry=0x62d000062170, end_of_records=end_of_records@entry=false) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:31511
#19 0x0000556107d439a6 in AGGR_OP::put_record (this=0x62d000062170) at /data/bld/10.6-asan-ubsan/sql/sql_select.h:1096
#20 sub_select_postjoin_aggr (join=0x62d00005d308, join_tab=0x62d00006ebb8, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22087
#21 0x0000556107ca6cec in evaluate_join_record (join=join@entry=0x62d00005d308, join_tab=join_tab@entry=0x62d00006e7f8, error=error@entry=0) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22623
#22 0x0000556107cd4c74 in sub_select (join=<optimized out>, join_tab=<optimized out>, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22393
#23 0x0000556107ca6cec in evaluate_join_record (join=join@entry=0x62d00005d308, join_tab=join_tab@entry=0x62d00006e438, error=error@entry=0) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22623
#24 0x0000556107cd5381 in sub_select (join=<optimized out>, join_tab=<optimized out>, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22432
#25 0x0000556107d7c0be in do_select (join=join@entry=0x62d00005d308, procedure=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:21916
#26 0x0000556107e36154 in JOIN::exec_inner (this=this@entry=0x62d00005d308) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:4939
#27 0x0000556107e36886 in JOIN::exec (this=this@entry=0x62d00005d308) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:4717
#28 0x0000556107e2d214 in mysql_select (thd=thd@entry=0x62b00007e218, tables=<optimized out>, fields=..., conds=conds@entry=0x0, og_num=og_num@entry=1, order=order@entry=0x62d00005d128, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:5196
#29 0x0000556107e2e3fe in handle_select (thd=thd@entry=0x62b00007e218, lex=lex@entry=0x62b000082538, result=result@entry=0x62d00005d2d8, setup_tables_done_option=setup_tables_done_option@entry=0) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:573
#30 0x0000556107b25136 in execute_sqlcom_select (thd=thd@entry=0x62b00007e218, all_tables=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:6422
#31 0x0000556107b68bc0 in mysql_execute_command (thd=thd@entry=0x62b00007e218, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:4013
#32 0x0000556107b8aada in mysql_parse (thd=thd@entry=0x62b00007e218, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7f0b74f69ac0) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:8200
#33 0x0000556107b93daa in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x62b00007e218, packet=packet@entry=0x6290000eb219 "", packet_length=packet_length@entry=77, blocking=blocking@entry=true) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:1908
#34 0x0000556107ba09fc in do_command (thd=thd@entry=0x62b00007e218, blocking=blocking@entry=true) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:1421
#35 0x000055610831ff36 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x608000002f38, put_in_cache=put_in_cache@entry=true) at /data/bld/10.6-asan-ubsan/sql/sql_connect.cc:1386
#36 0x0000556108321093 in handle_one_connection (arg=0x608000002f38) at /data/bld/10.6-asan-ubsan/sql/sql_connect.cc:1298
#37 0x0000556109d4bb11 in pfs_spawn_thread (arg=0x617000005b98) at /data/bld/10.6-asan-ubsan/storage/perfschema/pfs.cc:2201
#38 0x00007f0b7f2a81c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#39 0x00007f0b7f32885c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

all threads
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f0b7f31b1df in __GI___poll (fds=0x60e000003d98, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.

Thread 7 (Thread 0x7f0b73ad16c0 (LWP 1193130) "mariadbd"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f0b73ad0bb0, op=393, expected=0, futex_word=0x5561141c0628 <thread_cache+40>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5561141c0628 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f0b73ad0bb0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007f0b7f2a4f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5561141c0628 <thread_cache+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f0b73ad0bb0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f0b7f2a78bc in __pthread_cond_wait_common (abstime=0x7f0b73ad0bb0, clockid=0, mutex=0x5561141c0698 <thread_cache+152>, cond=0x5561141c0600 <thread_cache>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=cond@entry=0x5561141c0600 <thread_cache>, mutex=mutex@entry=0x5561141c0698 <thread_cache+152>, abstime=abstime@entry=0x7f0b73ad0bb0) at ./nptl/pthread_cond_wait.c:643
#5 0x000055610afa17d6 in safe_cond_timedwait (cond=cond@entry=0x5561141c0600 <thread_cache>, mp=mp@entry=0x5561141c0670 <thread_cache+112>, abstime=abstime@entry=0x7f0b73ad0bb0, file=file@entry=0x55610bce4d80 "/data/bld/10.6-asan-ubsan/mysys/my_thr_init.c", line=line@entry=611) at /data/bld/10.6-asan-ubsan/mysys/thr_mutex.c:541
#6 0x000055610af8bd6b in psi_cond_timedwait (that=0x5561141c0600 <thread_cache>, mutex=0x5561141c0670 <thread_cache+112>, abstime=<optimized out>, file=<optimized out>, line=<optimized out>) at /data/bld/10.6-asan-ubsan/mysys/my_thr_init.c:611
#7 0x00005561083211b1 in inline_mysql_cond_timedwait (that=that@entry=0x5561141c0600 <thread_cache>, mutex=mutex@entry=0x5561141c0670 <thread_cache+112>, abstime=abstime@entry=0x7f0b73ad0bb0, src_file=src_file@entry=0x55610b4d9de0 "/data/bld/10.6-asan-ubsan/sql/thread_cache.h", src_line=src_line@entry=176) at /data/bld/10.6-asan-ubsan/include/mysql/psi/mysql_thread.h:1086
#8 0x0000556108321747 in Thread_cache::park (this=0x5561141c0600 <thread_cache>) at /data/bld/10.6-asan-ubsan/sql/thread_cache.h:176
#9 0x000055610832065a in do_handle_one_connection (connect=<optimized out>, connect@entry=0x608000002fb8, put_in_cache=put_in_cache@entry=true) at /data/bld/10.6-asan-ubsan/sql/sql_connect.cc:1399
#10 0x0000556108321093 in handle_one_connection (arg=0x608000002fb8) at /data/bld/10.6-asan-ubsan/sql/sql_connect.cc:1298
#11 0x0000556109d4bb11 in pfs_spawn_thread (arg=0x617000005f18) at /data/bld/10.6-asan-ubsan/storage/perfschema/pfs.cc:2201
#12 0x00007f0b7f2a81c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f0b7f32885c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f0b74f6b6c0 (LWP 1193128) "mariadbd"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x61300004ffb0) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x61300004ffb0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007f0b7f2a4f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61300004ffb0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f0b7f2a75d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x556114231730 <maria_pagecache_var+240>, cond=0x61300004ff88) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=cond@entry=0x61300004ff88, mutex=mutex@entry=0x556114231730 <maria_pagecache_var+240>) at ./nptl/pthread_cond_wait.c:618
#5 0x000055610afa0d5f in safe_cond_wait (cond=cond@entry=0x61300004ff88, mp=mp@entry=0x556114231708 <maria_pagecache_var+200>, file=file@entry=0x55610bce4d80 "/data/bld/10.6-asan-ubsan/mysys/my_thr_init.c", line=line@entry=598) at /data/bld/10.6-asan-ubsan/mysys/thr_mutex.c:487
#6 0x000055610af8ba5c in psi_cond_wait (that=that@entry=0x61300004ff88, mutex=mutex@entry=0x556114231708 <maria_pagecache_var+200>, file=file@entry=0x55610b8625a0 "/data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c", line=line@entry=2149) at /data/bld/10.6-asan-ubsan/mysys/my_thr_init.c:598
#7 0x0000556109a79ed6 in inline_mysql_cond_wait (that=that@entry=0x61300004ff88, mutex=mutex@entry=0x556114231708 <maria_pagecache_var+200>, src_file=src_file@entry=0x55610b8625a0 "/data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c", src_line=src_line@entry=2149) at /data/bld/10.6-asan-ubsan/include/mysql/psi/mysql_thread.h:1070
#8 0x0000556109a860a5 in find_block (pagecache=pagecache@entry=0x556114231640 <maria_pagecache_var>, file=file@entry=0x629000127688, pageno=pageno@entry=7, init_hits_left=init_hits_left@entry=0, wrmode=wrmode@entry=1 '\001', block_is_copied=block_is_copied@entry=0 '\000', reg_req=<optimized out>, fast=<optimized out>, page_st=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c:2149
#9 0x0000556109a9c286 in pagecache_write_part (pagecache=pagecache@entry=0x556114231640 <maria_pagecache_var>, file=file@entry=0x629000127688, pageno=<optimized out>, level=level@entry=0, buff=<optimized out>, type=<optimized out>, lock=<optimized out>, pin=<optimized out>, write_mode=<optimized out>, page_link=0x7f0b74f66c60, first_REDO_LSN_for_page=<optimized out>, offset=<optimized out>, size=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_pagecache.c:4432
#10 0x0000556109b5a66f in write_tail (info=info@entry=0x629000127218, block=block@entry=0x61b000020f30, row_part=row_part@entry=0x62400006d918 'x' <repeats 200 times>..., org_length=org_length@entry=1024) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:2070
#11 0x0000556109b6b3db in write_block_record (info=info@entry=0x629000127218, old_record=old_record@entry=0x0, record=record@entry=0x62d00007b6b0 "", row=row@entry=0x629000127298, bitmap_blocks=bitmap_blocks@entry=0x629000127298, head_block_is_read=<optimized out>, row_pos=<optimized out>, undo_lsn=<optimized out>, old_record_checksum=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:2942
#12 0x0000556109b70988 in allocate_and_write_block_record (info=info@entry=0x629000127218, record=record@entry=0x62d00007b6b0 "", row=row@entry=0x629000127298, undo_lsn=undo_lsn@entry=1) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:3577
#13 0x0000556109b70ed2 in _ma_write_init_block_record (info=0x629000127218, record=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_blockrec.c:3617
#14 0x0000556109ba17f7 in maria_write (info=<optimized out>, record=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_write.c:157
#15 0x00005561099e92a4 in ha_maria::write_row (this=0x62d00007aeb0, buf=0x62d00007b6b0 "") at /data/bld/10.6-asan-ubsan/storage/maria/ha_maria.cc:1255
#16 0x0000556107e53f35 in handler::ha_write_tmp_row (this=this@entry=0x62d00007aeb0, buf=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_class.h:7713
#17 0x0000556107e0ecdd in end_write (join=0x62d00005d308, join_tab=0x62d00006ebb8, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:23949
#18 0x0000556107d41ce9 in AGGR_OP::put_record (this=this@entry=0x62d000062170, end_of_records=end_of_records@entry=false) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:31511
#19 0x0000556107d439a6 in AGGR_OP::put_record (this=0x62d000062170) at /data/bld/10.6-asan-ubsan/sql/sql_select.h:1096
#20 sub_select_postjoin_aggr (join=0x62d00005d308, join_tab=0x62d00006ebb8, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22087
#21 0x0000556107ca6cec in evaluate_join_record (join=join@entry=0x62d00005d308, join_tab=join_tab@entry=0x62d00006e7f8, error=error@entry=0) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22623
#22 0x0000556107cd4c74 in sub_select (join=<optimized out>, join_tab=<optimized out>, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22393
#23 0x0000556107ca6cec in evaluate_join_record (join=join@entry=0x62d00005d308, join_tab=join_tab@entry=0x62d00006e438, error=error@entry=0) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22623
#24 0x0000556107cd5381 in sub_select (join=<optimized out>, join_tab=<optimized out>, end_of_records=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:22432
#25 0x0000556107d7c0be in do_select (join=join@entry=0x62d00005d308, procedure=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:21916
#26 0x0000556107e36154 in JOIN::exec_inner (this=this@entry=0x62d00005d308) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:4939
#27 0x0000556107e36886 in JOIN::exec (this=this@entry=0x62d00005d308) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:4717
#28 0x0000556107e2d214 in mysql_select (thd=thd@entry=0x62b00007e218, tables=<optimized out>, fields=..., conds=conds@entry=0x0, og_num=og_num@entry=1, order=order@entry=0x62d00005d128, group=<optimized out>, having=<optimized out>, proc_param=<optimized out>, select_options=<optimized out>, result=<optimized out>, unit=<optimized out>, select_lex=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:5196
#29 0x0000556107e2e3fe in handle_select (thd=thd@entry=0x62b00007e218, lex=lex@entry=0x62b000082538, result=result@entry=0x62d00005d2d8, setup_tables_done_option=setup_tables_done_option@entry=0) at /data/bld/10.6-asan-ubsan/sql/sql_select.cc:573
#30 0x0000556107b25136 in execute_sqlcom_select (thd=thd@entry=0x62b00007e218, all_tables=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:6422
#31 0x0000556107b68bc0 in mysql_execute_command (thd=thd@entry=0x62b00007e218, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:4013
#32 0x0000556107b8aada in mysql_parse (thd=thd@entry=0x62b00007e218, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7f0b74f69ac0) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:8200
#33 0x0000556107b93daa in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x62b00007e218, packet=packet@entry=0x6290000eb219 "", packet_length=packet_length@entry=77, blocking=blocking@entry=true) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:1908
#34 0x0000556107ba09fc in do_command (thd=thd@entry=0x62b00007e218, blocking=blocking@entry=true) at /data/bld/10.6-asan-ubsan/sql/sql_parse.cc:1421
#35 0x000055610831ff36 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x608000002f38, put_in_cache=put_in_cache@entry=true) at /data/bld/10.6-asan-ubsan/sql/sql_connect.cc:1386
#36 0x0000556108321093 in handle_one_connection (arg=0x608000002f38) at /data/bld/10.6-asan-ubsan/sql/sql_connect.cc:1298
#37 0x0000556109d4bb11 in pfs_spawn_thread (arg=0x617000005b98) at /data/bld/10.6-asan-ubsan/storage/perfschema/pfs.cc:2201
#38 0x00007f0b7f2a81c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#39 0x00007f0b7f32885c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f0b75aca6c0 (LWP 1193127) "mariadbd"):
#0 0x00007f0b7f25bc82 in __GI___sigtimedwait (set=0x7f0b75ac9bd0, info=0x7f0b75ac9a80, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
#1 0x00007f0b7fc6e8ce in __interceptor_sigwaitinfo (set=0x7f0b75ac9bd0, info=0x7f0b75ac9a80) at ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4124
#2 0x0000556107530b7b in my_sigwait (set=set@entry=0x7f0b75ac9bd0, sig=sig@entry=0x7f0b75ac9ba0, code=code@entry=0x7f0b75ac9bb0) at /data/bld/10.6-asan-ubsan/include/my_pthread.h:192
#3 0x000055610753c11c in signal_hand () at /data/bld/10.6-asan-ubsan/sql/mysqld.cc:3157
#4 0x0000556109d4bb11 in pfs_spawn_thread (arg=0x617000003518) at /data/bld/10.6-asan-ubsan/storage/perfschema/pfs.cc:2201
#5 0x00007f0b7f2a81c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6 0x00007f0b7f32885c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f0b765e96c0 (LWP 1193126) "mariadbd"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5561141cb528 <COND_manager+40>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5561141cb528 <COND_manager+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007f0b7f2a4f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5561141cb528 <COND_manager+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f0b7f2a75d8 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5561141cb588 <LOCK_manager+40>, cond=0x5561141cb500 <COND_manager>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=cond@entry=0x5561141cb500 <COND_manager>, mutex=mutex@entry=0x5561141cb588 <LOCK_manager+40>) at ./nptl/pthread_cond_wait.c:618
#5 0x000055610afa0d5f in safe_cond_wait (cond=cond@entry=0x5561141cb500 <COND_manager>, mp=mp@entry=0x5561141cb560 <LOCK_manager>, file=file@entry=0x55610bce4d80 "/data/bld/10.6-asan-ubsan/mysys/my_thr_init.c", line=line@entry=598) at /data/bld/10.6-asan-ubsan/mysys/thr_mutex.c:487
#6 0x000055610af8ba5c in psi_cond_wait (that=0x5561141cb500 <COND_manager>, mutex=0x5561141cb560 <LOCK_manager>, file=<optimized out>, line=<optimized out>) at /data/bld/10.6-asan-ubsan/mysys/my_thr_init.c:598
#7 0x0000556107b101c2 in inline_mysql_cond_wait (that=that@entry=0x5561141cb500 <COND_manager>, mutex=mutex@entry=0x5561141cb560 <LOCK_manager>, src_file=src_file@entry=0x55610b3a0180 "/data/bld/10.6-asan-ubsan/sql/sql_manager.cc", src_line=src_line@entry=109) at /data/bld/10.6-asan-ubsan/include/mysql/psi/mysql_thread.h:1070
#8 0x0000556107b107ce in handle_manager (arg=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/sql_manager.cc:109
#9 0x0000556109d4bb11 in pfs_spawn_thread (arg=0x617000001598) at /data/bld/10.6-asan-ubsan/storage/perfschema/pfs.cc:2201
#10 0x00007f0b7f2a81c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007f0b7f32885c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f0b770ca6c0 (LWP 1193125) "mariadbd"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f0b770c9b40, op=393, expected=0, futex_word=0x556114a35ca8 <COND_checkpoint+40>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x556114a35ca8 <COND_checkpoint+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f0b770c9b40, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007f0b7f2a4f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x556114a35ca8 <COND_checkpoint+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f0b770c9b40, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f0b7f2a78bc in __pthread_cond_wait_common (abstime=0x7f0b770c9b40, clockid=0, mutex=0x556114a35d08 <LOCK_checkpoint+40>, cond=0x556114a35c80 <COND_checkpoint>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=cond@entry=0x556114a35c80 <COND_checkpoint>, mutex=mutex@entry=0x556114a35d08 <LOCK_checkpoint+40>, abstime=abstime@entry=0x7f0b770c9b40) at ./nptl/pthread_cond_wait.c:643
#5 0x000055610afa17d6 in safe_cond_timedwait (cond=cond@entry=0x556114a35c80 <COND_checkpoint>, mp=mp@entry=0x556114a35ce0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f0b770c9b40, file=file@entry=0x55610bce4d80 "/data/bld/10.6-asan-ubsan/mysys/my_thr_init.c", line=line@entry=611) at /data/bld/10.6-asan-ubsan/mysys/thr_mutex.c:541
#6 0x000055610af8bd6b in psi_cond_timedwait (that=that@entry=0x556114a35c80 <COND_checkpoint>, mutex=mutex@entry=0x556114a35ce0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f0b770c9b40, file=file@entry=0x55610b870e40 "/data/bld/10.6-asan-ubsan/storage/maria/ma_servicethread.c", line=line@entry=115) at /data/bld/10.6-asan-ubsan/mysys/my_thr_init.c:611
#7 0x0000556109ac57f1 in inline_mysql_cond_timedwait (that=0x556114a35c80 <COND_checkpoint>, mutex=mutex@entry=0x556114a35ce0 <LOCK_checkpoint>, abstime=abstime@entry=0x7f0b770c9b40, src_file=src_file@entry=0x55610b870e40 "/data/bld/10.6-asan-ubsan/storage/maria/ma_servicethread.c", src_line=src_line@entry=115) at /data/bld/10.6-asan-ubsan/include/mysql/psi/mysql_thread.h:1086
#8 0x0000556109ac67ce in my_service_thread_sleep (control=control@entry=0x556110dd1e80 <checkpoint_control>, sleep_time=sleep_time@entry=30000000000) at /data/bld/10.6-asan-ubsan/storage/maria/ma_servicethread.c:115
#9 0x0000556109aa518c in ma_checkpoint_background (arg=<optimized out>) at /data/bld/10.6-asan-ubsan/storage/maria/ma_checkpoint.c:725
#10 0x0000556109d4bb11 in pfs_spawn_thread (arg=0x617000000e98) at /data/bld/10.6-asan-ubsan/storage/perfschema/pfs.cc:2201
#11 0x00007f0b7f2a81c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007f0b7f32885c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f0b7781c6c0 (LWP 1193124) "mariadbd"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f0b7781bc50, op=393, expected=0, futex_word=0x556114b16ce8 <COND_timer+40>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x556114b16ce8 <COND_timer+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f0b7781bc50, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2 0x00007f0b7f2a4f7b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x556114b16ce8 <COND_timer+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f0b7781bc50, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f0b7f2a78bc in __pthread_cond_wait_common (abstime=0x7f0b7781bc50, clockid=0, mutex=0x556114b16d48 <LOCK_timer+40>, cond=0x556114b16cc0 <COND_timer>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=cond@entry=0x556114b16cc0 <COND_timer>, mutex=mutex@entry=0x556114b16d48 <LOCK_timer+40>, abstime=abstime@entry=0x7f0b7781bc50) at ./nptl/pthread_cond_wait.c:643
#5 0x000055610afa17d6 in safe_cond_timedwait (cond=cond@entry=0x556114b16cc0 <COND_timer>, mp=mp@entry=0x556114b16d20 <LOCK_timer>, abstime=abstime@entry=0x7f0b7781bc50, file=file@entry=0x55610bce4d80 "/data/bld/10.6-asan-ubsan/mysys/my_thr_init.c", line=line@entry=611) at /data/bld/10.6-asan-ubsan/mysys/thr_mutex.c:541
#6 0x000055610af8bd6b in psi_cond_timedwait (that=that@entry=0x556114b16cc0 <COND_timer>, mutex=mutex@entry=0x556114b16d20 <LOCK_timer>, abstime=abstime@entry=0x7f0b7781bc50, file=file@entry=0x55610bce8040 "/data/bld/10.6-asan-ubsan/mysys/thr_timer.c", line=line@entry=322) at /data/bld/10.6-asan-ubsan/mysys/my_thr_init.c:611
#7 0x000055610afa335b in inline_mysql_cond_timedwait (that=that@entry=0x556114b16cc0 <COND_timer>, mutex=mutex@entry=0x556114b16d20 <LOCK_timer>, abstime=abstime@entry=0x7f0b7781bc50, src_file=src_file@entry=0x55610bce8040 "/data/bld/10.6-asan-ubsan/mysys/thr_timer.c", src_line=src_line@entry=322) at /data/bld/10.6-asan-ubsan/include/mysql/psi/mysql_thread.h:1086
#8 0x000055610afa36a3 in timer_handler (arg=<optimized out>) at /data/bld/10.6-asan-ubsan/mysys/thr_timer.c:322
#9 0x0000556109d4bb11 in pfs_spawn_thread (arg=0x617000000418) at /data/bld/10.6-asan-ubsan/storage/perfschema/pfs.cc:2201
#10 0x00007f0b7f2a81c4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007f0b7f32885c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f0b7f980280 (LWP 1193123) "mariadbd"):
#0 0x00007f0b7f31b1df in __GI___poll (fds=0x60e000003d98, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f0b7fc755b2 in __interceptor_poll (fds=0x60e000003d98, nfds=3, timeout=-1) at ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4044
#2 0x00005561075576af in handle_connections_sockets () at /data/bld/10.6-asan-ubsan/sql/mysqld.cc:6220
#3 0x0000556107557d70 in run_main_loop () at /data/bld/10.6-asan-ubsan/sql/mysqld.cc:5519
#4 0x0000556107559125 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/mysqld.cc:5917
#5 0x000055610752c8e2 in main (argc=<optimized out>, argv=<optimized out>) at /data/bld/10.6-asan-ubsan/sql/main.cc:34

Originally the failure was observed in concurrent tests in the scope of ~~MDEV-24~~ feature testing, so Monty's comments below are based on the stack trace he was analyzing then:

Thread 12 is waiting for a page to become free when there is no available pages on the page cache.
This should in practice never happen as there should always be pages that are not in use.
One MUST have at least this many pages available in each page cache:
number of connected threads doing queries * ((3 (used when splitting pages) * b_tree_levels (always less than 10) *
number_of_indexes) + number of blobs + blob_size/8192)
The constants aboves are worst case scenarios.
Assuming 64 threads, having 64*3*4 = 768 pages should ensure this does not happen.
There may of course be some bug in the new code when using segments that we are mixing page caches. However, if this would happen, we should get failures on table close as we check that the used page caches are correct.
There should also be crashed tables if a table would use the wrong page cache as this would never be flushed to disk.
However, there is another problem:
select 16*1024*1024/128/8192=16
You have only 16 pages in each page cache. You are very close to the limit if you have many index, there is a few page splits and you have a big blob or several blobs.
I don't this second remember if the blog needs to fit into the page cache. Probably not, except that last page of the blob, which is exactly what the one thread above is waiting for.

Yes, full blob pages are not pinned, but the head and the end is. In other words, 2 blocks per blob has to be available
This is an old issue so not critical to fix just now but should be looked upon.
What can be done is to just wait a minute for a free page and if we do not get one we should assume a failure and give up.
This would result is a crashed table instead of a hang. Not much that can be done for this one.
Note that if we would crash the server, recovery would be able to fix the table.
Unfortunately we cannot yet to recovery of a single table while the server is running

However, if you have a few threads using the same pagecache of 16 blocks, this issue is certain to happen

Having 1M in each pagecache should ensure this doesn't happen. It is not likely that many threads are requiring a lot of blocks at the same time.
Almost all bugs with the segments will happen independent of the size of the page cache.
The bug scenarios are:

We are accidently using the wrong pagecache for a table

The table changes pagecache without some user not being aware of this (the only way a page can switch pagecache is if REPAIR TABLE is used as during repair the table can be moved to another cache).

We are using the wrong pagecache when flushing tables. The effect of this would be corrupted tables

S, in the case above the pagecache buffer size (128K) is practically meaningless, even although legally allowed, and at least the provided test case doesn't scale easily, that is doubling the data volume or the number of blobs doesn't make SELECT hang on 256K buffer size, etc. As I understand from Monty's comments, in general it can scale to some extent, but apparently it will still require unrealistic (or unreasonable) combination of the workload and the pagecache size.
So, I don't think it makes sense to add special logic (waiting and crashing instead of hanging forever, as Monty described) for the sake of such an extreme corner case. Thus closing as "won't fix".

If we ever encounter a more realistic scenario, we can always reconsider.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

threads_full.txt
2025-05-25 18:54
206 kB
Elena Stepanova
threads_short.txt
2025-05-25 18:54
57 kB
Elena Stepanova

Issue Links

relates to

MDEV-24 Segmented key cache for Aria

Closed

Activity

People

Assignee:: Unassigned

Reporter:: Elena Stepanova

Votes:: 1 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2025-05-25 18:54

Updated:: 2025-05-29 20:07

Resolved:: 2025-05-29 20:07

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.