[MDEV-26887] Assertion `(longlong) thd->status_var.local_memory_used >= 0 || !debug_assert_on_not_freed_memory' failed on slave, optimized builds stop replicating Created: 2021-10-22  Updated: 2023-11-28

Status: Confirmed
Project: MariaDB Server
Component/s: Data Definition - Temporary, Storage Engine - Aria
Affects Version/s: 10.4, 10.5, 10.6, 10.7, 10.9, 10.10, 10.11, 11.0, 11.1, 11.2
Fix Version/s: 10.4, 10.5, 10.6, 10.11, 11.0, 11.1, 11.2

Type: Bug Priority: Major
Reporter: Alice Sherepa Assignee: Andrei Elkin
Resolution: Unresolved Votes: 0
Labels: Memory_leak, affects-tests, replication_stops, sporadic

Issue Links:
Relates
relates to MDEV-23222 MDEV-23222 SIGSEG in maria_create() b... Closed
relates to MDEV-25064 rpl.rpl_parallel_temptable failed in ... Closed
relates to MDEV-24110 rpl.rpl_parallel (rpl.rpl_parallel_st... Confirmed

 Description   

Assertion appears with mixed/statement format +parallel rpl + setting slave-parallel-threads to smth (not 0 as default)
To repeat- run as

./mtr --mysqld=--binlog-format=MIXED --mysqld=--slave-parallel-threads=2  

-- source include/master-slave.inc
 
--connection master
CREATE TEMPORARY TABLE tmp (pk int) ENGINE=Aria AS SELECT 1;
 
--connection master1
DROP  TEMPORARY TABLE IF EXISTS tmp;
 
--sync_slave_with_master
 
--source include/rpl_end.inc

10.4 a736a3174a4e7c0d92a389

Warning:   16 bytes lost at 0x60d0000701e0, allocated by T@14 at mysys/my_malloc.c:101, mysys/my_malloc.c:155, maria/ma_open.c:1120, maria/ma_open.c:204, maria/ma_open.c:1053, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996
Warning: 2304 bytes lost at 0x61d0002bd4f0, allocated by T@14 at mysys/my_malloc.c:101, mysys/array.c:70, maria/ma_open.c:167, maria/ma_open.c:1053, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996, sql/temporary_tables.cc:1117
Warning:   48 bytes lost at 0x60f000070400, allocated by T@14 at mysys/my_malloc.c:101, maria/ma_blockrec.c:535, maria/ma_open.c:163, maria/ma_open.c:1053, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996, sql/temporary_tables.cc:1117
Warning:  144 bytes lost at 0x61200005c030, allocated by T@14 at mysys/my_malloc.c:101, mysys/array.c:70, maria/ma_blockrec.c:531, maria/ma_open.c:163, maria/ma_open.c:1053, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996
Warning:  272 bytes lost at 0x6140000700b0, allocated by T@14 at mysys/my_malloc.c:101, mysys/mulalloc.c:51, maria/ma_blockrec.c:492, maria/ma_open.c:163, maria/ma_open.c:1053, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996
Warning: 18296 bytes lost at 0x629000325270, allocated by T@14 at mysys/my_malloc.c:101, mysys/mulalloc.c:51, maria/ma_open.c:119, maria/ma_open.c:1053, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996, sql/temporary_tables.cc:1117
Warning:   24 bytes lost at 0x60d000070110, allocated by T@14 at mysys/my_malloc.c:101, mysys/array.c:70, maria/ma_bitmap.c:242, maria/ma_blockrec.c:436, maria/ma_open.c:915, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996
Warning: 16384 bytes lost at 0x629000320270, allocated by T@14 at mysys/my_malloc.c:101, maria/ma_bitmap.c:242, maria/ma_blockrec.c:436, maria/ma_open.c:915, maria/ha_maria.cc:1152, sql/handler.cc:2800, sql/table.cc:3996, sql/temporary_tables.cc:1117
Memory lost: 37488 bytes in 2414 chunks
mysqld: /10.4/src/sql/sql_class.cc:1731: virtual THD::~THD(): Assertion `status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory' failed.
211022 16:21:19 [ERROR] mysqld got signal 6 ;
 
Server version: 10.4.22-MariaDB-debug-log
 
linux/raise.c:51(__GI_raise)[0x7f4223c6c18b]
stdlib/abort.c:81(__GI_abort)[0x7f4223c4b859]
intl/loadmsgcat.c:509(get_sysdep_segment_value)[0x7f4223c4b729]
:0(__GI___assert_fail)[0x7f4223c5cf36]
sql/sql_class.cc:1734(THD::~THD())[0x5614b820f5c4]
sql/sql_class.cc:1737(THD::~THD())[0x5614b820f93a]
sql/rpl_parallel.cc:1506(handle_rpl_parallel_thread)[0x5614b885b7a8]
perfschema/pfs.cc:1871(pfs_spawn_thread)[0x5614b9df5679]
nptl/pthread_create.c:478(start_thread)[0x7f4224173609]
x86_64/clone.S:97(__GI___clone)[0x7f4223d48293]

mysqld: /10.6/src/sql/mysqld.cc:3647: void my_malloc_size_cb_func(long long int, my_bool): Assertion `(longlong) thd->status_var.local_memory_used >= 0 || !debug_assert_on_not_freed_memory' failed.
211022 14:26:08 [ERROR] mysqld got signal 6 ;
 
Server version: 10.6.5-MariaDB-debug-log
 
linux/raise.c:51(__GI_raise)[0x7eff40f3018b]
stdlib/abort.c:81(__GI_abort)[0x7eff40f0f859]
intl/loadmsgcat.c:509(get_sysdep_segment_value)[0x7eff40f0f729]
:0(__GI___assert_fail)[0x7eff40f20f36]
sql/mysqld.cc:3650(my_malloc_size_cb_func)[0x56159ce8a685]
mysys/my_malloc.c:211(my_free)[0x56159ef36b68]
maria/ma_close.c:262(maria_close)[0x56159e057801]
maria/ha_maria.cc:1250(ha_maria::close())[0x56159e070ab1]
sql/handler.cc:3382(handler::ha_close())[0x56159da15349]
sql/table.cc:4434(closefrm(TABLE*))[0x56159d514e71]
sql/temporary_tables.cc:1239(THD::close_temporary_table(TABLE*))[0x56159d84bd76]
sql/temporary_tables.cc:1490(THD::free_temporary_table(TABLE*))[0x56159d84d802]
sql/temporary_tables.cc:651(THD::drop_temporary_table(TABLE*, bool*, bool))[0x56159d847fdf]
sql/sql_table.cc:1297(mysql_rm_table_no_locks(THD*, TABLE_LIST*, st_mysql_const_lex_string const*, st_ddl_log_state*, bool, bool, bool, bool, bool, bool))[0x56159d40f8ae]
sql/sql_table.cc:1044(mysql_rm_table(THD*, TABLE_LIST*, bool, bool, bool, bool))[0x56159d40e2d9]
sql/sql_parse.cc:4949(mysql_execute_command(THD*, bool))[0x56159d1a6567]
sql/sql_parse.cc:8030(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x56159d1bb330]
sql/log_event_server.cc:1918(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x56159dd81a57]
sql/log_event_server.cc:1592(Query_log_event::do_apply_event(rpl_group_info*))[0x56159dd7ec75]
sql/log_event.h:1516(Log_event::apply_event(rpl_group_info*))[0x56159cf294dd]
sql/slave.cc:3881(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x56159cf0ac40]
sql/slave.cc:4068(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x56159cf0b9cc]
sql/rpl_parallel.cc:62(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x56159d765409]
sql/rpl_parallel.cc:1378(handle_rpl_parallel_thread)[0x56159d76d081]
perfschema/pfs.cc:2203(pfs_spawn_thread)[0x56159e313993]
nptl/pthread_create.c:478(start_thread)[0x7eff41437609]
x86_64/clone.S:97(__GI___clone)[0x7eff4100c293]
 
Query (0x61300009ec13): DROP TEMPORARY TABLE IF EXISTS `test`.`tmp` /* generated by server */

10.6 78dec1f199458cff300622

Warning:   40 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/my_malloc.c:141, maria/ma_open.c:1253, maria/ma_open.c:203, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning:  296 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/mulalloc.c:51, maria/ma_blockrec.c:495, maria/ma_open.c:161, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning:  168 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/array.c:71, maria/ma_blockrec.c:534, maria/ma_open.c:161, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning:   72 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, maria/ma_blockrec.c:539, maria/ma_open.c:161, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330, sql/temporary_tables.cc:1117
Warning:   48 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/array.c:71, maria/ma_bitmap.c:243, maria/ma_blockrec.c:436, maria/ma_open.c:1017, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning: 16408 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, maria/ma_bitmap.c:242, maria/ma_blockrec.c:436, maria/ma_open.c:1017, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330, sql/temporary_tables.cc:1117
Warning: 2328 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/array.c:71, maria/ma_open.c:165, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330, sql/temporary_tables.cc:1117
Warning: 18368 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/mulalloc.c:51, maria/ma_open.c:115, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330, sql/temporary_tables.cc:1117
Warning:   40 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/my_malloc.c:141, maria/ma_open.c:1253, maria/ma_open.c:203, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning:  296 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/mulalloc.c:51, maria/ma_blockrec.c:495, maria/ma_open.c:161, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning:  168 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/array.c:71, maria/ma_blockrec.c:534, maria/ma_open.c:161, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning:   72 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, maria/ma_blockrec.c:539, maria/ma_open.c:161, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330, sql/temporary_tables.cc:1117
Warning:   48 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/array.c:71, maria/ma_bitmap.c:243, maria/ma_blockrec.c:436, maria/ma_open.c:1017, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330
Warning: 16408 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, maria/ma_bitmap.c:242, maria/ma_blockrec.c:436, maria/ma_open.c:1017, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330, sql/temporary_tables.cc:1117
Warning: 2328 bytes freed by T@6, allocated by T@7 at mysys/my_malloc.c:90, mysys/array.c:71, maria/ma_open.c:165, maria/ma_open.c:1170, maria/ha_maria.cc:1160, sql/handler.cc:3313, sql/table.cc:4330, sql/temporary_tables.cc:1117
mysqld: /10.6/sql/mysqld.cc:3647: void my_malloc_size_cb_func(long long int, my_bool): Assertion `(longlong) thd->status_var.local_memory_used >= 0 || !debug_assert_on_not_freed_memory' failed.
211022 15:11:40 [ERROR] mysqld got signal 6 ;
 
Server version: 10.6.5-MariaDB-debug-log
 
linux/raise.c:51(__GI_raise)[0x7f90f813b18b]
stdlib/abort.c:81(__GI_abort)[0x7f90f811a859]
intl/loadmsgcat.c:509(get_sysdep_segment_value)[0x7f90f811a729]
:0(__GI___assert_fail)[0x7f90f812bf36]
sql/mysqld.cc:3650(my_malloc_size_cb_func)[0x55a479104685]
mysys/my_malloc.c:211(my_free)[0x55a47b1b210a]
maria/ma_close.c:262(maria_close)[0x55a47a2d13ef]
maria/ha_maria.cc:1250(ha_maria::close())[0x55a47a2ea69f]
sql/handler.cc:3382(handler::ha_close())[0x55a479c8f269]
sql/table.cc:4434(closefrm(TABLE*))[0x55a47978ee1d]
sql/temporary_tables.cc:1239(THD::close_temporary_table(TABLE*))[0x55a479ac5a50]
sql/temporary_tables.cc:1490(THD::free_temporary_table(TABLE*))[0x55a479ac74dc]
sql/temporary_tables.cc:651(THD::drop_temporary_table(TABLE*, bool*, bool))[0x55a479ac1cb9]
sql/sql_table.cc:4244(create_table_impl(THD*, st_ddl_log_state*, st_ddl_log_state*, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, st_mysql_const_lex_string const&, DDL_options_st, HA_CREATE_INFO*, Alter_info*, int, bool*, st_key**, unsigned int*, st_mysql_const_unsigned_lex_string*))[0x55a4796a0e85]
sql/sql_table.cc:4546(mysql_create_table_no_lock(THD*, st_ddl_log_state*, st_ddl_log_state*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, Table_specification_st*, Alter_info*, bool*, int, TABLE_LIST*))[0x55a4796a2c2f]
sql/sql_insert.cc:4509(select_create::create_table_from_items(THD*, List<Item>*, st_mysql_lock**, TABLEOP_HOOKS*))[0x55a47936fec3]
sql/sql_insert.cc:4711(select_create::prepare(List<Item>&, st_select_lex_unit*))[0x55a47937169b]
sql/sql_select.cc:1684(JOIN::prepare(TABLE_LIST*, Item*, unsigned int, st_order*, bool, st_order*, Item*, st_order*, st_select_lex*, st_select_lex_unit*))[0x55a4794d1bb9]
sql/sql_select.cc:4970(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55a4794f4967]
sql/sql_select.cc:545(handle_select(THD*, LEX*, select_result*, unsigned long))[0x55a4794c5121]
sql/sql_table.cc:11737(Sql_cmd_create_table_like::execute(THD*))[0x55a4796d8454]
sql/sql_parse.cc:5997(mysql_execute_command(THD*, bool))[0x55a479427dcb]
sql/sql_parse.cc:8030(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x55a47943559c]
sql/log_event_server.cc:1918(Query_log_event::do_apply_event(rpl_group_info*, char const*, unsigned int))[0x55a479ffb8b7]
sql/log_event_server.cc:1592(Query_log_event::do_apply_event(rpl_group_info*))[0x55a479ff8ad5]
sql/log_event.h:1516(Log_event::apply_event(rpl_group_info*))[0x55a4791a34dd]
sql/slave.cc:3881(apply_event_and_update_pos_apply(Log_event*, THD*, rpl_group_info*, int))[0x55a479184c40]
sql/slave.cc:4068(apply_event_and_update_pos_for_parallel(Log_event*, THD*, rpl_group_info*))[0x55a4791859cc]
sql/rpl_parallel.cc:62(rpt_handle_event(rpl_parallel_thread::queued_event*, rpl_parallel_thread*))[0x55a4799df1cd]
sql/rpl_parallel.cc:1378(handle_rpl_parallel_thread)[0x55a4799e6e45]
perfschema/pfs.cc:2203(pfs_spawn_thread)[0x55a47a58d581]
nptl/pthread_create.c:478(start_thread)[0x7f90f8642609]
x86_64/clone.S:97(__GI___clone)[0x7f90f8217293]
 
Query (0x6140000638d3): CREATE OR REPLACE TEMPORARY TABLE `tmp` (pk int) ENGINE=Aria AS SELECT * FROM ( VALUES (1),(2),(3),(4),(5) ) DT



 Comments   
Comment by Sergei Golubchik [ 2022-01-25 ]

besides asserts in the debug builds, this will show as invalid values for Used_mem in I_S.PROCESSLIST

Comment by Roel Van de Paar [ 2023-08-26 ]

I ran into this also, very regularly, during replication testing. MTR:

11.2.0 e81fa345020ec6a067583db6a7019d6404b26f93 (Debug)

Core was generated by `/test/MD010823-mariadb-11.2.0-linux-x86_64-dbg/bin/mariadbd --defaults-group-su'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=22813320312384) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x14bfa3da7640 (LWP 188418))]
 
Thread 30 (Thread 0x14bfa3e3d640 (LWP 188416)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x14bf8800c148) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x14bf8800c148) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x14bf8800c148, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a93ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x14bf8800c098, cond=0x14bf8800c120) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x14bf8800c120, mutex=mutex@entry=0x14bf8800c098) at ./nptl/pthread_cond_wait.c:627
#5  0x0000565522d593ef in safe_cond_wait (cond=0x14bf8800c120, mp=0x14bf8800c070, file=0x565522ef01e0 "/test/11.2_dbg/include/mysql/psi/mysql_thread.h", line=1072) at /test/11.2_dbg/mysys/thr_mutex.c:494
#6  0x00005655225c976f in inline_mysql_cond_wait (src_line=1710, src_file=0x565522f63f30 "/test/11.2_dbg/sql/rpl_parallel.cc", mutex=0x14bf8800c070, that=0x14bf8800c120) at /test/11.2_dbg/include/mysql/psi/mysql_thread.h:1072
#7  rpl_parallel_change_thread_count (pool=pool@entry=0x565523940e40 <global_rpl_thread_pool>, new_count=new_count@entry=0, force=force@entry=false) at /test/11.2_dbg/sql/rpl_parallel.cc:1710
#8  0x00005655225ca311 in rpl_parallel_inactivate_pool (pool=pool@entry=0x565523940e40 <global_rpl_thread_pool>) at /test/11.2_dbg/sql/rpl_parallel.cc:1849
#9  0x00005655225ca351 in rpl_parallel_resize_pool_if_no_slaves () at /test/11.2_dbg/sql/rpl_parallel.cc:1793
#10 0x0000565522314bd9 in handle_slave_sql (arg=0x565526380f30) at /test/11.2_dbg/sql/slave.cc:5838
#11 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 29 (Thread 0x14bfa3d11640 (LWP 188429)):
#0  0x000014bfd3b18d7f in __GI___poll (fds=fds@entry=0x14bfa3d10868, nfds=nfds@entry=1, timeout=timeout@entry=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000056552288dbe9 in poll (__timeout=28800000, __nfds=1, __fds=0x14bfa3d10868) at /usr/include/x86_64-linux-gnu/bits/poll2.h:39
#2  vio_io_wait (vio=vio@entry=0x14bf7800ae88, event=event@entry=VIO_IO_EVENT_READ, timeout=28800000) at /test/11.2_dbg/vio/viosocket.c:948
#3  0x000056552288dccc in vio_socket_io_wait (vio=vio@entry=0x14bf7800ae88, event=event@entry=VIO_IO_EVENT_READ) at /test/11.2_dbg/vio/viosocket.c:116
#4  0x000056552288dde3 in vio_read (vio=0x14bf7800ae88, buf=0x14bf7800b098 "\002", size=4) at /test/11.2_dbg/vio/viosocket.c:182
#5  0x000056552284490e in my_real_read (net=0x14bf78001098, complen=complen@entry=0x14bfa3d10a58, header=1 '\001') at /test/11.2_dbg/sql/net_serv.cc:1005
#6  0x0000565522845789 in my_net_read_packet_reallen (net=0x14bf78001098, read_from_server=<optimized out>, reallen=reallen@entry=0x14bfa3d10aa8) at /test/11.2_dbg/sql/net_serv.cc:1286
#7  0x00005655228459ce in my_net_read_packet (net=<optimized out>, read_from_server=<optimized out>) at /test/11.2_dbg/sql/net_serv.cc:1270
#8  0x00005655223e1455 in do_command (thd=0x14bf78000d58, blocking=blocking@entry=true) at /test/11.2_dbg/sql/sql_parse.cc:1261
#9  0x00005655225360d0 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x565525f22018, put_in_cache=put_in_cache@entry=true) at /test/11.2_dbg/sql/sql_connect.cc:1445
#10 0x000056552253632f in handle_one_connection (arg=0x565525f22018) at /test/11.2_dbg/sql/sql_connect.cc:1347
#11 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 28 (Thread 0x14bfa3fb4640 (LWP 188350)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5655239464cc <mysql_bin_log+4300>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5655239464cc <mysql_bin_log+4300>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5655239464cc <mysql_bin_log+4300>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a93ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x565523946418 <mysql_bin_log+4120>, cond=0x5655239464a0 <mysql_bin_log+4256>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x5655239464a0 <mysql_bin_log+4256>, mutex=mutex@entry=0x565523946418 <mysql_bin_log+4120>) at ./nptl/pthread_cond_wait.c:627
#5  0x0000565522d593ef in safe_cond_wait (cond=0x5655239464a0 <mysql_bin_log+4256>, mp=0x5655239463f0 <mysql_bin_log+4080>, file=0x565522ef01e0 "/test/11.2_dbg/include/mysql/psi/mysql_thread.h", line=1072) at /test/11.2_dbg/mysys/thr_mutex.c:494
#6  0x00005655228154e8 in inline_mysql_cond_wait (src_line=10789, src_file=0x5655230c7f7c "/test/11.2_dbg/sql/log.cc", mutex=0x5655239463f0 <mysql_bin_log+4080>, that=0x5655239464a0 <mysql_bin_log+4256>) at /test/11.2_dbg/include/mysql/psi/mysql_thread.h:1072
#7  binlog_background_thread (arg=<optimized out>) at /test/11.2_dbg/sql/log.cc:10789
#8  0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 27 (Thread 0x14bfa3d5c640 (LWP 188428)):
#0  0x000014bfd3b18d7f in __GI___poll (fds=fds@entry=0x14bfa3d5b868, nfds=nfds@entry=1, timeout=timeout@entry=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000056552288dbe9 in poll (__timeout=28800000, __nfds=1, __fds=0x14bfa3d5b868) at /usr/include/x86_64-linux-gnu/bits/poll2.h:39
#2  vio_io_wait (vio=vio@entry=0x14bf7400ae88, event=event@entry=VIO_IO_EVENT_READ, timeout=28800000) at /test/11.2_dbg/vio/viosocket.c:948
#3  0x000056552288dccc in vio_socket_io_wait (vio=vio@entry=0x14bf7400ae88, event=event@entry=VIO_IO_EVENT_READ) at /test/11.2_dbg/vio/viosocket.c:116
#4  0x000056552288dde3 in vio_read (vio=0x14bf7400ae88, buf=0x14bf7400b098 "\002", size=4) at /test/11.2_dbg/vio/viosocket.c:182
#5  0x000056552284490e in my_real_read (net=0x14bf74001098, complen=complen@entry=0x14bfa3d5ba58, header=1 '\001') at /test/11.2_dbg/sql/net_serv.cc:1005
#6  0x0000565522845789 in my_net_read_packet_reallen (net=0x14bf74001098, read_from_server=<optimized out>, reallen=reallen@entry=0x14bfa3d5baa8) at /test/11.2_dbg/sql/net_serv.cc:1286
#7  0x00005655228459ce in my_net_read_packet (net=<optimized out>, read_from_server=<optimized out>) at /test/11.2_dbg/sql/net_serv.cc:1270
#8  0x00005655223e1455 in do_command (thd=0x14bf74000d58, blocking=blocking@entry=true) at /test/11.2_dbg/sql/sql_parse.cc:1261
#9  0x00005655225360d0 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x5655263443b8, put_in_cache=put_in_cache@entry=true) at /test/11.2_dbg/sql/sql_connect.cc:1445
#10 0x000056552253632f in handle_one_connection (arg=0x5655263443b8) at /test/11.2_dbg/sql/sql_connect.cc:1347
#11 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 26 (Thread 0x14bfd2431640 (LWP 188309)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x565523866c78 <buf_pool+17464>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x565523866c78 <buf_pool+17464>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565523866c78 <buf_pool+17464>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a93ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x565523866b68 <buf_pool+17192>, cond=0x565523866c50 <buf_pool+17424>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x565523866c50 <buf_pool+17424>, mutex=mutex@entry=0x565523866b68 <buf_pool+17192>) at ./nptl/pthread_cond_wait.c:627
#5  0x0000565522d593ef in safe_cond_wait (cond=0x565523866c50 <buf_pool+17424>, mp=0x565523866b40 <buf_pool+17152>, file=0x565523183d90 "/test/11.2_dbg/storage/innobase/buf/buf0flu.cc", line=2355) at /test/11.2_dbg/mysys/thr_mutex.c:494
#6  0x0000565522c19843 in buf_flush_page_cleaner () at /test/11.2_dbg/storage/innobase/buf/buf0flu.cc:2355
#7  0x0000565522c1adbf in std::__invoke_impl<void, void (*)()> (__f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:60
#8  std::__invoke<void (*)()> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#9  std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#10 std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#12 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 25 (Thread 0x14bfa3ed3640 (LWP 188367)):
#0  0x000014bfd3b18d7f in __GI___poll (fds=fds@entry=0x14bfa3ed2868, nfds=nfds@entry=1, timeout=timeout@entry=28800000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000056552288dbe9 in poll (__timeout=28800000, __nfds=1, __fds=0x14bfa3ed2868) at /usr/include/x86_64-linux-gnu/bits/poll2.h:39
#2  vio_io_wait (vio=vio@entry=0x14bf9000ae88, event=event@entry=VIO_IO_EVENT_READ, timeout=28800000) at /test/11.2_dbg/vio/viosocket.c:948
#3  0x000056552288dccc in vio_socket_io_wait (vio=vio@entry=0x14bf9000ae88, event=event@entry=VIO_IO_EVENT_READ) at /test/11.2_dbg/vio/viosocket.c:116
#4  0x000056552288dde3 in vio_read (vio=0x14bf9000ae88, buf=0x14bf9000b098 "\002", size=4) at /test/11.2_dbg/vio/viosocket.c:182
#5  0x000056552284490e in my_real_read (net=0x14bf90001098, complen=complen@entry=0x14bfa3ed2a58, header=1 '\001') at /test/11.2_dbg/sql/net_serv.cc:1005
#6  0x0000565522845789 in my_net_read_packet_reallen (net=0x14bf90001098, read_from_server=<optimized out>, reallen=reallen@entry=0x14bfa3ed2aa8) at /test/11.2_dbg/sql/net_serv.cc:1286
#7  0x00005655228459ce in my_net_read_packet (net=<optimized out>, read_from_server=<optimized out>) at /test/11.2_dbg/sql/net_serv.cc:1270
#8  0x00005655223e1455 in do_command (thd=0x14bf90000d58, blocking=blocking@entry=true) at /test/11.2_dbg/sql/sql_parse.cc:1261
#9  0x00005655225360d0 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x565526332428, put_in_cache=put_in_cache@entry=true) at /test/11.2_dbg/sql/sql_connect.cc:1445
#10 0x000056552253632f in handle_one_connection (arg=0x565526332428) at /test/11.2_dbg/sql/sql_connect.cc:1347
#11 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 24 (Thread 0x14bfa3f69640 (LWP 188351)):
#0  0x000014bfd3a4321a in __GI___sigtimedwait (set=0x14bfa3f68a50, info=0x14bfa3f68ad0, timeout=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:61
#1  0x00005655222b8c05 in my_sigwait (code=<synthetic pointer>, sig=0x14bfa3f68a4c, set=0x14bfa3f68a50) at /test/11.2_dbg/include/my_pthread.h:195
#2  signal_hand (arg=<optimized out>) at /test/11.2_dbg/sql/mysqld.cc:3277
#3  0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 23 (Thread 0x14bfd0ca0640 (LWP 188340)):
#0  __futex_abstimed_wait_common64 (private=583505835, cancel=true, abstime=0x14bfd0c9fa90, op=137, expected=0, futex_word=0x565525f4818c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=583505835, abstime=0x14bfd0c9fa90, clockid=0, expected=0, futex_word=0x565525f4818c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565525f4818c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14bfd0c9fa90, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a9435d in __pthread_cond_wait_common (abstime=0x14bfd0c9fa90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48160) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=0x14bfd0c9fa90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48160) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x565525f48160, mutex=0x565525f47e50, clockid=1, abstime=0x14bfd0c9fa90) at ./nptl/pthread_cond_wait.c:679
#6  0x0000565522cf4438 in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=0x565525f48160) at /usr/include/c++/11/bits/std_mutex.h:169
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48160) at /usr/include/c++/11/condition_variable:201
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48160) at /usr/include/c++/11/condition_variable:111
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x565525f48160) at /usr/include/c++/11/condition_variable:163
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x565525f47cb0, lk=..., thread_data=thread_data@entry=0x565525f48160) at /test/11.2_dbg/tpool/tpool_generic.cc:480
#11 0x0000565522cf5465 in tpool::thread_pool_generic::get_task (this=this@entry=0x565525f47cb0, thread_var=thread_var@entry=0x565525f48160, t=t@entry=0x14bfd0c9fb38) at /test/11.2_dbg/tpool/tpool_generic.cc:533
#12 0x0000565522cf5582 in tpool::thread_pool_generic::worker_main (this=0x565525f47cb0, thread_var=0x565525f48160) at /test/11.2_dbg/tpool/tpool_generic.cc:578
#13 0x0000565522cf6404 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:74
#14 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 22 (LWP 188345):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
 
Thread 21 (LWP 188346):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
 
Thread 20 (Thread 0x14bfa3fff640 (LWP 188349)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x565523916b8c <COND_manager+44>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x565523916b8c <COND_manager+44>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565523916b8c <COND_manager+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a93ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x565523916bc8 <LOCK_manager+40>, cond=0x565523916b60 <COND_manager>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x565523916b60 <COND_manager>, mutex=mutex@entry=0x565523916bc8 <LOCK_manager+40>) at ./nptl/pthread_cond_wait.c:627
#5  0x0000565522d593ef in safe_cond_wait (cond=0x565523916b60 <COND_manager>, mp=0x565523916ba0 <LOCK_manager>, file=0x565522ef01e0 "/test/11.2_dbg/include/mysql/psi/mysql_thread.h", line=1072) at /test/11.2_dbg/mysys/thr_mutex.c:494
#6  0x00005655223ca256 in inline_mysql_cond_wait (src_line=103, src_file=0x565522f16740 "/test/11.2_dbg/sql/sql_manager.cc", mutex=0x565523916ba0 <LOCK_manager>, that=0x565523916b60 <COND_manager>) at /test/11.2_dbg/include/mysql/psi/mysql_thread.h:1072
#7  handle_manager (arg=<optimized out>) at /test/11.2_dbg/sql/sql_manager.cc:103
#8  0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 19 (Thread 0x14bfa3f1e640 (LWP 188358)):
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:375
#1  0x0000565522d3eb50 in memcpy (__len=2, __src=0x14bfa3f1c850, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29
#2  my_b_write (Count=2, Buffer=0x14bfa3f1c850 "33", info=0x565526382968) at /test/11.2_dbg/include/my_sys.h:528
#3  my_b_vprintf (info=0x565526382968, fmt=0x565522f42c49 "u\n%s\n%s\n%s\n%s\n%s\n%d\n%d\n%d\n%s\n%s\n%s\n%s\n%s\n%d\n%s\n%s\n%s\n%s\n%d\n%s\n%s", '\n' <repeats 12 times>, "using_gtid=%d\ndo_domain_ids=%s\nignore_domain_ids=%s\nEND_MARKER\n", args=args@entry=0x14bfa3f1c8c8) at /test/11.2_dbg/mysys/mf_iocache2.c:463
#4  0x0000565522d3ee2e in my_b_printf (info=<optimized out>, fmt=<optimized out>) at /test/11.2_dbg/mysys/mf_iocache2.c:298
#5  0x000056552252f3a3 in flush_master_info (mi=mi@entry=0x565526380f30, flush_relay_log_cache=flush_relay_log_cache@entry=true, need_lock_relay_log=need_lock_relay_log@entry=false) at /test/11.2_dbg/sql/rpl_mi.cc:814
#6  0x000056552230b5e2 in terminate_slave_threads (mi=mi@entry=0x565526380f30, thread_mask=thread_mask@entry=3, skip_lock=skip_lock@entry=false) at /test/11.2_dbg/sql/slave.cc:983
#7  0x0000565522410250 in stop_slave (thd=thd@entry=0x14bf8c000d58, mi=mi@entry=0x565526380f30, net_report=net_report@entry=true) at /test/11.2_dbg/sql/sql_repl.cc:3354
#8  0x00005655223d7c97 in mysql_execute_command (thd=thd@entry=0x14bf8c000d58, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/11.2_dbg/sql/sql_parse.cc:4282
#9  0x00005655223dd4c0 in mysql_parse (thd=thd@entry=0x14bf8c000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x14bfa3f1d200) at /test/11.2_dbg/sql/sql_parse.cc:7800
#10 0x00005655223df654 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x14bf8c000d58, packet=packet@entry=0x14bf8c175809 "STOP SLAVE", packet_length=packet_length@entry=10, blocking=blocking@entry=true) at /test/11.2_dbg/sql/sql_class.h:244
#11 0x00005655223e152f in do_command (thd=0x14bf8c000d58, blocking=blocking@entry=true) at /test/11.2_dbg/sql/sql_parse.cc:1405
#12 0x00005655225360d0 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x5655262655a8, put_in_cache=put_in_cache@entry=true) at /test/11.2_dbg/sql/sql_connect.cc:1445
#13 0x000056552253632f in handle_one_connection (arg=0x5655262655a8) at /test/11.2_dbg/sql/sql_connect.cc:1347
#14 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 18 (LWP 188339):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
 
Thread 17 (LWP 188337):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
 
Thread 16 (Thread 0x14bfd069d640 (LWP 188343)):
#0  __futex_abstimed_wait_common64 (private=-744, cancel=true, abstime=0x14bfd069ca90, op=137, expected=0, futex_word=0x565525f4830c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=-744, abstime=0x14bfd069ca90, clockid=0, expected=0, futex_word=0x565525f4830c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565525f4830c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14bfd069ca90, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a9435d in __pthread_cond_wait_common (abstime=0x14bfd069ca90, clockid=1, mutex=0x565525f47e50, cond=0x565525f482e0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=0x14bfd069ca90, clockid=1, mutex=0x565525f47e50, cond=0x565525f482e0) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x565525f482e0, mutex=0x565525f47e50, clockid=1, abstime=0x14bfd069ca90) at ./nptl/pthread_cond_wait.c:679
#6  0x0000565522cf4438 in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=0x565525f482e0) at /usr/include/c++/11/bits/std_mutex.h:169
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f482e0) at /usr/include/c++/11/condition_variable:201
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f482e0) at /usr/include/c++/11/condition_variable:111
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x565525f482e0) at /usr/include/c++/11/condition_variable:163
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x565525f47cb0, lk=..., thread_data=thread_data@entry=0x565525f482e0) at /test/11.2_dbg/tpool/tpool_generic.cc:480
#11 0x0000565522cf5465 in tpool::thread_pool_generic::get_task (this=this@entry=0x565525f47cb0, thread_var=thread_var@entry=0x565525f482e0, t=t@entry=0x14bfd069cb38) at /test/11.2_dbg/tpool/tpool_generic.cc:533
#12 0x0000565522cf5582 in tpool::thread_pool_generic::worker_main (this=0x565525f47cb0, thread_var=0x565525f482e0) at /test/11.2_dbg/tpool/tpool_generic.cc:578
#13 0x0000565522cf6404 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:74
#14 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 15 (LWP 188338):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
 
Thread 14 (Thread 0x14bfd089e640 (LWP 188342)):
#0  __futex_abstimed_wait_common64 (private=583505835, cancel=true, abstime=0x14bfd089da90, op=137, expected=0, futex_word=0x565525f48288) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=583505835, abstime=0x14bfd089da90, clockid=0, expected=0, futex_word=0x565525f48288) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565525f48288, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14bfd089da90, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a9435d in __pthread_cond_wait_common (abstime=0x14bfd089da90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48260) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=0x14bfd089da90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48260) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x565525f48260, mutex=0x565525f47e50, clockid=1, abstime=0x14bfd089da90) at ./nptl/pthread_cond_wait.c:679
#6  0x0000565522cf4438 in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=0x565525f48260) at /usr/include/c++/11/bits/std_mutex.h:169
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48260) at /usr/include/c++/11/condition_variable:201
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48260) at /usr/include/c++/11/condition_variable:111
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x565525f48260) at /usr/include/c++/11/condition_variable:163
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x565525f47cb0, lk=..., thread_data=thread_data@entry=0x565525f48260) at /test/11.2_dbg/tpool/tpool_generic.cc:480
#11 0x0000565522cf5465 in tpool::thread_pool_generic::get_task (this=this@entry=0x565525f47cb0, thread_var=thread_var@entry=0x565525f48260, t=t@entry=0x14bfd089db38) at /test/11.2_dbg/tpool/tpool_generic.cc:533
#12 0x0000565522cf5582 in tpool::thread_pool_generic::worker_main (this=0x565525f47cb0, thread_var=0x565525f48260) at /test/11.2_dbg/tpool/tpool_generic.cc:578
#13 0x0000565522cf6404 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:74
#14 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 13 (LWP 188336):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
 
Thread 12 (Thread 0x14bfd0a9f640 (LWP 188341)):
#0  __futex_abstimed_wait_common64 (private=583505835, cancel=true, abstime=0x14bfd0a9ea90, op=137, expected=0, futex_word=0x565525f48208) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=583505835, abstime=0x14bfd0a9ea90, clockid=0, expected=0, futex_word=0x565525f48208) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565525f48208, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14bfd0a9ea90, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a9435d in __pthread_cond_wait_common (abstime=0x14bfd0a9ea90, clockid=1, mutex=0x565525f47e50, cond=0x565525f481e0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=0x14bfd0a9ea90, clockid=1, mutex=0x565525f47e50, cond=0x565525f481e0) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x565525f481e0, mutex=0x565525f47e50, clockid=1, abstime=0x14bfd0a9ea90) at ./nptl/pthread_cond_wait.c:679
#6  0x0000565522cf4438 in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=0x565525f481e0) at /usr/include/c++/11/bits/std_mutex.h:169
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f481e0) at /usr/include/c++/11/condition_variable:201
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f481e0) at /usr/include/c++/11/condition_variable:111
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x565525f481e0) at /usr/include/c++/11/condition_variable:163
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x565525f47cb0, lk=..., thread_data=thread_data@entry=0x565525f481e0) at /test/11.2_dbg/tpool/tpool_generic.cc:480
#11 0x0000565522cf5465 in tpool::thread_pool_generic::get_task (this=this@entry=0x565525f47cb0, thread_var=thread_var@entry=0x565525f481e0, t=t@entry=0x14bfd0a9eb38) at /test/11.2_dbg/tpool/tpool_generic.cc:533
#12 0x0000565522cf5582 in tpool::thread_pool_generic::worker_main (this=0x565525f47cb0, thread_var=0x565525f481e0) at /test/11.2_dbg/tpool/tpool_generic.cc:578
#13 0x0000565522cf6404 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:74
#14 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 11 (LWP 188347):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0
 
Thread 10 (Thread 0x14bfa3e88640 (LWP 188415) (Exiting)):
#0  start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:513
#1  0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 9 (Thread 0x14bfd1a26640 (LWP 188331)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x14bfd1a25a90, op=137, expected=0, futex_word=0x565525f4808c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x14bfd1a25a90, clockid=0, expected=0, futex_word=0x565525f4808c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565525f4808c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14bfd1a25a90, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a9435d in __pthread_cond_wait_common (abstime=0x14bfd1a25a90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48060) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=0x14bfd1a25a90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48060) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x565525f48060, mutex=0x565525f47e50, clockid=1, abstime=0x14bfd1a25a90) at ./nptl/pthread_cond_wait.c:679
#6  0x0000565522cf4438 in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=0x565525f48060) at /usr/include/c++/11/bits/std_mutex.h:169
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48060) at /usr/include/c++/11/condition_variable:201
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48060) at /usr/include/c++/11/condition_variable:111
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x565525f48060) at /usr/include/c++/11/condition_variable:163
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x565525f47cb0, lk=..., thread_data=thread_data@entry=0x565525f48060) at /test/11.2_dbg/tpool/tpool_generic.cc:480
#11 0x0000565522cf5465 in tpool::thread_pool_generic::get_task (this=this@entry=0x565525f47cb0, thread_var=thread_var@entry=0x565525f48060, t=t@entry=0x14bfd1a25b38) at /test/11.2_dbg/tpool/tpool_generic.cc:533
#12 0x0000565522cf5582 in tpool::thread_pool_generic::worker_main (this=0x565525f47cb0, thread_var=0x565525f48060) at /test/11.2_dbg/tpool/tpool_generic.cc:578
#13 0x0000565522cf6404 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:74
#14 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 8 (Thread 0x14bfd2e32640 (LWP 188308)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000014bfd430ad43 in ____sys_io_uring_enter2 (sz=<optimized out>, sig=<optimized out>, flags=<optimized out>, min_complete=<optimized out>, to_submit=<optimized out>, fd=<optimized out>) at syscall.h:127
#2  _io_uring_get_cqe (ring=0x565525f8bf88, cqe_ptr=0x14bfd2e31b28, data=data@entry=0x14bfd2e31af0) at queue.c:111
#3  0x000014bfd430aebb in __io_uring_get_cqe (ring=<optimized out>, cqe_ptr=<optimized out>, submit=<optimized out>, wait_nr=<optimized out>, sigmask=<optimized out>) at queue.c:139
#4  0x0000565522cf79bd in io_uring_wait_cqe_nr (wait_nr=1, cqe_ptr=0x14bfd2e31b28, ring=0x565525f8bf88) at /usr/include/liburing.h:772
#5  io_uring_wait_cqe (cqe_ptr=0x14bfd2e31b28, ring=0x565525f8bf88) at /usr/include/liburing.h:792
#6  (anonymous namespace)::aio_uring::thread_routine (aio=0x565525f8bf80) at /test/11.2_dbg/tpool/aio_liburing.cc:145
#7  0x0000565522cf76a0 in std::__invoke_impl<void, void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> (__f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:61
#8  std::__invoke<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#9  std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> >::_M_invoke<0, 1> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#10 std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)((anonymous namespace)::aio_uring*), (anonymous namespace)::aio_uring*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#12 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 7 (Thread 0x14bfd3cdf640 (LWP 188307)):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x14bfd3cdeb50, op=393, expected=0, futex_word=0x565524190e0c <COND_timer+44>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x14bfd3cdeb50, clockid=40, expected=0, futex_word=0x565524190e0c <COND_timer+44>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565524190e0c <COND_timer+44>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x14bfd3cdeb50, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a93f1b in __pthread_cond_wait_common (abstime=0x14bfd3cdeb50, clockid=0, mutex=0x565524190e48 <LOCK_timer+40>, cond=0x565524190de0 <COND_timer>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x565524190de0 <COND_timer>, mutex=mutex@entry=0x565524190e48 <LOCK_timer+40>, abstime=abstime@entry=0x14bfd3cdeb50) at ./nptl/pthread_cond_wait.c:652
#5  0x0000565522d5968b in safe_cond_timedwait (cond=cond@entry=0x565524190de0 <COND_timer>, mp=mp@entry=0x565524190e20 <LOCK_timer>, abstime=abstime@entry=0x14bfd3cdeb50, file=file@entry=0x565522ef01e0 "/test/11.2_dbg/include/mysql/psi/mysql_thread.h", line=line@entry=1088) at /test/11.2_dbg/mysys/thr_mutex.c:548
#6  0x0000565522d59c3d in inline_mysql_cond_timedwait (src_line=321, src_file=0x5655231ada30 "/test/11.2_dbg/mysys/thr_timer.c", abstime=0x14bfd3cdeb50, mutex=0x565524190e20 <LOCK_timer>, that=0x565524190de0 <COND_timer>) at /test/11.2_dbg/include/mysql/psi/mysql_thread.h:1088
#7  timer_handler (arg=<optimized out>) at /test/11.2_dbg/mysys/thr_timer.c:321
#8  0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 6 (Thread 0x14bfd3ce33c0 (LWP 188301)):
#0  0x000014bfd3b18d7f in __GI___poll (fds=0x5655262f0d68, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00005655222c0546 in poll (__timeout=-1, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:39
#2  handle_connections_sockets () at /test/11.2_dbg/sql/mysqld.cc:6362
#3  0x00005655222c3fb1 in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /test/11.2_dbg/sql/mysqld.cc:6064
#4  0x00005655222b7c76 in main (argc=<optimized out>, argv=<optimized out>) at /test/11.2_dbg/sql/main.cc:34
 
Thread 5 (Thread 0x14bfd021a640 (LWP 188348)):
#0  __futex_abstimed_wait_common64 (private=-742300928, cancel=true, abstime=0x14bfd0219b00, op=393, expected=0, futex_word=0x56552414ab88 <COND_checkpoint+40>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=-742300928, abstime=0x14bfd0219b00, clockid=2072, expected=0, futex_word=0x56552414ab88 <COND_checkpoint+40>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x56552414ab88 <COND_checkpoint+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x14bfd0219b00, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a93f1b in __pthread_cond_wait_common (abstime=0x14bfd0219b00, clockid=0, mutex=0x56552414abc8 <LOCK_checkpoint+40>, cond=0x56552414ab60 <COND_checkpoint>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=cond@entry=0x56552414ab60 <COND_checkpoint>, mutex=mutex@entry=0x56552414abc8 <LOCK_checkpoint+40>, abstime=abstime@entry=0x14bfd0219b00) at ./nptl/pthread_cond_wait.c:652
#5  0x0000565522d5968b in safe_cond_timedwait (cond=0x56552414ab60 <COND_checkpoint>, mp=0x56552414aba0 <LOCK_checkpoint>, abstime=abstime@entry=0x14bfd0219b00, file=file@entry=0x565522ef01e0 "/test/11.2_dbg/include/mysql/psi/mysql_thread.h", line=line@entry=1088) at /test/11.2_dbg/mysys/thr_mutex.c:548
#6  0x00005655228eda7b in inline_mysql_cond_timedwait (src_line=115, src_file=0x565523107c78 "/test/11.2_dbg/storage/maria/ma_servicethread.c", abstime=0x14bfd0219b00, mutex=<optimized out>, that=<optimized out>) at /test/11.2_dbg/include/mysql/psi/mysql_thread.h:1088
#7  my_service_thread_sleep (control=control@entry=0x56552384d500 <checkpoint_control>, sleep_time=sleep_time@entry=29000000000) at /test/11.2_dbg/storage/maria/ma_servicethread.c:115
#8  0x00005655228e5ffd in ma_checkpoint_background (arg=0x1e) at /test/11.2_dbg/storage/maria/ma_checkpoint.c:725
#9  0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 4 (Thread 0x14bfd049c640 (LWP 188344)):
#0  __futex_abstimed_wait_common64 (private=-744, cancel=true, abstime=0x14bfd049ba90, op=137, expected=0, futex_word=0x565525f4838c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=-744, abstime=0x14bfd049ba90, clockid=0, expected=0, futex_word=0x565525f4838c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565525f4838c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14bfd049ba90, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a9435d in __pthread_cond_wait_common (abstime=0x14bfd049ba90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48360) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=0x14bfd049ba90, clockid=1, mutex=0x565525f47e50, cond=0x565525f48360) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x565525f48360, mutex=0x565525f47e50, clockid=1, abstime=0x14bfd049ba90) at ./nptl/pthread_cond_wait.c:679
#6  0x0000565522cf4438 in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=0x565525f48360) at /usr/include/c++/11/bits/std_mutex.h:169
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48360) at /usr/include/c++/11/condition_variable:201
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f48360) at /usr/include/c++/11/condition_variable:111
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x565525f48360) at /usr/include/c++/11/condition_variable:163
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x565525f47cb0, lk=..., thread_data=thread_data@entry=0x565525f48360) at /test/11.2_dbg/tpool/tpool_generic.cc:480
#11 0x0000565522cf5465 in tpool::thread_pool_generic::get_task (this=this@entry=0x565525f47cb0, thread_var=thread_var@entry=0x565525f48360, t=t@entry=0x14bfd049bb38) at /test/11.2_dbg/tpool/tpool_generic.cc:533
#12 0x0000565522cf5582 in tpool::thread_pool_generic::worker_main (this=0x565525f47cb0, thread_var=0x565525f48360) at /test/11.2_dbg/tpool/tpool_generic.cc:578
#13 0x0000565522cf6404 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:74
#14 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 3 (Thread 0x14bfd1825640 (LWP 188332)):
#0  __futex_abstimed_wait_common64 (private=-742300928, cancel=true, abstime=0x14bfd1824a90, op=137, expected=0, futex_word=0x565525f4810c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=-742300928, abstime=0x14bfd1824a90, clockid=0, expected=0, futex_word=0x565525f4810c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x565525f4810c, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x14bfd1824a90, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000014bfd3a9435d in __pthread_cond_wait_common (abstime=0x14bfd1824a90, clockid=1, mutex=0x565525f47e50, cond=0x565525f480e0) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_clockwait64 (abstime=0x14bfd1824a90, clockid=1, mutex=0x565525f47e50, cond=0x565525f480e0) at ./nptl/pthread_cond_wait.c:691
#5  ___pthread_cond_clockwait64 (cond=0x565525f480e0, mutex=0x565525f47e50, clockid=1, abstime=0x14bfd1824a90) at ./nptl/pthread_cond_wait.c:679
#6  0x0000565522cf4438 in std::__condvar::wait_until (__abs_time=..., __clock=1, __m=..., this=0x565525f480e0) at /usr/include/c++/11/bits/std_mutex.h:169
#7  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f480e0) at /usr/include/c++/11/condition_variable:201
#8  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x565525f480e0) at /usr/include/c++/11/condition_variable:111
#9  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x565525f480e0) at /usr/include/c++/11/condition_variable:163
#10 tpool::thread_pool_generic::wait_for_tasks (this=this@entry=0x565525f47cb0, lk=..., thread_data=thread_data@entry=0x565525f480e0) at /test/11.2_dbg/tpool/tpool_generic.cc:480
#11 0x0000565522cf5465 in tpool::thread_pool_generic::get_task (this=this@entry=0x565525f47cb0, thread_var=thread_var@entry=0x565525f480e0, t=t@entry=0x14bfd1824b38) at /test/11.2_dbg/tpool/tpool_generic.cc:533
#12 0x0000565522cf5582 in tpool::thread_pool_generic::worker_main (this=0x565525f47cb0, thread_var=0x565525f480e0) at /test/11.2_dbg/tpool/tpool_generic.cc:578
#13 0x0000565522cf6404 in std::__invoke_impl<void, void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/11/bits/invoke.h:74
#14 std::__invoke<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> (__fn=<optimized out>) at /usr/include/c++/11/bits/invoke.h:96
#15 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::_M_invoke<0ul, 1ul, 2ul> (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:259
#16 std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> >::operator() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:266
#17 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (tpool::thread_pool_generic::*)(tpool::worker_data*), tpool::thread_pool_generic*, tpool::worker_data*> > >::_M_run (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:211
#18 0x000014bfd3edc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#20 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 2 (Thread 0x14bfa3df2640 (LWP 188417) (Exiting)):
#0  start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:513
#1  0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 
Thread 1 (Thread 0x14bfa3da7640 (LWP 188418)):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=22813320312384) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=22813320312384) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=22813320312384, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x000014bfd3a42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x000014bfd3a287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x000014bfd3a2871b in __assert_fail_base (fmt=0x14bfd3bdd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x565522f0e340 "status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory", file=0x565522f0ccc8 "/test/11.2_dbg/sql/sql_class.cc", line=1744, function=<optimized out>) at ./assert/assert.c:92
#6  0x000014bfd3a39e96 in __GI___assert_fail (assertion=0x565522f0e340 "status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory", file=0x565522f0ccc8 "/test/11.2_dbg/sql/sql_class.cc", line=1744, function=0x565522f0ec1a "virtual THD::~THD()") at ./assert/assert.c:101
#7  0x000056552237c872 in THD::~THD (this=0x14bf80000f98, __in_chrg=<optimized out>) at /test/11.2_dbg/sql/sql_class.cc:1744
#8  0x000056552237c8a5 in THD::~THD (this=0x14bf80000f98, __in_chrg=<optimized out>) at /test/11.2_dbg/sql/sql_class.cc:1750
#9  0x00005655225cde72 in handle_rpl_parallel_thread (arg=<optimized out>) at /test/11.2_dbg/sql/rpl_parallel.cc:1572
#10 0x000014bfd3a94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x000014bfd3b26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Also note that even on optimized builds, this generates a memory leak. MTR output:

11.2.0 e81fa345020ec6a067583db6a7019d6404b26f93 (Optimized)

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
include/master-slave.inc
[connection master]
connection master;
CREATE TEMPORARY TABLE tmp (pk int) ENGINE=Aria AS SELECT 1;
connection master1;
DROP  TEMPORARY TABLE IF EXISTS tmp;
Warnings:
Note	1051	Unknown table 'test.tmp'
connection slave;
include/rpl_end.inc
***Warnings generated in error logs during shutdown after running tests: main.test
 
Warning: Memory not freed: -29472
 
main.test                                [ fail ]

Comment by Roel Van de Paar [ 2023-08-26 ]

In my runs, I ran into this using the following SQL:

SET SESSION binlog_format=MIXED;
CREATE TEMPORARY TABLE t1(c INT) ENGINE=Aria;
DROP TABLE t1;
CREATE TEMPORARY TABLE t2(c INT) ENGINE=INNODB;
LOAD DATA LOCAL INFILE 'nosuchfile.txt' INTO TABLE t1;
CREATE TEMPORARY TABLE t1 AS SELECT 1;
CREATE TABLE t1 (c INT);
RENAME TABLE t1 TO t5;
INSERT INTO t1 VALUES (1);

When this is executed on the master, the slave crashes.
mariadbd Options in use:

Master: --no-defaults --gtid_strict_mode=1 --log_bin=binlog --binlog_format=ROW --log_bin_trust_function_creators=1 --server_id=1
Slave: --no-defaults --relay-log=relaylog --slave-parallel-threads=65 --slave-parallel-mode=aggressive --slave-parallel-max-queued=1073741827 --slave_run_triggers_for_rbr=LOGGING --slave_skip_errors=ALL --server_id=2

I saw two different stacks:

11.2.0 e81fa345020ec6a067583db6a7019d6404b26f93 (Debug)

Core was generated by `/test/MD010823-mariadb-11.2.0-linux-x86_64-dbg/bin/mariadbd --no-defaults --rel'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=22970554230336)
    at ./nptl/pthread_kill.c:44
[Current thread is 1 (Thread 0x14e43fb9a640 (LWP 1190362))]
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=22970554230336) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=22970554230336) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=22970554230336, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x000014e4ee442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x000014e4ee4287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x000014e4ee42871b in __assert_fail_base (fmt=0x14e4ee5dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55c122d87340 "status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory", file=0x55c122d85cc8 "/test/11.2_dbg/sql/sql_class.cc", line=1744, function=<optimized out>) at ./assert/assert.c:92
#6  0x000014e4ee439e96 in __GI___assert_fail (assertion=0x55c122d87340 "status_var.local_memory_used == 0 || !debug_assert_on_not_freed_memory", file=0x55c122d85cc8 "/test/11.2_dbg/sql/sql_class.cc", line=1744, function=0x55c122d87c1a "virtual THD::~THD()") at ./assert/assert.c:101
#7  0x000055c1221f5872 in THD::~THD (this=0x14e374000f98, __in_chrg=<optimized out>) at /test/11.2_dbg/sql/sql_class.cc:1744
#8  0x000055c1221f58a5 in THD::~THD (this=0x14e374000f98, __in_chrg=<optimized out>) at /test/11.2_dbg/sql/sql_class.cc:1750
#9  0x000055c122446e72 in handle_rpl_parallel_thread (arg=<optimized out>) at /test/11.2_dbg/sql/rpl_parallel.cc:1572
#10 0x000014e4ee494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x000014e4ee526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

And

11.2.0 e81fa345020ec6a067583db6a7019d6404b26f93 (Debug)

Core was generated by `/test/MD010823-mariadb-11.2.0-linux-x86_64-dbg/bin/mariadbd --no-defaults --gti'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=22364721632832)
    at ./nptl/pthread_kill.c:44
[Current thread is 1 (Thread 0x14573149d640 (LWP 1014136))]
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=22364721632832) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=22364721632832) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=22364721632832, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x000014576c642476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x000014576c6287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x000014576c62871b in __assert_fail_base (fmt=0x14576c7dd150 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x5636b72bd6a8 "(longlong) thd->status_var.local_memory_used >= 0 || !debug_assert_on_not_freed_memory", file=0x5636b72ba004 "/test/11.2_dbg/sql/mysqld.cc", line=3816, function=<optimized out>) at ./assert/assert.c:92
#6  0x000014576c639e96 in __GI___assert_fail (assertion=0x5636b72bd6a8 "(longlong) thd->status_var.local_memory_used >= 0 || !debug_assert_on_not_freed_memory", file=0x5636b72ba004 "/test/11.2_dbg/sql/mysqld.cc", line=3816, function=0x5636b72bd630 "void my_malloc_size_cb_func(long long int, my_bool)") at ./assert/assert.c:101
#7  0x00005636b6687d64 in my_malloc_size_cb_func (size=<optimized out>, is_thread_specific=<optimized out>) at /test/11.2_dbg/sql/mysqld.cc:3816
#8  0x00005636b711f004 in my_free (ptr=0x1455d400ad48) at /test/11.2_dbg/mysys/my_malloc.c:208
#9  0x00005636b71166da in root_free (root=root@entry=0x1455d4006e70, ptr=<optimized out>, size=<optimized out>) at /test/11.2_dbg/mysys/my_alloc.c:83
#10 0x00005636b7116ee8 in free_root (root=0x1455d4006e70, MyFlags=<optimized out>) at /test/11.2_dbg/mysys/my_alloc.c:515
#11 0x00005636b674909f in THD::~THD (this=0x1455d4000f98, __in_chrg=<optimized out>) at /test/11.2_dbg/sql/sql_class.cc:1721
#12 0x00005636b67498a5 in THD::~THD (this=0x1455d4000f98, __in_chrg=<optimized out>) at /test/11.2_dbg/sql/sql_class.cc:1750
#13 0x00005636b699ae72 in handle_rpl_parallel_thread (arg=<optimized out>) at /test/11.2_dbg/sql/rpl_parallel.cc:1572
#14 0x000014576c694b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#15 0x000014576c726a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Comment by Roel Van de Paar [ 2023-08-26 ]

Note that even though optimized builds do not directly (or rather immediately) crash, replication stops:

11.2.0 e81fa345020ec6a067583db6a7019d6404b26f93 (Optimized)

2023-08-26 11:14:48 5 [Note] Slave I/O thread: connected to master 'repl_user@127.0.0.1:11612',replication starts at GTID position ''
2023-08-26 11:14:48 6 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './relaylog.000001' position: 4; GTID position ''
2023-08-26 11:14:50 69 [ERROR] Slave SQL: Error executing row event: 'Table 'test.t1' doesn't exist', Gtid 0-1-9, Internal MariaDB error code: 1146
2023-08-26 11:14:50 69 [Warning] Slave: Table 'test.t1' doesn't exist Error_code: 1146
2023-08-26 11:14:50 69 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'binlog.000001' position 1725; GTID position '0-1-8'
2023-08-26 11:14:50 6 [Note] Error reading relay log event: slave SQL thread was killed
2023-08-26 11:14:50 6 [Note] Slave SQL thread exiting, replication stopped in log 'binlog.000001' at position 1725; GTID position '0-1-8', master: 127.0.0.1:11612

Comment by Roel Van de Paar [ 2023-08-26 ]

The issue may be lightly sporadic (14 failures in 20 attempts)

Comment by Roel Van de Paar [ 2023-08-26 ]

ASAN/UBSAN does not immediately see the leak, though UBASAN debug builds crash with the regular stack(s) above.

Comment by Roel Van de Paar [ 2023-08-26 ]

I have also seen this UniqueID/stack in combination with this bug:

SIGABRT|my_malloc_size_cb_func|my_free|free_root|THD::~THD

Comment by Andrei Elkin [ 2023-08-28 ]

Highly likely this asserts is caused by MDEV-25064 (tmp tables + parallel on slave).

Generated at Thu Feb 08 09:48:42 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.